### What is Modin?
Modin is a drop-in replacement for [pandas](https://github.com/pandas-dev/pandas). While pandas is
single-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your
cores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs
[out of memory](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html).
Also, Modin comes with the [additional APIs](https://modin.readthedocs.io/en/latest/usage_guide/advanced_usage/index.html#additional-apis)
to improve user experience.
By simply replacing the import statement, Modin offers users effortless speed and scale for their pandas workflows:
In the GIFs below, Modin (left) and pandas (right) perform *the same pandas operations* on a 2GB dataset. The only difference between the two notebook examples is the import statement.
The charts below show the speedup you get by replacing pandas with Modin based on the examples above. The example notebooks can be found [here](examples/jupyter). To learn more about the speedups you could get with Modin and try out some examples on your own, check out our [10-minute quickstart guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html) to try out some examples on your own!
### Installation
#### From PyPI
Modin can be installed with `pip` on Linux, Windows and MacOS:
```bash
pip install "modin[all]" # (Recommended) Install Modin with Ray and Dask engines.
```
If you want to install Modin with a specific engine, we recommend:
```bash
pip install "modin[ray]" # Install Modin dependencies and Ray.
pip install "modin[dask]" # Install Modin dependencies and Dask.
pip install "modin[mpi]" # Install Modin dependencies and MPI through unidist.
```
To get Modin on MPI through unidist (as of unidist 0.5.0) fully working
it is required to have a working MPI implementation installed beforehand.
Otherwise, installation of `modin[mpi]` may fail. Refer to
[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)
section of the unidist documentation for more details about installation.
**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray` instead of `ray[default]`.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing `ray[default]` along with `modin[ray]`.
Modin automatically detects which engine(s) you have installed and uses that for scheduling computation.
#### From conda-forge
Installing from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`
will install Modin and three engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask) and
[MPI through unidist](https://github.com/modin-project/unidist).
```bash
conda install -c conda-forge modin-all
```
Each engine can also be installed individually (and also as a combination of several engines):
```bash
conda install -c conda-forge modin-ray # Install Modin dependencies and Ray.
conda install -c conda-forge modin-dask # Install Modin dependencies and Dask.
conda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.
```
**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray-core` instead of `ray-default`.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing `ray-default` along with `modin-ray`.
Refer to
[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)
section of the unidist documentation for more details on how to install a specific MPI implementation to run on.
To speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:
```bash
conda install -n base conda-libmamba-solver
```
and then use it during istallation either like:
```bash
conda install -c conda-forge modin-ray --experimental-solver=libmamba
```
or starting from conda 22.11 and libmamba solver 22.12 versions:
```bash
conda install -c conda-forge modin-ray --solver=libmamba
```
#### Choosing a Compute Engine
If you want to choose a specific compute engine to run on, you can set the environment
variable `MODIN_ENGINE` and Modin will do computation with that engine:
```bash
export MODIN_ENGINE=ray # Modin will use Ray
export MODIN_ENGINE=dask # Modin will use Dask
export MODIN_ENGINE=unidist # Modin will use Unidist
```
If you want to choose the Unidist engine, you should set the additional environment
variable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:
```bash
export UNIDIST_BACKEND=mpi # Unidist will use MPI backend
```
This can also be done within a notebook/interpreter before you import Modin:
```python
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put("ray") # Modin will use Ray
modin_cfg.Engine.put("dask") # Modin will use Dask
modin_cfg.Engine.put('unidist') # Modin will use Unidist
unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend
```
_Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior._
#### Which engine should I use?
On Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required
to use either of these engines as Modin abstracts away all of the complexity, so feel
free to pick either!
### Pandas API Coverage
Some pandas APIs are easier to implement than others, so if something is missing feel
free to open an issue!
### More about Modin
For the complete documentation on Modin, visit our [ReadTheDocs](https://modin.readthedocs.io/en/latest/index.html) page.
#### Scale your pandas workflow by changing a single line of code.
_Note: In local mode (without a cluster), Modin will create and manage a local (Dask or Ray) cluster for the execution._
To use Modin, you do not need to specify how to distribute the data, or even know how many
cores your system has. In fact, you can continue using your previous
pandas notebooks while experiencing a considerable speedup from Modin, even on a single
machine. Once you've changed your import statement, you're ready to use Modin just like
you would with pandas!
#### Faster pandas, even on your laptop
The `modin.pandas` DataFrame is an extremely light-weight parallel DataFrame.
Modin transparently distributes the data and computation so that you can continue using the same pandas API
while working with more data faster. Because it is so light-weight,
Modin provides speed-ups of up to 4x on a laptop with 4 physical cores.
In pandas, you are only able to use one core at a time when you are doing computation of
any kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a
traditionally synchronous task like `read_csv`, we see large speedups by efficiently
distributing the work across your entire machine.
```python
import modin.pandas as pd
df = pd.read_csv("my_dataset.csv")
```
#### Modin can handle the datasets that pandas can't
Often data scientists have to switch between different tools
for operating on datasets of different sizes. Processing large dataframes with pandas
is slow, and pandas does not support working with dataframes that are too large to fit
into the available memory. As a result, pandas workflows that work well
for prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size
of your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably
work with hundreds of GBs without worrying about substantial slowdown or memory errors.
With [cluster](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html)
and [out of core](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html)
support, Modin is a DataFrame library with both great single-node performance and high
scalability in a cluster.
#### Modin Architecture
We designed [Modin's architecture](https://modin.readthedocs.io/en/latest/development/architecture.html)
to be modular so we can plug in different components as they develop and improve:
### Other Resources
#### Getting Started with Modin
- [Documentation](https://modin.readthedocs.io/en/latest/)
- [10-min Quickstart Guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html)
- [Examples and Tutorials](https://modin.readthedocs.io/en/latest/getting_started/examples.html)
- [Videos and Blogposts](https://modin.readthedocs.io/en/latest/getting_started/examples.html#talks-podcasts)
- [Benchmarking Modin](https://modin.readthedocs.io/en/latest/usage_guide/benchmarking.html)
#### Modin Community
- [Slack](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA)
- [Twitter](https://twitter.com/modin_project)
- [Mailing List](https://groups.google.com/g/modin-dev)
- [GitHub Issues](https://github.com/modin-project/modin/issues)
- [StackOverflow](https://stackoverflow.com/questions/tagged/modin)
#### Learn More about Modin
- [Frequently Asked Questions (FAQs)](https://modin.readthedocs.io/en/latest/getting_started/faq.html)
- [Troubleshooting Guide](https://modin.readthedocs.io/en/latest/getting_started/troubleshooting.html)
- [Development Guide](https://modin.readthedocs.io/en/latest/development/index.html)
- Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:
- [Flexible Rule-Based Decomposition and Metadata Independence in Modin](https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) (VLDB 2021)
- [Dataframe Systems: Theory, Architecture, and Implementation](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.pdf) (PhD Dissertation 2021)
- [Towards Scalable Dataframe Systems](https://arxiv.org/pdf/2001.00888.pdf) (VLDB 2020)
#### Getting Involved
***`modin.pandas` is currently under active development. Requests and contributions are welcome!***
For more information on how to contribute to Modin, check out the
[Modin Contribution Guide](https://modin.readthedocs.io/en/latest/development/contributing.html).
### License
[Apache License 2.0](LICENSE)
================================================
FILE: asv_bench/README.md
================================================
# Modin ASV benchmarks
## Here are some scenarios in which [ASV](https://asv.readthedocs.io/en/stable/index.html) can be used:
* Check the impact of the new patch on the performance of a certain set of operations:
`asv continuous -f 1.05 src/main HEAD -b TimeGroupBy --launch-method=spawn`
* Check for presence of errors inside of benchmarks after changing them or writing new ones:
`asv run --quick --show-stderr --python=same --launch-method=spawn`
* Run entire benchmark suite to get the current times:
`asv run --launch-method=spawn`
* Check the range of commits for performance degradation:
```
asv run [start_hash]..[end_hash] --launch-method=spawn
asv publish
asv preview
```
For more consistent results, you may need to use the following parameters which
description is in [ASV docs](https://asv.readthedocs.io/en/stable/benchmarks.html?highlight=sample_time#timing-benchmarks):
* `-a sample_time=1`
* `-a warmup_time=1`
### Notes about using Modin on Ray with Asv:
* `--launch-method=forkserver` is not working;
* Each set of parameters for each test is launched in its own process, which brings
a large overhead, since for each process redis server and other necessary processes
from ray initialization are started and destroyed.
## Adding new benchmark
Basic information on writing benchmarks is present [in ASV documentation](https://asv.readthedocs.io/en/stable/writing_benchmarks.html)
Benchmarks from `benchmarks/benchmarks.py`, `benchmarks/scalability/scalability_benchmarks.py` or `benchmarks/io/csv.py`
could be used as a starting point.
Requirements:
* the benchmark should be able to run both on Modin and on Pandas when the appropriate value
of the environment variable `MODIN_ASV_USE_IMPL` is selected.
* the size of the benchmark dataset should depend on the environment variable `MODIN_TEST_DATASET_SIZE`.
## Changing existing benchmark
It should be remembered that the hash calculated from the benchmark source code is used to display the results.
When changing the benchmark, the old results will no longer be displayed in the dashboard. In general, this is the correct
behavior so as not to get a situation when incomparable numbers are displayed in the dashboard.
But it should be noted that there could be changes in the source code when it is still correct to compare
the "before" and "after" versions, for example, name of a variable changed, comment added, etc.
In this case you must either run a new version of the benchmark for all the commits ever accounted for or manually change
the hash in the corresponding result files.
## Pipeline for displaying results in a dashboard
Step 1: checking benchmarks for validity, runs in PRs CI.
During the test, the benchmarks are run once on small data.
The implementation can be found in `test-asv-benchmarks` job of [ci.yml](https://github.com/modin-project/modin/blob/main/.github/workflows/ci.yml)
Step 2: running benchmarks with saving the results in [modin-bench@master](https://github.com/modin-project/modin-bench).
The launch takes place on internal server using specific TeamCity configuration.
The description of the server can be found in the ["Benchmark list"](https://modin.org/modin-bench/#summarylist?sort=0&dir=asc) tab,
on the left when you hover the mouse over the machine name.
This step starts as scheduled (now every half hour), subject to the presence of new commits in the Modin `main` branch.
Command to run benchmarks: `asv run HASHFILE:hashfile.txt --show-stderr --machine xeon-e5 --launch-method=spawn`.
In the file `hashfile.txt` is the last modin commit hash.
Writing to a `modin-bench@master` triggers 3 step of the pipeline.
Step 3: converting the results to html representation, which is saved in [modin-bench@gh-pages](https://github.com/modin-project/modin-bench)
The implementation can be found in `deploy-gh-pages` job of [push.yml](https://github.com/modin-project/modin-bench/blob/master/.github/workflows/push.yml)
Basic actions for step 2:
* setup environment variable:
* export MODIN_TEST_DATASET=Big
* export MODIN_CPUS=44
* setup git client
* prepare json file with machine description
* This file should be placed in the user's home directory.
* ASV does not always automatically create the file with the description of the machine correctly (e.g. due to being run in a container).
It is recommended to create a file using [asv machine](https://asv.readthedocs.io/en/stable/commands.html?highlight=machine%20description#asv-machine) command, and manually check the result.
[Example](https://github.com/modin-project/modin-bench/blob/master/results/xeon-e5/machine.json)
* copy old result to folder where new result will appear
(conflict resolution will be performed by ASV itself instead of git)
* push performance result to modin-bench repository
================================================
FILE: asv_bench/asv.conf.dask.json
================================================
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,
// The name of the project being benchmarked
"project": "modin",
// The project's homepage
"project_url": "https://modin.readthedocs.io/",
// The URL or local path of the source code repository for the
// project being benchmarked
"repo": "..",
// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["main"],
// Customizable commands for building, installing, and
// uninstalling the project. See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[dask]"],
// The tool to use to create environments. May be "conda",
// "virtualenv" or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",
// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
"install_timeout": 6000,
// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/modin-project/modin/commit/",
// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.9"],
// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": ".asv/results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": ".asv/html",
}
================================================
FILE: asv_bench/asv.conf.json
================================================
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,
// The name of the project being benchmarked
"project": "modin",
// The project's homepage
"project_url": "https://modin.readthedocs.io/",
// The URL or local path of the source code repository for the
// project being benchmarked
"repo": "..",
// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["main"],
// Customizable commands for building, installing, and
// uninstalling the project. See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[ray]"],
// The tool to use to create environments. May be "conda",
// "virtualenv" or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",
// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
"install_timeout": 6000,
// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/modin-project/modin/commit/",
// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.9"],
// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": ".asv/results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": ".asv/html",
}
================================================
FILE: asv_bench/asv.conf.unidist.json
================================================
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,
// The name of the project being benchmarked
"project": "modin",
// The project's homepage
"project_url": "https://modin.readthedocs.io/",
// The URL or local path of the source code repository for the
// project being benchmarked
"repo": "..",
// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["main"],
// Customizable commands for building, installing, and
// uninstalling the project. See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[unidist]"],
// The tool to use to create environments. May be "conda",
// "virtualenv" or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",
// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
"install_timeout": 6000,
// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/modin-project/modin/commit/",
// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.9"],
// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": ".asv/results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": ".asv/html",
}
================================================
FILE: asv_bench/benchmarks/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin benchmarks."""
================================================
FILE: asv_bench/benchmarks/benchmarks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""General Modin benchmarks."""
# define `MODIN_CPUS` env var to control the number of partitions
# it should be defined before modin.pandas import (in case of using os.environ)
# define `MODIN_ASV_USE_IMPL` env var to choose library for using in performance
# measurements
import math
import numpy as np
from .utils import (
GROUPBY_NGROUPS,
IMPL,
RAND_HIGH,
RAND_LOW,
execute,
gen_nan_data,
generate_dataframe,
get_benchmark_shapes,
random_booleans,
random_columns,
random_string,
translator_groupby_ngroups,
)
class BaseTimeGroupBy:
def setup(self, shape, ngroups=5, groupby_ncols=1):
ngroups = translator_groupby_ngroups(ngroups, shape)
self.df, self.groupby_columns = generate_dataframe(
"int",
*shape,
RAND_LOW,
RAND_HIGH,
groupby_ncols,
count_groups=ngroups,
)
class TimeGroupByMultiColumn(BaseTimeGroupBy):
param_names = ["shape", "ngroups", "groupby_ncols"]
params = [
get_benchmark_shapes("TimeGroupByMultiColumn"),
GROUPBY_NGROUPS,
[6],
]
def time_groupby_agg_quan(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).agg("quantile"))
def time_groupby_agg_mean(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).apply(lambda df: df.mean()))
class TimeGroupByDefaultAggregations(BaseTimeGroupBy):
param_names = ["shape", "ngroups"]
params = [
get_benchmark_shapes("TimeGroupByDefaultAggregations"),
GROUPBY_NGROUPS,
]
def time_groupby_count(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).count())
def time_groupby_size(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).size())
def time_groupby_sum(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).sum())
def time_groupby_mean(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).mean())
class TimeGroupByDictionaryAggregation(BaseTimeGroupBy):
param_names = ["shape", "ngroups", "operation_type"]
params = [
get_benchmark_shapes("TimeGroupByDictionaryAggregation"),
GROUPBY_NGROUPS,
["reduce", "aggregation"],
]
operations = {
"reduce": ["sum", "count", "prod"],
"aggregation": ["quantile", "std", "median"],
}
def setup(self, shape, ngroups, operation_type):
super().setup(shape, ngroups)
self.cols_to_agg = self.df.columns[1:4]
operations = self.operations[operation_type]
self.agg_dict = {
c: operations[i % len(operations)] for i, c in enumerate(self.cols_to_agg)
}
def time_groupby_dict_agg(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).agg(self.agg_dict))
class TimeJoin:
param_names = ["shapes", "how", "sort"]
params = [
get_benchmark_shapes("TimeJoin"),
["left", "inner"],
[False],
]
def setup(self, shapes, how, sort):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_join(self, shapes, how, sort):
# join dataframes on index to get the predictable shape
execute(self.df1.join(self.df2, how=how, lsuffix="left_", sort=sort))
class TimeJoinStringIndex:
param_names = ["shapes", "sort"]
params = [
get_benchmark_shapes("TimeJoinStringIndex"),
[True, False],
]
def setup(self, shapes, sort):
assert shapes[0] % 100 == 0, "implementation restriction"
level1 = IMPL.Index([f"i-{i}" for i in range(10)], dtype=object).values
level2 = IMPL.Index(
[f"i-{i}" for i in range(shapes[0] // 100)], dtype=object
).values
codes1 = np.arange(10).repeat(shapes[0] // 100)
codes2 = np.tile(np.arange(shapes[0] // 100), 10)
index2 = IMPL.MultiIndex(levels=[level1, level2], codes=[codes1, codes2])
self.df_multi = IMPL.DataFrame(
np.random.randn(len(index2), 4), index=index2, columns=["A", "B", "C", "D"]
)
self.key1 = np.tile(level1.take(codes1), 10)
self.key2 = np.tile(level2.take(codes2), 10)
self.df = generate_dataframe("int", *shapes, RAND_LOW, RAND_HIGH)
# just to keep source shape
self.df = self.df.drop(columns=self.df.columns[-2:])
self.df["key1"] = self.key1
self.df["key2"] = self.key2
execute(self.df)
self.df_key1 = IMPL.DataFrame(
np.random.randn(len(level1), 4), index=level1, columns=["A", "B", "C", "D"]
)
self.df_key2 = IMPL.DataFrame(
np.random.randn(len(level2), 4), index=level2, columns=["A", "B", "C", "D"]
)
def time_join_dataframe_index_multi(self, shapes, sort):
execute(self.df.join(self.df_multi, on=["key1", "key2"], sort=sort))
def time_join_dataframe_index_single_key_bigger(self, shapes, sort):
execute(self.df.join(self.df_key2, on="key2", sort=sort))
def time_join_dataframe_index_single_key_small(self, shapes, sort):
execute(self.df.join(self.df_key1, on="key1", sort=sort))
class TimeMergeDefault:
param_names = ["shapes", "how", "sort"]
params = [
get_benchmark_shapes("TimeMergeDefault"),
["left", "inner"],
[True, False],
]
def setup(self, shapes, how, sort):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_merge(self, shapes, how, sort):
execute(IMPL.merge(self.df1, self.df2, how=how, sort=sort))
class TimeMerge:
param_names = ["shapes", "how", "sort"]
params = [
get_benchmark_shapes("TimeMerge"),
["left", "inner"],
[True, False],
]
def setup(self, shapes, how, sort):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_merge(self, shapes, how, sort):
# merge dataframes by index to get the predictable shape
execute(
self.df1.merge(
self.df2, left_index=True, right_index=True, how=how, sort=sort
)
)
def time_merge_dataframe_empty_right(self, shapes, how, sort):
# Getting an empty dataframe using `iloc` should be very fast,
# so the impact on the time of the merge operation should be negligible.
execute(IMPL.merge(self.df1, self.df2.iloc[:0], how=how, sort=sort))
def time_merge_dataframe_empty_left(self, shapes, how, sort):
# Getting an empty dataframe using `iloc` should be very fast,
# so the impact on the time of the merge operation should be negligible.
execute(IMPL.merge(self.df1.iloc[:0], self.df2, how=how, sort=sort))
class TimeMergeCategoricals:
param_names = ["shapes", "data_type"]
params = [
get_benchmark_shapes("MergeCategoricals"),
["object", "category"],
]
def setup(self, shapes, data_type):
assert len(shapes) == 2
assert shapes[1] == 2
size = (shapes[0],)
self.left = IMPL.DataFrame(
{
"X": np.random.choice(range(0, 10), size=size),
"Y": np.random.choice(["one", "two", "three"], size=size),
}
)
self.right = IMPL.DataFrame(
{
"X": np.random.choice(range(0, 10), size=size),
"Z": np.random.choice(["jjj", "kkk", "sss"], size=size),
}
)
if data_type == "category":
self.left = self.left.assign(Y=self.left["Y"].astype("category"))
execute(self.left)
self.right = self.right.assign(Z=self.right["Z"].astype("category"))
execute(self.right)
def time_merge_categoricals(self, shapes, data_type):
execute(IMPL.merge(self.left, self.right, on="X"))
class TimeConcat:
param_names = ["shapes", "how", "axis", "ignore_index"]
params = [
get_benchmark_shapes("TimeConcat"),
["inner", "outer"],
[0, 1],
[True, False],
]
def setup(self, shapes, how, axis, ignore_index):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_concat(self, shapes, how, axis, ignore_index):
execute(
IMPL.concat(
[self.df1, self.df2], axis=axis, join=how, ignore_index=ignore_index
)
)
class TimeBinaryOp:
param_names = ["shapes", "binary_op", "axis"]
params = [
get_benchmark_shapes("TimeBinaryOp"),
["mul"],
[0, 1],
]
def setup(self, shapes, binary_op, axis):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
self.op = getattr(self.df1, binary_op)
def time_binary_op(self, shapes, binary_op, axis):
execute(self.op(self.df2, axis=axis))
class TimeBinaryOpSeries:
param_names = ["shapes", "binary_op"]
params = [
get_benchmark_shapes("TimeBinaryOpSeries"),
["mul"],
]
def setup(self, shapes, binary_op):
df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
self.series1 = df1[df1.columns[0]]
self.series2 = df2[df2.columns[0]]
self.op = getattr(self.series1, binary_op)
execute(self.series1)
execute(self.series2)
def time_binary_op_series(self, shapes, binary_op):
execute(self.op(self.series2))
class BaseTimeSetItem:
param_names = ["shape", "item_length", "loc", "is_equal_indices"]
@staticmethod
def get_loc(df, loc, axis, item_length):
locs_dict = {
"zero": 0,
"middle": len(df.axes[axis]) // 2,
"last": len(df.axes[axis]) - 1,
}
base_loc = locs_dict[loc]
range_based_loc = np.arange(
base_loc, min(len(df.axes[axis]), base_loc + item_length)
)
return (
(df.axes[axis][base_loc], base_loc)
if len(range_based_loc) == 1
else (df.axes[axis][range_based_loc], range_based_loc)
)
def setup(self, shape, item_length, loc, is_equal_indices):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH).copy()
self.loc, self.iloc = self.get_loc(
self.df, loc, item_length=item_length, axis=1
)
self.item = self.df[self.loc] + 1
self.item_raw = self.item.to_numpy()
if not is_equal_indices:
self.item.index = reversed(self.item.index)
class TimeSetItem(BaseTimeSetItem):
params = [
get_benchmark_shapes("TimeSetItem"),
[1],
["zero", "middle", "last"],
[True, False],
]
def time_setitem_qc(self, *args, **kwargs):
self.df[self.loc] = self.item
execute(self.df)
def time_setitem_raw(self, *args, **kwargs):
self.df[self.loc] = self.item_raw
execute(self.df)
class TimeInsert(BaseTimeSetItem):
params = [
get_benchmark_shapes("TimeInsert"),
[1],
["zero", "middle", "last"],
[True, False],
]
def time_insert_qc(self, *args, **kwargs):
self.df.insert(loc=self.iloc, column=random_string(), value=self.item)
execute(self.df)
def time_insert_raw(self, *args, **kwargs):
self.df.insert(loc=self.iloc, column=random_string(), value=self.item_raw)
execute(self.df)
class TimeArithmetic:
param_names = ["shape", "axis"]
params = [
get_benchmark_shapes("TimeArithmetic"),
[0, 1],
]
def setup(self, shape, axis):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
def time_sum(self, shape, axis):
execute(self.df.sum(axis=axis))
def time_count(self, shape, axis):
execute(self.df.count(axis=axis))
def time_median(self, shape, axis):
execute(self.df.median(axis=axis))
def time_nunique(self, shape, axis):
execute(self.df.nunique(axis=axis))
def time_apply(self, shape, axis):
execute(self.df.apply(lambda df: df.sum(), axis=axis))
def time_mean(self, shape, axis):
execute(self.df.mean(axis=axis))
def time_mode(self, shape, axis):
execute(self.df.mode(axis=axis))
def time_add(self, shape, axis):
execute(self.df.add(2, axis=axis))
def time_mul(self, shape, axis):
execute(self.df.mul(2, axis=axis))
def time_mod(self, shape, axis):
execute(self.df.mod(2, axis=axis))
def time_abs(self, shape, axis):
execute(self.df.abs())
def time_aggregate(self, shape, axis):
execute(self.df.aggregate(lambda df: df.sum(), axis=axis))
def time_is_in(self, shape, axis):
execute(self.df.isin([0, 2]))
def time_transpose(self, shape, axis):
execute(self.df.transpose())
class TimeSortValues:
param_names = ["shape", "columns_number", "ascending_list"]
params = [
get_benchmark_shapes("TimeSortValues"),
[1, 2, 10, 100],
[False, True],
]
def setup(self, shape, columns_number, ascending_list):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.columns = random_columns(self.df.columns, columns_number)
self.ascending = (
random_booleans(columns_number)
if ascending_list
else bool(random_booleans(1)[0])
)
def time_sort_values(self, shape, columns_number, ascending_list):
execute(self.df.sort_values(self.columns, ascending=self.ascending))
class TimeDrop:
param_names = ["shape", "axis", "drop_ncols"]
params = [
get_benchmark_shapes("TimeDrop"),
[0, 1],
[1, 0.8],
]
def setup(self, shape, axis, drop_ncols):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
drop_count = (
int(len(self.df.axes[axis]) * drop_ncols)
if isinstance(drop_ncols, float)
else drop_ncols
)
self.labels = self.df.axes[axis][:drop_count]
def time_drop(self, shape, axis, drop_ncols):
execute(self.df.drop(self.labels, axis=axis))
class TimeHead:
param_names = ["shape", "head_count"]
params = [
get_benchmark_shapes("TimeHead"),
[5, 0.8],
]
def setup(self, shape, head_count):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.head_count = (
int(head_count * len(self.df.index))
if isinstance(head_count, float)
else head_count
)
def time_head(self, shape, head_count):
execute(self.df.head(self.head_count))
class TimeTail:
param_names = ["shape", "tail_count"]
params = [
get_benchmark_shapes("TimeTail"),
[5, 0.8],
]
def setup(self, shape, tail_count):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.tail_count = (
int(tail_count * len(self.df.index))
if isinstance(tail_count, float)
else tail_count
)
def time_tail(self, shape, tail_count):
execute(self.df.tail(self.tail_count))
class TimeExplode:
param_names = ["shape"]
params = [
get_benchmark_shapes("TimeExplode"),
]
def setup(self, shape):
self.df = generate_dataframe(
"int", *shape, RAND_LOW, RAND_HIGH, gen_unique_key=True
)
def time_explode(self, shape):
execute(self.df.explode("col1"))
class TimeFillnaSeries:
param_names = ["value_type", "shape", "limit"]
params = [
["scalar", "dict", "Series"],
get_benchmark_shapes("TimeFillnaSeries"),
[None, 0.8],
]
def setup(self, value_type, shape, limit):
self.series = gen_nan_data(*shape)
if value_type == "scalar":
self.value = 18.19
elif value_type == "dict":
self.value = {k: k * 1.23 for k in range(shape[0])}
elif value_type == "Series":
self.value = IMPL.Series(
[k * 1.23 for k in range(shape[0])], index=IMPL.RangeIndex(shape[0])
)
else:
assert False
limit = int(limit * shape[0]) if limit else None
self.kw = {"value": self.value, "limit": limit}
def time_fillna(self, value_type, shape, limit):
execute(self.series.fillna(**self.kw))
def time_fillna_inplace(self, value_type, shape, limit):
self.series.fillna(inplace=True, **self.kw)
execute(self.series)
class TimeFillnaDataFrame:
param_names = ["value_type", "shape", "limit"]
params = [
["scalar", "dict", "DataFrame", "Series"],
get_benchmark_shapes("TimeFillnaDataFrame"),
[None, 0.8],
]
def setup(self, value_type, shape, limit):
self.df = gen_nan_data(*shape)
columns = self.df.columns
if value_type == "scalar":
self.value = 18.19
elif value_type == "dict":
self.value = {k: i * 1.23 for i, k in enumerate(columns)}
elif value_type == "Series":
self.value = IMPL.Series(
[i * 1.23 for i in range(len(columns))], index=columns
)
elif value_type == "DataFrame":
self.value = IMPL.DataFrame(
{
k: [i + j * 1.23 for j in range(shape[0])]
for i, k in enumerate(columns)
},
index=IMPL.RangeIndex(shape[0]),
columns=columns,
)
else:
assert False
limit = int(limit * shape[0]) if limit else None
self.kw = {"value": self.value, "limit": limit}
def time_fillna(self, value_type, shape, limit):
execute(self.df.fillna(**self.kw))
def time_fillna_inplace(self, value_type, shape, limit):
self.df.fillna(inplace=True, **self.kw)
execute(self.df)
class BaseTimeValueCounts:
def setup(self, shape, ngroups=5, subset=1):
ngroups = translator_groupby_ngroups(ngroups, shape)
self.df, self.subset = generate_dataframe(
"int",
*shape,
RAND_LOW,
RAND_HIGH,
groupby_ncols=subset,
count_groups=ngroups,
)
class TimeValueCountsFrame(BaseTimeValueCounts):
param_names = ["shape", "ngroups", "subset"]
params = [
get_benchmark_shapes("TimeValueCountsFrame"),
GROUPBY_NGROUPS,
[2, 10],
]
def time_value_counts(self, *args, **kwargs):
execute(self.df.value_counts(subset=self.subset))
class TimeValueCountsSeries(BaseTimeValueCounts):
param_names = ["shape", "ngroups", "bins"]
params = [
get_benchmark_shapes("TimeValueCountsSeries"),
GROUPBY_NGROUPS,
[None, 3],
]
def setup(self, shape, ngroups, bins):
super().setup(ngroups=ngroups, shape=shape)
self.df = self.df[self.subset[0]]
def time_value_counts(self, shape, ngroups, bins):
execute(self.df.value_counts(bins=bins))
class TimeIndexing:
param_names = ["shape", "indexer_type"]
params = [
get_benchmark_shapes("TimeIndexing"),
[
"bool_array",
"bool_series",
"scalar",
"slice",
"continuous_slice",
"numpy_array_take_all_values",
"python_list_take_10_values",
"function",
],
]
indexer_getters = {
"bool_array": lambda df: np.array([False, True] * (len(df) // 2)),
# This boolean-Series is a projection of the source frame, it shouldn't
# be reimported or triggered to execute:
"bool_series": lambda df: df.iloc[:, 0] > 50,
"scalar": lambda df: len(df) // 2,
"slice": lambda df: slice(0, len(df), 2),
"continuous_slice": lambda df: slice(len(df) // 2),
"numpy_array_take_all_values": lambda df: np.arange(len(df)),
"python_list_take_10_values": lambda df: list(range(min(10, len(df)))),
"function": lambda df: (lambda df: df.index[::-2]),
}
def setup(self, shape, indexer_type):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.indexer = self.indexer_getters[indexer_type](self.df)
if isinstance(self.indexer, (IMPL.Series, IMPL.DataFrame)):
# HACK: Triggering `dtypes` meta-data computation in advance,
# so it won't affect the `loc/iloc` time:
self.indexer.dtypes
def time_iloc(self, shape, indexer_type):
# Pandas doesn't implement `df.iloc[series boolean_mask]` and raises an exception on it.
# Replacing this with the semantically equivalent construction:
if indexer_type != "bool_series":
execute(self.df.iloc[self.indexer])
else:
execute(self.df[self.indexer])
def time_loc(self, shape, indexer_type):
execute(self.df.loc[self.indexer])
class TimeIndexingColumns:
param_names = ["shape"]
params = [get_benchmark_shapes("TimeIndexing")]
def setup(self, shape):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.numeric_indexer = [0, 1]
self.labels_indexer = self.df.columns[self.numeric_indexer].tolist()
def time_iloc(self, shape):
execute(self.df.iloc[:, self.numeric_indexer])
def time_loc(self, shape):
execute(self.df.loc[:, self.labels_indexer])
def time___getitem__(self, shape):
execute(self.df[self.labels_indexer])
class TimeMultiIndexing:
param_names = ["shape"]
params = [get_benchmark_shapes("TimeMultiIndexing")]
def setup(self, shape):
df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
index = IMPL.MultiIndex.from_product(
[df.index[: shape[0] // 2], ["bar", "foo"]]
)
columns = IMPL.MultiIndex.from_product(
[df.columns[: shape[1] // 2], ["buz", "fuz"]]
)
df.index = index
df.columns = columns
self.df = df.sort_index(axis=1)
def time_multiindex_loc(self, shape):
execute(
self.df.loc[
self.df.index[2] : self.df.index[-2],
self.df.columns[2] : self.df.columns[-2],
]
)
class TimeResetIndex:
param_names = ["shape", "drop", "level"]
params = [
get_benchmark_shapes("TimeResetIndex"),
[False, True],
[None, "level_1"],
]
def setup(self, shape, drop, level):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
if level:
index = IMPL.MultiIndex.from_product(
[self.df.index[: shape[0] // 2], ["bar", "foo"]],
names=["level_1", "level_2"],
)
self.df.index = index
def time_reset_index(self, shape, drop, level):
execute(self.df.reset_index(drop=drop, level=level))
class TimeAstype:
param_names = ["shape", "dtype", "astype_ncolumns"]
params = [
get_benchmark_shapes("TimeAstype"),
["float64", "category"],
["one", "all"],
]
def setup(self, shape, dtype, astype_ncolumns):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
if astype_ncolumns == "all":
self.astype_arg = dtype
elif astype_ncolumns == "one":
self.astype_arg = {"col1": dtype}
else:
raise ValueError(f"astype_ncolumns: {astype_ncolumns} isn't supported")
def time_astype(self, shape, dtype, astype_ncolumns):
execute(self.df.astype(self.astype_arg))
class TimeDescribe:
param_names = ["shape"]
params = [
get_benchmark_shapes("TimeDescribe"),
]
def setup(self, shape):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
def time_describe(self, shape):
execute(self.df.describe())
class TimeProperties:
param_names = ["shape"]
params = [
get_benchmark_shapes("TimeProperties"),
]
def setup(self, shape):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
def time_shape(self, shape):
return self.df.shape
def time_columns(self, shape):
return self.df.columns
def time_index(self, shape):
return self.df.index
class TimeIndexingNumericSeries:
param_names = ["shape", "dtype", "index_structure"]
params = [
get_benchmark_shapes("TimeIndexingNumericSeries"),
(np.int64, np.uint64, np.float64),
("unique_monotonic_inc", "nonunique_monotonic_inc"),
]
def setup(self, shape, dtype, index_structure):
N = shape[0]
indices = {
"unique_monotonic_inc": IMPL.Index(range(N), dtype=dtype),
"nonunique_monotonic_inc": IMPL.Index(
list(range(N // 100)) + [(N // 100) - 1] + list(range(N // 100, N - 1)),
dtype=dtype,
),
}
self.data = IMPL.Series(np.random.rand(N), index=indices[index_structure])
self.array = np.arange(N // 2)
self.index_to_query = N // 2
self.array_list = self.array.tolist()
execute(self.data)
def time_getitem_scalar(self, shape, index, index_structure):
# not calling execute as execute function fails for scalar
self.data[self.index_to_query]
def time_getitem_slice(self, shape, index, index_structure):
execute(self.data[: self.index_to_query])
def time_getitem_list_like(self, shape, index, index_structure):
execute(self.data[[self.index_to_query]])
def time_getitem_array(self, shape, index, index_structure):
execute(self.data[self.array])
def time_getitem_lists(self, shape, index, index_structure):
execute(self.data[self.array_list])
def time_iloc_array(self, shape, index, index_structure):
execute(self.data.iloc[self.array])
def time_iloc_list_like(self, shape, index, index_structure):
execute(self.data.iloc[[self.index_to_query]])
def time_iloc_scalar(self, shape, index, index_structure):
# not calling execute as execute function fails for scalar
self.data.iloc[self.index_to_query]
def time_iloc_slice(self, shape, index, index_structure):
execute(self.data.iloc[: self.index_to_query])
def time_loc_array(self, shape, index, index_structure):
execute(self.data.loc[self.array])
def time_loc_list_like(self, shape, index, index_structure):
execute(self.data.loc[[self.index_to_query]])
def time_loc_scalar(self, shape, index, index_structure):
self.data.loc[self.index_to_query]
def time_loc_slice(self, shape, index, index_structure):
execute(self.data.loc[: self.index_to_query])
class TimeReindex:
param_names = ["shape"]
params = [get_benchmark_shapes("TimeReindex")]
def setup(self, shape):
rows, cols = shape
rng = IMPL.date_range(start="1/1/1970", periods=rows, freq="1min")
self.df = IMPL.DataFrame(
np.random.rand(rows, cols), index=rng, columns=range(cols)
)
self.df["foo"] = "bar"
self.rng_subset = IMPL.Index(rng[::2])
self.df2 = IMPL.DataFrame(
index=range(rows), data=np.random.rand(rows, cols), columns=range(cols)
)
level1 = IMPL.Index(
[f"i-{i}" for i in range(rows // 10)], dtype=object
).values.repeat(10)
level2 = np.tile(
IMPL.Index([f"i-{i}" for i in range(10)], dtype=object).values, rows // 10
)
index = IMPL.MultiIndex.from_arrays([level1, level2])
self.s = IMPL.Series(np.random.randn(rows), index=index)
self.s_subset = self.s[::2]
self.s_subset_no_cache = self.s[::2].copy()
mi = IMPL.MultiIndex.from_product([rng[: len(rng) // 10], range(10)])
self.s2 = IMPL.Series(np.random.randn(len(mi)), index=mi)
self.s2_subset = self.s2[::2].copy()
execute(self.df), execute(self.df2)
execute(self.s), execute(self.s_subset)
execute(self.s2), execute(self.s2_subset)
execute(self.s_subset_no_cache)
def time_reindex_dates(self, shape):
execute(self.df.reindex(self.rng_subset))
def time_reindex_columns(self, shape):
execute(self.df2.reindex(columns=self.df.columns[1:5]))
def time_reindex_multiindex_with_cache(self, shape):
# MultiIndex._values gets cached (pandas specific)
execute(self.s.reindex(self.s_subset.index))
def time_reindex_multiindex_no_cache(self, shape):
# Copy to avoid MultiIndex._values getting cached (pandas specific)
execute(self.s.reindex(self.s_subset_no_cache.index.copy()))
def time_reindex_multiindex_no_cache_dates(self, shape):
# Copy to avoid MultiIndex._values getting cached (pandas specific)
execute(self.s2_subset.reindex(self.s2.index.copy()))
class TimeReindexMethod:
params = [
get_benchmark_shapes("TimeReindexMethod"),
["pad", "backfill"],
[IMPL.date_range, IMPL.period_range],
]
param_names = ["shape", "method", "constructor"]
def setup(self, shape, method, constructor):
N = shape[0]
self.idx = constructor("1/1/2000", periods=N, freq="1min")
self.ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]
execute(self.ts)
def time_reindex_method(self, shape, method, constructor):
execute(self.ts.reindex(self.idx, method=method))
class TimeFillnaMethodSeries:
params = [get_benchmark_shapes("TimeFillnaMethodSeries"), ["pad", "backfill"]]
param_names = ["shape", "method"]
def setup(self, shape, method):
N = shape[0]
self.idx = IMPL.date_range("1/1/2000", periods=N, freq="1min")
ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]
self.ts_reindexed = ts.reindex(self.idx)
self.ts_float32 = self.ts_reindexed.astype("float32")
execute(self.ts_reindexed), execute(self.ts_float32)
def time_reindexed(self, shape, method):
execute(self.ts_reindexed.fillna(method=method))
def time_float_32(self, shape, method):
execute(self.ts_float32.fillna(method=method))
class TimeFillnaMethodDataframe:
params = [get_benchmark_shapes("TimeFillnaMethodDataframe"), ["pad", "backfill"]]
param_names = ["shape", "method"]
def setup(self, shape, method):
self.idx = IMPL.date_range("1/1/2000", periods=shape[0], freq="1min")
df_ts = IMPL.DataFrame(np.random.randn(*shape), index=self.idx)[::2]
self.df_ts_reindexed = df_ts.reindex(self.idx)
self.df_ts_float32 = self.df_ts_reindexed.astype("float32")
execute(self.df_ts_reindexed), execute(self.df_ts_float32)
def time_reindexed(self, shape, method):
execute(self.df_ts_reindexed.fillna(method=method))
def time_float_32(self, shape, method):
execute(self.df_ts_float32.fillna(method=method))
class TimeLevelAlign:
params = [get_benchmark_shapes("TimeLevelAlign")]
param_names = ["shapes"]
def setup(self, shapes):
rows, cols = shapes[0]
rows_sqrt = round(math.sqrt(rows))
# the new number of rows may differ from the requested (slightly, so ok)
rows = rows_sqrt * rows_sqrt
self.index = IMPL.MultiIndex(
levels=[np.arange(10), np.arange(rows_sqrt), np.arange(rows_sqrt)],
codes=[
np.arange(10).repeat(rows),
np.tile(np.arange(rows_sqrt).repeat(rows_sqrt), 10),
np.tile(np.tile(np.arange(rows_sqrt), rows_sqrt), 10),
],
)
self.df1 = IMPL.DataFrame(
np.random.randn(len(self.index), cols), index=self.index
)
self.df2 = IMPL.DataFrame(np.random.randn(*shapes[1]))
execute(self.df1), execute(self.df2)
def time_align_level(self, shapes):
left, right = self.df1.align(self.df2, level=1, copy=False)
execute(left), execute(right)
def time_reindex_level(self, shapes):
# `reindex` returns the same result here as `align`.
# Approximately the same performance is expected.
execute(self.df2.reindex(self.index, level=1))
class TimeDropDuplicatesDataframe:
params = [get_benchmark_shapes("TimeDropDuplicatesDataframe")]
param_names = ["shape"]
def setup(self, shape):
rows, cols = shape
N = rows // 10
K = 10
data = {}
# dataframe would have cols-1 keys(strings) and one value(int) column
for col in range(cols - 1):
data["key" + str(col + 1)] = IMPL.Index(
[f"i-{i}" for i in range(N)], dtype=object
).values.repeat(K)
data["value"] = np.random.randn(N * K)
self.df = IMPL.DataFrame(data)
execute(self.df)
def time_drop_dups(self, shape):
execute(self.df.drop_duplicates(self.df.columns[:-1]))
def time_drop_dups_inplace(self, shape):
self.df.drop_duplicates(self.df.columns[:-1], inplace=True)
execute(self.df)
class TimeDropDuplicatesSeries:
params = [get_benchmark_shapes("TimeDropDuplicatesSeries")]
param_names = ["shape"]
def setup(self, shape):
rows = shape[0]
self.series = IMPL.Series(
np.tile(
IMPL.Index([f"i-{i}" for i in range(rows // 10)], dtype=object).values,
10,
)
)
execute(self.series)
def time_drop_dups(self, shape):
execute(self.series.drop_duplicates())
def time_drop_dups_string(self, shape):
self.series.drop_duplicates(inplace=True)
execute(self.series)
class TimeDatetimeAccessor:
params = [get_benchmark_shapes("TimeDatetimeAccessor")]
param_names = ["shape"]
def setup(self, shape):
self.series = IMPL.Series(
IMPL.timedelta_range("1 days", periods=shape[0], freq="h")
)
execute(self.series)
def time_dt_accessor(self, shape):
execute(self.series.dt)
def time_timedelta_days(self, shape):
execute(self.series.dt.days)
def time_timedelta_seconds(self, shape):
execute(self.series.dt.seconds)
class BaseCategories:
def setup(self, shape):
rows = shape[0]
arr = [f"s{i:04d}" for i in np.random.randint(0, rows // 10, size=rows)]
self.ts = IMPL.Series(arr).astype("category")
execute(self.ts)
class TimeSetCategories(BaseCategories):
params = [get_benchmark_shapes("TimeSetCategories")]
param_names = ["shape"]
def time_set_categories(self, shape):
execute(self.ts.cat.set_categories(self.ts.cat.categories[::2]))
class TimeRemoveCategories(BaseCategories):
params = [get_benchmark_shapes("TimeRemoveCategories")]
param_names = ["shape"]
def time_remove_categories(self, shape):
execute(self.ts.cat.remove_categories(self.ts.cat.categories[::2]))
class BaseReshape:
def setup(self, shape):
rows, cols = shape
k = 10
arrays = [
np.arange(rows // k).repeat(k),
np.roll(np.tile(np.arange(rows // k), k), 25),
]
index = IMPL.MultiIndex.from_arrays(arrays)
self.df = IMPL.DataFrame(np.random.randn(rows, cols), index=index)
execute(self.df)
class TimeStack(BaseReshape):
params = [get_benchmark_shapes("TimeStack")]
param_names = ["shape"]
def setup(self, shape):
super().setup(shape)
self.udf = self.df.unstack(1)
execute(self.udf)
def time_stack(self, shape):
execute(self.udf.stack())
class TimeUnstack(BaseReshape):
params = [get_benchmark_shapes("TimeUnstack")]
param_names = ["shape"]
def time_unstack(self, shape):
execute(self.df.unstack(1))
class TimeReplace:
params = [get_benchmark_shapes("TimeReplace")]
param_names = ["shape"]
def setup(self, shape):
rows, cols = shape
self.to_replace = {i: getattr(IMPL, "Timestamp")(i) for i in range(rows)}
self.df = IMPL.DataFrame(np.random.randint(rows, size=(rows, cols)))
execute(self.df)
def time_replace(self, shape):
execute(self.df.replace(self.to_replace))
class TimeGroups:
params = [get_benchmark_shapes("TimeGroups")]
param_names = ["shape"]
def setup(self, shape):
self.series = IMPL.Series(np.random.randint(0, 100, size=shape[0]))
execute(self.series)
# returns a pretty dict thus not calling execute
def time_series_groups(self, shape):
self.series.groupby(self.series).groups
# returns a dict thus not calling execute
def time_series_indices(self, shape):
self.series.groupby(self.series).indices
class TimeRepr:
params = [get_benchmark_shapes("TimeRepr")]
param_names = ["shape"]
def setup(self, shape):
self.df = IMPL.DataFrame(np.random.randn(*shape))
execute(self.df)
# returns a string thus not calling execute
def time_repr(self, shape):
repr(self.df)
class TimeMaskBool:
params = [get_benchmark_shapes("TimeMaskBool")]
param_names = ["shape"]
def setup(self, shape):
self.df = IMPL.DataFrame(np.random.randn(*shape))
self.mask = self.df < 0
execute(self.df), execute(self.mask)
def time_frame_mask(self, shape):
execute(self.df.mask(self.mask))
class TimeIsnull:
params = [get_benchmark_shapes("TimeIsnull")]
param_names = ["shape"]
def setup(self, shape):
sample = np.array([np.nan, 1.0])
data = np.random.choice(sample, (shape[0], shape[1]))
self.df = IMPL.DataFrame(data)
execute(self.df)
def time_isnull(self, shape):
execute(IMPL.isnull(self.df))
class TimeDropna:
params = (["all", "any"], [0, 1], get_benchmark_shapes("TimeDropna"))
param_names = ["how", "axis", "shape"]
def setup(self, how, axis, shape):
row, col = shape
self.df = IMPL.DataFrame(np.random.randn(row, col))
self.df.iloc[row // 20 : row // 10, col // 3 : col // 2] = np.nan
self.df["foo"] = "bar"
execute(self.df)
def time_dropna(self, how, axis, shape):
execute(self.df.dropna(how=how, axis=axis))
class TimeEquals:
params = [get_benchmark_shapes("TimeEquals")]
param_names = ["shape"]
def setup(self, shape):
self.df = IMPL.DataFrame(np.random.randn(*shape))
self.df.iloc[-1, -1] = np.nan
execute(self.df)
# returns a boolean thus not calling execute
def time_frame_float_equal(self, shape):
self.df.equals(self.df)
from .utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/io/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""IO Modin benchmarks."""
================================================
FILE: asv_bench/benchmarks/io/csv.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import numpy as np
from ..utils import (
ASV_USE_IMPL,
IMPL,
RAND_HIGH,
RAND_LOW,
execute,
generate_dataframe,
get_benchmark_shapes,
get_shape_id,
prepare_io_data,
)
class BaseReadCsv:
# test data file should be created only once
def setup_cache(self, test_filename="io_test_file"):
test_filenames = prepare_io_data(
test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)
)
return test_filenames
def setup(self, test_filenames, shape, *args, **kwargs):
# ray init
if ASV_USE_IMPL == "modin":
IMPL.DataFrame([])
self.shape_id = get_shape_id(shape)
class TimeReadCsvSkiprows(BaseReadCsv):
shapes = get_benchmark_shapes("TimeReadCsvSkiprows")
skiprows_mapping = {
"lambda_even_rows": lambda x: x % 2,
"range_uniform": np.arange(1, shapes[0][0] // 10),
"range_step2": np.arange(1, shapes[0][0], 2),
}
data_type = "str_int"
param_names = ["shape", "skiprows"]
params = [
shapes,
[None, "lambda_even_rows", "range_uniform", "range_step2"],
]
def setup(self, test_filenames, shape, skiprows):
super().setup(test_filenames, shape, skiprows)
self.skiprows = self.skiprows_mapping[skiprows] if skiprows else None
def time_skiprows(self, test_filenames, shape, skiprows):
execute(IMPL.read_csv(test_filenames[self.shape_id], skiprows=self.skiprows))
class TimeReadCsvTrueFalseValues(BaseReadCsv):
data_type = "true_false_int"
param_names = ["shape"]
params = [get_benchmark_shapes("TimeReadCsvTrueFalseValues")]
def time_true_false_values(self, test_filenames, shape):
execute(
IMPL.read_csv(
test_filenames[self.shape_id],
true_values=["Yes", "true"],
false_values=["No", "false"],
),
)
class TimeReadCsvNamesDtype:
shapes = get_benchmark_shapes("TimeReadCsvNamesDtype")
_dtypes_params = ["Int64", "Int64_Timestamp"]
_timestamp_columns = ["col1", "col2"]
param_names = ["shape", "names", "dtype"]
params = [
shapes,
["array-like"],
_dtypes_params,
]
def _get_file_id(self, shape, dtype):
return get_shape_id(shape) + dtype
def _add_timestamp_columns(self, df):
df = df.copy()
date_column = IMPL.date_range("2000", periods=df.shape[0], freq="ms")
for col in self._timestamp_columns:
df[col] = date_column
return df
def setup_cache(self, test_filename="io_test_file_csv_names_dtype"):
# filenames with a metadata of saved dataframes
cache = {}
for shape in self.shapes:
for dtype in self._dtypes_params:
df = generate_dataframe(
"int", *shape, RAND_LOW, RAND_HIGH, impl="pandas"
)
if dtype == "Int64_Timestamp":
df = self._add_timestamp_columns(df)
file_id = self._get_file_id(shape, dtype)
cache[file_id] = (
f"{test_filename}_{file_id}.csv",
df.columns.to_list(),
df.dtypes.to_dict(),
)
df.to_csv(cache[file_id][0], index=False)
return cache
def setup(self, cache, shape, names, dtype):
# ray init
if ASV_USE_IMPL == "modin":
IMPL.DataFrame([])
file_id = self._get_file_id(shape, dtype)
self.filename, self.names, self.dtype = cache[file_id]
self.parse_dates = None
if dtype == "Int64_Timestamp":
# cached version of dtype should not change
self.dtype = self.dtype.copy()
for col in self._timestamp_columns:
del self.dtype[col]
self.parse_dates = self._timestamp_columns
def time_read_csv_names_dtype(self, cache, shape, names, dtype):
execute(
IMPL.read_csv(
self.filename,
names=self.names,
header=0,
dtype=self.dtype,
parse_dates=self.parse_dates,
)
)
from ..utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/io/parquet.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
from ..utils import (
ASV_USE_IMPL,
IMPL,
execute,
get_benchmark_shapes,
get_shape_id,
prepare_io_data_parquet,
)
class TimeReadParquet:
shapes = get_benchmark_shapes("TimeReadParquet")
data_type = "str_int"
param_names = ["shape"]
params = [
shapes,
]
# test data file should be created only once
def setup_cache(self, test_filename="io_test_file"):
test_filenames = prepare_io_data_parquet(
test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)
)
return test_filenames
def setup(self, test_filenames, shape):
# ray init
if ASV_USE_IMPL == "modin":
IMPL.DataFrame([])
self.shape_id = get_shape_id(shape)
def time_read_parquet(self, test_filenames, shape):
execute(
IMPL.read_parquet(
test_filenames[self.shape_id],
)
)
from ..utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/scalability/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Benchmarks measuring how Modin performance scales when MODIN_CPUS are changed."""
================================================
FILE: asv_bench/benchmarks/scalability/scalability_benchmarks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""These benchmarks are supposed to be run only for modin, since they do not make sense for pandas."""
import modin.pandas as pd
try:
from modin.pandas.io import from_pandas
except ImportError:
from modin.pandas.utils import from_pandas
try:
from modin.pandas.io import to_numpy, to_pandas
except ImportError:
try:
from modin.utils import to_numpy, to_pandas
except ImportError:
# This provides compatibility with older versions of the Modin, allowing us to test old commits.
from modin.pandas.utils import to_pandas
import pandas
from ..utils import (
RAND_HIGH,
RAND_LOW,
execute,
gen_data,
generate_dataframe,
get_benchmark_shapes,
)
class TimeFromPandas:
param_names = ["shape", "cpus"]
params = [
get_benchmark_shapes("TimeFromPandas"),
[4, 16, 32],
]
def setup(self, shape, cpus):
self.data = pandas.DataFrame(gen_data("int", *shape, RAND_LOW, RAND_HIGH))
from modin.config import NPartitions
NPartitions.get = lambda: cpus
# trigger ray init
pd.DataFrame([])
def time_from_pandas(self, shape, cpus):
execute(from_pandas(self.data))
class TimeToPandas:
param_names = ["shape", "cpus"]
params = [
get_benchmark_shapes("TimeToPandas"),
[4, 16, 32],
]
def setup(self, shape, cpus):
from modin.config import NPartitions
NPartitions.get = lambda: cpus
self.data = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH, impl="modin")
def time_to_pandas(self, shape, cpus):
# to_pandas is already synchronous
to_pandas(self.data)
class TimeToNumPy:
param_names = ["shape", "cpus"]
params = [
get_benchmark_shapes("TimeToNumPy"),
[4, 16, 32],
]
def setup(self, shape, cpus):
from modin.config import NPartitions
NPartitions.get = lambda: cpus
self.data = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH, impl="modin")
def time_to_numpy(self, shape, cpus):
# to_numpy is already synchronous
to_numpy(self.data)
from ..utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/utils/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin benchmarks utils."""
from .common import (
IMPL,
execute,
gen_data,
gen_nan_data,
generate_dataframe,
get_shape_id,
prepare_io_data,
prepare_io_data_parquet,
random_booleans,
random_columns,
random_string,
setup,
translator_groupby_ngroups,
)
from .compatibility import ASV_USE_IMPL, ASV_USE_STORAGE_FORMAT
from .data_shapes import GROUPBY_NGROUPS, RAND_HIGH, RAND_LOW, get_benchmark_shapes
__all__ = [
"ASV_USE_IMPL",
"ASV_USE_STORAGE_FORMAT",
"RAND_LOW",
"RAND_HIGH",
"GROUPBY_NGROUPS",
"get_benchmark_shapes",
"IMPL",
"execute",
"get_shape_id",
"gen_data",
"gen_nan_data",
"generate_dataframe",
"prepare_io_data",
"prepare_io_data_parquet",
"random_string",
"random_columns",
"random_booleans",
"translator_groupby_ngroups",
"setup",
]
================================================
FILE: asv_bench/benchmarks/utils/common.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
The module contains the functionality that is used when benchmarking Modin commits.
In the case of using utilities from the main Modin code, there is a chance that when
benchmarking old commits, the utilities changed, which in turn can unexpectedly affect
the performance results, hence some utility functions are duplicated here.
"""
import logging
import uuid
from typing import Optional, Union
import numpy as np
import pandas
import modin.pandas
from .compatibility import ASV_DATASET_SIZE, ASV_USE_ENGINE, ASV_USE_IMPL
from .data_shapes import RAND_HIGH, RAND_LOW
POSSIBLE_IMPL = {
"modin": modin.pandas,
"pandas": pandas,
}
IMPL = POSSIBLE_IMPL[ASV_USE_IMPL]
def translator_groupby_ngroups(groupby_ngroups: Union[str, int], shape: tuple) -> int:
"""
Translate a string representation of the number of groups, into a number.
Parameters
----------
groupby_ngroups : str or int
Number of groups that will be used in `groupby` operation.
shape : tuple
Same as pandas.Dataframe.shape.
Returns
-------
int
"""
if ASV_DATASET_SIZE == "big":
if groupby_ngroups == "huge_amount_groups":
return min(shape[0] // 2, 5000)
return groupby_ngroups
else:
return groupby_ngroups
class weakdict(dict): # noqa: GL08
__slots__ = ("__weakref__",)
data_cache = dict()
dataframes_cache = dict()
def gen_nan_data(nrows: int, ncols: int) -> dict:
"""
Generate nan data with caching.
The generated data are saved in the dictionary and on a subsequent call,
if the keys match, saved data will be returned. Therefore, we need
to carefully monitor the changing of saved data and make its copy if needed.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
Returns
-------
modin.pandas.DataFrame or pandas.DataFrame or modin.pandas.Series or pandas.Series
DataFrame or Series with shape (nrows, ncols) or (nrows,), respectively.
"""
cache_key = (ASV_USE_IMPL, nrows, ncols)
if cache_key in data_cache:
return data_cache[cache_key]
logging.info("Generating nan data {} rows and {} columns".format(nrows, ncols))
if ncols > 1:
columns = [f"col{x}" for x in range(ncols)]
data = IMPL.DataFrame(np.nan, index=IMPL.RangeIndex(nrows), columns=columns)
elif ncols == 1:
data = IMPL.Series(np.nan, index=IMPL.RangeIndex(nrows))
else:
assert False, "Number of columns (ncols) should be >= 1"
data_cache[cache_key] = data
return data
def gen_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:
"""
Generate int data.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
"""
data = {
"col{}".format(i): np.random.randint(rand_low, rand_high, size=(nrows))
for i in range(ncols)
}
return data
def gen_str_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:
"""
Generate int data and string data.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
One of the columns with string values.
"""
data = gen_int_data(nrows, ncols, rand_low, rand_high).copy()
# convert values in arbitary column to string type
key = list(data.keys())[0]
data[key] = [f"str_{x}" for x in data[key]]
return data
def gen_true_false_int_data(nrows, ncols, rand_low, rand_high):
"""
Generate int data and string data "true" and "false" values.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
One half of the columns with integer values, another half - with "true" and
"false" string values.
"""
data = gen_int_data(nrows // 2, ncols // 2, rand_low, rand_high)
data_true_false = {
"tf_col{}".format(i): np.random.choice(
["Yes", "true", "No", "false"], size=(nrows - nrows // 2)
)
for i in range(ncols - ncols // 2)
}
data.update(data_true_false)
return data
def gen_data(
data_type: str,
nrows: int,
ncols: int,
rand_low: int,
rand_high: int,
) -> dict:
"""
Generate data with caching.
The generated data are saved in the dictionary and on a subsequent call,
if the keys match, saved data will be returned. Therefore, we need
to carefully monitor the changing of saved data and make its copy if needed.
Parameters
----------
data_type : {"int", "str_int", "true_false_int"}
Type of data generation.
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
Notes
-----
Returned data type depends on the `data_type` parameter in the next way:
- `data_type`=="int" - all columns will be contain only integer values;
- `data_type`=="str_int" some of the columns will be of string type;
- `data_type`=="true_false_int" half of the columns will be filled with
string values representing "true" and "false" values and another half - with
integers.
"""
type_to_generator = {
"int": gen_int_data,
"str_int": gen_str_int_data,
"true_false_int": gen_true_false_int_data,
}
cache_key = (data_type, nrows, ncols, rand_low, rand_high)
if cache_key in data_cache:
return data_cache[cache_key]
logging.info(
"Generating {} data {} rows and {} columns [{}-{}]".format(
data_type, nrows, ncols, rand_low, rand_high
)
)
assert data_type in type_to_generator
data_generator = type_to_generator[data_type]
data = data_generator(nrows, ncols, rand_low, rand_high)
data_cache[cache_key] = weakdict(data)
return data
def generate_dataframe(
data_type: str,
nrows: int,
ncols: int,
rand_low: int,
rand_high: int,
groupby_ncols: Optional[int] = None,
count_groups: Optional[int] = None,
gen_unique_key: bool = False,
cache_prefix: str = None,
impl: str = None,
) -> Union[modin.pandas.DataFrame, pandas.DataFrame]:
"""
Generate DataFrame with caching.
The generated dataframes are saved in the dictionary and on a subsequent call,
if the keys match, one of the saved dataframes will be returned. Therefore, we need
to carefully monitor that operations that change the dataframe work with its copy.
Parameters
----------
data_type : str
Type of data generation;
supported types: {"int", "str_int"}.
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
groupby_ncols : int, default: None
Number of columns for which `groupby` will be called in the future;
to get more stable performance results, we need to have the same number of values
in each group every benchmarking time.
count_groups : int, default: None
Count of groups in groupby columns.
gen_unique_key : bool, default: False
Generate `col1` column where all elements are unique.
cache_prefix : str, optional
Prefix to add to the cache key of the requested frame.
impl : str, optional
Implementation used to create the dataframe;
supported implemetations: {"modin", "pandas"}.
Returns
-------
modin.pandas.DataFrame or pandas.DataFrame [and list]
Notes
-----
The list of groupby columns names returns when groupby columns are generated.
"""
assert not (
(groupby_ncols is None) ^ (count_groups is None)
), "You must either specify both parameters 'groupby_ncols' and 'count_groups' or none of them."
if groupby_ncols and count_groups:
ncols -= groupby_ncols
if impl is None:
impl = ASV_USE_IMPL
cache_key = (
impl,
data_type,
nrows,
ncols,
rand_low,
rand_high,
groupby_ncols,
count_groups,
gen_unique_key,
)
if cache_prefix is not None:
cache_key = (cache_prefix, *cache_key)
if cache_key in dataframes_cache:
return dataframes_cache[cache_key]
logging.info(
"Allocating {} DataFrame {}: {} rows and {} columns [{}-{}]".format(
impl, data_type, nrows, ncols, rand_low, rand_high
)
)
data = gen_data(data_type, nrows, ncols, rand_low, rand_high)
if groupby_ncols and count_groups:
groupby_columns = [f"groupby_col{x}" for x in range(groupby_ncols)]
for groupby_col in groupby_columns:
data[groupby_col] = np.tile(np.arange(count_groups), nrows // count_groups)
if gen_unique_key:
data["col1"] = np.arange(nrows)
df = POSSIBLE_IMPL[impl].DataFrame(data)
if groupby_ncols and count_groups:
dataframes_cache[cache_key] = df, groupby_columns
return df, groupby_columns
dataframes_cache[cache_key] = df
return df
def random_string() -> str:
"""
Create a 36-character random string.
Returns
-------
str
"""
return str(uuid.uuid4())
def random_columns(df_columns: list, columns_number: int) -> list:
"""
Pick sublist of random columns from a given sequence.
Parameters
----------
df_columns : list
Columns to choose from.
columns_number : int
How many columns to pick.
Returns
-------
list
"""
return list(np.random.choice(df_columns, size=columns_number))
def random_booleans(number: int) -> list:
"""
Create random list of booleans with `number` elements.
Parameters
----------
number : int
Count of booleans in result list.
Returns
-------
list
"""
return list(np.random.choice([True, False], size=number))
def execute(df: Union[modin.pandas.DataFrame, pandas.DataFrame]):
"""
Make sure the calculations are finished.
Parameters
----------
df : modin.pandas.DataFrame or pandas.Datarame
DataFrame to be executed.
"""
if ASV_USE_IMPL == "modin":
partitions = df._query_compiler._modin_frame._partitions.flatten()
mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls
if len(partitions) and hasattr(mgr_cls, "wait_partitions"):
mgr_cls.wait_partitions(partitions)
return
# compatibility with old Modin versions
all(
map(
lambda partition: partition.drain_call_queue() or True,
partitions,
)
)
if ASV_USE_ENGINE == "ray":
from ray import wait
all(map(lambda partition: wait([partition._data]), partitions))
elif ASV_USE_ENGINE == "dask":
from dask.distributed import wait
all(map(lambda partition: wait(partition._data), partitions))
elif ASV_USE_ENGINE == "python":
pass
elif ASV_USE_IMPL == "pandas":
pass
def get_shape_id(shape: tuple) -> str:
"""
Join shape numbers into a string with `_` delimiters.
Parameters
----------
shape : tuple
Same as pandas.Dataframe.shape.
Returns
-------
str
"""
return "_".join([str(element) for element in shape])
def prepare_io_data(test_filename: str, data_type: str, shapes: list):
"""
Prepare data for IO tests with caching.
Parameters
----------
test_filename : str
Unique file identifier that is used to distinguish data
for different tests.
data_type : {"int", "str_int", "true_false_int"}
Type of data generation.
shapes : list
Data shapes to prepare.
Returns
-------
test_filenames : dict
Dictionary that maps dataset shape to the file on disk.
"""
test_filenames = {}
for shape in shapes:
shape_id = get_shape_id(shape)
test_filenames[shape_id] = f"{test_filename}_{shape_id}_{data_type}.csv"
df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl="pandas")
df.to_csv(test_filenames[shape_id], index=False)
return test_filenames
def prepare_io_data_parquet(test_filename: str, data_type: str, shapes: list):
"""
Prepare data for IO tests with caching.
Parameters
----------
test_filename : str
Unique file identifier that is used to distinguish data
for different tests.
data_type : "str_int"
Type of data generation.
shapes : list
Data shapes to prepare.
Returns
-------
test_filenames : dict
Dictionary that maps dataset shape to the file on disk.
"""
test_filenames = {}
for shape in shapes:
shape_id = get_shape_id(shape)
test_filenames[shape_id] = f"{test_filename}_{shape_id}_{data_type}.parquet"
df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl="pandas")
df.to_parquet(test_filenames[shape_id], index=False)
return test_filenames
def setup(*args, **kwargs): # noqa: GL08
# This function just needs to be imported into each benchmark file to
# set up the random seed before each function. ASV run it automatically.
# https://asv.readthedocs.io/en/latest/writing_benchmarks.html
np.random.seed(42)
================================================
FILE: asv_bench/benchmarks/utils/compatibility.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Compatibility layer for parameters used by ASV."""
import os
import modin.pandas as pd
try:
from modin.config import NPartitions
NPARTITIONS = NPartitions.get()
except ImportError:
NPARTITIONS = pd.DEFAULT_NPARTITIONS
try:
from modin.config import AsvImplementation, Engine, StorageFormat, TestDatasetSize
ASV_USE_IMPL = AsvImplementation.get()
ASV_DATASET_SIZE = TestDatasetSize.get() or "Small"
ASV_USE_ENGINE = Engine.get()
ASV_USE_STORAGE_FORMAT = StorageFormat.get()
except ImportError:
# The same benchmarking code can be run for different versions of Modin, so in
# case of an error importing important variables, we'll just use predefined values
ASV_USE_IMPL = os.environ.get("MODIN_ASV_USE_IMPL", "modin")
ASV_DATASET_SIZE = os.environ.get("MODIN_TEST_DATASET_SIZE", "Small")
ASV_USE_ENGINE = os.environ.get("MODIN_ENGINE", "Ray")
ASV_USE_STORAGE_FORMAT = os.environ.get("MODIN_STORAGE_FORMAT", "Pandas")
ASV_USE_IMPL = ASV_USE_IMPL.lower()
ASV_DATASET_SIZE = ASV_DATASET_SIZE.lower()
ASV_USE_ENGINE = ASV_USE_ENGINE.lower()
ASV_USE_STORAGE_FORMAT = ASV_USE_STORAGE_FORMAT.lower()
assert ASV_USE_IMPL in ("modin", "pandas")
assert ASV_DATASET_SIZE in ("big", "small")
assert ASV_USE_ENGINE in ("ray", "dask", "python", "unidist")
assert ASV_USE_STORAGE_FORMAT in ("pandas")
================================================
FILE: asv_bench/benchmarks/utils/data_shapes.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Define data shapes."""
import json
import os
from .compatibility import ASV_DATASET_SIZE
RAND_LOW = 0
RAND_HIGH = 100
BINARY_OP_DATA_SIZE = {
"big": [
[[5000, 5000], [5000, 5000]],
# the case extremely inefficient
# [[20, 500_000], [10, 1_000_000]],
[[500_000, 20], [1_000_000, 10]],
],
"small": [[[250, 250], [250, 250]], [[10_000, 20], [25_000, 10]]],
}
UNARY_OP_DATA_SIZE = {
"big": [
[5000, 5000],
# the case extremely inefficient
# [10, 1_000_000],
[1_000_000, 10],
],
"small": [[250, 250], [10_000, 10]],
}
SERIES_DATA_SIZE = {
"big": [[100_000, 1]],
"small": [[10_000, 1]],
}
BINARY_OP_SERIES_DATA_SIZE = {
"big": [
[[500_000, 1], [1_000_000, 1]],
[[500_000, 1], [500_000, 1]],
],
"small": [[[5_000, 1], [10_000, 1]]],
}
DEFAULT_GROUPBY_NGROUPS = {
"big": [100, "huge_amount_groups"],
"small": [5],
}
GROUPBY_NGROUPS = DEFAULT_GROUPBY_NGROUPS[ASV_DATASET_SIZE]
_DEFAULT_CONFIG_T = [
(
UNARY_OP_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeGroupByMultiColumn",
"TimeGroupByDefaultAggregations",
"TimeGroupByDictionaryAggregation",
"TimeSetItem",
"TimeInsert",
"TimeArithmetic",
"TimeSortValues",
"TimeDrop",
"TimeHead",
"TimeTail",
"TimeExplode",
"TimeFillna",
"TimeFillnaDataFrame",
"TimeValueCountsFrame",
"TimeValueCountsSeries",
"TimeIndexing",
"TimeMultiIndexing",
"TimeResetIndex",
"TimeAstype",
"TimeDescribe",
"TimeProperties",
"TimeReindex",
"TimeReindexMethod",
"TimeFillnaMethodDataframe",
"TimeDropDuplicatesDataframe",
"TimeStack",
"TimeUnstack",
"TimeRepr",
"TimeMaskBool",
"TimeIsnull",
"TimeDropna",
"TimeEquals",
# IO benchmarks
"TimeReadCsvSkiprows",
"TimeReadCsvTrueFalseValues",
"TimeReadCsvNamesDtype",
"TimeReadParquet",
# Scalability benchmarks
"TimeFromPandas",
"TimeToPandas",
"TimeToNumPy",
],
),
(
BINARY_OP_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeJoin",
"TimeMerge",
"TimeMergeDefault",
"TimeConcat",
"TimeAppend",
"TimeBinaryOp",
"TimeLevelAlign",
],
),
(
SERIES_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeFillnaSeries",
"TimeGroups",
"TimeIndexingNumericSeries",
"TimeFillnaMethodSeries",
"TimeDatetimeAccessor",
"TimeSetCategories",
"TimeRemoveCategories",
"TimeDropDuplicatesSeries",
],
),
(
BINARY_OP_SERIES_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeBinaryOpSeries",
],
),
]
DEFAULT_CONFIG = {}
DEFAULT_CONFIG["MergeCategoricals"] = (
[[10_000, 2]] if ASV_DATASET_SIZE == "big" else [[1_000, 2]]
)
DEFAULT_CONFIG["TimeJoinStringIndex"] = (
[[100_000, 64]] if ASV_DATASET_SIZE == "big" else [[1_000, 4]]
)
DEFAULT_CONFIG["TimeReplace"] = (
[[10_000, 2]] if ASV_DATASET_SIZE == "big" else [[1_000, 2]]
)
for config in (_DEFAULT_CONFIG_T,):
for _shape, _names in config:
DEFAULT_CONFIG.update({_name: _shape for _name in _names})
# Correct shapes in the case when the operation ended with a timeout error
if ASV_DATASET_SIZE == "big":
DEFAULT_CONFIG["TimeMergeDefault"] = [
[[1000, 1000], [1000, 1000]],
[[500_000, 20], [1_000_000, 10]],
]
DEFAULT_CONFIG["TimeLevelAlign"] = [
[[2500, 2500], [2500, 2500]],
[[250_000, 20], [500_000, 10]],
]
DEFAULT_CONFIG["TimeStack"] = [
[1500, 1500],
[100_000, 10],
]
DEFAULT_CONFIG["TimeUnstack"] = DEFAULT_CONFIG["TimeStack"]
CONFIG_FROM_FILE = None
def get_benchmark_shapes(bench_id: str):
"""
Get custom benchmark shapes from a json file stored in MODIN_ASV_DATASIZE_CONFIG.
If `bench_id` benchmark is not found in the file, then the default value will
be used.
Parameters
----------
bench_id : str
Unique benchmark identifier that is used to get shapes.
Returns
-------
list
Benchmark shapes.
"""
global CONFIG_FROM_FILE
if not CONFIG_FROM_FILE:
try:
from modin.config import AsvDataSizeConfig
filename = AsvDataSizeConfig.get()
except ImportError:
filename = os.environ.get("MODIN_ASV_DATASIZE_CONFIG", None)
if filename:
# should be json
with open(filename) as _f:
CONFIG_FROM_FILE = json.load(_f)
if CONFIG_FROM_FILE and bench_id in CONFIG_FROM_FILE:
return CONFIG_FROM_FILE[bench_id]
return DEFAULT_CONFIG[bench_id]
================================================
FILE: asv_bench/test/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
================================================
FILE: asv_bench/test/test_utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
from unittest.mock import Mock, mock_open, patch
import numpy as np
import pytest
from benchmarks.utils import data_shapes, execute, get_benchmark_shapes
import modin.pandas as pd
from modin.config import AsvDataSizeConfig
@pytest.mark.parametrize(
"asv_config_content, result",
[
(
'{"TimeJoin": [[[10, 10], [15, 15]], [[11, 11], [13, 13]]], \
"TimeGroupBy": [[11, 11], [13, 13]]}',
[
[
# binary shapes
[[10, 10], [15, 15]],
[[11, 11], [13, 13]],
],
[
# unary shapes
[11, 11],
[13, 13],
],
],
),
],
)
@patch.object(data_shapes, "CONFIG_FROM_FILE", new=None)
def test_get_benchmark_shapes(asv_config_content, result):
AsvDataSizeConfig.put("mock_filename")
with patch("builtins.open", mock_open(read_data=asv_config_content)):
assert result[0] == get_benchmark_shapes("TimeJoin")
assert result[1] == get_benchmark_shapes("TimeGroupBy")
@pytest.mark.parametrize(
"asv_config_content, result",
[
(
'{"TimeJoin": [[[10, 10], [15, 15]]]',
[[100, 100]],
),
],
)
@patch.object(data_shapes, "CONFIG_FROM_FILE", new=None)
def test_get_benchmark_shapes_default(asv_config_content, result):
AsvDataSizeConfig.put(None)
with patch.object(data_shapes, "DEFAULT_CONFIG", new={"TimeJoin": result}):
assert result == get_benchmark_shapes("TimeJoin")
def test_execute():
df = pd.DataFrame(np.random.rand(100, 64))
partitions = df._query_compiler._modin_frame._partitions.flatten()
mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls
with patch.object(mgr_cls, "wait_partitions", new=Mock()):
execute(df)
mgr_cls.wait_partitions.assert_called_once()
assert (mgr_cls.wait_partitions.call_args[0] == partitions).all()
================================================
FILE: ci/teamcity/Dockerfile.teamcity-ci
================================================
# Create images from this container like this (in modin repo root):
#
# git rev-parse HEAD > ci/teamcity/git-rev
#
# tar cf ci/teamcity/modin.tar .
#
# docker build --build-arg ENVIRONMENT=environment-dev.yml -t modin-project/teamcity-ci:${BUILD_NUMBER} -f ci/teamcity/Dockerfile.teamcity-ci ci/teamcity
FROM rayproject/ray:latest
ARG ENVIRONMENT=environment-dev.yml
ADD modin.tar /modin
ADD git-rev /modin/git-rev
WORKDIR /modin
RUN sudo chown -R ray /modin
# Make RUN commands use `bash --login`:
SHELL ["/bin/bash", "--login", "-c"]
# Initialize conda in bash config files:
RUN conda init bash
ENV PATH /home/ray/anaconda3/envs/modin/bin:$PATH
RUN conda config --set channel_priority strict
RUN conda update python -y
RUN conda env create -f ${ENVIRONMENT}
RUN conda install curl PyGithub
# Activate the environment, and make sure it's activated:
# The following line also removed conda initialization from
# ~/.bashrc so conda starts complaining that it should be
# initialized for bash. But it is necessary to do it because
# activation is not always executed when "docker exec" is used
# and then conda initialization overwrites PATH with its base
# environment where python doesn't have any packages installed.
RUN echo "conda activate modin" > ~/.bashrc
RUN echo "Make sure environment is activated"
RUN conda list -n modin
================================================
FILE: ci/teamcity/build-docker.py
================================================
import os
import sys
def execute_command(cmd):
status = os.system(cmd)
ec = os.WEXITSTATUS(status)
if ec != 0:
raise SystemExit('Command "{}" failed'.format(cmd))
if sys.platform.startswith("linux"):
execute_command("git rev-parse HEAD > git-rev")
execute_command(
"(cd ../.. && git archive -o ci/teamcity/modin.tar $(cat ci/teamcity/git-rev))"
)
base_image = "ray-project/deploy"
requirements = "requirements-dev.txt"
execute_command(
"docker build -f Dockerfile.modin-base --build-arg BASE_IMAGE={} -t modin-project/modin-base .".format(
base_image
)
)
else:
raise SystemExit(
"TeamCity CI in Docker containers is supported only on Linux at the moment."
)
execute_command(
"docker build -f Dockerfile.teamcity-ci --build-arg REQUIREMENTS={} -t modin-project/teamcity-ci .".format(
requirements
)
)
if sys.platform.startswith("linux"):
execute_command("rm ./modin.tar ./git-rev")
================================================
FILE: ci/teamcity/comment_on_pr.py
================================================
"""
Post the comment like the following to the PR:
```
:robot: TeamCity test results bot :robot:
```
"""
import os
import sys
from github import Github
# Check if this is a pull request or not based on the environment variable
try:
pr_id = int(os.environ["GITHUB_PR_NUMBER"].split("/")[-1])
except Exception:
sys.exit(0)
engine = os.environ["MODIN_ENGINE"]
header = """
TeamCity {} test results bot
\n\n""".format(
engine.title()
)
if engine == "ray":
pytest_outputs = ["ray_tests.log"]
elif engine == "dask":
pytest_outputs = ["dask_tests.log"]
elif engine == "python":
pytest_outputs = ["python_tests.log"]
else:
raise Exception("Unknown Engine, set `MODIN_ENGINE` environment variable")
full_comment = ""
# Do not include coverage info in PR comment
split_by_first = (
"----------- coverage: platform linux, python 3.7.5-final-0 -----------"
)
split_by_second = "--------------------------------------------------------------------------------------"
tests_failed = False
for out in pytest_outputs:
content = open(out, "r").read()
full_comment += "".join(
"".join(
[
i.split(split_by_first)[0],
i.split(split_by_first)[-1].split(split_by_second)[-1],
]
)
for i in content.split("+ python3 -m pytest ")
)
tests_failed = tests_failed or ("FAILURES" in full_comment)
if len(full_comment) > 65_000:
full_comment = (
full_comment[-65_000:] + "\n\nRemaining output truncated\n\n"
)
full_comment = "Tests Logs\n\n\n```\n" + full_comment
full_comment += "\n```\n\n\n"
if not tests_failed:
header += '
Tests PASSed
\n\n'
else:
header += '
Tests FAILed
\n\n'
full_comment = header + full_comment
token = os.environ["GITHUB_TOKEN"]
g = Github(token)
repo = g.get_repo("modin-project/modin")
pr = repo.get_pull(pr_id)
if any(
i.user.login == "modin-bot"
and "TeamCity {} test results bot".format(engine).lower() in i.body.lower()
for i in pr.get_issue_comments()
):
pr_comment_list = [
i
for i in list(pr.get_issue_comments())
if i.user.login == "modin-bot"
and "TeamCity {} test results bot".format(engine).lower() in i.body.lower()
]
assert len(pr_comment_list) == 1, "Too many comments from modin-bot already"
pr_comment_list[0].edit(full_comment)
else:
pr.create_issue_comment(full_comment)
================================================
FILE: codecov.yml
================================================
comment: false
coverage:
status:
project:
default:
branches:
- main
target: 85%
patch:
default:
target: 30%
================================================
FILE: contributing/contributing.md
================================================
# Modin dev onboarding
1. [Set up git](https://docs.github.com/en/get-started/quickstart/set-up-git)
1. [install anaconda](https://www.anaconda.com/products/individual#macos). Once installed,
you should reopen your terminal to find "(base)" next to your prompt: 
1. [Generate an SSH key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) for GitHub
1. Fork the [modin repo](https://github.com/modin-project/modin) on GitHub
1. Clone the forked repo in a local directory of your choice:
```
git clone ${PATH_TO_REPO}
```
where the path can be found here: 
4. Inside the cloned "modin" directory, add a remote branch called "upstream":
```
git remote add upstream git@github.com:modin-project/modin.git
```
where the upstream link comes from here: 
1. Fetch the upstream branch:
```
git fetch upstream
```
1. Set the default remote branch for your local main branch.
```
git branch --set-upstream-to=upstream/main main
```
1. Install modin from local source code, and install all its dependencies:
```
pip install -e ".[all]"
```
1. Install ipython:
```
pip install ipython
```
1. If you ever want to install modin at a release version (not the editable version from your machine):
```
pip install modin
```
1. If you want a specific version:
```
pip install modin==0.11
```
1. To upgrade modin to the newest available version:
```
pip install -U modin
```
1. Now go back to local modin.
```
pip install -e .
```
1. Try out modin in ipython:
```
ipython
import modin
modin.__version__
```
You should see the Modin version, which consists of the version, the last commit number, and the last commit hash.
1. List Modin versions:
```
git tag
```
1. Get a summary of a particular release:
```
git tag -l --format='%(contents)' 0.11.0
```
1. Check out the developer requirements in `requirements-dev.txt`. Install them with:
```
pip install -r requirements-dev.txt
```
1. Try a unit test:
```
pytest modin/tests/pandas/test_concat.py
```
1. [Add a GPG key](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-new-gpg-key-to-your-github-account ) to your Modin account. Your commits need to be signed with a GPG key. For mac, you can use [Mac GPG](https://gpgtools.org/).
1. (Optional) We recommend a few workflow settings:
1. If you use Visual Studio Code, auto-format with [black](https://black.readthedocs.io/en/stable/) every time you save changes:
1. Install [Microsoft's Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
1. Open your VSCode settings, in `Code -> Preferences -> Settings`.
1. Search for "python formatting provider" and select "black" from the dropdown menu.
1. Again in settings, search for "format on save" and enable the "Editor: Format on Save" option.
2. Add a pre-commit hook:
1. In your modin repository, copy [this pre-commit file](pre-commit) to `.git/hooks/pre-commit`
1. Every time you try to commit, git will try to run flake8 and mypy, and abort the commit if either one fails. This lets you make sure your commits pass these tests before you push to GitHub.
1. To bypass the pre-commit hook (e.g. if you don't want to create a pull request, or you already know your code will pass the tests), commit with the flag `--no-verify`.
================================================
FILE: contributing/pre-commit
================================================
#!/bin/sh
#
# Called by "git commit" with no arguments. The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
set -e
printf "running black. This script will preempt the commit if black fails.\n"
black --check --diff modin/ asv_bench/benchmarks scripts/doc_checker.py
printf 'black passed!\n'
printf "running isort. This script will preempt the commit if isort fails.\n"
isort . --check-only
printf 'isort passed!\n'
printf "running flake8. This script will preempt the commit if flake8 fails.\n"
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
printf "flake8 passed!\n"
printf "running mypy. This script will preempt the commit if mypy fails.\n"
mypy --config-file mypy.ini
printf "mypy passed!\n"
printf "pre-commit hook finished!\n"
================================================
FILE: docker/Dockerfile
================================================
FROM continuumio/miniconda3
RUN conda install -c conda-forge psutil setproctitle
RUN pip install modin
================================================
FILE: docs/_static/custom.js
================================================
document.addEventListener("DOMContentLoaded", function () {
var script = document.createElement("script");
script.type = "module";
script.id = "runllm-widget-script"
script.src = "https://widget.runllm.com";
script.setAttribute("runllm-keyboard-shortcut", "Mod+j"); // cmd-j or ctrl-j to open the widget.
script.setAttribute("runllm-name", "Modin");
script.setAttribute("runllm-position", "BOTTOM_RIGHT");
script.setAttribute("runllm-assistant-id", "164");
script.async = true;
document.head.appendChild(script);
});
================================================
FILE: docs/_templates/layout.html
================================================
{% extends "!layout.html" %}
{% block footer %} {{ super() }}
{% endblock %}
================================================
FILE: docs/conf.py
================================================
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/stable/config
import os
# -- Project information -----------------------------------------------------
import sys
import types
import ray
# stub ray.remote to be a no-op so it doesn't shadow docstrings
def noop_decorator(*args, **kwargs):
if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):
# This is the case where the decorator is just @ray.remote without parameters.
return args[0]
return lambda cls_or_func: cls_or_func
ray.remote = noop_decorator
# fake modules if they're missing
for mod_name in (
"xgboost",
"unidist",
"unidist.config",
):
try:
__import__(mod_name)
except ImportError:
sys.modules[mod_name] = types.ModuleType(
mod_name, f"fake {mod_name} for building docs"
)
if not hasattr(sys.modules["xgboost"], "Booster"):
sys.modules["xgboost"].Booster = type("Booster", (object,), {})
if not hasattr(sys.modules["unidist"], "remote"):
sys.modules["unidist"].remote = noop_decorator
if not hasattr(sys.modules["unidist"], "core"):
sys.modules["unidist"].core = type("core", (object,), {})
if not hasattr(sys.modules["unidist"].core, "base"):
sys.modules["unidist"].core.base = type("base", (object,), {})
if not hasattr(sys.modules["unidist"].core.base, "object_ref"):
sys.modules["unidist"].core.base.object_ref = type("object_ref", (object,), {})
if not hasattr(sys.modules["unidist"].core.base.object_ref, "ObjectRef"):
sys.modules["unidist"].core.base.object_ref.ObjectRef = type("ObjectRef", (object,), {})
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
import modin
from modin.config.__main__ import export_config_help
configs_file_path = os.path.abspath(
os.path.join(os.path.dirname(__file__), "flow/modin/configs_help.csv")
)
# Export configs help to create configs table in the docs/flow/modin/config.rst
export_config_help(configs_file_path)
project = "Modin"
copyright = "2018-2024, Modin Developers."
author = "Modin contributors"
# The short X.Y version
version = "{}".format(modin.__version__)
# The full version, including alpha/beta/rc tags
release = version
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.intersphinx",
"sphinx.ext.todo",
"sphinx.ext.mathjax",
"sphinx.ext.githubpages",
"sphinx.ext.graphviz",
"sphinxcontrib.plantuml",
"sphinx_issues",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = ".rst"
# The master toctree document.
master_doc = "index"
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = "en"
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
html_static_path = ["_static"]
html_js_files = ["custom.js"]
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
# -- Options for HTML output -------------------------------------------------
# Maps git branches to Sphinx themes
default_html_theme = "pydata_sphinx_theme"
current_branch = "nature"
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "pydata_sphinx_theme"
html_favicon = "img/MODIN_ver2.ico"
html_logo = "img/MODIN_ver2.png"
html_context = {"default_mode": "light"}
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {
"navbar_end": ["navbar-icon-links"],
"sidebarwidth": 270,
"collapse_navigation": False,
"navigation_depth": 4,
"show_toc_level": 2,
"github_url": "https://github.com/modin-project/modin",
"icon_links": [
{
"name": "PyPI",
"url": "https://pypi.org/project/modin",
"icon": "fab fa-python",
},
{
"name": "conda-forge",
"url": "https://anaconda.org/conda-forge/modin",
"icon": "fas fa-circle-notch",
},
{
"name": "Join the Slack",
"url": "https://modin.org/slack.html",
"icon": "fab fa-slack",
},
{
"name": "Mailing List",
"url": "https://groups.google.com/forum/#!forum/modin-dev",
"icon": "fas fa-envelope-square",
},
],
"navigation_with_keys": True,
}
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# The default pydata_sphinx_theme sidebar templates are
# sidebar-nav-bs.html and search-field.html.
html_sidebars = {}
issues_github_path = "modin-project/modin"
================================================
FILE: docs/contact.rst
================================================
Contact
=======
Slack
-----
Join our `Slack`_ community to connect with Modin users and contributors,
discuss, and ask questions about all things Modin-related.
Mailing List
------------
General questions, potential contributors, and ideas can be directed to the
`developer mailing list`_. It is an open Google Group, so feel free to join anytime! If
you are unsure about where to ask or post something, the mailing list is a good place to
ask as well.
Issues
------
Bug reports and feature requests can be directed to the issues_ page of the Modin
GitHub repo.
.. _Slack: https://modin.org/slack.html
.. _developer mailing list: https://groups.google.com/forum/#!forum/modin-dev
.. _issues: https://github.com/modin-project/modin/issues
================================================
FILE: docs/development/architecture.rst
================================================
System Architecture
===================
In this section, we will lay out the overall system architecture for
Modin, as well as go into detail about the component design, implementation and
other important details. This document also contains important reference
information for those interested in contributing new functionality, bugfixes
and enhancements.
High-Level Architectural View
-----------------------------
The diagram below outlines the general layered view to the components of Modin
with a short description of each major section of the documentation following.
.. image:: /img/modin_architecture.png
:align: center
Modin is logically separated into different layers that represent the hierarchy of a
typical Database Management System. Abstracting out each component allows us to
individually optimize and swap out components without affecting the rest of the system.
We can implement, for example, new compute kernels that are optimized for a certain type
of data and can simply plug it in to the existing infrastructure by implementing a small
interface. It can still be distributed by our choice of compute engine with the
logic internally.
System View
-----------
A top-down view of Modin’s architecture is detailed below:
.. image:: /img/10000_meter.png
:align: center
The user - Data Scientist interacts with the Modin system by sending interactive or
batch commands through API and Modin executes them using various execution
engines: Ray, Dask and MPI are currently supported.
Subsystem/Container View
------------------------
If we click down to the next level of details we will see that inside Modin the layered
architecture is implemented using several interacting components:
.. image:: /img/component_view.png
:align: center
For the simplicity the other execution systems - Dask and MPI are omitted and only Ray execution is shown.
* Dataframe subsystem is the backbone of the dataframe holding and query compilation. It is responsible for
dispatching the ingress/egress to the appropriate module, getting the pandas API and calling the query
compiler to convert calls to the internal intermediate Dataframe Algebra.
* Data Ingress/Egress Module is working in conjunction with Dataframe and Partitions subsystem to read data
split into partitions and send data into the appropriate node for storing.
* Query Planner is subsystem that translates the pandas API to intermediate Dataframe Algebra representation
DAG and performs an initial set of optimizations.
* Query Executor is responsible for getting the Dataframe Algebra DAG, performing further optimizations based
on a selected storage format and mapping or compiling the Dataframe Algebra DAG to and actual
execution sequence.
* Storage formats module is responsible for mapping the abstract operation to an actual executor call, e.g. pandas,
custom format.
* Orchestration subsystem is responsible for spawning and controlling the actual execution environment for the
selected execution. It spawns the actual nodes, fires up the execution environment, e.g. Ray, monitors the state
of executors and provides telemetry
Component View
--------------
User queries which perform data transformation, data ingress or data egress pass through the Modin components
detailed below. The path the query takes is mostly similar across execution systems.
Data Transformation
'''''''''''''''''''
.. image:: /img/generic_data_transform.svg
:align: center
Query Compiler
""""""""""""""
The :ref:`Query Compiler ` receives queries from the pandas API layer. The API layer is
responsible for ensuring a clean input to the Query Compiler. The Query Compiler must
have knowledge of the compute kernels and in-memory format of the data in order to
efficiently compile the query.
The Query Compiler is responsible for sending the compiled query to the Core Modin Dataframe.
In this design, the Query Compiler does not have information about where or when the
query will be executed, and gives the control of the partition layout to the Modin
Dataframe.
In the interest of reducing the pandas API, the Query Compiler layer closely follows the
pandas API, but cuts out a large majority of the repetition.
.. _auto-switch architecture:
Automatic Engine Switching and Casting
""""""""""""""""""""""""""""""""""""""
QueryCompilers which are derived from QueryCompilerCaster can participate in automatic casting when
different query compilers, representing different underlying engines, are used together in a
function. A relative "cost" of casting is used to determine which query compiler everything should
be moved to. Each query compiler must implement the functions, `move_to_cost`, `move_to_me_cost`,
`max_cost` and `stay_cost` to provide information and query costs associated with different decision
points in cost opimization. With the exception of `max_cost` these methods need to return a
QCCoercionCost in the range of 0-1000.
These functions have precise meanings:
* `move_to_cost` is the transmission cost of moving the data, including known serialization costs
from the perspective of that particular compiler. Colloquially, the question being asked of the
query compiler is, "What is the normalized cost of moving my data to the other engine?"
* `move_to_me_cost` is the execution cost for the data and operation on the proposed *destination*
query compiler. Since this method is called before the data has been migrated this is a class
method and the destination query_compiler may have very limited information on the possible cost
after migration. Factors that may be considered here include available memory, cpu, and the
unique characteristics of the engine. The question being asked is, "If this data were moved to
me, what would be the normalized execution cost to perform that operation?"
* `stay_cost` is the execution cost on the current query compilier ( where the data is ). The question
asked of the query compiler is, "If I were to keep this data on my engine, what would be the normalized
execution cost?"
* `max_cost` is the maximum cost allowed by this query compiler across all data movements. This method
sets a normalized upper bound for situations where multiple data frames from different engines all
need to move to the same engine. The value returned by this method can exceed
QCCoercionCost.COST_IMPOSSIBLE
There are generally two places where automatic casting is considered: When two or more DataFrames on
different engines are participating in an operation ( such as pd.concat ) or at registered functions
for particular engines through the `register_function_for_pre_op_switch` and
`register_function_for_post_op_switch` methods.
Core Modin Dataframe
""""""""""""""""""""
At this layer, operations can be performed lazily. Currently, Modin executes most
operations eagerly in an attempt to behave as pandas does. Some operations, e.g.
``transpose`` are expensive and create full copies of the data in-memory. In these
cases, we can wait until another operation triggers computation. In the future, we plan
to add additional query planning and laziness to Modin to ensure that queries are
performed efficiently.
The structure of the Core Modin Dataframe is extensible, such that any operation that could
be better optimized for a given execution can be overridden and optimized in that way.
This layer has a significantly reduced API from the QueryCompiler and the user-facing
API. Each of these APIs represents a single way of performing a given operation or
behavior.
Core Modin Dataframe API
""""""""""""""""""""""""
More documentation can be found internally in the code_. This API is not complete, but
represents an overwhelming majority of operations and behaviors.
This API can be implemented by other distributed/parallel DataFrame libraries and
plugged in to Modin as well. Create an issue_ or discuss
on our `Slack `_ for more information!
The :doc:`Core Modin Dataframe ` is responsible for the data layout and shuffling, partitioning,
and serializing the tasks that get sent to each partition. Other implementations of the
Modin Dataframe interface will have to handle these as well.
Partition Manager
"""""""""""""""""
The Partition Manager can change the size and shape of the partitions based on the type
of operation. For example, certain operations are complex and require access to an
entire column or row. The Partition Manager can convert the block partitions to row
partitions or column partitions. This gives Modin the flexibility to perform operations
that are difficult in row-only or column-only partitioning schemas.
Another important component of the Partition Manager is the serialization and shipment
of compiled queries to the Partitions. It maintains metadata for the length and width of
each partition, so when operations only need to operate on or extract a subset of the
data, it can ship those queries directly to the correct partition. This is particularly
important for some operations in pandas which can accept different arguments and
operations for different columns, e.g. ``fillna`` with a dictionary.
This abstraction separates the actual data movement and function application from the
Dataframe layer to keep the Core Dataframe API small and separately optimize the data
movement and metadata management.
Partitions
""""""""""
Partitions are responsible for managing a subset of the Dataframe. As mentioned
below, the Dataframe is partitioned both row and column-wise. This gives Modin
scalability in both directions and flexibility in data layout. There are a number of
optimizations in Modin that are implemented in the partitions. Partitions are specific
to the execution framework and in-memory format of the data, allowing Modin to
exploit potential optimizations across both. These optimizations are explained
further on the pages specific to the execution framework.
Execution Engine
''''''''''''''''
This layer performs computation on partitions of the data. The
Modin Dataframe is designed to work with `task parallel`_ frameworks, but integration with
data parallel frameworks should be possible with some effort.
Storage Format
''''''''''''''
The :doc:`storage format ` describes the in-memory partition type.
The base storage format in Modin is pandas. In the default case, the Modin Dataframe operates on partitions that contain ``pandas.DataFrame`` objects.
Data Ingress
''''''''''''
.. note::
Data ingress operations (e.g. ``read_csv``) in Modin load data from the source into
partitions and vice versa for data egress (e.g. ``to_csv``) operation.
Improved performance is achieved by reading/writing in partitions in parallel.
Data ingress starts with a function in the pandas API layer (e.g. ``read_csv``). Then the user's
query is passed to the :doc:`Factory Dispatcher `,
which defines a factory specific for the execution. The factory for execution contains an IO class
(e.g. ``PandasOnRayIO``) whose responsibility is to perform a parallel read/write from/to a file.
This IO class contains class methods with interfaces and names that are similar to pandas IO functions
(e.g. ``PandasOnRayIO.read_csv``). The IO class declares the Modin Dataframe and Query Compiler
classes specific for the execution engine and storage format to ensure the correct object is constructed.
It also declares IO methods that are mix-ins containing a combination of the engine-specific class for
deploying remote tasks, the class for parsing the given file format and the class handling the chunking
of the format-specific file on the head node (see dispatcher classes implementation
:doc:`details `). The output from the IO class data ingress function is
a :doc:`Modin Dataframe `.
.. image:: /img/generic_data_ingress.svg
:align: center
Data Egress
'''''''''''
Data egress operations (e.g. ``to_csv``) are similar to data ingress operations up to
execution-specific IO class functions construction. Data egress functions of the IO class
are defined slightly different from data ingress functions and created only
specifically for the engine since partitions already have information about its storage
format. Using the IO class, data is exported from partitions to the target file.
.. image:: /img/generic_data_egress.svg
:align: center
Supported Execution Engines and Storage Formats
'''''''''''''''''''''''''''''''''''''''''''''''
This is a list of execution engines and in-memory formats supported in Modin. If you
would like to contribute a new execution engine or in-memory format, please see the
documentation page on :doc:`contributing `.
- :doc:`pandas on Ray `
- Uses the Ray_ execution framework.
- The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
- For more information on the execution path, see the :doc:`pandas on Ray ` page.
- :doc:`pandas on Dask `
- Uses the `Dask Futures`_ execution framework.
- The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
- For more information on the execution path, see the :doc:`pandas on Dask ` page.
- :doc:`pandas on MPI `
- Uses MPI_ through the Unidist_ execution framework.
- The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
- For more information on the execution path, see the :doc:`pandas on Unidist ` page.
- :doc:`pandas on Python `
- Uses native python execution - mainly used for debugging.
- The storage format is `pandas` and the in-memory partition type is a pandas DataFrame.
- For more information on the execution path, see the :doc:`pandas on Python ` page.
- pandas on Snowflake
- Uses the Snowpark Python library to transpile pandas API calls to SQL queries.
- The storage format is the custom-defined `Snowflake` format; data remains within Snowflake warehouses until retrieved by pandas API calls.
- For more information on pandas on Snowflake, refer to Snowflake's `documentation `_ (external link).
.. _directory-tree:
DataFrame Partitioning
----------------------
The Modin DataFrame architecture follows in the footsteps of modern architectures for
database and high performance matrix systems. We chose a partitioning schema that
partitions along both columns and rows because it gives Modin flexibility and
scalability in both the number of columns and the number of rows. The
following figure illustrates this concept.
.. image:: /img/block_partitions_diagram.png
:align: center
Currently, the main in-memory format of each partition is a
`pandas DataFrame`_ (:doc:`pandas storage format `).
Index
-----
We currently use the ``pandas.Index`` object for indexing both columns and rows. In the
future, we will implement a distributed, pandas-compatible Index object in order to remove
this scaling limitation from the system. Most workloads will not be affected by this scalability limit
since it only appears when operating on more than 10's of billions of columns or rows.
**Important note**: If you are using the
default index (``pandas.RangeIndex``) there is a fixed memory overhead (~200 bytes) and
there will be no scalability issues with the index.
API
---
The API is the outer-most layer that faces users. The following classes contain Modin's implementation of the pandas API:
.. toctree::
/flow/modin/pandas/base
/flow/modin/pandas/dataframe
/flow/modin/pandas/series
Module/Class View
-----------------
Modin's modules layout is shown below. Click on the links to deep dive into Modin's internal implementation
details. The documentation covers most modules, with more docs being added everyday!
.. parsed-literal::
├───.github
├───asv_bench
├───ci
├───docker
├───docs
├───examples
├───modin
│ ├─── :doc:`config `
| ├─── :doc:`utils `
│ ├───core
│ │ ├─── :doc:`dataframe `
│ │ │ ├─── :doc:`algebra `
│ │ │ ├─── :doc:`base `
│ │ │ └─── :doc:`pandas `
│ │ ├───execution
│ │ │ ├───dask
│ │ │ │ ├───common
│ │ │ │ └───implementations
│ │ │ │ └─── :doc:`pandas_on_dask `
│ │ │ ├─── :doc:`dispatching `
│ │ │ ├───python
│ │ │ │ └───implementations
│ │ │ │ └─── :doc:`pandas_on_python `
│ │ │ ├───ray
│ │ │ │ ├───common
│ │ │ │ ├─── :doc:`generic `
│ │ │ │ └───implementations
│ │ │ │ └─── :doc:`pandas_on_ray `
│ │ │ └───unidist
│ │ │ ├───common
│ │ │ ├─── :doc:`generic `
│ │ │ └───implementations
│ │ │ └─── :doc:`pandas_on_unidist `
│ │ ├─── :doc:`io `
│ │ └─── :doc:`storage_formats `
│ │ ├─── :doc:`base `
│ │ └─── :doc:`pandas `
│ ├───distributed
│ │ ├───dataframe
│ │ │ └─── :doc:`pandas `
│ ├─── :doc:`experimental `
│ │ ├───core
| | | └─── :doc:`io `
│ │ ├─── :doc:`pandas `
│ │ ├─── :doc:`sklearn `
│ │ ├───spreadsheet
│ │ ├─── :doc:`xgboost `
│ │ └─── :doc:`batch `
│ └───pandas
│ ├─── :doc:`dataframe `
│ └─── :doc:`series `
├───requirements
├───scripts
└───stress_tests
.. _pandas Dataframe: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html
.. _Ray: https://github.com/ray-project/ray
.. _Unidist: https://github.com/modin-project/unidist
.. _MPI: https://www.mpi-forum.org/
.. _code: https://github.com/modin-project/modin/blob/main/modin/core/dataframe
.. _Dask: https://github.com/dask/dask
.. _Dask Futures: https://docs.dask.org/en/latest/futures.html
.. _issue: https://github.com/modin-project/modin/issues
.. _task parallel: https://en.wikipedia.org/wiki/Task_parallelism
.. _experimental features: /usage_guide/advanced_usage/index.html
================================================
FILE: docs/development/contributing.rst
================================================
Contributing
============
Getting Started
---------------
If you're interested in getting involved in the development of Modin, but aren't sure
where start, take a look at the issues tagged `Good first issue`_ or Documentation_.
These are issues that would be good for getting familiar with the codebase and better
understanding some of the more complex components of the architecture. There is
documentation here about the :doc:`architecture ` that you will
want to review in order to get started.
Also, feel free to join the discussions on the `developer mailing list`_.
If you want a quick guide to getting your development environment setup, please
use `the contributing instructions on GitHub`_.
Certificate of Origin
---------------------
To keep a clear track of who did what, we use a `sign-off` procedure (same requirements
for using the signed-off-by process as the Linux kernel has
https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html) on patches or pull
requests that are being sent. The sign-off is a simple line at the end of the explanation
for the patch, which certifies that you wrote it or otherwise have the right to pass it
on as an open-source patch. The rules are pretty simple: if you can certify the below:
CERTIFICATE OF ORIGIN V 1.1
^^^^^^^^^^^^^^^^^^^^^^^^^^^
"By making a contribution to this project, I certify that:
1.) The contribution was created in whole or in part by me and I have the right to
submit it under the open source license indicated in the file; or
2.) The contribution is based upon previous work that, to the best of my knowledge, is
covered under an appropriate open source license and I have the right under that license
to submit that work with modifications, whether created in whole or in part by me, under
the same open source license (unless I am permitted to submit under a different
license), as indicated in the file; or
3.) The contribution was provided directly to me by some other person who certified (a),
(b) or (c) and I have not modified it.
4.) I understand and agree that this project and the contribution are public and that a
record of the contribution (including all personal information I submit with it,
including my sign-off) is maintained indefinitely and may be redistributed consistent
with this project or the open source license(s) involved."
.. code-block:: bash
This is my commit message
Signed-off-by: Awesome Developer
Code without a proper signoff cannot be merged into the
main branch. Note: You must use your real name (sorry, no pseudonyms or anonymous
contributions.)
The text can either be manually added to your commit body, or you can add either ``-s``
or ``--signoff`` to your usual ``git commit`` commands:
.. code-block:: bash
git commit --signoff -m "This is my commit message"
git commit -s -m "This is my commit message"
This will use your default git configuration which is found in .git/config. To change
this, you can use the following commands:
.. code-block:: bash
git config --global user.name "Awesome Developer"
git config --global user.email "awesome.developer.@example.org"
If you have authored a commit that is missing the signed-off-by line, you can amend your
commits and push them to GitHub.
.. code-block:: bash
git commit --amend --signoff
If you've pushed your changes to GitHub already you'll need to force push your branch
after this with ``git push -f``.
Commit Message formatting
-------------------------
We request that your first commit follow a particular format, and we
**require** that your PR title follow the format. The format is:
.. code-block:: bash
FEAT-#9999: Add `DataFrame.rolling` functionality, to enable rolling window operations
The ``FEAT`` component represents the type of commit. This component of the commit
message can be one of the following:
* FEAT: A new feature that is added
* DOCS: Documentation improvements or updates
* FIX: A bugfix contribution
* REFACTOR: Moving or removing code without change in functionality
* TEST: Test updates or improvements
* PERF: Performance enhancements
The ``#9999`` component of the commit message should be the issue number in the Modin
GitHub issue tracker: https://github.com/modin-project/modin/issues. This is important
because it links commits to their issues.
The commit message should follow a colon (:) and be descriptive and succinct.
A Modin CI job on GitHub will enforce that your pull request title follows the
format we suggest. Note that if you update the PR title, you have to push
another commit (even if it's empty) or amend your last commit for the job to
pick up the new PR title. Re-running the job in Github Actions won't work.
General Rules for committers
----------------------------
- Try to write a PR name as descriptive as possible.
- Try to keep PRs as small as possible. One PR should be making one semantically atomic change.
- Don't merge your own PRs even if you are technically able to do it.
Development Dependencies
------------------------
We recommend doing development in a virtualenv or conda environment, though this decision
is ultimately yours. You will want to run the following in order to install all of the required
dependencies for running the tests and formatting the code:
.. code-block:: bash
conda env create --file environment-dev.yml
# or
pip install -r requirements-dev.txt
Code Formatting and Lint
------------------------
We use black_ for code formatting. Before you submit a pull request, please make sure
that you run the following from the project root:
.. code-block:: bash
black modin/ asv_bench/benchmarks scripts/doc_checker.py
We also use flake8_ to check linting errors. Running the following from the project root
will ensure that it passes the lint checks on Github Actions:
.. code-block:: bash
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
We test that this has been run on our `Github Actions`_ test suite. If you do this and find
that the tests are still failing, try updating your version of black and flake8.
Adding a test
-------------
If you find yourself fixing a bug or adding a new feature, don't forget to add a test to
the test suite to verify its correctness! More on testing and the layout of the tests
can be found in our testing documentation. We ask that you follow the existing
structure of the tests for ease of maintenance.
Running the tests
-----------------
To run the entire test suite, run the following from the project root:
.. code-block:: bash
pytest modin/pandas/test
The test suite is very large, and may take a long time if you run every test. If you've
only modified a small amount of code, it may be sufficient to run a single test or some
subset of the test suite. In order to run a specific test run:
.. code-block:: bash
pytest modin/pandas/test::test_new_functionality
The entire test suite is automatically run for each pull request.
Performance measurement
-----------------------
We use Asv_ tool for performance tracking of various Modin functionality. The results
can be viewed here: `Asv dashboard`_.
More information can be found in the `Asv readme`_.
Building documentation
----------------------
To build the documentation, please follow the steps below from the project root:
.. code-block:: bash
pip install -r docs/requirements-doc.txt
sphinx-build -b html docs docs/build
To visualize the documentation locally, run the following from `build` folder:
.. code-block:: bash
python -m http.server
# python -m http.server 1234
then open the browser at `0.0.0.0:` (e.g. `0.0.0.0:1234`).
Contributing a new execution framework or in-memory format
----------------------------------------------------------
If you are interested in contributing support for a new execution framework or in-memory
format, please make sure you understand the :doc:`architecture ` of Modin.
The best place to start the discussion for adding a new execution framework or in-memory
format is the `developer mailing list`_.
More docs on this coming soon...
.. _Good first issue: https://github.com/modin-project/modin/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue+%3Abeginner%3A%22
.. _Documentation: https://github.com/modin-project/modin/issues?q=is%3Aissue+is%3Aopen+label%3A%22documentation+%3Abookmark_tabs%3A%22
.. _black: https://github.com/ambv/black
.. _flake8: http://flake8.pycqa.org/en/latest/
.. _Github Actions: https://github.com/features/actions
.. _Asv: https://github.com/airspeed-velocity/asv#airspeed-velocity
.. _developer mailing list: https://groups.google.com/forum/#!forum/modin-dev
.. _Asv dashboard: https://modin.org/modin-bench/#/
.. _Asv readme: https://github.com/modin-project/modin/blob/main/asv_bench/README.md
.. _the contributing instructions on GitHub: https://github.com/modin-project/modin/blob/main/contributing/contributing.md
================================================
FILE: docs/development/index.rst
================================================
Development
===========
.. toctree::
:maxdepth: 4
contributing
architecture
partition_api
using_pandas_on_ray
using_pandas_on_dask
using_pandas_on_python
using_pandas_on_mpi
.. meta::
:description lang=en:
Development-specific documentation.
================================================
FILE: docs/development/partition_api.rst
================================================
Partition API in Modin
======================
When you are working with a :py:class:`~modin.pandas.dataframe.DataFrame`, you can unwrap its remote partitions
to get the raw futures objects compatible with the execution engine (e.g. ``ray.ObjectRef`` for Ray).
In addition to unwrapping of the remote partitions we also provide an API to construct a ``modin.pandas.DataFrame``
from raw futures objects.
Partition IPs
-------------
For finer grained placement control, Modin also provides an API to get the IP addresses of the nodes that hold each partition.
You can pass the partitions having needed IPs to your function. It can help with minimizing of data movement between nodes.
Partition API implementations
-----------------------------
By default, a :py:class:`~modin.pandas.dataframe.DataFrame` stores underlying partitions as ``pandas.DataFrame`` objects.
You can find the specific implementation of Modin's Partition Interface in :doc:`pandas Partition API `.
.. toctree::
:hidden:
/flow/modin/distributed/dataframe/pandas
Ray engine
----------
However, it is worth noting that for Modin on ``Ray`` engine with ``pandas`` in-memory format IPs of the remote partitions may not match
actual locations if the partitions are lower than 100 kB. Ray saves such objects (<= 100 kB, by default) in in-process store
of the calling process (please, refer to `Ray documentation`_ for more information). We can't get IPs for such objects while maintaining good performance.
So, you should keep in mind this for unwrapping of the remote partitions with their IPs. Several options are provided to handle the case in
``How to handle Ray objects that are lower 100 kB`` section.
Dask engine
-----------
There is no mentioned above issue for Modin on ``Dask`` engine with ``pandas`` in-memory format because ``Dask`` saves any objects
in the worker process that processes a function (please, refer to `Dask documentation`_ for more information).
Unidist engine
--------------
Currently, Modin only supports MPI through unidist. There is no mentioned above issue for
Modin on ``Unidist`` engine using ``MPI`` backend with ``pandas`` in-memory format
because ``Unidist`` saves any objects in the MPI worker process that processes a function
(please, refer to `Unidist documentation`_ for more information).
How to handle Ray objects that are lower than 100 kB
----------------------------------------------------
* If you are sure that each of the remote partitions being unwrapped is higher than 100 kB, you can just import Modin or perform ``ray.init()`` manually.
* If you don't know partition sizes you can pass the option ``_system_config={"max_direct_call_object_size": ,}``, where ``nbytes`` is threshold for objects that will be stored in in-process store, to ``ray.init()``.
* You can also start Ray as follows: ``ray start --head --system-config='{"max_direct_call_object_size":}'``.
Note that when specifying the threshold the performance of some Modin operations may change.
.. _`Ray documentation`: https://docs.ray.io/en/master/index.html#
.. _`Dask documentation`: https://distributed.dask.org/en/latest/index.html
.. _`Unidist documentation`: https://unidist.readthedocs.io/en/latest/index.html
================================================
FILE: docs/development/using_pandas_on_dask.rst
================================================
pandas on Dask
==============
This section describes usage related documents for the pandas on Dask component of Modin.
Modin uses pandas as a primary memory format of the underlying partitions and optimizes queries
ingested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it
but you can explicitly specify it anyway as shown below.
One of the execution engines that Modin uses is Dask. To enable the pandas on Dask execution you should set the following environment variables:
.. code-block:: bash
export MODIN_ENGINE=dask
export MODIN_STORAGE_FORMAT=pandas
or turn them on in source code:
.. code-block:: python
import modin.config as cfg
cfg.Engine.put('dask')
cfg.StorageFormat.put('pandas')
Using Modin on Dask locally
---------------------------
If you want to run Modin on Dask locally using a single node, just set Modin engine to ``Dask`` and
continue working with a Modin DataFrame as if it was a pandas DataFrame.
You can either initialize a Dask client on your own and Modin connects to the existing Dask cluster or
allow Modin itself to initialize a Dask client.
.. code-block:: python
import modin.pandas as pd
import modin.config as modin_cfg
modin_cfg.Engine.put("dask")
df = pd.DataFrame(...)
Using Modin on Dask in a Cluster
--------------------------------
If you want to run Modin on Dask in a cluster, you should set up a Dask cluster and initialize a Dask client.
Once the Dask client is initialized, Modin will be able to connect to it and use the Dask cluster.
.. code-block:: python
from distributed import Client
import modin.pandas as pd
import modin.config as modin_cfg
# Define your cluster here
cluster = ...
client = Client(cluster)
modin_cfg.Engine.put("dask")
df = pd.DataFrame(...)
To get more information on how to deploy and run a Dask cluster, visit the `Deploy Dask Clusters`_ page.
Conversion between Modin DataFrame and Dask DataFrame
-----------------------------------------------------
Modin DataFrame can be converted to/from Dask DataFrame with no-copy partition conversion.
This allows you to take advantage of both Modin and Dask libraries for maximum performance.
.. code-block:: python
import modin.pandas as pd
import modin.config as modin_cfg
from modin.pandas.io import to_dask, from_dask
modin_cfg.Engine.put("dask")
df = pd.DataFrame(...)
# Convert Modin to Dask DataFrame
dask_df = to_dask(df)
# Convert Dask to Modin DataFrame
modin_df = from_dask(dask_df)
.. _Deploy Dask Clusters: https://docs.dask.org/en/stable/deploying.html
================================================
FILE: docs/development/using_pandas_on_mpi.rst
================================================
pandas on MPI through unidist
=============================
This section describes usage related documents for the pandas on MPI through unidist component of Modin.
Modin uses pandas as a primary memory format of the underlying partitions and optimizes queries
ingested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it
but you can explicitly specify it anyway as shown below.
One of the execution engines that Modin uses is MPI through unidist.
To enable the pandas on MPI through unidist execution you should set the following environment variables:
.. code-block:: bash
export MODIN_ENGINE=unidist
export MODIN_STORAGE_FORMAT=pandas
export UNIDIST_BACKEND=mpi
or turn it on in source code:
.. code-block:: python
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put('unidist')
modin_cfg.StorageFormat.put('pandas')
unidist_cfg.Backend.put('mpi')
To run a python application you should use ``mpiexec -n 1 python `` command.
.. code-block:: bash
mpiexec -n 1 python script.py
For more information on how to run a python application with unidist on MPI backend
please refer to `Unidist on MPI`_ section of the unidist documentation.
As of unidist 0.5.0 there is support for a shared object store for MPI backend.
The feature allows to improve performance in the workloads,
where workers use same data multiple times by reducing data copies.
You can enable the feature by setting the following environment variable:
.. code-block:: bash
export UNIDIST_MPI_SHARED_OBJECT_STORE=True
or turn it on in source code:
.. code-block:: python
import unidist.config as unidist_cfg
unidist_cfg.MpiSharedObjectStore.put(True)
.. _`Unidist on MPI`: https://unidist.readthedocs.io/en/latest/using_unidist/unidist_on_mpi.html
================================================
FILE: docs/development/using_pandas_on_python.rst
================================================
pandas on Python
================
This section describes usage related documents for the pandas on Python component of Modin.
Modin uses pandas as the primary memory format of the underlying partitions and optimizes queries
from the API layer in a specific way to this format. Since it is a default, you do not need to specify
the pandas memory format, but we show how to explicitly set it below.
One of the execution engines that Modin uses is Python. This engine is sequential and used for debugging.
To enable the pandas on Python execution you should set the following environment variables:
.. code-block:: bash
export MODIN_ENGINE=python
export MODIN_STORAGE_FORMAT=pandas
or turn a debug mode on:
.. code-block:: bash
export MODIN_DEBUG=True
export MODIN_STORAGE_FORMAT=pandas
or do the same in source code:
.. code-block:: python
import modin.config as cfg
cfg.Engine.put('python')
cfg.StorageFormat.put('pandas')
.. code-block:: python
import modin.config as cfg
cfg.IsDebug.put(True)
cfg.StorageFormat.put('pandas')
================================================
FILE: docs/development/using_pandas_on_ray.rst
================================================
pandas on Ray
=============
This section describes usage related documents for the pandas on Ray component of Modin.
Modin uses pandas as a primary memory format of the underlying partitions and optimizes queries
ingested from the API layer in a specific way to this format. Thus, there is no need to care of choosing it
but you can explicitly specify it anyway as shown below.
One of the execution engines that Modin uses is Ray. If you have Ray installed in your system,
Modin also uses it by default to distribute computations.
If you want to be explicit, you could set the following environment variables:
.. code-block:: bash
export MODIN_ENGINE=ray
export MODIN_STORAGE_FORMAT=pandas
or turn it on in source code:
.. code-block:: python
import modin.config as cfg
cfg.Engine.put('ray')
cfg.StorageFormat.put('pandas')
================================================
FILE: docs/ecosystem.rst
================================================
Ecosystem
=========
There is a constantly growing number of users and packages using pandas
to address their specific needs in data preparation, analysis and visualization.
pandas is being used ubiquitously and is a good choise to handle small-sized data.
However, pandas scales poorly and is non-interactive on moderate to large datasets.
Modin provides a drop-in replacement API for pandas and scales computation across nodes and
CPUs available. What you need to do to switch to Modin is just replace a single line of code.
.. code-block:: python
# import pandas as pd
import modin.pandas as pd
While most packages can consume a pandas DataFrame and operate it efficiently,
this is not the case with a Modin DataFrame due to its distributed nature.
Thus, some packages may lack support for handling Modin DataFrame(s) correctly and,
moreover, efficiently. Modin implements such methods as ``__array__``, ``__dataframe__``, etc.
to facilitate other libraries to consume a Modin DataFrame. If you feel that a certain library
can operate efficiently with a specific format of data, it is possible to convert a Modin DataFrame
to the format preferred.
to_pandas
---------
You can refer to `pandas ecosystem`_ page to get more details on
where pandas can be used and what libraries it powers.
.. code-block:: python
from modin.pandas.io import to_pandas
pandas_df = to_pandas(modin_df)
to_numpy
--------
You can refer to `NumPy ecosystem`_ section of NumPy documentation to get more details on
where NumPy can be used and what libraries it powers.
.. code-block:: python
from modin.pandas.io import to_numpy
numpy_arr = to_numpy(modin_df)
to_ray
------
You can refer to `Ray Data`_ page to get more details on
where Ray Dataset can be used and what libraries it powers.
.. code-block:: python
from modin.pandas.io import to_ray
ray_dataset = to_ray(modin_df)
to_dask
-------
You can refer to `Dask DataFrame`_ page to get more details on
where Dask DataFrame can be used and what libraries it powers.
.. code-block:: python
from modin.pandas.io import to_dask
dask_df = to_dask(modin_df)
.. _pandas ecosystem: https://pandas.pydata.org/community/ecosystem.html
.. _NumPy ecosystem: https://numpy.org
.. _Ray Data: https://docs.ray.io/en/latest/data/data.html
.. _Dask DataFrame: https://docs.dask.org/en/stable/dataframe.html
================================================
FILE: docs/flow/modin/config.rst
================================================
:orphan:
Modin Configuration Settings
""""""""""""""""""""""""""""
To adjust Modin's default behavior, you can set the value of Modin
configs by setting an environment variable or by using the
``modin.config`` API. To list all available configs in Modin, please
run ``python -m modin.config`` to print all
Modin configs with descriptions.
Public API
''''''''''
Potentially, the source of configs can be any, but for now only environment
variables are implemented. Any environment variable originate from
:class:`~modin.config.envvars.EnvironmentVariable`, which contains most of
the config API implementation.
.. autoclass:: modin.config.envvars.EnvironmentVariable
:members: get, put, get_help, get_value_source, once, subscribe
Modin Configs List
''''''''''''''''''
.. csv-table::
:file: configs_help.csv
:header-rows: 1
Usage Guide
'''''''''''
See example of interaction with Modin configs below, as it can be seen config
value can be set either by setting the environment variable or by using config
API.
.. code-block:: python
import os
# Setting `MODIN_ENGINE` environment variable.
# Also can be set outside the script.
os.environ["MODIN_ENGINE"] = "Dask"
import modin.config
import modin.pandas as pd
# Checking initially set `Engine` config,
# which corresponds to `MODIN_ENGINE` environment
# variable
print(modin.config.Engine.get()) # prints 'Dask'
# Checking default value of `NPartitions`
print(modin.config.NPartitions.get()) # prints '8'
# Changing value of `NPartitions`
modin.config.NPartitions.put(16)
print(modin.config.NPartitions.get()) # prints '16'
One can also use config variables with a context manager in order to use
some config only for a certain part of the code:
.. code-block:: python
import modin.config as cfg
# Default value for this config is 'False'
print(cfg.RangePartitioning.get()) # False
# Set the config to 'True' inside of the context-manager
with cfg.context(RangePartitioning=True):
print(cfg.RangePartitioning.get()) # True
df.merge(...) # will use range-partitioning impl
# Once the context is over, the config gets back to its previous value
print(cfg.RangePartitioning.get()) # False
# You can also set multiple config at once when you pass a dictionary to 'cfg.context'
print(cfg.AsyncReadMode.get()) # False
with cfg.context(RangePartitioning=True, AsyncReadMode=True):
print(cfg.RangePartitioning.get()) # True
print(cfg.AsyncReadMode.get()) # True
print(cfg.RangePartitioning.get()) # False
print(cfg.AsyncReadMode.get()) # False
================================================
FILE: docs/flow/modin/core/dataframe/algebra.rst
================================================
:orphan:
Operators Module Description
""""""""""""""""""""""""""""
Brief description
'''''''''''''''''
Most of the functions that are evaluated by `QueryCompiler` can be categorized into
one of the patterns: Map, TreeReduce, Binary, Reduce, etc., called core operators. The ``modin.core.dataframe.algebra``
module provides templates to easily build such types of functions. These templates
are supposed to be used at the `QueryCompiler` level since each built function accepts
and returns `QueryCompiler`.
High-Level Module Overview
''''''''''''''''''''''''''
Each template class implements a
``register`` method, which takes functions to apply and
instantiate the related template. Functions that are passed to ``register`` will be executed
against converted to pandas and preprocessed in a template-specific way partition, so the function
would take one of the pandas object: ``pandas.DataFrame``, ``pandas.Series`` or ``pandas.DataFrameGroupbyObject``.
.. note::
Currently, functions that are built in that way are supported only in a pandas
storage format (i.e. can be used only in `PandasQueryCompiler`).
Algebra module provides templates for this type of function:
Map operator
-------------
Uniformly apply a function argument to each partition in parallel.
**Note**: map function should not change the shape of the partitions.
.. figure:: /img/map_evaluation.svg
:align: center
This operator performs best when the number of partitions equals to the number of CPUs
so that each single partition gets processed in parallel. When the number of partitions is 1.5x greater than
the number of CPUs, Modin applies a heuristic to join some partitions to get "ideal" partitioning so that
each new partition gets processed in parallel.
Reduce operator
---------------
Applies an argument function that reduces each column or row on the specified axis into a scalar, but requires knowledge about the whole axis.
Be aware that providing this knowledge may be expensive because the execution engine has to
concatenate partitions along the specified axis. Also, note that the execution engine expects
that the reduce function returns a one dimensional frame.
.. figure:: /img/reduce_evaluation.svg
:align: center
This operator performs best when the number of partitions (row or column partitions in depend on the specified axis)
equals to the number of CPUs so that each single axis partition gets processed in parallel.
TreeReduce operator
-------------------
Applies an argument function that reduces specified axis into a scalar. First applies map function to each partition
in parallel, then concatenates resulted partitions along the specified axis and applies reduce
function. In contrast with `Map function` template, here you're allowed to change partition shape
in the map phase. Note that the execution engine expects that the reduce function returns a one dimensional frame.
This operator performs best when the number of partitions (including the initial and intermediate stages)
equals to the number of CPUs so that each single axis partition gets processed in parallel.
Binary operator
---------------
Applies an argument function, that takes exactly two operands (first is always `QueryCompiler`).
If both operands are query compilers then the execution engine broadcasts partitions of
the right operand to the left.
.. figure:: /img/binary_evaluation.svg
:align: center
.. warning::
To be able to do frame broadcasting, partitioning along the index axis of both frames
has to be equal, otherwise they need to be aligned first. The execution engine will do
it automatically but note that this requires repartitioning, which is a much
more expensive operation than the binary function itself.
This operator performs best when both operands have identical partitioning and the number of partitions of an operand
equals to the number of CPUs so that each single partition gets processed in parallel.
Fold operator
-------------
Applies an argument function that requires knowledge of the whole axis. Be aware that providing this knowledge may be
expensive because the execution engine has to concatenate partitions along the specified axis.
This operator performs best when the number of partitions (row or column partitions in depend on the specified axis)
equals to the number of CPUs so that each single axis partition gets processed in parallel.
GroupBy operator
----------------
Evaluates GroupBy aggregation for that type of functions that can be executed via TreeReduce approach.
To be able to form groups engine broadcasts ``by`` partitions to each partition of the source frame.
This operator performs best when the cardinality of ``by`` columns is low (small number of output groups).
At the ``Map`` stage, the operator computes the aggregation for each row partition individually, meaning,
that the ``Reduce`` stage takes a dataframe with the following number of rows:
``num_groups * n_row_parts``. If the number of groups is too high, there's a risk of getting a dataframe
with even bigger than the initial shape at the ``Reduce`` stage.
Default-to-pandas operator
--------------------------
Do :doc:`fallback to pandas ` for passed function.
This operator has a performance penalty for going from a partitioned Modin DataFrame to pandas because of
the communication cost and single-threaded nature of pandas.
How to register your own function
'''''''''''''''''''''''''''''''''
Let's examine an example of how to use the algebra module to create your own
new functions.
Imagine you have a complex aggregation that can be implemented into a single query but
doesn't have any implementation in pandas API. If you know how to implement this
aggregation efficiently in a distributed frame, you may want to use one of the above described
patterns (e.g. ``TreeReduce``).
Let's implement a function that counts non-NA values for each column or row
(``pandas.DataFrame.count``). First, we need to determine the function type.
TreeReduce approach would be great: in a map phase, we'll count non-NA cells in each
partition in parallel and then just sum its results in the reduce phase.
To define the TreeReduce function that does `count` + `sum` we just need to register the
appropriate functions and then assign the result to the picked `QueryCompiler`
(`PandasQueryCompiler` in our case):
.. code-block:: python
from modin.core.storage_formats import PandasQueryCompiler
from modin.core.dataframe.algebra import TreeReduce
PandasQueryCompiler.custom_count = TreeReduce.register(pandas.DataFrame.count, pandas.DataFrame.sum)
Then, we want to handle it from the :py:class:`~modin.pandas.dataframe.DataFrame`, so we need to create a way to do that:
.. code-block:: python
import modin.pandas as pd
def count_func(self, **kwargs):
# The constructor allows you to pass in a query compiler as a keyword argument
return self.__constructor__(query_compiler=self._query_compiler.custom_count(**kwargs))
pd.DataFrame.count_custom = count_func
And then you can use it like you usually would:
.. code-block:: python
df.count_custom(axis=1)
Many of the `pandas` API functions can be easily implemented this way, so if you find
out that one of your favorite function is still defaulted to pandas and decide to
contribute to Modin to add its implementation, you may use this example as a reference.
================================================
FILE: docs/flow/modin/core/dataframe/base/dataframe.rst
================================================
ModinDataframe
""""""""""""""
The :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` is the parent class for all dataframes - regardless of what storage format they are backed by. Its purpose is to define the algebra operators that must be exposed by a dataframe.
This class exposes the dataframe algebra and is meant to be subclassed by all dataframe implementations.
Descendants of this class implement the algebra, and act as the intermediate level
between the query compiler and the underlying execution details (e.g. the conforming partition manager). The class provides
a significantly reduced set of operations that can be composed to form any pandas query.
The :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` is an example of a descendant of this class. It currently has implementations for some of the operators
exposed in this class, and is currently being refactored to include implementations for all of the algebra operators. Please
refer to the :doc:`PandasDataframe documentation ` for more information.
The :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` is independent of implementation specific details such as partitioning, storage format, or execution engine.
Public API
----------
.. autoclass:: modin.core.dataframe.base.dataframe.dataframe.ModinDataframe
:members:
================================================
FILE: docs/flow/modin/core/dataframe/base/index.rst
================================================
Purpose
=======
The :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` serves the purpose of describing and defining the :doc:`Core Dataframe Algebra `.
It is the core construction element and serves as the client for the :doc:`Modin Query Compiler`. Descendants that offer implementations execute the queries from the compiler by invoking functions over partitions via a partition manager.
The partitions and partition manager interfaces are currently implementation-specific, but may
be standardized in the future.
The :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe` and axis partitions are the interfaces that must be implemented by any :doc:`execution backend` in order for it to be plugged in to Modin.
These classes are mostly abstract, however very simple and generic enough methods like
:py:meth:`~modin.core.dataframe.base.partitioning.BaseDataframeAxisPartition.force_materialization` can be implemented at the base level because for now we do not expect them to differ in any implementation.
ModinDataframe Interface
========================
* :doc:`ModinDataframe ` is an abstract class which represents the algebra operators a dataframe must expose.
* :doc:`BaseDataframeAxisPartition ` is an abstract class, representing a joined group of partitions along some axis (either rows or labels).
.. toctree::
:hidden:
dataframe
partitioning/axis_partition
================================================
FILE: docs/flow/modin/core/dataframe/base/partitioning/axis_partition.rst
================================================
BaseDataframeAxisPartition
""""""""""""""""""""""""""
The class is base for any axis partition class and serves as the last level on which
operations that were conveyed from the partition manager are being performed on an entire column or row.
**Note**: ``modin.core.dataframe.base`` intentionally does not describe any particular partition interface,
as it is the partition manager responsibility (if said partition manager is implemented), i.e. it is
too low-level to be present on the base, abstract level.
The class provides an API that has to be overridden by the child classes in order to manipulate
on a list of block partitions (making up column or row partition) they store.
The procedures that use this class and its methods assume that they have some global knowledge
about the entire axis. This may require the implementation to use concatenation or append on the
list of block partitions.
Public API
----------
.. autoclass:: modin.core.dataframe.base.partitioning.axis_partition.BaseDataframeAxisPartition
:members:
================================================
FILE: docs/flow/modin/core/dataframe/index.rst
================================================
:orphan:
Core Modin Dataframe Objects
============================
Modin partitions data to scale efficiently.
To keep track of everything a few key classes are introduced: ``Dataframe``, ``Partition``, ``AxisPartiton`` and ``PartitionManager``.
* ``Dataframe`` is the class conforming to Dataframe Algebra.
* ``Partition`` is an element of a NxM grid which, when combined, represents the ``Dataframe``
* ``AxisPartition`` is a joined group of ``Partition``-s along some axis (either rows or columns)
* ``PartitionManager`` is the manager that implements the primitives used for Dataframe Algebra operations over ``Partition``-s
Each :doc:`storage format `, execution engine, and each execution system (storage format + execution engine)
may have its own implementations of these Core Dataframe's entities.
Current stable implementations are the following:
* :doc:`Base ModinDataframe ` defines a common interface and algebra operators for `Dataframe` implementations.
Storage format specific:
* :doc:`Modin PandasDataframe ` is an implementation for any frame class of :doc:`pandas storage format `.
Engine specific:
* :doc:`Modin GenericRayDataframe ` is an implementation for any frame class that works on Ray execution engine.
* :doc:`Modin GenericUnidistDataframe ` is an implementation for any frame class that works on Unidist execution engine.
Execution system specific:
* :doc:`Modin PandasOnRayDataframe ` is a specialization of the Core Modin Dataframe for ``PandasOnRay`` execution.
* :doc:`Modin PandasOnDaskDataframe ` is specialization of the Core Modin Dataframe for ``PandasOnDask`` execution.
* :doc:`Modin PandasOnPythonDataframe ` is a specialization of the Core Modin Dataframe for ``PandasOnPython`` execution.
* :doc:`Modin PandasOnUnidistDataframe ` is a specialization of the Core Modin Dataframe for ``PandasOnUnidist`` execution.
.. note::
At the current stage of Modin development, the base interfaces of the Dataframe objects are not defined yet.
So for now the origin of all changes in the Dataframe interfaces is the :doc:`Dataframe for pandas storage format`.
.. toctree::
:hidden:
base/index
pandas/index
================================================
FILE: docs/flow/modin/core/dataframe/pandas/dataframe.rst
================================================
PandasDataframe
"""""""""""""""
:py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` is a direct descendant of :py:class:`~modin.core.dataframe.base.dataframe.dataframe.ModinDataframe`. Its purpose is to implement the abstract interfaces for usage with all ``pandas``-based :doc:`storage formats`.
:py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` could be inherited and augmented further by any specific implementation which needs it to take special care of some behavior or to improve performance for certain execution engine.
The class serves as the intermediate level
between ``pandas`` query compiler and conforming partition manager. All queries formed
at the query compiler layer are ingested by this class and then conveyed jointly with the stored partitions
into the partition manager for processing. Direct partitions manipulation by this class is prohibited except
cases if an operation is strictly private or protected and called inside of the class only. The class provides
significantly reduced set of operations that fit plenty of pandas operations.
Main tasks of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` are storage of partitions, manipulation with labels of axes and
providing set of methods to perform operations on the internal data.
As mentioned above, ``PandasDataframe`` shouldn't work with stored partitions directly and
the responsibility for modifying partitions array has to lay on :doc:`partitioning/partition_manager`. For example, method
:meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.broadcast_apply_full_axis` redirects applying
function to :meth:`~PandasDataframePartitionManager.broadcast_axis_partitions` method.
``Modin PandasDataframe`` can be created from ``pandas.DataFrame``, ``pyarrow.Table``
(methods :meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.from_pandas`,
:meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.from_arrow` are used respectively). Also,
``PandasDataframe`` can be converted to ``np.array``, ``pandas.DataFrame``
(methods :meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.to_numpy`,
:meth:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe.to_pandas` are used respectively).
Manipulation with labels of axes happens using internal methods for changing labels on the new,
adding prefixes/suffixes etc.
Public API
----------
.. autoclass:: modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe
:members:
================================================
FILE: docs/flow/modin/core/dataframe/pandas/index.rst
================================================
Modin PandasDataframe Objects
=============================
``modin.core.dataframe.pandas`` is the package which houses common implementations
of different Modin internal classes used by most `pandas`-based :doc:`storage formats`.
It also double-serves as the full example of how to implement Modin execution backend pieces (sans the :doc:`execution part` which is absent here),
as it implements everything an execution backend needs to be fully conformant to Modin expectations.
* :doc:`PandasDataframe ` is the class conforming to Dataframe Algebra.
* :doc:`PandasDataframePartition ` implements ``Partition`` interface holding ``pandas.DataFrame``.
* :doc:`PandasDataframeAxisPartition ` is a joined group of ``PandasDataframePartition``-s along some axis (either rows or labels)
* :doc:`PandasDataframePartitionManager ` is the manager that implements the primitives used for Dataframe Algebra operations over ``PandasDataframePartition``-s
* :doc:`ModinDtypes `
* :doc:`ModinIndex `
.. toctree::
:hidden:
dataframe
partitioning/partition
partitioning/axis_partition
partitioning/partition_manager
metadata/dtypes
metadata/index
================================================
FILE: docs/flow/modin/core/dataframe/pandas/metadata/dtypes.rst
================================================
ModinDtypes
"""""""""""
Public API
----------
.. autoclass:: modin.core.dataframe.pandas.metadata.dtypes.ModinDtypes
:members:
================================================
FILE: docs/flow/modin/core/dataframe/pandas/metadata/index.rst
================================================
ModinIndex
""""""""""
Public API
----------
.. autoclass:: modin.core.dataframe.pandas.metadata.index.ModinIndex
:members:
================================================
FILE: docs/flow/modin/core/dataframe/pandas/partitioning/axis_partition.rst
================================================
PandasDataframeAxisPartition
""""""""""""""""""""""""""""
The class implements abstract interface methods from :py:class:`~modin.core.dataframe.base.partitioning.axis_partition.BaseDataframeAxisPartition`
giving the means for a sibling :doc:`partition manager` to actually work with the axis-wide partitions.
The class is base for any axis partition class of ``pandas`` storage format.
Subclasses must implement ``list_of_blocks`` which represents data wrapped by the :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`
objects and creates something interpretable as a ``pandas.DataFrame``.
See :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.axis_partition.PandasOnRayDataframeAxisPartition`
for an example on how to override/use this class when the implementation needs to be augmented.
The :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition` object has an invariant that requires that this
object is never returned from a function. It assumes that there will always be
``PandasDataframeAxisPartition`` object stored and structures itself accordingly.
Public API
----------
.. autoclass:: modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition
:members:
================================================
FILE: docs/flow/modin/core/dataframe/pandas/partitioning/partition.rst
================================================
PandasDataframePartition
""""""""""""""""""""""""
The class is base for any partition class of ``pandas`` storage format and serves as the last level
on which operations that were conveyed from the partition manager are being performed on an
individual block partition.
The class provides an API that has to be overridden by child classes in order to manipulate
on data and metadata they store.
The public API exposed by the children of this class is used in :py:class:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager`.
The objects wrapped by the child classes are treated as immutable by ``PandasDataframePartitionManager`` subclasses
and no logic for updating inplace.
Public API
----------
.. autoclass:: modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition
:members:
================================================
FILE: docs/flow/modin/core/dataframe/pandas/partitioning/partition_manager.rst
================================================
PandasDataframePartitionManager
"""""""""""""""""""""""""""""""
The class is base for any partition manager class of ``pandas`` storage format and serves as
intermediate level between :doc:`Modin PandasDataframe <../dataframe>` and conforming :doc:`partition ` class.
The class is responsible for partitions manipulation and applying a function to individual partitions:
block partitions, row partitions or column partitions, i.e. the class can form axis partitions from
block partitions to apply a function if an operation requires access to an entire column or row.
The class translates frame API into partition API and also can have some preprocessing operations
depending on the partition type for improving performance (for example,
:meth:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager.preprocess_func`).
Main task of partition manager is to keep knowledge of how partitions are stored and managed
internal to itself, so surrounding code could use it via lean enough API without worrying about
implementation details.
Partition manager can apply user-passed (arbitrary) function in different modes:
* block-wise (apply a function to individual block partitions):
* optionally accepting partition indices along each axis
* optionally accepting an item to be split so parts of it would be sent to each partition
* along a full axis (apply a function to an entire column or row made up of block partitions when user function needs information about the whole axis)
It can also broadcast partitions from `right` to `left` when executing certain operations making
`right` partitions available for functions executed where `left` live.
..
TODO: insert more text explaining "broadcast" term
Partition manager also is used to create "logical" partitions, or :doc:`axis partitions `
by joining existing partitions along specified axis (either rows or labels),
and to concatenate different partition sets along given axis.
It also maintains mapping from "external" (end user-visible) indices along all axes to internal
indices which are actually pairs of indices of partitions and indices inside the partitions,
as well as manages conversion to numpy and pandas representations.
Public API
----------
.. autoclass:: modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/dataframe.rst
================================================
PandasOnDaskDataframe
"""""""""""""""""""""
The class is the specific implementation of the dataframe algebra for the `Dask` execution engine.
It serves as an intermediate level between ``pandas`` query compiler and
:py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartitionManager`.
Public API
----------
.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.dataframe.PandasOnDaskDataframe
:members:
================================================
FILE: docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/index.rst
================================================
:orphan:
PandasOnDask Execution
======================
Queries that perform data transformation, data ingress or data egress using the `pandas on Dask` execution
pass through the Modin components detailed below.
To enable `pandas on Dask` execution, please refer to the usage section in :doc:`pandas on Dask `.
Data Transformation
'''''''''''''''''''
.. image:: /img/pandas_on_dask_data_transform.svg
:align: center
When a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer
to be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,
for example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query
and defining specific intermediate values to provide more context to the query compiler.
The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for
processing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,
to determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`
storage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of
the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.dataframe.PandasOnDaskDataframe` which inherits
generic functionality from the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.
PandasOnDask Dataframe implementation
-------------------------------------
Modin implements ``Dataframe``, ``PartitionManager``, ``AxisPartition`` and ``Partition`` classes
specifically for the `PandasOnDask` execution.
* :doc:`PandasOnDaskDataframe `
* :doc:`PandasOnDaskDataframePartition `
* :doc:`PandasOnDaskDataframeVirtualPartition `
* :doc:`PandasOnDaskDataframePartitionManager `
.. toctree::
:hidden:
dataframe
partitioning/partition
partitioning/virtual_partition
partitioning/partition_manager
Data Ingress
''''''''''''
.. image:: /img/pandas_on_dask_data_ingress.svg
:align: center
Data Egress
'''''''''''
.. image:: /img/pandas_on_dask_data_egress.svg
:align: center
When a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the
:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for
the execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnDaskFactory`. The factory, in turn,
exposes the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` class
whose responsibility is to perform a parallel read/write from/to a file.
When reading data from a CSV file, for example, the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` class forwards
the user query to the :meth:`~modin.core.io.text.CSVDispatcher._read` method of :py:class:`~modin.core.io.text.CSVDispatcher`, where the query's parameters are preprocessed
to check if they are supported by the execution (defaulting to pandas if they are not) and computes some metadata
common for all partitions to be read. Then, the file is split into row chunks, and this data is used to launch remote tasks on the Dask workers
via the :meth:`~modin.core.execution.dask.common.engine_wrapper.DaskWrapper.deploy` method of :py:class:`~modin.core.execution.dask.common.engine_wrapper.DaskWrapper`.
On each Dask worker, the :py:class:`~modin.core.storage_formats.pandas.parsers.PandasCSVParser` parses data.
After the remote tasks are finished, additional result postprocessing is performed,
and a new query compiler with the data read is returned.
When writing data to a CSV file, for example, the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` processes
the user query to execute it on Dask workers. Then, the :py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO` asks the
:py:class:`~modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskDataframe` to decompose the data into row-wise partitions
that will be written into the file in parallel in Dask workers.
.. note::
Currently, data egress uses default `pandas` implementation for `pandas on Dask` execution.
================================================
FILE: docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.rst
================================================
PandasOnDaskDataframePartition
""""""""""""""""""""""""""""""
The class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,
providing the API to perform operations on a block partition, namely, ``pandas.DataFrame``, using Dask as the execution engine.
In addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:
* ``length`` - length of ``pandas.DataFrame`` wrapped
* ``width`` - width of ``pandas.DataFrame`` wrapped
* ``ip`` - node IP address that holds ``pandas.DataFrame`` wrapped
An operation on a block partition can be performed in two modes:
* asynchronously_ - via :meth:`~modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartition.apply`
* lazily_ - via :meth:`~modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartition.add_to_apply_calls`
Public API
----------
.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartition
:members:
.. _asynchronously: https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)
.. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation
================================================
FILE: docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.rst
================================================
PandasOnDaskDataframePartitionManager
"""""""""""""""""""""""""""""""""""""
This class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager`
using Dask as the execution engine. This class is responsible for partition manipulation and applying a function to
block/row/column partitions.
Public API
----------
.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/virtual_partition.rst
================================================
PandasOnDaskDataframeVirtualPartition
"""""""""""""""""""""""""""""""""""""
The class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.virtual_partition.PandasOnDaskDataframeVirtualPartition`,
providing the API to perform operations on an axis (column or row) partition using Dask as the execution engine.
The axis partition is a wrapper over a list of block partitions that are stored in this class.
Public API
----------
.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframeVirtualPartition
:members:
PandasOnDaskDataframeColumnPartition
""""""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframeColumnPartition
:members:
PandasOnDaskDataframeRowPartition
"""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.dask.implementations.pandas_on_dask.partitioning.PandasOnDaskDataframeRowPartition
:members:
================================================
FILE: docs/flow/modin/core/execution/dispatching.rst
================================================
:orphan:
..
TODO: add links to documentation for mentioned modules.
Factories Module Description
""""""""""""""""""""""""""""
Brief description
'''''''''''''''''
Modin has several execution engines and storage formats, combining them together forms certain executions.
Calling any :py:class:`~modin.pandas.dataframe.DataFrame` API function will end up in some execution-specific method. The responsibility of dispatching high-level API calls to
execution-specific function belongs to the :ref:`QueryCompiler `, which is determined at the time of the dataframe's creation by the factory of
the corresponding execution. The mission of this module is to route IO function calls from
the API level to its actual execution-specific implementations, which builds the
`QueryCompiler` of the appropriate execution.
Execution representation via Factories
''''''''''''''''''''''''''''''''''''''
Execution is a combination of the :doc:`storage format ` and an actual execution engine.
For example, ``PandasOnRay`` execution means the combination of the `pandas storage format` and `Ray` engine.
Each storage format has its own :ref:`Query Compiler ` which compiles the most efficient queries
for the corresponding :doc:`Core Modin Dataframe ` implementation. Speaking about ``PandasOnRay``
execution, its Query Compiler is :doc:`PandasQueryCompiler ` and the
Dataframe implementation is :doc:`PandasDataframe `,
which is general implementation for every execution of the pandas storage format. The actual implementation of ``PandasOnRay`` dataframe
is defined by the :doc:`PandasOnRayDataframe ` class that
extends ``PandasDataframe``.
In the scope of this module, each execution is represented with a factory class located in
``modin/core/execution/dispatching/factories/factories.py``. Each factory contains a field that identifies the IO module of the corresponding execution. This IO module is
responsible for dispatching calls of IO functions to their actual implementations in the
underlying IO module. For more information about IO module visit :doc:`IO ` page.
Factory Dispatcher
''''''''''''''''''
The :py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` class provides
public methods whose interface corresponds to pandas IO functions, the only difference is that they return `QueryCompiler` of the
selected storage format instead of high-level :py:class:`~modin.pandas.dataframe.DataFrame`. ``FactoryDispatcher`` is responsible for routing
these IO calls to the factory which represents the selected execution.
So when you call ``read_csv()`` function and your execution is ``PandasOnRay`` then the
trace would be the following:
.. figure:: /img/factory_dispatching.svg
:align: center
``modin.pandas.read_csv`` calls ``FactoryDispatcher.read_csv``, which calls ``._read_csv``
function of the factory of the selected execution, in our case it's ``PandasOnRayFactory._read_csv``,
which in turn forwards this call to the actual implementation of ``read_csv`` — to the
``PandasOnRayIO.read_csv``. The result of ``modin.pandas.read_csv`` will return a high-level Modin
DataFrame with the appropriate `QueryCompiler` bound to it, which is responsible for
dispatching all of the further function calls.
Public API
''''''''''
.. automodule:: modin.core.execution.dispatching.factories.factories
:members:
================================================
FILE: docs/flow/modin/core/execution/python/implementations/pandas_on_python/dataframe.rst
================================================
PandasOnPythonDataframe
"""""""""""""""""""""""
The class is specific implementation of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`
for `Python` execution engine. It serves as an intermediate level between
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` and
:py:class:`~modin.core.execution.python.implementations.pandas_on_python.partitioning.partition_manager.PandasOnPythonDataframePartitionManager`.
Public API
----------
.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.dataframe.dataframe.PandasOnPythonDataframe
:members:
================================================
FILE: docs/flow/modin/core/execution/python/implementations/pandas_on_python/index.rst
================================================
:orphan:
PandasOnPython Execution
========================
Queries that perform data transformation, data ingress or data egress using the `pandas on Python` execution
pass through the Modin components detailed below.
`pandas on Python` execution is sequential and it's used for the debug purposes. To enable `pandas on Python` execution,
please refer to the usage section in :doc:`pandas on Python `.
Data Transformation
'''''''''''''''''''
.. image:: /img/pandas_on_python_data_transform.svg
:align: center
When a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer
to be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,
for example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query
and defining specific intermediate values to provide more context to the query compiler.
The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for
processing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,
to determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`
storage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of
the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.dataframe.dataframe.PandasOnPythonDataframe` which inherits
generic functionality from the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.
PandasOnPython Dataframe implementation
---------------------------------------
This page describes implementation of :doc:`Modin PandasDataframe Objects `
specific for `PandasOnPython` execution. Since Python engine doesn't allow computation parallelization,
operations on partitions are performed sequentially. The absence of parallelization doesn't give any
performance speed-up, so ``PandasOnPython`` is used for testing purposes only.
* :doc:`PandasOnPythonDataframe `
* :doc:`PandasOnPythonDataframePartition `
* :doc:`PandasOnPythonDataframeAxisPartition `
* :doc:`PandasOnPythonDataframePartitionManager `
.. toctree::
:hidden:
dataframe
partitioning/partition
partitioning/axis_partition
partitioning/partition_manager
Data Ingress
''''''''''''
.. image:: /img/pandas_on_python_data_ingress.svg
:align: center
Data Egress
'''''''''''
.. image:: /img/pandas_on_python_data_egress.svg
:align: center
When a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the
:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for
the execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnPythonFactory`. The factory, in turn,
exposes the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.io.PandasOnPythonIO` class
whose responsibility is a read/write from/to a file.
When reading data from a CSV file, for example, the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.io.io.PandasOnPythonIO` class
reads the data using corresponding `pandas` function (``pandas.read_csv()`` in this case). After the reading is complete, a new query compiler is created from `pandas` object
using :py:meth:`~modin.core.execution.python.implementations.pandas_on_python.io.io.PandasOnPythonIO.from_pandas` and returned.
When writing data to a CSV file, for example, the :py:class:`~modin.core.execution.python.implementations.pandas_on_python.io.PandasOnPythonIO` converts a query compiler
to `pandas` object using :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.to_pandas`. After that, `pandas` writes the data to the file using
corresponding function (``pandas.to_csv()`` in this case).
================================================
FILE: docs/flow/modin/core/execution/python/implementations/pandas_on_python/partitioning/axis_partition.rst
================================================
PandasOnPythonDataframeAxisPartition
""""""""""""""""""""""""""""""""""""
The class is specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition`,
providing the API to perform operations on an axis partition, using Python
as the execution engine. The axis partition is made up of list of block
partitions that are stored in this class.
Public API
----------
.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.virtual_partition.PandasOnPythonDataframeAxisPartition
PandasOnPythonFrameColumnPartition
""""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.virtual_partition.PandasOnPythonDataframeColumnPartition
:members:
PandasOnPythonFrameRowPartition
"""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.virtual_partition.PandasOnPythonDataframeRowPartition
:members:
================================================
FILE: docs/flow/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.rst
================================================
PandasOnPythonDataframePartition
""""""""""""""""""""""""""""""""
The class is specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,
providing the API to perform operations on a block partition using Python as the execution engine.
In addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:
* ``length`` - length of ``pandas.DataFrame`` wrapped
* ``width`` - width of ``pandas.DataFrame`` wrapped
An operation on a block partition can be performed in two modes:
* immediately via :meth:`~modin.core.execution.python.implementations.pandas_on_python.partitioning.partition.PandasOnPythonDataframePartition.apply` -
in this case accumulated call queue and new function will be executed
immediately.
* lazily_ via :meth:`~modin.core.execution.python.implementations.pandas_on_python.partitioning.partition.PandasOnPythonDataframePartition.add_to_apply_calls` -
in this case function will be added to the call queue and no computations
will be done at the moment.
Public API
----------
.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.partition.PandasOnPythonDataframePartition
:members:
.. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation
================================================
FILE: docs/flow/modin/core/execution/python/implementations/pandas_on_python/partitioning/partition_manager.rst
================================================
PandasOnPythonDataframePartition
""""""""""""""""""""""""""""""""
The class is specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition_manager.PandasDataframePartitionManager`
using Python as the execution engine. This class is responsible for partitions manipulation and applying
a function to block/row/column partitions.
Public API
----------
.. autoclass:: modin.core.execution.python.implementations.pandas_on_python.partitioning.partition_manager.PandasOnPythonDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/execution/ray/generic.rst
================================================
:orphan:
Generic Ray-based members
=========================
Objects which are storage format agnostic but require specific Ray implementation
are placed in ``modin.core.execution.ray.generic``.
Their purpose is to implement certain parallel I/O operations and to serve
as a foundation for building storage format specific objects:
.. autoclass:: modin.core.execution.ray.generic.io.RayIO
:members:
.. autoclass:: modin.core.execution.ray.generic.partitioning.GenericRayDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/dataframe.rst
================================================
PandasOnRayDataframe
""""""""""""""""""""
The class is specific implementation of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`
class using Ray distributed engine. It serves as an intermediate level between
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` and
:py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartitionManager`.
Public API
----------
.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.dataframe.PandasOnRayDataframe
:members:
================================================
FILE: docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/index.rst
================================================
:orphan:
PandasOnRay Execution
=====================
Queries that perform data transformation, data ingress or data egress using the `pandas on Ray` execution
pass through the Modin components detailed below.
To enable `pandas on Ray` execution, please refer to the usage section in :doc:`pandas on Ray `.
Data Transformation
'''''''''''''''''''
.. image:: /img/pandas_on_ray_data_transform.svg
:align: center
When a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer
to be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,
for example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query
and defining specific intermediate values to provide more context to the query compiler.
The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for
processing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,
to determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`
storage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of
the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.dataframe.PandasOnRayDataframe` which inherits
generic functionality from the ``GenericRayDataframe`` and the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.
..
TODO: insert a link to ``GenericRayDataframe`` once we add an implementatiton of the class
PandasOnRay Dataframe implementation
------------------------------------
Modin implements ``Dataframe``, ``PartitionManager``, ``VirtualPartition`` (a specific kind of ``AxisPartition`` with the capability
to combine smaller partitions into the one "virtual") and ``Partition`` classes specifically for the ``PandasOnRay`` execution:
* :doc:`PandasOnRayDataframe `
* :doc:`PandasOnRayDataframePartition `
* :doc:`PandasOnRayDataframeVirtualPartition `
* :doc:`PandasOnRayDataframePartitionManager `
.. toctree::
:hidden:
dataframe
partitioning/partition
partitioning/axis_partition
partitioning/partition_manager
Data Ingress
''''''''''''
.. image:: /img/pandas_on_ray_data_ingress.svg
:align: center
Data Egress
'''''''''''
.. image:: /img/pandas_on_ray_data_egress.svg
:align: center
When a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the
:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for
the execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnRayFactory`. The factory, in turn,
exposes the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` class
whose responsibility is to perform a parallel read/write from/to a file.
When reading data from a CSV file, for example, the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` class forwards
the user query to the :meth:`~modin.core.io.text.CSVDispatcher._read` method of :py:class:`~modin.core.io.text.CSVDispatcher`, where the query's parameters are preprocessed
to check if they are supported by the execution (defaulting to pandas if they are not) and computes some metadata
common for all partitions to be read. Then, the file is split into row chunks, and this data is used to launch remote tasks on the Ray workers
via the :meth:`~modin.core.execution.ray.common.RayWrapper.deploy` method of :py:class:`~modin.core.execution.ray.common.RayWrapper`.
On each Ray worker, the :py:class:`~modin.core.storage_formats.pandas.parsers.PandasCSVParser` parses data.
After the remote tasks are finished, additional result postprocessing is performed,
and a new query compiler with the data read is returned.
When writing data to a CSV file, for example, the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` processes
the user query to execute it on Ray workers. Then, the :py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO` asks the
:py:class:`~modin.core.execution.ray.implementations.pandas_on_ray.dataframe.PandasOnRayDataframe` to decompose the data into row-wise partitions
that will be written into the file in parallel in Ray workers.
================================================
FILE: docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/axis_partition.rst
================================================
PandasOnRayDataframeVirtualPartition
""""""""""""""""""""""""""""""""""""
This class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition`,
providing the API to perform operations on an axis partition, using Ray as an execution engine. The virtual partition is
a wrapper over a list of block partitions, which are stored in this class, with the capability to combine the smaller partitions into the one "virtual".
Public API
----------
.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframeVirtualPartition
:members:
PandasOnRayDataframeColumnPartition
"""""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframeColumnPartition
:members:
PandasOnRayDataframeRowPartition
""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframeRowPartition
:members:
================================================
FILE: docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.rst
================================================
PandasOnRayDataframePartition
"""""""""""""""""""""""""""""
The class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,
providing the API to perform operations on a block partition, namely, ``pandas.DataFrame``, using Ray as an execution engine.
In addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:
* ``length`` - length of ``pandas.DataFrame`` wrapped
* ``width`` - width of ``pandas.DataFrame`` wrapped
* ``ip`` - node IP address that holds ``pandas.DataFrame`` wrapped
An operation on a block partition can be performed in two modes:
* asynchronously_ - via :meth:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartition.apply`
* lazily_ - via :meth:`~modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartition.add_to_apply_calls`
Public API
----------
.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartition
:members:
.. _asynchronously: https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)
.. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation
================================================
FILE: docs/flow/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.rst
================================================
PandasOnRayDataframePartitionManager
""""""""""""""""""""""""""""""""""""
This class is the specific implementation of :py:class:`~modin.core.execution.ray.generic.partitioning.GenericRayDataframePartitionManager`
using Ray distributed engine. This class is responsible for partition manipulation and applying a function to
block/row/column partitions.
Public API
----------
.. autoclass:: modin.core.execution.ray.implementations.pandas_on_ray.partitioning.PandasOnRayDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/execution/unidist/generic.rst
================================================
:orphan:
Generic Unidist-based members
=============================
Objects which are storage format agnostic but require specific Unidist implementation
are placed in ``modin.core.execution.unidist.generic``.
Their purpose is to implement certain parallel I/O operations and to serve
as a foundation for building storage format specific objects:
.. autoclass:: modin.core.execution.unidist.generic.io.UnidistIO
:members:
.. autoclass:: modin.core.execution.unidist.generic.partitioning.GenericUnidistDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/dataframe.rst
================================================
PandasOnUnidistDataframe
""""""""""""""""""""""""
The class is specific implementation of :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`
class using Unidist distributed engine. It serves as an intermediate level between
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` and
:py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartitionManager`.
Public API
----------
.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.dataframe.PandasOnUnidistDataframe
:members:
================================================
FILE: docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/index.rst
================================================
:orphan:
PandasOnUnidist Execution
=========================
Queries that perform data transformation, data ingress or data egress using the `pandas on Unidist` execution
pass through the Modin components detailed below.
To enable `pandas on MPI through unidist` execution,
please refer to the usage section in :doc:`pandas on MPI through unidist `.
Data Transformation
'''''''''''''''''''
.. image:: /img/pandas_on_unidist_data_transform.svg
:align: center
When a user calls any :py:class:`~modin.pandas.dataframe.DataFrame` API, a query starts forming at the `API` layer
to be executed at the `Execution` layer. The `API` layer is responsible for processing the query appropriately,
for example, determining whether the final result should be a ``DataFrame`` or ``Series`` object. This layer is also responsible for sanitizing the input to the
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler`, e.g. validating a parameter from the query
and defining specific intermediate values to provide more context to the query compiler.
The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for
processing the query, received from the :py:class:`~modin.pandas.dataframe.DataFrame` `API` layer,
to determine how to apply it to a subset of the data - either cell-wise or along an axis-wise partition backed by the `pandas`
storage format. The :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` maps the query to one of the :doc:`Core Algebra Operators ` of
the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.dataframe.PandasOnUnidistDataframe` which inherits
generic functionality from the ``GenericUnidistDataframe`` and the :py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe`.
..
TODO: insert a link to ``GenericUnidistDataframe`` once we add an implementatiton of the class
PandasOnUnidist Dataframe implementation
----------------------------------------
Modin implements ``Dataframe``, ``PartitionManager``, ``VirtualPartition`` (a specific kind of ``AxisPartition`` with the capability
to combine smaller partitions into the one "virtual") and ``Partition`` classes specifically for the ``PandasOnUnidist`` execution:
* :doc:`PandasOnUnidistDataframe `
* :doc:`PandasOnUnidistDataframePartition `
* :doc:`PandasOnUnidistDataframeVirtualPartition `
* :doc:`PandasOnUnidistDataframePartitionManager `
.. toctree::
:hidden:
dataframe
partitioning/partition
partitioning/axis_partition
partitioning/partition_manager
Data Ingress
''''''''''''
.. image:: /img/pandas_on_unidist_data_ingress.svg
:align: center
Data Egress
'''''''''''
.. image:: /img/pandas_on_unidist_data_egress.svg
:align: center
When a user calls any IO function from the ``modin.pandas.io`` module, the `API` layer queries the
:py:class:`~modin.core.execution.dispatching.factories.dispatcher.FactoryDispatcher` which defines a factory specific for
the execution, namely, the :py:class:`~modin.core.execution.dispatching.factories.factories.PandasOnUnidistFactory`. The factory, in turn,
exposes the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` class
whose responsibility is to perform a parallel read/write from/to a file.
When reading data from a CSV file, for example, the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` class forwards
the user query to the :meth:`~modin.core.io.text.CSVDispatcher._read` method of :py:class:`~modin.core.io.text.CSVDispatcher`, where the query's parameters are preprocessed
to check if they are supported by the execution (defaulting to pandas if they are not) and computes some metadata
common for all partitions to be read. Then, the file is split into row chunks, and this data is used to launch remote tasks on the Unidist workers
via the :meth:`~modin.core.execution.unidist.common.UnidistWrapper.deploy` method of :py:class:`~modin.core.execution.unidist.common.UnidistWrapper`.
On each Unidist worker, the :py:class:`~modin.core.storage_formats.pandas.parsers.PandasCSVParser` parses data.
After the remote tasks are finished, additional result postprocessing is performed,
and a new query compiler with the data read is returned.
When writing data to a CSV file, for example, the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` processes
the user query to execute it on Unidist workers. Then, the :py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.io.PandasOnUnidistIO` asks the
:py:class:`~modin.core.execution.unidist.implementations.pandas_on_unidist.dataframe.PandasOnUnidistDataframe` to decompose the data into row-wise partitions
that will be written into the file in parallel in Unidist workers.
================================================
FILE: docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/partitioning/axis_partition.rst
================================================
PandasOnUnidistDataframeVirtualPartition
""""""""""""""""""""""""""""""""""""""""
This class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.axis_partition.PandasDataframeAxisPartition`,
providing the API to perform operations on an axis partition, using Unidist as an execution engine. The virtual partition is
a wrapper over a list of block partitions, which are stored in this class, with the capability to combine the smaller partitions into the one "virtual".
Public API
----------
.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframeVirtualPartition
:members:
PandasOnUnidistDataframeColumnPartition
"""""""""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframeColumnPartition
:members:
PandasOnUnidistDataframeRowPartition
""""""""""""""""""""""""""""""""""""
Public API
----------
.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframeRowPartition
:members:
================================================
FILE: docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/partitioning/partition.rst
================================================
PandasOnUnidistDataframePartition
"""""""""""""""""""""""""""""""""
The class is the specific implementation of :py:class:`~modin.core.dataframe.pandas.partitioning.partition.PandasDataframePartition`,
providing the API to perform operations on a block partition, namely, ``pandas.DataFrame``, using Unidist as an execution engine.
In addition to wrapping a ``pandas.DataFrame``, the class also holds the following metadata:
* ``length`` - length of ``pandas.DataFrame`` wrapped
* ``width`` - width of ``pandas.DataFrame`` wrapped
* ``ip`` - node IP address that holds ``pandas.DataFrame`` wrapped
An operation on a block partition can be performed in two modes:
* asynchronously_ - via :meth:`~modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartition.apply`
* lazily_ - via :meth:`~modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartition.add_to_apply_calls`
Public API
----------
.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartition
:members:
.. _asynchronously: https://en.wikipedia.org/wiki/Asynchrony_(computer_programming)
.. _lazily: https://en.wikipedia.org/wiki/Lazy_evaluation
================================================
FILE: docs/flow/modin/core/execution/unidist/implementations/pandas_on_unidist/partitioning/partition_manager.rst
================================================
PandasOnUnidistDataframePartitionManager
""""""""""""""""""""""""""""""""""""""""
This class is the specific implementation of :py:class:`~modin.core.execution.unidist.generic.partitioning.GenericUnidistDataframePartitionManager`
using Unidist distributed engine. This class is responsible for partition manipulation and applying a function to
block/row/column partitions.
Public API
----------
.. autoclass:: modin.core.execution.unidist.implementations.pandas_on_unidist.partitioning.PandasOnUnidistDataframePartitionManager
:members:
================================================
FILE: docs/flow/modin/core/io/index.rst
================================================
:orphan:
IO Module Description
"""""""""""""""""""""
Dispatcher Classes Workflow Overview
''''''''''''''''''''''''''''''''''''
Calls from ``read_*`` functions of execution-specific IO classes (for example, ``PandasOnRayIO`` for
Ray engine and pandas storage format) are forwarded to the ``_read`` function of the file
format-specific class (for example ``CSVDispatcher`` for CSV files), where function parameters are
preprocessed to check if they are supported (defaulting to pandas if not)
and common metadata is computed for all partitions. The file is then split
into chunks (splitting mechanism described below) and the data is used to launch tasks
on the remote workers. After the remote tasks finish, additional
postprocessing is performed on the results, and a new query compiler with the imported data will
be returned.
Data File Splitting Mechanism
'''''''''''''''''''''''''''''
Modin's file splitting mechanism differs depending on the data format type:
* text format type - the file is split into bytes according to user specified arguments.
In the simplest case, when no row related parameters (such as ``nrows`` or
``skiprows``) are passed, data chunk limits (start and end bytes) are derived
by dividing the file size by the number of partitions (chunks can
slightly differ between each other because usually end byte may occurs inside a
line and in that case the last byte of the line should be used instead of initial
value). In other cases the same splitting mechanism is used, but chunks sizes are
defined according to the number of lines that each partition should contain.
* columnar store type - the file is split so that each chunk contains approximately the same number of columns.
* SQL type - chunking is obtained by wrapping initial SQL query with a query that
specifies initial row offset and number of rows in the chunk.
After file splitting is complete, chunks data is passed to the parser functions
(``PandasCSVParser.parse`` for ``read_csv`` function with pandas storage format) for
further processing on each worker.
Submodules Description
''''''''''''''''''''''
``modin.core.io`` module is used mostly for storing utils and dispatcher
classes for reading files of different formats.
* ``io.py`` - class containing basic utils and default implementation of IO functions.
* ``file_dispatcher.py`` - class reading data from different kinds of files and
handling some util functions common for all formats. Also this class contains ``read``
function which is entry point function for all dispatchers ``_read`` functions.
* text - directory for storing all text file format dispatcher classes
* ``text_file_dispatcher.py`` - class for reading text formats files. This class
holds ``partitioned_file`` function for splitting text format files into chunks,
``offset`` function for moving file offset at the specified amount of bytes,
``_read_rows`` function for moving file offset at the specified amount of rows
and many other functions.
* format/feature specific dispatchers: ``csv_dispatcher.py``, ``excel_dispatcher.py``,
``fwf_dispatcher.py`` and ``json_dispatcher.py``.
* column_stores - directory for storing all columnar store file format dispatcher classes
* ``column_store_dispatcher.py`` - class for reading columnar type files. This class
holds ``build_query_compiler`` function that performs file splitting, deploying remote
tasks and results postprocessing and many other functions.
* format/feature specific dispatchers: ``feather_dispatcher.py``, ``hdf_dispatcher.py``
and ``parquet_dispatcher.py``.
* sql - directory for storing SQL dispatcher class
* ``sql_dispatcher.py`` - class for reading SQL queries or database tables.
Public API
''''''''''
.. automodule:: modin.core.io
:members:
Handling ``skiprows`` Parameter
'''''''''''''''''''''''''''''''
Handling ``skiprows`` parameter by pandas import functions can be very tricky, especially
for ``read_csv`` function because of interconnection with ``header`` parameter. In this section
the techniques of ``skiprows`` processing by both pandas and Modin are covered.
Processing ``skiprows`` by pandas
=================================
Let's consider a simple snippet with ``pandas.read_csv`` in order to understand interconnection
of ``header`` and ``skiprows`` parameters:
.. code-block:: python
import pandas
from io import StringIO
data = """0
1
2
3
4
5
6
7
8
"""
# `header` parameter absence is equivalent to `header="infer"` or `header=0`
# rows 1, 5, 6, 7, 8 are read with header "0"
df = pandas.read_csv(StringIO(data), skiprows=[2, 3, 4])
# rows 5, 6, 7, 8 are read with header "1", row 0 is skipped additionally
df = pandas.read_csv(StringIO(data), skiprows=[2, 3, 4], header=1)
# rows 6, 7, 8 are read with header "5", rows 0, 1 are skipped additionally
df = pandas.read_csv(StringIO(data), skiprows=[2, 3, 4], header=2)
In the examples above list-like ``skiprows`` values are fixed and ``header`` is varied. In the first
example with no ``header`` provided, rows 2, 3, 4 are skipped and row 0 is considered as the header.
In the second example ``header == 1``, so the zeroth row is skipped and the next available row is
considered the header. The third example illustrates when the ``header`` and ``skiprows`` parameters
values are both present - in this case ``skiprows`` rows are dropped first and then the ``header`` is derived
from the remaining rows (rows before header are skipped too).
In the examples above only list-like ``skiprows`` and integer ``header`` parameters are considered,
but the same logic is applicable for other types of the parameters.
Processing ``skiprows`` by Modin
================================
As it can be seen, skipping rows in the pandas import functions is complicated and distributing
this logic across multiple workers can complicate it even more. Thus in some rare corner cases
default pandas implementation is used in Modin to avoid excessive Modin code complication.
Modin uses two techniques for skipping rows:
1) During file partitioning (setting file limits that should be read by each partition)
exact rows can be excluded from partitioning scope, thus they won't be read at all and can be
considered as skipped. This is the most effective way of skipping rows since it doesn't require
any actual data reading and postprocessing, but in this case ``skiprows`` parameter can be an
integer only. When it is possible Modin always uses this approach.
2) Rows for skipping can be dropped after full dataset import. This is more expensive way since
it requires extra IO work and postprocessing afterwards, but ``skiprows`` parameter can be of any
non-integer type supported by ``pandas.read_csv``.
In some cases, if ``skiprows`` is uniformly distributed array (e.g. [1, 2, 3]), ``skiprows`` can be
"squashed" and represented as an integer to make a fastpath by skipping these rows during file partitioning
(using the first option). But if there is a gap between the first row for skipping
and the last line of the header (that will be skipped too since header is read by each partition
to ensure metadata is defined properly), then this gap should be assigned for reading first
by assigning the first partition to read these rows by setting ``pre_reading`` parameter.
Let's consider an example of skipping rows during partitioning when ``header="infer"`` and
``skiprows=[3, 4, 5]``. In this specific case fastpath can be done since ``skiprows`` is uniformly
distributed array, so we can "squash" it to an integer and set "partitioning" skiprows to 3. But
if no additional action is done, these three rows will be skipped right after header line,
that corresponds to ``skiprows=[1, 2, 3]``. To avoid this discrepancy, we need to assign the first
partition to read data between header line and the first row for skipping by setting special
``pre_reading`` parameter to 2. Then, after the skipping of rows considered to be skipped during
partitioning, the rest data will be divided between the rest of partitions, see rows assignment
below:
.. code-block::
0 - header line (skip during partitioning)
1 - pre reading (assign to read by the first partition)
2 - pre reading (assign to read by the first partition)
3 - "partitioning" skiprows (skip during partitioning)
4 - "partitioning" skiprows (skip during partitioning)
5 - "partitioning" skiprows (skip during partitioning)
6 - data to partition (divide between the rest of partitions)
7 - data to partition (divide between the rest of partitions)
================================================
FILE: docs/flow/modin/core/storage_formats/base/query_compiler.rst
================================================
BaseQueryCompiler
"""""""""""""""""
Brief description
'''''''''''''''''
:py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` is an abstract class of query compiler, and sets a common interface
that every other query compiler implementation in Modin must follow. The Base class contains a basic
implementations for most of the interface methods, all of which
:doc:`fallback to pandas `.
Subclassing :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler`
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
If you want to add new type of query compiler to Modin the new class needs to inherit
from :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` and implement the abstract methods:
- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.from_pandas` build query compiler from pandas DataFrame.
- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.from_arrow` build query compiler from Arrow Table.
- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.to_pandas` get query compiler representation as pandas DataFrame.
- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.default_to_pandas` do :doc:`fallback to pandas ` for the passed function.
- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.finalize` finalize object constructing.
- :py:meth:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler.free` trigger memory cleaning.
(Please refer to the code documentation to see the full documentation for these functions).
This is a minimum set of operations to ensure a new query compiler will function in the Modin architecture,
and the rest of the API can safely default to the pandas implementation via the base class implementation.
To add a storage format specific implementation for some of the query compiler operations, just override
the corresponding method in your query compiler class.
Example
'''''''
As an exercise let's define a new query compiler in `Modin`, just to see how easy it is.
Usually, the query compiler routes formed queries to the underlying :doc:`frame ` class,
which submits operators to an execution engine. For the sake
of simplicity and independence of this example, our execution engine will be the pandas itself.
We need to inherit a new class from :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` and implement all of the abstract methods.
In this case, with `pandas` as an execution engine, it's trivial:
.. code-block:: python
from modin.core.storage_formats import BaseQueryCompiler
class DefaultToPandasQueryCompiler(BaseQueryCompiler):
def __init__(self, pandas_df):
self._pandas_df = pandas_df
@classmethod
def from_pandas(cls, df, *args, **kwargs):
return cls(df)
@classmethod
def from_arrow(cls, at, *args, **kwargs):
return cls(at.to_pandas())
def to_pandas(self):
return self._pandas_df.copy()
def default_to_pandas(self, pandas_op, *args, **kwargs):
return type(self)(pandas_op(self.to_pandas(), *args, **kwargs))
def finalize(self):
pass
def free(self):
pass
All done! Now you've got a fully functional query compiler, which is ready for extensions
and already can be used in Modin DataFrame:
.. code-block:: python
import pandas
pandas_df = pandas.DataFrame({"col1": [1, 2, 2, 1], "col2": [10, 2, 3, 40]})
# Building our query compiler from pandas object
qc = DefaultToPandasQueryCompiler.from_pandas(pandas_df)
import modin.pandas as pd
# Building Modin DataFrame from newly created query compiler
modin_df = pd.DataFrame(query_compiler=qc)
# Got fully functional Modin DataFrame
>>> print(modin_df.groupby("col1").sum().reset_index())
col1 col2
0 1 50
1 2 5
To be able to select this query compiler as default via ``modin.config`` you also need
to define the combination of your query compiler and pandas engine as an execution
by adding the corresponding factory. To find more information about factories,
visit :doc:`dispatching ` page.
Query Compiler API
''''''''''''''''''
.. autoclass:: modin.core.storage_formats.base.query_compiler.BaseQueryCompiler
:members:
================================================
FILE: docs/flow/modin/core/storage_formats/index.rst
================================================
:orphan:
Storage Formats
===============
Storage format is one of the components that form Modin's execution, it describes the type(s)
of objects that are stored in the partitions of the selected Core Modin Dataframe implementation.
The base storage format in Modin is pandas. In that format, Modin Dataframe operates with
partitions that hold ``pandas.DataFrame`` objects. Pandas is the most natural storage format
since high-level DataFrame objects mirror its API.
The storage format + execution engine (Ray, Dask, etc.) form the execution backend.
The Query Compiler (QC) converts high-level pandas API calls to queries that are understood
by the execution backend.
.. _query_compiler_def:
Query Compiler
==============
.. toctree::
:hidden:
base/query_compiler
pandas/index
Modin supports several execution backends (storage format + execution engine). Calling any
DataFrame API function will end up in some execution-specific method. The query compiler is
a bridge between pandas DataFrame API and the actual Core Modin Dataframe implementation for the
corresponding execution.
.. image:: /img/simplified_query_flow.svg
:align: right
:width: 300px
Each storage format has its own Query Compiler class that implements the most optimal
query routing for the selected format.
Query compilers of all storage formats implement a common API, which is used by the high-level Modin DataFrame
to support dataframe queries. The role of the query compiler is to translate its API into
a pairing of known user-defined functions and dataframe algebra operators. Each query compiler instance contains a
:doc:`Core Modin Dataframe ` of the selected execution implementation and queries
it with the compiled queries to get the result. The query compiler object is immutable,
so the result of every method is a new query compiler.
The query compilers API is defined by the :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler` class
and may resemble the pandas API, however, they're not equal. The query compilers API
is significantly reduced in comparison with pandas, since many corner cases or even the
whole methods can be handled at the API layer with the existing API.
The query compiler is the level where Modin stops distinguishing DataFrame and Series (or column) objects.
A Series is represented by a `1xN` query compiler, where the Series name is the column label.
If Series is unnamed, then the label is ``MODIN_UNNAMED_SERIES_LABEL``, which is equal to ``"__reduced__"``. The high-level DataFrame API layer
interprets a one-column query compiler as Series or DataFrame depending on the operation context.
.. note::
Although we're declaring that there is no difference between DataFrame and Series at the query compiler,
you still may find methods like ``method_ser`` and ``method_df`` which are implemented differently because they're
emulating either Series or DataFrame logic, or you may find parameters, which indicates whether this one-column
query compiler is representing Series or not. All of these are hacks, and we're working on getting rid of them.
High-level module overview
''''''''''''''''''''''''''
This module houses submodules of all of the stable storage formats:
- :doc:`Base module ` contains an abstract query compiler class which defines common API.
- :doc:`Pandas module ` contains query compiler and text parsers for pandas storage format.
================================================
FILE: docs/flow/modin/core/storage_formats/pandas/index.rst
================================================
:orphan:
Pandas storage format
"""""""""""""""""""""
.. toctree::
:hidden:
query_compiler
parsers
High-Level Module Overview
''''''''''''''''''''''''''
This module houses submodules which are responsible for communication between
the query compiler level and execution implementation level for pandas storage format:
- :doc:`Query compiler ` is responsible for compiling efficient queries for :doc:`PandasDataframe `.
- :doc:`Parsers ` are responsible for parsing data on workers during IO operations.
================================================
FILE: docs/flow/modin/core/storage_formats/pandas/parsers.rst
================================================
Pandas Parsers Module Description
"""""""""""""""""""""""""""""""""
High-Level Module Overview
''''''''''''''''''''''''''
This module houses parser classes (classes that are used for data parsing on the workers)
and util functions for handling parsing results. ``PandasParser`` is base class for parser
classes with pandas storage format, that contains methods common for all child classes. Other
module classes implement ``parse`` function that performs parsing of specific format data
basing on the chunk information computed in the ``modin.core.io`` module. After
the chunk is parsed, the resulting ``DataFrame``-s will be split into smaller
``DataFrame``-s according to the ``num_splits`` parameter, data type, or number of
rows/columns in the parsed chunk. These frames, along with some additional metadata, are then returned.
.. note::
If you are interested in the data parsing mechanism implementation details, please refer
to the source code documentation.
Public API
''''''''''
.. automodule:: modin.core.storage_formats.pandas.parsers
:members:
================================================
FILE: docs/flow/modin/core/storage_formats/pandas/query_compiler.rst
================================================
PandasQueryCompiler
"""""""""""""""""""
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` is responsible for compiling
a set of known predefined functions and pairing those with dataframe algebra operators in the
:doc:`PandasDataframe `, specifically for dataframes backed by
``pandas.DataFrame`` objects.
Each :py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` contains an instance of
:py:class:`~modin.core.dataframe.pandas.dataframe.dataframe.PandasDataframe` which it queries to get the result.
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` supports methods built by
the :doc:`algebra module `.
If you want to add an implementation for a query compiler method, visit the algebra module documentation
to see whether the new operation fits one of the existing function templates and can be easily implemented
with them.
Public API
''''''''''
:py:class:`~modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler` implements common query compilers API
defined by the :py:class:`~modin.core.storage_formats.base.query_compiler.BaseQueryCompiler`. Some functionalities
are inherited from the base class, in the following section only overridden methods are presented.
.. autoclass:: modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler
:members:
================================================
FILE: docs/flow/modin/distributed/dataframe/pandas.rst
================================================
Pandas partitioning API
=======================
This page contains a description of the API to extract partitions from and build Modin Dataframes.
unwrap_partitions
-----------------
.. autofunction:: modin.distributed.dataframe.pandas.unwrap_partitions
from_partitions
---------------
.. autofunction:: modin.distributed.dataframe.pandas.from_partitions
Example
-------
.. code-block:: python
import modin.pandas as pd
from modin.distributed.dataframe.pandas import unwrap_partitions, from_partitions
import numpy as np
data = np.random.randint(0, 100, size=(2 ** 10, 2 ** 8))
df = pd.DataFrame(data)
partitions = unwrap_partitions(df, axis=0, get_ip=True)
print(partitions)
new_df = from_partitions(partitions, axis=0)
print(new_df)
================================================
FILE: docs/flow/modin/experimental/batch.rst
================================================
Batch Pipeline API
""""""""""""""""""
This API exposes the ability to pipeline row-parallel batch queries on a Modin DataFrame. Currently,
this feature is only supported for the ``PandasOnRay`` execution.
API
'''
.. automodule:: modin.experimental.batch.pipeline
:members:
================================================
FILE: docs/flow/modin/experimental/core/io/index.rst
================================================
:orphan:
Experimental IO Module Description
""""""""""""""""""""""""""""""""""
The module is used mostly for storing experimental utils and
dispatcher classes for reading/writing files of different formats.
Submodules Description
''''''''''''''''''''''
* text - directory for storing all text file format dispatcher classes
* format/feature specific dispatchers: ``csv_glob_dispatcher.py``,
``custom_text_dispatcher.py``.
* sql - directory for storing SQL dispatcher class
* format/feature specific dispatchers: ``sql_dispatcher.py``
* pickle - directory for storing Pickle dispatcher class
* format/feature specific dispatchers: ``pickle_dispatcher.py``
Public API
''''''''''
.. automodule:: modin.experimental.core.io
:members:
================================================
FILE: docs/flow/modin/experimental/index.rst
================================================
:orphan:
Experimental Modules Overview
"""""""""""""""""""""""""""""
In some cases Modin can give the user the opportunity to extend (not modify) typical pandas
API or to try new functionality in order to get more flexibility. Depending on the exact
experimental feature user may need to install additional packages, change configurations or
replace the standard Modin import statement ``import modin.pandas as pd`` with modified version
``import modin.experimental.pandas as pd``.
``modin.experimental`` holds experimental functionality that is under development right now
and provides a limited set of functionality:
* :doc:`xgboost `
* :doc:`sklearn `
* :doc:`batch `
.. toctree::
:hidden:
sklearn
xgboost
batch
================================================
FILE: docs/flow/modin/experimental/pandas.rst
================================================
:orphan:
Experimental Pandas API
"""""""""""""""""""""""
.. automodule:: modin.experimental.pandas
:noindex:
Experimental API Reference
''''''''''''''''''''''''''
.. autofunction:: read_sql
.. autofunction:: read_csv_glob
.. autofunction:: read_custom_text
.. autofunction:: read_pickle_glob
.. autofunction:: read_parquet_glob
.. autofunction:: read_json_glob
.. autofunction:: read_xml_glob
.. automethod:: modin.pandas.DataFrame.modin::to_pandas
.. automethod:: modin.pandas.DataFrame.modin::to_ray
.. automethod:: modin.pandas.DataFrame.modin::to_pickle_glob
.. automethod:: modin.pandas.DataFrame.modin::to_parquet_glob
.. automethod:: modin.pandas.DataFrame.modin::to_json_glob
.. automethod:: modin.pandas.DataFrame.modin::to_xml_glob
================================================
FILE: docs/flow/modin/experimental/range_partitioning_groupby.rst
================================================
:orphan:
.. redirect to the new page
.. raw:: html
================================================
FILE: docs/flow/modin/experimental/reshuffling_groupby.rst
================================================
:orphan:
.. redirect to the new page
.. raw:: html
================================================
FILE: docs/flow/modin/experimental/sklearn.rst
================================================
Scikit-learn module description
"""""""""""""""""""""""""""""""
This module holds experimental scikit-learn-specific functionality for Modin.
API
'''
.. automodule:: modin.experimental.sklearn.model_selection
:members:
================================================
FILE: docs/flow/modin/experimental/xgboost.rst
================================================
Modin XGBoost module description
""""""""""""""""""""""""""""""""
High-level Module Overview
''''''''''''''''''''''''''
This module holds classes, public interface and internal functions for distributed XGBoost in Modin.
Public classes :py:class:`~modin.experimental.xgboost.Booster`, :py:class:`~modin.experimental.xgboost.DMatrix`
and function :py:func:`~modin.experimental.xgboost.train` provide the user with familiar XGBoost interfaces.
They are located in the ``modin.experimental.xgboost.xgboost`` module.
The internal module ``modin.experimental.xgboost.xgboost.xgboost_ray`` contains the implementation of Modin XGBoost
for the Ray execution engine. This module mainly consists of the Ray actor-class :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor`,
a function to distribute Modin's partitions between actors :py:func:`~modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors`,
an internal :py:func:`~modin.experimental.xgboost.xgboost_ray._train`/:py:func:`~modin.experimental.xgboost.xgboost_ray._predict`
function used from the public interfaces and additional util functions for computing cluster resources, actor creations etc.
Public interfaces
'''''''''''''''''
:py:class:`~modin.experimental.xgboost.DMatrix` inherits original class ``xgboost.DMatrix`` and overrides
its constructor, which currently supports only `data` and `label` parameters. Both of the parameters must
be ``modin.pandas.DataFrame``, which will be internally unwrapped to lists of delayed objects of Modin's
row partitions using the function :py:func:`~modin.distributed.dataframe.pandas.unwrap_partitions`.
.. autoclass:: modin.experimental.xgboost.DMatrix
:members:
:py:class:`~modin.experimental.xgboost.Booster` inherits original class ``xgboost.Booster`` and
overrides method ``predict``. The difference from original class interface for ``predict``
method is changing the type of the `data` parameter to :py:class:`~modin.experimental.xgboost.DMatrix`.
.. autoclass:: modin.experimental.xgboost.Booster
:members:
:py:func:`~modin.experimental.xgboost.train` function has 2 differences from the original ``train`` function - (1) the
data type of `dtrain` parameter is :py:class:`~modin.experimental.xgboost.DMatrix` and (2) a new parameter `num_actors`.
.. autofunction:: modin.experimental.xgboost.train
Internal execution flow on Ray engine
'''''''''''''''''''''''''''''''''''''
Internal functions :py:func:`~modin.experimental.xgboost.xgboost_ray._train` and
:py:func:`~modin.experimental.xgboost.xgboost_ray._predict` work similar to xgboost.
Training
********
1. The data is passed to the :py:func:`~modin.experimental.xgboost.xgboost_ray._train`
function as a :py:class:`~modin.experimental.xgboost.DMatrix` object. Lists of ``ray.ObjectRef``
corresponding to row partitions of Modin DataFrames are extracted by iterating over the
:py:class:`~modin.experimental.xgboost.DMatrix`. Example:
.. code-block:: python
# Extract lists of row partitions from dtrain (DMatrix object)
X_row_parts, y_row_parts = dtrain
..
2. On this step, the parameter `num_actors` is processed. The internal function :py:func:`~modin.experimental.xgboost.xgboost_ray._get_num_actors`
examines the value provided by the user. In case the value isn't provided, the `num_actors` will be computed using condition that 1 actor should use maximum 2 CPUs.
This condition was chosen for using maximum parallel workers with multithreaded XGBoost training (2 threads
per worker will be used in this case).
.. note:: `num_actors` parameter is made available for public function :py:func:`~modin.experimental.xgboost.train` to allow
fine-tuning for obtaining the best performance in specific use cases.
3. :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor` objects are created.
4. Data `dtrain` is split between actors evenly. The internal function
:py:func:`~modin.experimental.xgboost.xgboost_ray._split_data_across_actors` runs assigning row partitions to actors
using internal function :py:func:`~modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors`.
This function creates a dictionary in the form: `{actor_rank: ([part_i0, part_i3, ..], [0, 3, ..]), ..}`.
.. note:: :py:func:`~modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors` takes into account IP
addresses of row partitions of `dtrain` data to minimize excess data transfer.
5. For each :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor` object ``set_train_data`` method is
called remotely. This method runs loading row partitions in actor according to the dictionary with partitions
distribution from previous step. When data is passed to the actor, the row partitions are automatically materialized
(``ray.ObjectRef`` -> ``pandas.DataFrame``).
6. ``train`` method of :py:class:`~modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor` class object is called remotely. This method
runs XGBoost training on local data of actor, connects to ``Rabit Tracker`` for sharing training state between
actors and returns dictionary with `booster` and `evaluation results`.
7. At the final stage results from actors are returned. `booster` and `evals_result` are returned using ``ray.get``
function from remote actor.
Prediction
**********
1. The data is passed to :py:func:`~modin.experimental.xgboost.xgboost_ray._predict`
function as a :py:class:`~modin.experimental.xgboost.DMatrix` object.
2. :py:func:`~modin.experimental.xgboost.xgboost_ray._map_predict` function is applied remotely for each partition
of the data to make a partial prediction.
3. Result ``modin.pandas.DataFrame`` is created from ``ray.ObjectRef`` objects, obtained in the previous step.
Internal API
''''''''''''
.. autoclass:: modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor
:members:
:private-members:
.. autofunction:: modin.experimental.xgboost.xgboost_ray._assign_row_partitions_to_actors
.. autofunction:: modin.experimental.xgboost.xgboost_ray._train
.. autofunction:: modin.experimental.xgboost.xgboost_ray._predict
.. autofunction:: modin.experimental.xgboost.xgboost_ray._get_num_actors
.. autofunction:: modin.experimental.xgboost.xgboost_ray._split_data_across_actors
.. autofunction:: modin.experimental.xgboost.xgboost_ray._map_predict
================================================
FILE: docs/flow/modin/pandas/base.rst
================================================
Base pandas Dataset API
"""""""""""""""""""""""
The class implements functionality that is common to Modin's pandas API for both ``DataFrame`` and ``Series`` classes.
Public API
----------
.. autoclass:: modin.pandas.base.BasePandasDataset
:noindex:
:members:
================================================
FILE: docs/flow/modin/pandas/dataframe.rst
================================================
:orphan:
DataFrame Module Overview
"""""""""""""""""""""""""
Modin's ``pandas.DataFrame`` API
''''''''''''''''''''''''''''''''
Modin's ``pandas.DataFrame`` API is backed by a distributed object providing an identical
API to pandas. After the user calls some ``DataFrame`` function, this call is internally
rewritten into a representation that can be processed in parallel by the partitions. These
results can be e.g., reduced to single output, identical to the single threaded
pandas ``DataFrame`` method output.
..
TODO: add link to the docs with detailed description of queries compilation
and execution ater DOCS-#2996 is merged.
Public API
----------
.. autoclass:: modin.pandas.dataframe.DataFrame
Usage Guide
'''''''''''
The most efficient way to create Modin ``DataFrame`` is to import data from external
storage using the highly efficient Modin IO methods (for example using ``pd.read_csv``,
see details for Modin IO methods in the :doc:`IO ` page),
but even if the data does not originate from a file, any pandas supported data type or
``pandas.DataFrame`` can be used. Internally, the ``DataFrame`` data is divided into
partitions, which number along an axis usually corresponds to the number of the user's hardware CPUs. If needed,
the number of partitions can be changed by setting ``modin.config.NPartitions``.
Let's consider simple example of creation and interacting with Modin ``DataFrame``:
.. code-block:: python
import modin.config
# This explicitly sets the number of partitions
modin.config.NPartitions.put(4)
import modin.pandas as pd
import pandas
# Create Modin DataFrame from the external file
pd_dataframe = pd.read_csv("test_data.csv")
# Create Modin DataFrame from the python object
# data = {f'col{x}': [f'col{x}_{y}' for y in range(100, 356)] for x in range(4)}
# pd_dataframe = pd.DataFrame(data)
# Create Modin DataFrame from the pandas object
# pd_dataframe = pd.DataFrame(pandas.DataFrame(data))
# Show created DataFrame
print(pd_dataframe)
# List DataFrame partitions. Note, that internal API is intended for
# developers needs and was used here for presentation purposes
# only.
partitions = pd_dataframe._query_compiler._modin_frame._partitions
print(partitions)
# Show the first DataFrame partition
print(partitions[0][0].get())
Output:
# created DataFrame
col0 col1 col2 col3
0 col0_100 col1_100 col2_100 col3_100
1 col0_101 col1_101 col2_101 col3_101
2 col0_102 col1_102 col2_102 col3_102
3 col0_103 col1_103 col2_103 col3_103
4 col0_104 col1_104 col2_104 col3_104
.. ... ... ... ...
251 col0_351 col1_351 col2_351 col3_351
252 col0_352 col1_352 col2_352 col3_352
253 col0_353 col1_353 col2_353 col3_353
254 col0_354 col1_354 col2_354 col3_354
255 col0_355 col1_355 col2_355 col3_355
[256 rows x 4 columns]
# List of DataFrame partitions
[[]
[]
[]
[]]
# The first DataFrame partition
col0 col1 col2 col3
0 col0_100 col1_100 col2_100 col3_100
1 col0_101 col1_101 col2_101 col3_101
2 col0_102 col1_102 col2_102 col3_102
3 col0_103 col1_103 col2_103 col3_103
4 col0_104 col1_104 col2_104 col3_104
.. ... ... ... ...
60 col0_160 col1_160 col2_160 col3_160
61 col0_161 col1_161 col2_161 col3_161
62 col0_162 col1_162 col2_162 col3_162
63 col0_163 col1_163 col2_163 col3_163
64 col0_164 col1_164 col2_164 col3_164
[65 rows x 4 columns]
As we show in the example above, Modin ``DataFrame`` can be easily created, and supports any input that pandas ``DataFrame`` supports.
Also note that tuning of the ``DataFrame`` partitioning can be done by just setting a single config.
================================================
FILE: docs/flow/modin/pandas/series.rst
================================================
:orphan:
Series Module Overview
""""""""""""""""""""""
Modin's ``pandas.Series`` API
'''''''''''''''''''''''''''''
Modin's ``pandas.Series`` API is backed by a distributed object providing an identical
API to pandas. After the user calls some ``Series`` function, this call is internally rewritten
into a representation that can be processed in parallel by the partitions. These
results can be e.g., reduced to single output, identical to the single threaded
pandas ``Series`` method output.
..
TODO: add link to the docs with detailed description of queries compilation
and execution ater DOCS-#2996 is merged.
Public API
----------
.. autoclass:: modin.pandas.series.Series
Usage Guide
'''''''''''
The most efficient way to create Modin ``Series`` is to import data from external
storage using the highly efficient Modin IO methods (for example using ``pd.read_csv``,
see details for Modin IO methods in the :doc:`IO ` page),
but even if the data does not originate from a file, any pandas supported data type or
``pandas.Series`` can be used. Internally, the ``Series`` data is divided into
partitions, which number along an axis usually corresponds to the number of the user's hardware CPUs. If needed,
the number of partitions can be changed by setting ``modin.config.NPartitions``.
Let's consider simple example of creation and interacting with Modin ``Series``:
.. code-block:: python
import modin.config
# This explicitly sets the number of partitions
modin.config.NPartitions.put(4)
import modin.pandas as pd
import pandas
# Create Modin Series from the external file
pd_series = pd.read_csv("test_data.csv", header=None).squeeze()
# Create Modin Series from the python object
# pd_series = pd.Series([x for x in range(256)])
# Create Modin Series from the pandas object
# pd_series = pd.Series(pandas.Series([x for x in range(256)]))
# Show created `Series`
print(pd_series)
# List `Series` partitions. Note, that internal API is intended for
# developers needs and was used here for presentation purposes
# only.
partitions = pd_series._query_compiler._modin_frame._partitions
print(partitions)
# Show the first `Series` partition
print(partitions[0][0].get())
Output:
# created `Series`
0 100
1 101
2 102
3 103
4 104
...
251 351
252 352
253 353
254 354
255 355
Name: 0, Length: 256, dtype: int64
# List of `Series` partitions
[[]
[]
[]
[]]
# The first `Series` partition
0
0 100
1 101
2 102
3 103
4 104
.. ...
60 160
61 161
62 162
63 163
64 164
[65 rows x 1 columns]
As we show in the example above, Modin ``Series`` can be easily created, and supports any input that pandas ``Series`` supports.
Also note that tuning of the ``Series`` partitioning can be done by just setting a single config.
================================================
FILE: docs/flow/modin/utils.rst
================================================
:orphan:
Modin Utils
"""""""""""
Here are utilities that can be useful when working with Modin.
Public API
''''''''''
.. autofunction:: modin.utils.try_cast_to_pandas
.. autofunction:: modin.utils.execute
================================================
FILE: docs/getting_started/examples.rst
================================================
Examples and Resources
======================
Here you can find additional resources to learn about Modin. To learn more about
advanced usage for Modin, please refer to :doc:`Usage Guide ` section..
Usage Examples
''''''''''''''
The following notebooks demonstrate how Modin can be used for scalable data science:
- Quickstart Guide to Modin [`Source `__]
- Using Modin with the NYC Taxi Dataset [`Source `__]
- Modin for Machine Learning with scikit-learn [`Source `__]
Tutorials
'''''''''
The following tutorials cover the basic usage of Modin. `Here `__ is a one hour video tutorial that walks through these basic exercises.
- Exercise 1: Introduction to Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]
- Exercise 2: Speed Improvements with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]
- Exercise 3: Defaulting to pandas with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]
The following tutorials covers more advanced features in Modin:
- Exercise 4: Experimental Features in Modin (Spreadsheet, Progress Bar) [`Source PandasOnRay `__, `Source PandasOnDask `__]
- Exercise 5: Setting up Modin in a Cluster Environment [`Source PandasOnRay `__]
- Exercise 6: Running Modin in a Cluster Environment [`Source PandasOnRay `__]
How to get required dependencies for the tutorial notebooks and to run them please refer to the respective `README.md `__ file.
Talks & Podcasts
''''''''''''''''
- `Scaling Interactive Data Science with Modin and Ray `_ (20 minute, Ray Summit 2021)
- `Unleash The Power Of Dataframes At Any Scale With Modin `_ (40 minute, Python Podcast 2021)
- `[Russian] Distributed Data Processing and XGBoost Training and Prediction with Modin `_ (30 minute, PyCon Russia 2021)
- `[Russian] Efficient Data Science with Modin `_ (30 minute, ISP RAS Open 2021)
- `Modin: Scaling the Capabilities of the Data Scientist, not the Machine `_ (1 hour, RISE Camp 2020)
- `Modin: Pandas Scalability with Devin Petersohn `_ (1 hour, Software Engineering Daily Podcast 2020)
- `Introduction to the DataFrame and Modin `_ (20 minute, RISECamp 2019)
- `Scaling Interactive Pandas Workflows with Modin `_ (40 minute, PyData NYC 2018)
Community contributions
'''''''''''''''''''''''
Here are some blogposts and articles about Modin:
- `Anaconda Blog: Scale your pandas workflow with Modin by Vasilij Litvinov `_
- `The Modin view of Scaling Pandas by Devin Petersohn `_
- `Data Science at Scale with Modin by Areg Melik-Adamyan `_
- `Speed up Pandas using Modin by Eric D. Brown, D.Sc. `_
- `Explore Python Libraries: Make Your DataFrames Parallel With Modin by Zachary Bennett `_
- `Get faster pandas with Modin, even on your laptops by Parul Pandey `_
- `How to speedup pandas by changing one line of code by Shrivarsheni `_
- `How To Accelerate Pandas With Just One Line Of Code by Analytics India `_
- `An Easy Introduction to Modin: A Step-by-Step Guide to Accelerating Pandas by Intel `_
Here are some articles contributed by the international community:
- `[Chinese] 用 Modin 来提速 pandas 工作流程 by Python Chinese Community `_
- `[German] Was ist Modin? by Dipl.-Ing. (FH) Stefan Luber `_
- `[Russian] Ускоряем Pandas при помощи модуля modin by Разработка `_
- `[Korean] modin 으로 pandas 더 빠르게 사용하기 by 분석뉴비 `_
If you would like your articles to be featured here, please `submit a pull request `_ to let us know!
================================================
FILE: docs/getting_started/faq.rst
================================================
Frequently Asked Questions (FAQs)
=================================
Below, you will find answers to the most commonly asked questions about
Modin. If you still cannot find the answer you are looking for, please post your
question on the #support channel on our Slack_ community or open a Github issue_.
FAQs: Why choose Modin?
-----------------------
What’s wrong with pandas and why should I use Modin?
""""""""""""""""""""""""""""""""""""""""""""""""""""
While pandas works extremely well on small datasets, as soon as you start working with
medium to large datasets that are more than a few GBs, pandas can become painfully
slow or run out of memory. This is because pandas is single-threaded. In other words,
you can only process your data with one core at a time. This approach does not scale to
larger data sets and adding more hardware does not lead to more performance gain.
The :py:class:`~modin.pandas.dataframe.DataFrame` is a highly
scalable, parallel DataFrame. Modin transparently distributes the data and computation so
that you can continue using the same pandas API while being able to work with more data faster.
Modin lets you use all the CPU cores on your machine, and because it is lightweight, it
often has less memory overhead than pandas. See :doc:` Why Modin? `
page to learn more about how Modin is different from pandas.
Why not just improve pandas?
""""""""""""""""""""""""""""
pandas is a massive community and well established codebase. Many of the issues
we have identified and resolved with pandas are fundamental to its current
implementation. While we would be happy to donate parts of Modin that
make sense in pandas, many of these components would require significant (or
total) redesign of the pandas architecture. Modin's architecture goes beyond
pandas, which is why the pandas API is just a thin layer at the user level. To learn
more about Modin's architecture, see the :doc:`architecture ` documentation.
How much faster can I go with Modin compared to pandas?
"""""""""""""""""""""""""""""""""""""""""""""""""""""""
Modin is designed to scale with the amount of hardware available.
Even in a traditionally serial task like ``read_csv``, we see large gains by efficiently
distributing the work across your entire machine. Because it is so light-weight,
Modin provides speed-ups of up to 4x on a laptop with 4 physical cores. This speedup scales
efficiently to larger machines with more cores. We have several published papers_ that
include performance results and comparisons against pandas.
How much more data would I be able to process with Modin?
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Often data scientists have to use different tools for operating on datasets of different sizes.
This is not only because processing large dataframes is slow, but also pandas does not support working
with dataframes that don't fit into the available memory. As a result, pandas workflows that work well
for prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size
of your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably
work with hundreds of GBs without worrying about substantial slowdown or memory errors. For more information,
see :doc:`out-of-memory support ` for Modin.
How does Modin compare to Dask DataFrame and Koalas?
""""""""""""""""""""""""""""""""""""""""""""""""""""
TLDR: Modin has better coverage of the pandas API, has a flexible backend, better ordering semantics,
and supports both row and column-parallel operations.
Check out :doc:`Modin vs Dask vs Koalas ` page detailing
the differences!
How does Modin work under the hood?
"""""""""""""""""""""""""""""""""""
Modin is logically separated into different layers that represent the hierarchy of a
typical Database Management System. User queries which perform data transformation,
data ingress or data egress pass through the Modin Query Compiler which translates
queries from the top-level pandas API Layer that users interact with to the Modin Core
Dataframe layer.
The Modin Core DataFrame is our efficient DataFrame implementation that utilizes a partitioning schema
which allows for distributing tasks and queries. From here, the Modin DataFrame works with engines like
Ray, Dask or Unidist to execute computation, and then return the results to the user.
For more details, take a look at our system :doc:`architecture `.
FAQs: How to use Modin?
-----------------------
If I’m only using my laptop, can I still get the benefits of Modin?
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Absolutely! Unlike other parallel DataFrame systems, Modin is an extremely
light-weight, robust DataFrame. Because it is so light-weight, Modin provides
speed-ups of up to 4x on a laptop with 4 physical cores
and allows you to work on data that doesn't fit in your laptop's RAM.
How do I use Jupyter or Colab notebooks with Modin?
"""""""""""""""""""""""""""""""""""""""""""""""""""
You can take a look at this Google Colab installation guide_ and
this notebook tutorial_. Once Modin is installed, simply replace your pandas
import with Modin import:
.. code-block:: python
# import pandas as pd
import modin.pandas as pd
Which execution engine (Ray, Dask or Unidist) should I use for Modin?
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Modin lets you effortlessly speed up your pandas workflows with either Ray_'s, Dask_'s or Unidist_'s execution engine.
You don't need to know anything about either engine in order to use it with Modin. If you only have one engine
installed, Modin will automatically detect which engine you have installed and use that for scheduling computation.
If you don't have a preference, we recommend starting with Modin's default Ray engine.
If you want to use a specific compute engine, you can set the environment variable ``MODIN_ENGINE``
and Modin will do computation with that engine:
.. code-block:: bash
pip install "modin[ray]" # Install Modin dependencies and Ray to run on Ray
export MODIN_ENGINE=ray # Modin will use Ray
pip install "modin[dask]" # Install Modin dependencies and Dask to run on Dask
export MODIN_ENGINE=dask # Modin will use Dask
pip install "modin[mpi]" # Install Modin dependencies and MPI to run on MPI through unidist.
export MODIN_ENGINE=unidist # Modin will use Unidist
export UNIDIST_BACKEND=mpi # Unidist will use MPI backend.
This can also be done with:
.. code-block:: python
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put("ray") # Modin will use Ray
modin_cfg.Engine.put("dask") # Modin will use Dask
modin_cfg.Engine.put('unidist') # Modin will use Unidist
unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend
We plan to support more execution engines in future. If you have a specific request,
please post on the #feature-requests channel on our Slack_ community.
How do I connect Modin to a database via `read_sql`?
""""""""""""""""""""""""""""""""""""""""""""""""""""
To read from a SQL database, you have two options:
1) Pass a connection string, e.g. ``postgresql://reader:NWDMCE5xdipIjRrp@hh-pgsql-public.ebi.ac.uk:5432/pfmegrnargs``
2) Pass an open database connection, e.g. for psycopg2, ``psycopg2.connect("dbname=pfmegrnargs user=reader password=NWDMCE5xdipIjRrp host=hh-pgsql-public.ebi.ac.uk")``
The first option works with both Modin and pandas. If you try the second option
in Modin, Modin will default to pandas because open database connections cannot be pickled.
Pickling is required to send connection details to remote workers.
To handle the unique requirements of distributed database access, Modin has a distributed
database connection called ``ModinDatabaseConnection``:
.. code-block:: python
import modin.pandas as pd
from modin.db_conn import ModinDatabaseConnection
con = ModinDatabaseConnection(
'psycopg2',
host='hh-pgsql-public.ebi.ac.uk',
dbname='pfmegrnargs',
user='reader',
password='NWDMCE5xdipIjRrp')
df = pd.read_sql("SELECT * FROM rnc_database",
con,
index_col=None,
coerce_float=True,
params=None,
parse_dates=None,
chunksize=None)
The ``ModinDatabaseConnection`` will save any arguments you supply it and forward
them to the workers to make their own connections.
How can I contribute to Modin?
""""""""""""""""""""""""""""""
**Modin is currently under active development. Requests and contributions are welcome!**
If you are interested in contributing please check out the :doc:`Contributing Guide`
and then refer to the :doc:`Development Documentation`,
where you can find system architecture, internal implementation details, and other useful information.
Also check out the `Github`_ to view open issues and make contributions.
.. _issue: https://github.com/modin-project/modin/issues
.. _Slack: https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA
.. _Github: https://github.com/modin-project/modin
.. _Ray: https://github.com/ray-project/ray/
.. _Dask: https://github.com/dask/dask
.. _Unidist: https://github.com/modin-project/unidist
.. _papers: https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf
.. _guide: https://modin.readthedocs.io/en/latest/getting_started/installation.html#installing-on-google-colab
.. _tutorial: https://github.com/modin-project/modin/tree/main/examples/tutorial
================================================
FILE: docs/getting_started/installation.rst
================================================
=============
Installation
=============
.. note::
| *Estimated Reading Time: 15 minutes*
| If you already installed Modin on your machine, you can skip this section.
There are several ways to install Modin. Most users will want to install with
``pip`` or using ``conda`` tool, but some users may want to build from the main branch
on the `GitHub repo`_. The main branch has the most recent patches, but may be less
stable than a release installed from ``pip`` or ``conda``.
Installing with pip
-------------------
Stable version
""""""""""""""
Modin can be installed with ``pip`` on Linux, Windows and MacOS.
To install the most recent stable release run the following:
.. code-block:: bash
pip install -U modin # -U for upgrade in case you have an older version
Modin can be used with :doc:`Ray`, :doc:`Dask`,
:doc:`Unidist` engines.
If you don't have Ray_, Dask_ or Unidist_ installed, you will need to install Modin with one of the targets:
.. code-block:: bash
pip install "modin[ray]" # Install Modin dependencies and Ray to run on Ray
pip install "modin[dask]" # Install Modin dependencies and Dask to run on Dask
pip install "modin[mpi]" # Install Modin dependencies and MPI to run on MPI through unidist
pip install "modin[all]" # Install Ray and Dask
To get Modin on MPI through unidist (as of unidist 0.5.0) fully working
it is required to have a working MPI implementation installed beforehand.
Otherwise, installation of ``modin[mpi]`` may fail. Refer to
`Installing with pip`_ section of the unidist documentation for more details about installation.
**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: ``ray`` instead of ``ray[default]``.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing ``ray[default]`` along with ``modin[ray]``.
Modin will automatically detect which engine you have installed and use that for
scheduling computation!
Release candidates
""""""""""""""""""
Before most major releases, we will upload a release candidate to test and check if there are any problems. If you would like to install a pre-release of Modin, run the following:
.. code-block:: bash
pip install --pre modin
These pre-releases are uploaded for dependencies and users to test their existing code
to ensure that it still works. If you find something wrong, please raise an issue_ or
email the bug reporter: bug_reports@modin.org.
Installing specific dependency sets
"""""""""""""""""""""""""""""""""""
Modin has a number of specific dependency sets for running Modin on different execution engines and
storage formats or for different functionalities of Modin. Here is a list of dependency sets for Modin:
.. code-block:: bash
pip install "modin[ray]" # If you want to use the Ray execution engine
.. code-block:: bash
pip install "modin[dask]" # If you want to use the Dask execution engine
.. code-block:: bash
pip install "modin[mpi]" # If you want to use MPI through unidist execution engine
Consortium Standard-compatible implementation based on Modin
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.. code-block:: bash
pip install "modin[consortium-standard]"
Installing on Google Colab
"""""""""""""""""""""""""""
Modin can be used with Google Colab_ via the ``pip`` command, by running the following code in a new cell:
.. code-block:: bash
!pip install "modin[all]"
Since Colab preloads several of Modin's dependencies by default, we need to restart the Colab environment once Modin is installed by either clicking on the :code:`"RESTART RUNTIME"` button in the installation output or by run the following code:
.. code-block:: python
# Post-install automatically kill and restart Colab environment
import os
os.kill(os.getpid(), 9)
Once you have restarted the Colab environment, you can use Modin in Colab in subsequent sessions.
Note that on the free version of Colab, there is a `limit on the compute resource `_. To leverage the full power of Modin, you may have to upgrade to Colab Pro to get access to more compute resources.
Installing with conda
---------------------
Using conda-forge channel
"""""""""""""""""""""""""
Modin releases can be installed using ``conda`` from conda-forge channel. Starting from 0.10.1
it is possible to install modin with chosen engine(s) alongside. Current options are:
+---------------------------------+---------------------------+-----------------------------+
| **Package name in conda-forge** | **Engine(s)** | **Supported OSs** |
+---------------------------------+---------------------------+-----------------------------+
| modin | Dask_ | Linux, Windows, MacOS |
+---------------------------------+---------------------------+-----------------------------+
| modin-dask | Dask | Linux, Windows, MacOS |
+---------------------------------+---------------------------+-----------------------------+
| modin-ray | Ray_ | Linux, Windows |
+---------------------------------+---------------------------+-----------------------------+
| modin-mpi | MPI_ through unidist_ | Linux, Windows, MacOS |
+---------------------------------+---------------------------+-----------------------------+
| modin-all | Dask, Ray, Unidist | Linux |
+---------------------------------+---------------------------+-----------------------------+
**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: ``ray-core`` instead of ``ray-default``.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing ``ray-default`` along with ``modin-ray``.
For installing Dask, Ray and MPI through unidist engines into conda environment following command should be used:
.. code-block:: bash
conda install -c conda-forge modin-ray modin-dask modin-mpi
All set of engines could be available in conda environment by specifying:
.. code-block:: bash
conda install -c conda-forge modin-all
or explicitly:
.. code-block:: bash
conda install -c conda-forge modin-ray modin-dask modin-mpi
Refer to `Installing with conda`_ section of the unidist documentation
for more details on how to install a specific MPI implementation to run on.
``conda`` may be slow installing ``modin-all`` or combitations of execution engines so we currently recommend using libmamba solver for the installation process.
To do this install it in a base environment:
.. code-block:: bash
conda install -n base conda-libmamba-solver
Then it can be used during installation either like
.. code-block:: bash
conda install -c conda-forge modin-ray modin- --experimental-solver=libmamba
or starting from conda 22.11 and libmamba solver 22.12 versions
.. code-block:: bash
conda install -c conda-forge modin-ray --solver=libmamba
Installing from the GitHub main branch
--------------------------------------
If you'd like to try Modin using the most recent updates from the main branch, you can
also use ``pip``.
.. code-block:: bash
pip install "modin[all] @ git+https://github.com/modin-project/modin"
This will install directly from the repo without you having to manually clone it! Please be aware
that these changes have not made it into a release and may not be completely stable.
If you would like to install Modin with a specific engine, you can use ``modin[ray]`` or ``modin[dask]`` or ``modin[mpi]`` instead of ``modin[all]`` in the command above.
Windows
-------
All Modin engines are available both on Windows and Linux as mentioned above.
Default engine on Windows is :doc:`Ray`.
It is also possible to use Windows Subsystem For Linux (WSL_), but this is generally
not recommended due to the limitations and poor performance of Ray on WSL, a roughly
2-3x worse than native Windows.
Building Modin from Source
--------------------------
If you're planning on :doc:`contributing ` to Modin, you will need to ensure that you are
building Modin from the local repository that you are working off of. Occasionally,
there are issues in overlapping Modin installs from pypi and from source. To avoid these
issues, we recommend uninstalling Modin before you install from source:
.. code-block:: bash
pip uninstall modin
To build from source, you first must clone the repo. We recommend forking the repository first
through the GitHub interface, then cloning as follows:
.. code-block:: bash
git clone https://github.com//modin.git
Once cloned, ``cd`` into the ``modin`` directory and use ``pip`` to install:
.. code-block:: bash
cd modin
pip install -e .
pip install -e ".[all]" # will install dependencies for all engines
.. _`GitHub repo`: https://github.com/modin-project/modin/tree/main
.. _issue: https://github.com/modin-project/modin/issues
.. _WSL: https://docs.microsoft.com/en-us/windows/wsl/install-win10
.. _Ray: http://ray.readthedocs.io
.. _Dask: https://github.com/dask/dask
.. _MPI: https://www.mpi-forum.org/
.. _Unidist: https://github.com/modin-project/unidist
.. _`Installing with pip`: https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip
.. _`Installing with conda`: https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda
.. _`Intel Distribution of Modin`: https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/distribution-of-modin.html#gs.86stqv
.. _`Intel Distribution of Modin Getting Started`: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-distribution-of-modin-getting-started-guide.html
.. |reg| unicode:: U+000AE .. REGISTERED SIGN
.. _Colab: https://colab.research.google.com/
================================================
FILE: docs/getting_started/quickstart.rst
================================================
Getting Started
===============
.. note::
| *Estimated Reading Time: 10 minutes*
| You can follow along this tutorial in a Jupyter notebook `here `_.
.. toctree::
:hidden:
:maxdepth: 4
10-min Quickstart Guide
installation
using_modin/using_modin
why_modin/why_modin
examples
faq
troubleshooting
.. meta::
:description lang=en:
Introduction to Modin.
Quick Start Guide
-----------------
To install the most recent stable release for Modin run the following:
.. code-block:: bash
pip install "modin[all]"
For further instructions on how to install Modin with conda or for specific platforms
or engines, see our detailed `installation guide <../getting_started/installation.html>`_.
Modin acts as a drop-in replacement for pandas so you simply have to replace the import
of pandas with the import of Modin as follows to speed up your pandas workflows:
.. code-block:: bash
# import pandas as pd
import modin.pandas as pd
Example: Instant Scalability with No Extra Effort
-------------------------------------------------
When working on large datasets, pandas becomes painfully slow or :doc:`runs out of memory`. Modin automatically scales up your
pandas workflows by parallelizing the dataframe operations, so that you can more
effectively leverage the compute resources available.
For the purpose of demonstration, we will load in modin as ``pd`` and pandas as
``pandas``.
.. code-block:: python
import modin.pandas as pd
import pandas
#############################################
### For the purpose of timing comparisons ###
#############################################
import time
import ray
# Look at the Ray documentation with respect to the Ray configuration suited to you most.
ray.init()
#############################################
In this toy example, we look at the NYC taxi dataset, which is around 200MB in size. You can download `this dataset `_ to run the example locally.
.. code-block:: python
# This may take a few minutes to download
import urllib.request
dataset_url = "https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv"
urllib.request.urlretrieve(dataset_url, "taxi.csv")
Faster Data Loading with ``read_csv``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: python
start = time.time()
pandas_df = pandas.read_csv(dataset_url, parse_dates=["tpep_pickup_datetime", "tpep_dropoff_datetime"], quoting=3)
end = time.time()
pandas_duration = end - start
print("Time to read with pandas: {} seconds".format(round(pandas_duration, 3)))
By running the same command ``read_csv`` with Modin, we generally get around 4X speedup
for loading in the data in parallel.
.. code-block:: python
start = time.time()
modin_df = pd.read_csv(dataset_url, parse_dates=["tpep_pickup_datetime", "tpep_dropoff_datetime"], quoting=3)
end = time.time()
modin_duration = end - start
print("Time to read with Modin: {} seconds".format(round(modin_duration, 3)))
print("Modin is {}x faster than pandas at `read_csv`!".format(round(pandas_duration / modin_duration, 2)))
Faster ``concat`` across multiple dataframes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Our previous ``read_csv`` example operated on a relatively small dataframe. In the
following example, we duplicate the same taxi dataset 100 times and then concatenate
them together, resulting in a dataset around 19GB in size.
.. code-block:: python
start = time.time()
big_pandas_df = pandas.concat([pandas_df for _ in range(25)])
end = time.time()
pandas_duration = end - start
print("Time to concat with pandas: {} seconds".format(round(pandas_duration, 3)))
.. code-block:: python
start = time.time()
big_modin_df = pd.concat([modin_df for _ in range(25)])
end = time.time()
modin_duration = end - start
print("Time to concat with Modin: {} seconds".format(round(modin_duration, 3)))
print("Modin is {}x faster than pandas at `concat`!".format(round(pandas_duration / modin_duration, 2)))
Modin speeds up the ``concat`` operation by more than 60X, taking less than a second to
create the large dataframe, while pandas took close to a minute.
Faster ``apply`` over a single column
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The performance benefits of Modin become apparent when we operate on large
gigabyte-scale datasets. Let's say we want to round up values
across a single column via the ``apply`` operation.
.. code-block:: python
start = time.time()
rounded_trip_distance_pandas = big_pandas_df["trip_distance"].apply(round)
end = time.time()
pandas_duration = end - start
print("Time to apply with pandas: {} seconds".format(round(pandas_duration, 3)))
.. code-block:: python
start = time.time()
rounded_trip_distance_modin = big_modin_df["trip_distance"].apply(round)
end = time.time()
modin_duration = end - start
print("Time to apply with Modin: {} seconds".format(round(modin_duration, 3)))
print("Modin is {}x faster than pandas at `apply` on one column!".format(round(pandas_duration / modin_duration, 2)))
Modin is more than 30X faster at applying a single column of data, operating on 130+
million rows in a second.
In short, Modin provides orders of magnitude speed up over pandas for a variety of operations out of the box.
.. figure:: ../img/quickstart_speedup.svg
:align: center
Summary
-------
Hopefully, this tutorial demonstrated how Modin delivers significant speedup on pandas
operations without the need for any extra effort. Throughout example, we moved from
working with 100MBs of data to 20GBs of data all without having to change anything or
manually optimize our code to achieve the level of scalable performance that Modin
provides.
Note that in this quickstart example, we've only shown ``read_csv``, ``concat``,
``apply``, but these are not the only pandas operations that Modin optimizes for. In
fact, Modin covers `more than 90\% of the pandas API `_, yielding considerable speedups for
many common operations.
================================================
FILE: docs/getting_started/troubleshooting.rst
================================================
Troubleshooting
===============
We hope your experience with Modin is bug-free, but there are some quirks about Modin
that may require troubleshooting. If you are still having issues, please post on
the #support channel on our Slack_ community or open a Github issue_.
Frequently encountered issues
-----------------------------
This is a list of the most frequently encountered issues when using Modin. Some of these
are working as intended, while others are known bugs that are being actively worked on.
Warning during execution: ``defaulting to pandas``
""""""""""""""""""""""""""""""""""""""""""""""""""
Please note, that while Modin covers a large portion of the pandas API, not all functionality is implemented. For methods that are not yet implemented, such as ``asfreq``, you may see the following:
.. code-block:: text
UserWarning: `DataFrame.asfreq` defaulting to pandas implementation.
To understand which functions will lead to this warning, we have compiled a list of :doc:`currently supported methods `. When you see this warning, Modin defaults to pandas by converting the Modin dataframe to pandas to perform the operation. Once the operation is complete in pandas, it is converted back to a Modin dataframe. These operations will have a high overhead due to the communication involved and will take longer than pandas. When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. You can learn more about this :doc:`here `.
If you would like to request a particular method be implemented, feel free to open an
`issue`_. Before you open an issue please make sure that someone else has not already
requested that functionality.
Hanging on ``import modin.pandas as pd``
""""""""""""""""""""""""""""""""""""""""
This can happen when Ray fails to start. It will keep retrying, but often it is faster
to just restart the notebook or interpreter. Generally, this should not happen. Most
commonly this is encountered when starting multiple notebooks or interpreters in quick
succession.
**Solution**
Restart your interpreter or notebook kernel.
**Avoiding this Error**
Avoid starting many Modin notebooks or interpreters in quick succession. Wait 2-3
seconds before starting the next one.
Importing heterogeneous data using ``read_csv``
"""""""""""""""""""""""""""""""""""""""""""""""
Since Modin's ``read_csv`` imports data in parallel, it is possible for data across
partitions to be heterogeneously typed (this can happen when columns contain
heterogeneous data, i.e. values in the same column are of different types). An example
of how this is handled is shown below.
.. code-block:: python
import os
import pandas
import modin.pandas as pd
from modin.config import NPartitions
NPartitions.put(2)
test_filename = "test.csv"
# data with heterogeneous values in the first column
data = """one,2
3,4
5,6
7,8
9.0,10
"""
kwargs = {
# names of the columns to set, if `names` parameter is set,
# header inffering from the first data row/rows will be disabled
"names": ["col1", "col2"],
# explicit setting of data type of column/columns with heterogeneous
# data will force partitions to read data with correct dtype
# "dtype": {"col1": str},
}
try :
with open(test_filename, "w") as f:
f.write(data)
pandas_df = pandas.read_csv(test_filename, **kwargs)
pd_df = pd.read_csv(test_filename, **kwargs)
print(pandas_df)
print(pd_df)
finally:
os.remove(test_filename)
Output:
pandas_df:
col1 col2
0 one 2
1 3 4
2 5 6
3 7 8
4 9.0 10
pd_df:
col1 col2
0 one 2
1 3 4
2 5 6
3 7.0 8
4 9.0 10
In this case, ``col1`` of the `DataFrame` read by pandas contains only ``str`` data
because the first value ("one") is inferred to have type ``str``, which forces pandas to handle the rest of the values in the column
as strings. The first Modin partition (the first three rows) handles the data as pandas does,
but the second partition (the last two rows) reads the data as floats. This is because the
second column contains an int and a float, and thus the column type is inferred to be float. As a
result, `7` is interpreted as `7.0`, which differs from the pandas output.
The above example demonstrates heterogenous data import with str, int, and float types,
but heterogeneous data consisting of other data/parameter combinations can also result in
data type mismatches with pandas.
**Solution**
When heterogeneous data is detected, a warning will be raised.
Currently, these discrepancies aren't properly handled
by Modin, so to avoid this issue, you need to set the ``dtype`` parameter of ``read_csv``
manually to force the correct data type coercion during data import. Note that
to avoid excessive performance degradation, the ``dtype`` value should only be set for columns that may contain heterogenous data.
as possible (specify ``dtype`` parameter only for columns with heterogeneous data).
Specifying the ``dtype`` parameter will work well in most cases. If the file
contains a column that should be interpreted as the index
(the ``index_col`` parameter is specified) there may still be type discrepancies in the index, since the ``dtype`` parameter is only responsible for data
fields. If in the above example, ``kwargs`` was set like so:
.. code-block:: python
kwargs = {
"names": ["col1", "col2"],
"dtype": {"col1": str},
"index_col": "col1",
}
The resulting Modin DataFrame will contain incorrect values - just as if ``dtype``
had not been specified:
.. code-block:: python
col1
one 2
3 4
5 6
7.0 8
9.0 10
One workaround is to import the data without setting the ``index_col`` parameter, and then
set the index column using the ``DataFrame.set_index`` function as shown in
the example below:
.. code-block:: python
pd_df = pd.read_csv(filename, dtype=data_dtype, index_col=None)
pd_df = pd_df.set_index(index_col_name)
pd_df.index.name = None
Using Modin with python multiprocessing
"""""""""""""""""""""""""""""""""""""""
We strongly recommend against using a distributed execution engine (e.g. Ray or Dask)
in conjunction with Python multiprocessing because that can lead to undefined behavior.
One such example is shown below:
.. code-block:: python
import modin.pandas as pd
# Ray engine is used by default
df = pandas.DataFrame([1, 2, 3])
def f(arg):
return df + arg
if __name__ == '__main__':
from multiprocessing import Pool
with Pool(5) as p:
print(p.map(f, [1]))
Although this example may work on your machine, we do not recommend it, because
the Python multiprocessing library will duplicate Ray clusters, causing both
excessive resource usage and conflict over the available resources.
Poor performance of the first operation with Modin on Ray engine
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
There might be cases when the first operation with Modin on Ray engine is much slower than the subsequent calls of the operation.
That happens because Ray workers may not be fully set up yet to perform computation after initialization of the engine
with ``ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})``, which is the default behavior of Modin on Ray engine
if Ray has not been initialised yet. Modin intentionaly initializes Ray this way to import ``pandas`` in workers
once Python interpreter is started in them so that to avoid a race condition in Ray between the import thread and the thread executing the code.
..
See more details on why we started using ``ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})` in
https://github.com/modin-project/modin/pull/4603.
.. code-block:: python
import time
import pandas
import numpy as np
import ray
import modin.pandas as pd
import modin.config as cfg
# Look at the Ray documentation with respect to the Ray configuration suited to you most.
ray.init(runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})
pandas_df = pandas.DataFrame(
np.random.randint(0, 100, size=(1000000, 13))
)
pandas_df.to_csv("foo.csv", index=False)
def read_csv_with_pandas():
start_time = time.time()
pandas_df = pandas.read_csv("foo.csv", index_col=0)
end_time = time.time()
pandas_duration = end_time - start_time
print("Time to read_csv with pandas: {} seconds".format(round(pandas_duration, 3)))
return pandas_df
def read_csv_with_modin():
start_time = time.time()
modin_df = pd.read_csv("foo.csv", index_col=0)
end_time = time.time()
modin_duration = end_time - start_time
print("Time to read_csv with Modin: {} seconds".format(round(modin_duration, 3)))
return modin_df
for i in range(5):
pandas_df = read_csv_with_pandas()
modin_df = read_csv_with_modin()
Time to read_csv with pandas: 0.708 seconds
Time to read_csv with Modin: 4.132 seconds
Time to read_csv with pandas: 0.735 seconds
Time to read_csv with Modin: 0.37 seconds
Time to read_csv with pandas: 0.646 seconds
Time to read_csv with Modin: 0.377 seconds
Time to read_csv with pandas: 0.673 seconds
Time to read_csv with Modin: 0.371 seconds
Time to read_csv with pandas: 0.672 seconds
Time to read_csv with Modin: 0.379 seconds
**Solution**
So far there is no a solution to fix or work around the problem rather than not to pass a non-empty runtime_env to ``ray.init()``.
However, this may lead to other problem regarding a race condition in Ray between the import thread and the thread executing the code.
So for now we just highlight the problem in hope of a future fix in Ray itself.
Also, it is worth noting that every distributed engine by its nature has a little overhead for the first operation being called,
which may be important for microbenchmarks. What you likely want to do is warm up worker processes
either by excluding the time of the first iteration from your measurements or execute a simple function in workers to fully set up them.
Common errors
-------------
Error when using Dask engine: ``RuntimeError: if __name__ == '__main__':``
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The following `script.py` uses Modin with Dask as an execution engine and produces errors:
.. code-block:: python
# script.py
import modin.pandas as pd
import modin.config as cfg
cfg.Engine.put("dask")
df = pd.DataFrame([0,1,2,3])
print(df)
A part of the produced errors by the script above would be the following:
.. code-block::
File "/path/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
This happens because Dask Client uses `fork `_
to start processes.
**Solution**
To avoid the problem the Dask Client creation code needs to be moved into the ``__main__`` scope of the module.
The corrected `script.py` would look like:
.. code-block:: python
# script.py
import modin.pandas as pd
import modin.config as cfg
cfg.Engine.put("dask")
if __name__ == "__main__":
df = pd.DataFrame([0, 1, 2, 3]) # Dask Client creation is hidden in the first call of Modin functionality.
print(df)
or
.. code-block:: python
# script.py
from distributed import Client
import modin.pandas as pd
import modin.config as cfg
cfg.Engine.put("dask")
if __name__ == "__main__":
# Explicit Dask Client creation.
# Look at the Dask Distributed documentation with respect to the Client configuration suited to you most.
client = Client()
df = pd.DataFrame([0, 1, 2, 3])
print(df)
Spurious error "cannot import partially initialised pandas module" on custom Ray cluster
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
If you're using some pre-configured Ray cluster to run Modin, it's possible you would
be seeing spurious errors like
.. code-block::
ray.exceptions.RaySystemError: System error: partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)
traceback: Traceback (most recent call last):
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
return self._deserialize_msgpack_data(data, metadata_fields)
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
python_objects = self._deserialize_pickle5_data(pickle5_data)
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/ray/serialization.py", line 180, in _deserialize_pickle5_data
obj = pickle.loads(in_band, buffers=buffers)
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/pandas/__init__.py", line 135, in
from pandas import api, arrays, errors, io, plotting, testing, tseries
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/pandas/testing.py", line 6, in
from pandas._testing import (
File "/usr/share/miniconda/envs/modin/lib/python3.8/site-packages/pandas/_testing/__init__.py", line 979, in
cython_table = pd.core.common._cython_table.items()
AttributeError: partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)
**Solution**
Modin contains a workaround that should automatically do ``import pandas`` upon worker process starts.
It is triggered by the presence of non-empty ``__MODIN_AUTOIMPORT_PANDAS__`` environment variable which
Modin sets up automatically on the Ray clusters it spawns, but it might be missing on pre-configured clusters.
So if you're seeing the issue like shown above, please make sure you set this environment variable on all
worker nodes of your cluster before actually spawning the workers.
.. _issue: https://github.com/modin-project/modin/issues
.. _Slack: https://modin.org/slack.html
================================================
FILE: docs/getting_started/using_modin/using_modin.rst
================================================
Using Modin
===========
In this section, we show how Modin can be used to accelerate your pandas workflows on a
single machine up to multiple machines in a cluster setting.
.. toctree::
:maxdepth: 4
using_modin_locally
using_modin_cluster
================================================
FILE: docs/getting_started/using_modin/using_modin_cluster.rst
================================================
Using Modin in a Cluster
========================
.. note::
| *Estimated Reading Time: 15 minutes*
Often in practice we have a need to exceed the capabilities of a single machine.
Modin works and performs well in both local mode and in a cluster environment.
The key advantage of Modin is that your python code does not change between
local development and cluster execution. Users are not required to think about
how many workers exist or how to distribute and partition their data;
Modin handles all of this seamlessly and transparently.
.. note::
It is possible to use a Jupyter notebook, but you will have to deploy a Jupyter server
on the remote cluster head node and connect to it.
.. image:: ../../img/modin_cluster.png
:alt: Modin cluster
:align: center
Extra requirements for AWS authentication
-----------------------------------------
First of all, install the necessary dependencies in your environment:
.. code-block:: bash
pip install boto3
The next step is to setup your AWS credentials. One can set ``AWS_ACCESS_KEY_ID``,
``AWS_SECRET_ACCESS_KEY`` and ``AWS_SESSION_TOKEN`` (Optional)
(refer to `AWS CLI environment variables`_ to get more insight on this) or
just run the following command:
.. code-block:: bash
aws configure
Starting and connecting to the cluster
--------------------------------------
This example starts 1 head node (m5.24xlarge) and 5 worker nodes (m5.24xlarge), 576 total CPUs.
You can check the `Amazon EC2 pricing`_ page.
It is possble to manually create AWS EC2 instances and configure them or just use the `Ray CLI`_ to
create and initialize a Ray cluster on AWS using `Modin's Ray cluster setup config`_,
which we are going to utilize in this example.
Refer to `Ray's autoscaler options`_ page on how to modify the file.
More details on how to launch a Ray cluster can be found on `Ray's cluster docs`_.
To start up the Ray cluster, run the following command in your terminal:
.. code-block:: bash
ray up modin-cluster.yaml
Once the head node has completed initialization, you can optionally connect to it by running the following command.
.. code-block:: bash
ray attach modin-cluster.yaml
To exit the ssh session and return back into your local shell session, type:
.. code-block:: bash
exit
Executing in a cluster environment
----------------------------------
.. note::
Be careful when using the `Ray client`_ to connect to a remote cluster.
We don't recommend this connection mode, beacuse it may not work. Known bugs:
- https://github.com/ray-project/ray/issues/38713,
- https://github.com/modin-project/modin/issues/6641.
Modin lets you instantly speed up your workflows with a large data by scaling pandas
on a cluster. In this tutorial, we will use a 12.5 GB ``big_yellow.csv`` file that was
created by concatenating a 200MB `NYC Taxi dataset`_ file 64 times. Preparing this
file was provided as part of our `Modin's Ray cluster setup config`_.
If you want to use the other dataset, you should provide it to each of
the cluster nodes with the same path. We recomnend doing this by customizing the
``setup_commands`` section of the `Modin's Ray cluster setup config`_.
To run any script in a remote cluster, you need to submit it to the Ray. In this way,
the script file is sent to the the remote cluster head node and executed there.
In this tutorial, we provide the `exercise_5.py`_ script, which reads the data from the
CSV file and executes such pandas operations as count, groupby and map.
As the result, you will see the size of the file being read and the execution time of the entire script.
You can submit this script to the existing remote cluster by running the following command.
.. code-block:: bash
ray submit modin-cluster.yaml exercise_5.py
To download or upload files to the cluster head node, use ``ray rsync_down`` or ``ray rsync_up``.
It may help if you want to use some other Python modules that should be available to
execute your own script or download a result file after executing the script.
.. code-block:: bash
# download a file from the cluster to the local machine:
ray rsync_down modin-cluster.yaml '/path/on/cluster' '/local/path'
# upload a file from the local machine to the cluster:
ray rsync_up modin-cluster.yaml '/local/path' '/path/on/cluster'
Shutting down the cluster
--------------------------
Now that we have finished the computation, we need to shut down the cluster with `ray down` command.
.. code-block:: bash
ray down modin-cluster.yaml
.. _`Ray's autoscaler options`: https://docs.ray.io/en/latest/cluster/vms/references/ray-cluster-configuration.html#cluster-config
.. _`Ray's cluster docs`: https://docs.ray.io/en/latest/cluster/getting-started.html
.. _`NYC Taxi dataset`: https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv
.. _`Modin's Ray cluster setup config`: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_ray/cluster/modin-cluster.yaml
.. _`Amazon EC2 pricing`: https://aws.amazon.com/ec2/pricing/on-demand/
.. _`exercise_5.py`: https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_ray/cluster/exercise_5.py
.. _`Ray client`: https://docs.ray.io/en/latest/cluster/running-applications/job-submission/ray-client.html
.. _`Ray CLI`: https://docs.ray.io/en/latest/cluster/vms/getting-started.html#running-applications-on-a-ray-cluster
.. _`AWS CLI environment variables`: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
================================================
FILE: docs/getting_started/using_modin/using_modin_locally.rst
================================================
===================
Using Modin Locally
===================
.. note::
| *Estimated Reading Time: 5 minutes*
| You can follow along this tutorial in the `Jupyter notebook`_.
In our quickstart example, we have already seen how you can achieve considerable
speedup from Modin, even on a single machine. Users do not need to know how many
cores their system has, nor do they need to specify how to distribute the data. In fact,
users can **continue using their existing pandas code** while experiencing a
considerable speedup from Modin, even on a single machine.
To use Modin on a single machine, only a modification of the import statement is needed.
Once you've changed your import statement, you're ready to use Modin
just like you would pandas, since the API is identical to pandas.
.. code-block:: python
# import pandas as pd
import modin.pandas as pd
**That's it. You're ready to use Modin on your previous pandas workflows!**
Advanced: Configuring the resources Modin uses
----------------------------------------------
Modin automatically check the number of CPUs available on your machine and sets the
number of partitions to be equal to the number of CPUs. You can verify this by running
the following code:
.. code-block:: python
import modin
print(modin.config.NPartitions.get()) #prints 16 on a laptop with 16 physical cores
Modin fully utilizes the resources on your machine. To read more about how this works,
see :doc:`Why Modin? ` page for more details.
Since Modin will use all of the resources available on your machine by default, at
times, it is possible that you may like to limit the amount of resources Modin uses to
free resources for another task or user. Here is how you would limit the number of CPUs
Modin used in your bash environment variables:
.. code-block:: bash
export MODIN_CPUS=4
You can also specify this in your python script with ``os.environ``:
.. code-block:: python
import os
os.environ["MODIN_CPUS"] = "4"
import modin.pandas as pd
If you're using a specific engine and want more control over the environment Modin
uses, you can start Ray or Dask in your environment and Modin will connect to it.
.. code-block:: python
import ray
ray.init(num_cpus=4)
import modin.pandas as pd
Specifying ``num_cpus`` limits the number of processors that Modin uses. You may also
specify more processors than you have available on your machine; however this will not
improve the performance (and might end up hurting the performance of the system).
.. note::
Make sure to update the ``MODIN_CPUS`` configuration and initialize your preferred
engine before you start working with the first operation using Modin! Otherwise,
Modin will opt for the default setting.
.. _`Jupyter notebook`: https://github.com/modin-project/modin/tree/main/examples/quickstart.ipynb
================================================
FILE: docs/getting_started/why_modin/modin_vs_dask_vs_koalas.rst
================================================
Modin vs. Dask DataFrame vs. Koalas
===================================
Libraries such as `Dask DataFrame `_ (DaskDF for short) and `Koalas `_ aim to support the pandas API on top of distributed computing frameworks, Dask and Spark respectively. Instead, Modin aims to preserve the pandas API and behavior as is, while abstracting away the details of the distributed computing framework underneath. Thus, the aims of these libraries are fundamentally different.
Specifically, Modin enables pandas-like
* row and column-parallel operations, unlike DaskDF and Koalas that only support row-parallel operations
* indexing & ordering semantics, unlike DaskDF and Koalas that deviate from these semantics
* eager execution, unlike DaskDF and Koalas that provide lazy execution
As a result, Modin's coverage is `more than 90% `_ of the pandas API, while DaskDF and Koalas' coverage is about 55%.
.. figure:: ../../img/api_coverage_comparison.svg
:align: center
:alt: Percentage coverage of the pandas API after deduplication
For more technical details please see our VLDB 2022 research paper, referenced `here `_.
Brief Overview of DaskDF and Koalas
-----------------------------------
Dask's `DataFrame `_ (DaskDF) is effectively a meta-DataFrame, partitioning and scheduling many smaller ``pandas.DataFrame`` objects. Users construct a task graph of dataframe computation step by step and then trigger computation using the ``compute`` function.
Spark's `Koalas `_ provides the pandas API on Spark, leveraging the preexisting Spark SQL optimizer to execute select pandas commands. Like DaskDF, Koalas also employs lazy computation, only triggering computation when the user requests to see the results.
Partitioning and Parallelization
--------------------------------
Modin, DaskDF, Koalas are all examples of parallel dataframe systems. Parallelism is achieved by partitioning a large dataframe into smaller ones that can be operated on in parallel. As a result, the partitioning scheme chosen by the system dictates the pandas functions that can or can not be supported.
**DaskDF and Koalas only support row-oriented partitioning and parallelism.** This approach is analogous to relational databases. The dataframe is conceptually broken down into horizontal partitions along rows, where each partition is independently processed if possible. When DaskDF or Koalas are required to perform column-parallel operations that to be done on columns independently (e.g., dropping columns with null values via ``dropna`` on the column ``axis``), they either perform very poorly with no parallelism or do not support that operation.
**Modin supports both row, column, and cell-oriented partitioning and parallelism**. That is, the dataframe can be conceptually broken down as groups of rows, groups of columns, or both groups of rows and groups of columns (effectively a block or sub-matrix). Modin will transparently reshape the partitioning as necessary for the corresponding operation, based on whether the operation is row-parallel, column-parallel, or cell-parallel (independently applied to each unit cell). This allows Modin to support more of the pandas API and do so efficiently. Due to the finer-grained control over the partitioning, Modin can support a number of operations that are very challenging to parallelize in row-oriented systems (e.g., ``transpose``, ``median``, ``quantile``). This flexibility in partitioning also gives Modin tremendous power to implement efficient straggler mitigation and improve utilization over the entire cluster.
API Coverage
------------
One of the key benefits of pandas is its versatility, due to the wide array of operations, with more than 600+ API operations for data cleaning, feature engineering, data transformation, data summarization, data exploration, and machine learning. However, it is not trivial to develop scalable implementations of each of these operations in a dataframe system.
**DaskDF and Koalas only implements about** `55% `_ **of the pandas API**; they do not implement certain APIs that would deviate from the row-wise partitioning approach, or would be inefficient with the row-wise parallelization. For example, Dask does not implement ``iloc``, ``MultiIndex``, ``apply(axis=0)``, ``quantile`` (only approximate quantile is available), ``median``, and more. Given DaskDF's row-oriented architecture, ``iloc``, for example, can technically be implemented, but it would be inefficient, and column-wise operations such as ``apply(axis=0)`` would be impossible to implement. Similarly, Koalas does not implement ``apply(axis=0)`` (it only applies the function per row partition, giving a different result), ``quantile``, ``median`` (only approximate quantile/median is available), ``MultiIndex``, ``combine``, ``compare`` and more.
**Modin supports all of the above pandas API functions, as well as others, with** `more than 90% `_ **coverage of the pandas API.** Modin additionally acts as a drop-in replacement for pandas, such that even if the API is not yet supported, it still works by falling back to running vanilla pandas. One of the key features of being a drop-in replacement is that not only will it work for existing code, if a user wishes to go back to running pandas directly, they are not locked in to using Modin and can switch between Modin and pandas at no cost. In other words, scripts and notebooks written in Modin can be converted to and from pandas as the user desires by simply replacing the import statement.
Execution Semantics
---------------------
**DaskDF and Koalas make use of lazy evaluation, which means that the computation is delayed until users explicitly evaluate the results.** This mode of evaluation places a lot of optimization responsibility on the user, forcing them to think about when it would be useful to inspect the intermediate results or delay doing so. Specifically, DaskDF's API differs from pandas in that it requires users to explicitly call ``.compute()`` to materialize the result of the computation. Often if that computation corresponds to a long chain of operators, this call can take a very long time to execute. Overall, the need to explicitly trigger computation makes the API less convenient to work with, but gives DaskDF and Koalas the opportunity to perform holistic optimizations over the entire dataflow graph. However, to the best of our knowledge, neither DaskDF nor Koalas actually leverage holistic optimizations.
**Modin employs eager evaluation, like pandas.** Eager evaluation is the default mode of operation for data scientists when working with pandas in an interactive environment, such as Jupyter Notebooks. Modin reproduces this familiar behavior by performing all computations eagerly as soon as it is issued, so that users can inspect intermediate results and quickly see the results of their computations without having to wait or explicitly trigger computation. This is especially useful during interactive data analysis, where users often iterate on their dataframe workflows or build up their dataframe queries in an incremental fashion. We also have developed techniques for `opportunistic evaluation `_ that bridges the gap between lazy and eager evaluation that will be incorporated in Modin in the future.
Ordering Semantics
------------------
By default, pandas preserves the order of the dataframe, so that users can expect a consistent, ordered view as they are operating on their dataframe.
**Both DaskDF and Koalas make no guarantees about the order of rows in the DataFrame.** This is because DaskDF sorts the ``index`` for optimization purposes to speed up computations that involve the row index; and as a result, it does not support user-specified order. Likewise, Koalas `does not support ordering `_ by default because it will lead to a performance overhead when operating on distributed datasets.
**DaskDF additionally does not support multi-indexing or sorting.**
DaskDF sorts the data based on a single set of row labels for fast row lookups, and builds an indexing structure based on these row labels. Data is both logically and physically stored in the same order. As a result, DaskDF does not support a `sort` function.
**Modin reproduces the intuitive behavior in pandas where the order of the DataFrame is preserved, and supports multi-indexing.** Enforcing ordering on a parallel dataframe system like Modin requires non-trivial effort that involves decoupling of the logical and physical representation of the data, enabling the order to be lazily kept up-to-date, but eagerly computed based on user needs (See Section 4.2 in `our recent paper `_). Modin abstracts away the physical representation of the data and provides an ordered view that is consistent with user's expectations.
Compatibility with Computational Frameworks
-------------------------------------------
**DaskDF and Koalas are meant to be run on Dask and Spark respectively.** They are highly tuned to the corresponding frameworks, and cannot be ported to other computational frameworks.
**Modin's highly modular design is architected to run on a variety of systems, and support a variety of APIs.** The goal for the extensible design is that users can take the same notebook or script and seamlessly move between different clusters and environments, with Modin being able to support the pandas API on your preexisting infrastructure. Currently, Modin support running on Dask's compute engine in addition to Ray. The modular design makes it easier for developers to different execution engines or compile to different memory formats. Modin can run on a Dask cluster in the same way that DaskDF can, but they differ in the ways described above. In addition, Modin is continually expanding to support popular data processing APIs (SQL in addition to pandas, among other DSLs for data processing) while leveraging the same underlying execution framework. Modin's flexible architecture also means that as the `pandas API continues to evolve `_, Modin can quickly move towards supporting new versions of the pandas API.
.. figure:: ../../img/performance-all-supported.svg
:align: center
:alt: Scalability of operators supported by Modin and other systems
:width: 95%
Performance Comparison
----------------------
**On operations supported by all systems, Modin provides substantial speedups.** Thanks to its optimized design, Modin is able to take advantage of multiple cores relative to both Koalas and DaskDF to efficiently execute pandas operations. It is notable that Koalas is often slower than pandas, due to the overhead of Spark.
.. figure:: ../../img/performance-not-all-supported.svg
:align: center
:alt: Scalability of operators supported by Modin but not by other systems
**Modin provides substantial speedups even on operators not supported by other systems.** Thanks to its flexible partitioning schemes that enable it to support the vast majority of pandas operations — be it row, column, or cell-oriented - Modin provides benefits on operations such as ``join``, ``median``, and ``infer_types``. While Koalas performs ``join`` slower than Pandas, Dask failed to support ``join`` on more than 20M rows, likely due poor support for `shuffles `_. Details of the benchmark and additional join experiments can be found in `our paper `_.
.. _documentation: http://docs.dask.org/en/latest/DataFrame.html#design.
.. _Modin's documentation: https://modin.readthedocs.io/en/latest/development/architecture.html
================================================
FILE: docs/getting_started/why_modin/out_of_core.rst
================================================
Out-of-memory data with Modin
=============================
.. note::
| *Estimated Reading Time: 10 minutes*
When using pandas, you might run into a memory error if you are working with large datasets that cannot fit in memory or perform certain memory-intensive operations (e.g., joins).
Modin solves this problem by spilling over to disk, in other words, it uses your disk as an overflow for memory so that you can work with datasets that are too large to fit in memory. By default, Modin leverages out-of-core methods to handle datasets that don't fit in memory for both Ray and Dask engines.
.. note::
Object spilling is disabled in a multi-node Ray cluster by default. To enable object spilling
use `Ray instruction `_.
Motivating Example: Memory error with pandas
--------------------------------------------
pandas makes use of in-memory data structures to store and operate on data, which means that if you have a dataset that is too large to fit in memory, it will cause an error on pandas. As an example, let's creates a 80GB DataFrame by appending together 40 different 2GB DataFrames.
.. code-block:: python
import pandas
import numpy as np
df = pandas.concat([pandas.DataFrame(np.random.randint(0, 100, size=(2**20, 2**8))) for _ in range(40)]) # Memory Error!
When we run this on a laptop with 32GB of RAM, pandas will run out of memory and throw an error (e.g., :code:`MemoryError` , :code:`Killed: 9`).
The `pandas documentation `_ has a great section on recommendations for scaling your analysis to these larger datasets. However, this generally involves loading in less data or rewriting your pandas code to process the data in smaller chunks.
Operating on out-of-memory data with Modin
------------------------------------------
In order to work with data that exceeds memory constraints, you can use Modin to handle these large datasets.
.. code-block:: python
import modin.pandas as pd
import numpy as np
df = pd.concat([pd.DataFrame(np.random.randint(0, 100, size=(2**20, 2**8))) for _ in range(40)]) # 40x2GB frames -- Working!
df.info()
Not only does Modin let you work with datasets that are too large to fit in memory, we can perform various operations on them without worrying about memory constraints.
Advanced: Configuring out-of-core settings
------------------------------------------
.. why would you want to disable out of core?
By default, out-of-core functionality is enabled by the compute engine selected.
To disable it, start your preferred compute engine with the appropriate arguments. For example:
.. code-block:: python
import modin.pandas as pd
import ray
ray.init(_plasma_directory="/tmp") # setting to disable out of core in Ray
df = pd.read_csv("some.csv")
If you are using Dask, you have to modify local configuration files. Visit the
Dask documentation_ on object spilling for more details.
.. _documentation: https://distributed.dask.org/en/latest/worker.html#memory-management
================================================
FILE: docs/getting_started/why_modin/pandas.rst
================================================
How does Modin differ from pandas?
==================================
.. note::
| *Estimated Reading Time: 10 minutes*
In the earlier tutorials, we have seen how Modin can be used to speed up pandas workflows. Here, we discuss at a high level how Modin works, in particular, how Modin's dataframe implementation differs from pandas.
Scalablity of implementation
----------------------------
Modin exposes the pandas API through ``modin.pandas``, but it does not inherit the same pitfalls and design decisions that make it difficult to scale.
The pandas implementation is inherently single-threaded. This means that only one of
your CPU cores can be utilized at any given time. In a laptop, it would look something
like this with pandas:
.. image:: /img/pandas_multicore.png
:alt: pandas is single threaded!
:align: center
:scale: 80%
However, Modin's implementation enables you to use all of the cores on your machine, or
all of the cores in an entire cluster. On a laptop, it will look something like this:
.. image:: /img/modin_multicore.png
:alt: modin uses all of the cores!
:align: center
:scale: 80%
The additional utilization leads to improved performance, however if you want to scale
to an entire cluster, Modin suddenly looks something like this:
.. image:: /img/modin_cluster.png
:alt: modin works on a cluster too!
:align: center
:scale: 30%
Modin is able to efficiently make use of all of the hardware available to it!
Memory usage and immutability
-----------------------------
The pandas API contains many cases of "inplace" updates, which are known to be
controversial. This is due in part to the way pandas manages memory: the user may
think they are saving memory, but pandas is usually copying the data whether an
operation was inplace or not.
Modin allows for inplace semantics, but the underlying data structures within Modin's
implementation are immutable, unlike pandas. This immutability gives Modin the ability
to internally chain operators and better manage memory layouts, because they will not
be changed. This leads to improvements over pandas in memory usage in many common cases,
due to the ability to share common memory blocks among all dataframes.
Modin provides the inplace semantics by having a mutable pointer to the immutable
internal Modin dataframe. This pointer can change, but the underlying data cannot, so
when an inplace update is triggered, Modin will treat it as if it were not inplace and
just update the pointer to the resulting Modin dataframe.
API vs implementation
---------------------
It is well known that the pandas API contains many duplicate ways of performing the same
operation. Modin instead enforces that any one behavior have one and only one
implementation internally. This guarantee enables Modin to focus on and optimize a
smaller code footprint while still guaranteeing that it covers the entire pandas API.
Modin has an internal algebra, which is roughly 15 operators, narrowed down from the
original >200 that exist in pandas. The algebra is grounded in both practical and
theoretical work. Learn more in our `VLDB 2020 paper`_. More information about this
algebra can be found in the :doc:`architecture ` documentation.
.. _VLDB 2020 paper: https://arxiv.org/abs/2001.00888
================================================
FILE: docs/getting_started/why_modin/why_modin.rst
================================================
Why Modin?
==========
In this section, we explain the design and motivation behind Modin and why you should use Modin to scale up your pandas workflows. We first describe the architectural differences between pandas and Modin. Then we describe how Modin can also help resolve out-of-memory issues common to pandas. Finally, we look at the key differences between Modin and other distributed dataframe libraries.
.. toctree::
:maxdepth: 4
pandas
out_of_core
modin_vs_dask_vs_koalas
Modin is built on many years of research and development at UC Berkeley. For more information on how this works underneath the hoods, check out our publications in this space:
- `Flexible Rule-Based Decomposition and Metadata Independence in Modin `_ (VLDB 2021)
- `Enhancing the Interactivity of Dataframe Queries by Leveraging Think Time `_ (IEEE Data Eng 2021)
- `Dataframe Systems: Theory, Architecture, and Implementation `_ (PhD Dissertation 2021)
- `Scaling Data Science does not mean Scaling Machines `_ (CIDR 2021)
- `Towards Scalable Dataframe Systems `_ (VLDB 2020)
================================================
FILE: docs/index.rst
================================================
.. image:: img/MODIN_ver2_hrz.png
:width: 400px
:alt: modin logo
:align: center
====
.. toctree::
:hidden:
getting_started/quickstart
usage_guide/index
supported_apis/index
development/index
ecosystem
contact
.. raw:: html
To use Modin, replace the pandas import:
.. figure:: img/Modin_Pandas_Import.gif
:align: center
Scale your pandas workflow by changing a single line of code
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Modin uses Ray_, Dask_ or Unidist_ to provide an effortless way to speed up your pandas notebooks,
scripts, and libraries. Unlike other distributed DataFrame libraries, Modin provides
seamless integration and compatibility with existing pandas code. Even using the
DataFrame constructor is identical.
.. code-block:: python
import modin.pandas as pd
import numpy as np
frame_data = np.random.randint(0, 100, size=(2**10, 2**8))
df = pd.DataFrame(frame_data)
It is not necessary to know in advance the available hardware resources in order to use Modin.
Additionally, it is not necessary to specify how to distribute or place data.
Modin acts as a drop-in replacement for pandas, which means that you can continue using your previous
pandas notebooks, *unchanged*, while experiencing a considerable speedup thanks to Modin, even on a single
machine. Once you've changed your import statement, you’re ready to use Modin just like
you would pandas.
Installation and choosing your compute engine
"""""""""""""""""""""""""""""""""""""""""""""
Modin can be installed from PyPI:
.. code-block:: bash
pip install modin
If you don't have Ray_, Dask_ or Unidist_ installed, you will need to install Modin with one
of the targets:
.. code-block:: bash
pip install "modin[ray]" # Install Modin dependencies and Ray to run on Ray
pip install "modin[dask]" # Install Modin dependencies and Dask to run on Dask
pip install "modin[mpi]" # Install Modin dependencies and MPI to run on MPI through unidist
pip install "modin[all]" # Install all of the above
Modin will automatically detect which engine you have installed and use that for
scheduling computation!
If you want to choose a specific compute engine to run on, you can set the environment
variable ``MODIN_ENGINE`` and Modin will do computation with that engine:
.. code-block:: bash
export MODIN_ENGINE=ray # Modin will use Ray
export MODIN_ENGINE=dask # Modin will use Dask
export MODIN_ENGINE=unidist # Modin will use Unidist
If you want to choose the Unidist engine, you should set the additional environment
variable ``UNIDIST_BACKEND``, because currently Modin only supports MPI through unidist:
.. code-block:: bash
export UNIDIST_BACKEND=mpi # Unidist will use MPI backend
This can also be done within a notebook/interpreter before you import Modin:
.. code-block:: python
import os
os.environ["MODIN_ENGINE"] = "ray" # Modin will use Ray
os.environ["MODIN_ENGINE"] = "dask" # Modin will use Dask
os.environ["MODIN_ENGINE"] = "unidist" # Modin will use Unidist
os.environ["UNIDIST_BACKEND"] = "mpi" # Unidist will use MPI backend
import modin.pandas as pd
Faster pandas, even on your laptop
""""""""""""""""""""""""""""""""""
.. image:: img/read_csv_benchmark.png
:height: 350px
:width: 300px
:alt: Plot of read_csv
:align: right
The ``modin.pandas`` `DataFrame`_ is an extremely light-weight parallel DataFrame. Modin
transparently distributes the data and computation so that all you need to do is
continue using the pandas API as you were before installing Modin. Unlike other parallel
DataFrame systems, Modin is an extremely light-weight, robust DataFrame. Because it is so
light-weight, Modin provides speed-ups of up to 4x on a laptop with 4 physical cores.
In pandas, you are only able to use one core at a time when you are doing computation of
any kind. With Modin, you are able to use all of the CPU cores on your machine. Even in
``read_csv``, we see large gains by efficiently distributing the work across your entire
machine.
.. code-block:: python
import modin.pandas as pd
df = pd.read_csv("my_dataset.csv")
Modin is a DataFrame for datasets from 1MB to 1TB+
""""""""""""""""""""""""""""""""""""""""""""""""""
We have focused heavily on bridging the solutions between DataFrames for small data
(e.g. pandas) and large data. Often data scientists require different tools for doing
the same thing on different sizes of data. The DataFrame solutions that exist for 1MB do
not scale to 1TB+, and the overheads of the solutions for 1TB+ are too costly for
datasets in the 1KB range. With Modin, because of its light-weight, robust, and scalable
nature, you get a fast DataFrame at 1MB and 1TB+.
**Modin is currently under active development. Requests and contributions are welcome!**
If you are interested in learning more about Modin, please check out the :doc:`Getting Started`
guide then refer to the :doc:`Developer Documentation` section,
where you can find system architecture, internal implementation details, and other useful information.
Also check out the `Github`_ to view open issues and make contributions.
.. _Dataframe: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html
.. _Ray: https://github.com/ray-project/ray/
.. _Dask: https://dask.org/
.. _Unidist: https://github.com/modin-project/unidist/
.. _Github: https://github.com/modin-project/modin
================================================
FILE: docs/release-procedure.md
================================================
## Versioning
### Patch release
Modin uses semantic versioning. So when doing a patch release, please make a separate branch
off the previous release tag, and `git cherry-pick` **only** the commits we would like to have in our
patch release (assuming previous release was versioned `X.Y.Z`):
git checkout -b release-X.Y.Z+1 X.Y.Z
### Major and Minor releases
A major (`xx.0.0`) or minor (`0.xx.0`) release could be done by branching from `main`:
git checkout -b release-X.Y.0 main
## Preparing the release
Before continuing with the release process, make sure that automated CI which runs on each commit passed successfully with the commit you deem as a "release candidate".
Modin follows the "no push" logic, which is _only_ circumvented for cherry-picked commits,
as reviewing them again would not add a lot of value but would add lots of excess work.
Hence non-cherry-pick commits should happen in a separate branch in your own fork, and
be delivered to the release branch by using a PR.
Note that Modin uses fully signed commits, so you have to have GPG keys set up. See [onboarding instructions](https://github.com/modin-project/modin/blob/main/contributing/contributing.md) on where to get started.
To update Modin version, follow the instructions below.
### Preparing the repo for a Major or Minor Version
**Note**: this should be done in your fork of Modin.
First, update your fork of Modin's main with the main repo's main. From your main, create a new
branch called `release-X.Y.0` off of main. Create an empty commit in your new branch with the message
`Release version X.Y.0`. Make sure to sign this commit with both your GPG key
and with the conventional `git commit -s` (so `git commit -s -S`). Open a PR against modin-project/modin with just this commit.
### Preparing the repo for a Patch Version
**Note**: this should be done in the original Modin repository (in `upstream`) .
First, you must create a new branch in the upstream (main modin-project/modin) repo for the new release.
This branch must be named `release-X.Y.Z`, and should be made off of the tag for the last release. To
do this, use `git checkout -b release-X.Y.Z+1 X.Y.Z` to create the branch for the new release. Once
this branch has been created, cherry-pick the commits that will go into this release, and push this
branch to `upstream`.
**Note**: now you must switch to your fork of Modin.
From your fork of Modin, fetch the upstream repo, and checkout the release branch you made above.
From this release branch, create a new branch.
From your new branch, edit the `README.md` so that the PyPi badge will
point to the badge for this specific version (instead of latest) and so that the docs link will point
to the docs for this specific version (rather than latest).
Once the badges have been edited, create a commit, the same as for a major or minor version,
with the message `Release version X.Y.Z`, and make sure to sign it with both your GPG key, and the
traditional git sign-off. Create a PR using your branch against the `release-X.Y.Z` branch in the
original Modin repo.
### Tag commit
After the PR has been merged, clone a clean copy of the Modin repo from the modin-project organization.
You now need to tag the commit that corresponds to the above PR with the appropriate tag for this release.
**Note**: from now on you work on the `main` branch (in `upstream`) for a major or minor release,
or the `release-X.Y.Z` branch (in `upstream`) for a patch release.
git tag -as X.Y.Z
* Use `scripts/release.py` to draft the release notes (might be as simple as `python scripts/release.py notes > draft.txt`)
* If you're experiencing [rate limiting by GitHub](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting) during username resolving, pass a token via `--token` option to the script
* Fill in the placeholder for summary of the release
* Please look into PR sections and split them if necessary into smaller but better fitting ones, as the script only categorizes by prefix (`FIX-`, `TEST-`, etc.)
* Make sure to correctly resolve contributors whom script failed to transform to GitHub usernames if there are any!
* Include release documentation in the annotation and make sure it is signed.
* Push the tag to `main` or `release-X.Y.Z` branch: `git push upstream X.Y.Z`
* If you're re-pushing a tag (beware! you shouldn't be doing that, no, _really_!), you can remove remote tag and push a local one by `git push upstream :refs/tags/X.Y.Z`
### Build wheel:
**Note**: This should be done from your clean clone of the `upstream` Modin
repository from the modin-project organization, where you made the release tag.
```bash
# Install/update tools
pip install --upgrade build twine
# Build a pure Python wheel.
python3 setup.py sdist bdist_wheel
```
You may see the wheel in the `dist` folder: `ls -l dist`. Make sure the version is correct.
Also make sure there is a `tar` file that contains the source.
### Upload wheels:
Make sure you have an active PyPI account which has write access to Modin pypi repo, and make sure you have a pypi token set up.
Use `twine` to upload wheels:
```bash
twine upload dist/*
```
When asked for account, provide `__token__` (exactly as stated), when asked for password, present your token from pypi.
### Check with `pip install`:
Run `pip install -U "modin[all]"` on Linux, Mac, and Windows systems in a new environment
to test that the wheels were uploaded correctly.
## Make Github and conda-forge release
### Github
Once the tag has been published, we need to make the release on GitHub. Go to the
[Release page](https://github.com/modin-project/modin/releases), and click on `Draft a new release`.
Choose the tag you made above from the dropdown menu, and copy paste the name of the release
in the `Release title` box. Next, copy paste the release notes from above into the box labelled
`Describe this release`. This will ensure that the release notes on GitHub are Markdown formatted.
Double check that everything looks good by clicking `Preview`, and then hit the green `Publish release`
button!
### Conda-forge
Conda-forge has a bot which watches for new releases of software packaged through it,
and in case of Modin it waits either for Github releases or for tags and then makes
a new automatic PR with version increment.
You should watch for that PR and, fixing any issues if there are some, merge it
to make new Modin release appear in `conda-forge` channel. For detailed instructions
on how to ensure the PR passes CI and is merge-able, check out [the how-to page in the modin-feedstock repo](https://github.com/conda-forge/modin-feedstock/blob/main/HOWTO.md)!
## Publicize Release
Once the release has been finalized, make sure to post an announcement
in the #general channel of the public Modin Slack!
================================================
FILE: docs/release_notes/release_notes-0.14.0.rst
================================================
:orphan:
Modin 0.14.0
Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-#4058: Allow pickling empty dataframes and series (#4095)
* FIX-#4136: Fix exercise_3.ipynb example notebook (#4137)
* FIX-#4105: Fix names of pandas options to avoid `OptionError` (#4109)
* FIX-#3417: Fix read_csv with skiprows and header parameters (#3419)
* FIX-#4142: Fix OmniSci enabling (#4146)
* FIX-#4162: Use `skipif` instead of `skip` for compatibility with pytest 7.0 (#4163)
* FIX-#4158: Do not print OmniSci logs to stdout by default (#4159)
* FIX-#4177: Support read_feather from pathlike objects (#4177)
* FIX-#4234: Upgrade pandas to 1.4.1 (#4235)
* FIX-#3368: support unsigned integers in OmniSci backend (#4256)
* FIX-#4057: Allow reading an empty parquet file (#4075)
* FIX-#3884: Fix read_excel() dropping empty rows (#4161)
* FIX-#4257: Fix Categorical() for scalar categories (#4258)
* FIX-#4300: Fix Modin Categorical column dtype categories (#4276)
* FIX-#4208: Fix lazy metadata update for `PandasDataFrame.from_labels` (#4209)
* FIX-#3981, FIX-#3801, FIX-#4149: Stop broadcasting scalars to set items (#4160)
* FIX-#4185: Fix rolling across column partitions (#4262)
* FIX-#4303: Fix the syntax error in reading from postgres (#4304)
* FIX-#4308: Add proper error handling in df.set_index (#4309)
* FIX-#4056: Allow an empty parse_date list in `read_csv_glob` (#4074)
* FIX-#4312: Fix constructing categorical frame with duplicate column names (#4313).
* FIX-#4314: Allow passing a series of dtypes to astype (#4318)
* FIX-#4310: Handle lists of lists of ints in read_csv_glob (#4319)
* Performance enhancements
* FIX-#4138, FIX-#4009: remove redundant sorting in the internal '.mask()' flow (#4140)
* FIX-#4183: Stop shallow copies from creating global shared state. (#4184)
* Benchmarking enhancements
* FIX-#4221: add `wait` method for `PandasOnRayDataframeColumnPartition` class (#4231)
* Refactor Codebase
* REFACTOR-#3990: remove code duplication in `PandasDataframePartition` hierarchy (#3991)
* REFACTOR-#4229: remove unused `dask_client` global variable in `modin\pandas\__init__.py` (#4230)
* REFACTOR-#3997: remove code duplication for `broadcast_apply` method (#3996)
* REFACTOR-#3994: remove code duplication for `get_indices` function (#3995)
* REFACTOR-#4331: remove code duplication for `to_pandas`, `to_numpy` functions in `QueryCompiler` hierarchy (#4332)
* REFACTOR-#4213: Refactor `modin/examples/tutorial/` directory (#4214)
* REFACTOR-#4206: add assert check into `__init__` method of `PandasOnDaskDataframePartition` class (#4207)
* REFACTOR-#3900: add flake8-no-implicit-concat plugin and refactor flake8 error codes (#3901)
* REFACTOR-#4093: Refactor base to be smaller (#4220)
* REFACTOR-#4047: Rename `cluster` directory to `cloud` in examples (#4212)
* REFACTOR-#3853: interacting with Dask interface through `DaskWrapper` class (#3854)
* REFACTOR-#4322: Move is_reduce_fn outside of groupby_agg (#4323)
* Pandas API implementations and improvements
* FEAT-#3603: add experimental `read_custom_text` function that can read custom line-by-line text files (#3441)
* FEAT-#979: Enable reading from SQL server (#4279)
* OmniSci enhancements
*
* XGBoost enhancements
*
* Developer API enhancements
* FEAT-#4245: Define base interface for dataframe exchange protocol (#4246)
* FEAT-#4244: Implement dataframe exchange protocol for HdkOnNative execution (#4269)
* FEAT-#4144: Implement dataframe exchange protocol for pandas storage format (#4150)
* FEAT-#4342: Support `from_dataframe`` for pandas storage format (#4343)
* Update testing suite
* TEST-#3628: Report coverage data for `test-internals` CI job (#4198)
* TEST-#3938: Test tutorial notebooks in CI (#4145)
* TEST-#4153: Fix condition of running lint-commit and set of CI triggers (#4156)
* TEST-#4201: Add read_parquet, explode, tail, and various arithmetic functions to asv_bench (#4203)
* Documentation improvements
* DOCS-#4077: Add release notes template to docs folder (#4078)
* DOCS-#4082: Add pdf/epub/htmlzip formats for doc builds (#4083)
* DOCS-#4168: Fix rendering the examples on troubleshooting page (#4169)
* DOCS-#4151: Add info in troubleshooting page related to Dask engine usage (#4152)
* DOCS-#4172: Refresh Intel Distribution of Modin paragraph (#4175)
* DOCS-#4173: Mention strict channel priority in conda install section (#4178)
* DOCS-#4176: Update OmniSci usage section (#4192)
* DOCS-#4027: Add GIF images and chart to Modin README demonstrating speedups (#4232)
* DOCS-#3954: Add Dask example notebooks (#4139)
* DOCS-#4272: Add bar chart comparisons to quick start guide (#4277)
* DOCS-#3953: Add docs and notebook examples on running Modin with OmniSci (#4001)
* DOCS-#4280: Change links in jupyter notebooks (#4281)
* DOCS-#4290: Add changes for OmniSci notebooks (#4291)
* DOCS-#4241: Update warnings and docs regarding defaulting to pandas (#4242)
* DOCS-#3099: Fix `BasePandasDataSet` docstrings warnings (#4333)
* DOCS-#4339: Reformat I/O functions docstrings (#4341)
* DOCS-#4336: Reformat general utilities docstrings (#4338)
* Dependencies
* FIX-#4113, FIX-#4116, FIX-#4115: Apply new `black` formatting, fix pydocstyle check and readthedocs build (#4114)
* TEST-#3227: Use codecov github action instead of bash form in GA workflows (#3226)
* FIX-#4115: Unpin `pip` in readthedocs deps list (#4170)
* TEST-#4217: Pin `Dask<2022.2.0` as a temporary fix of CI (#4218)
Contributors
------------
@prutskov
@amyskov
@paulovn
@anmyachev
@YarShev
@RehanSD
@devin-petersohn
@dchigarev
@Garra1980
@mvashishtha
@naren-ponder
@jeffreykennethli
@dorisjlee
@Rubtsowa
================================================
FILE: docs/release_notes/release_notes-0.15.0.rst
================================================
:orphan:
Modin 0.15.0
Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-#4376: Upgrade pandas to 1.4.2 (#4377)
* FIX-#3615: Relax some deps in development env (#4365)
* FIX-#4370: Fix broken docstring links (#4375)
* FIX-#4392: Align Modin XGBoost with xgb>=1.6 (#4393)
* FIX-#4385: Get rid of `use-deprecated` option in `pip` (#4386)
* FIX-#3527: Fix parquet partitioning issue causing negative row length partitions (#4368)
* FIX-#4330: Override the memory limit to start ray 1.11.0 on Macs (#4335)
* FIX-#4407: Align `insert` function with pandas in case of numpy array with several columns (#4408)
* FIX-#4373: Fix invalid file path when trying `read_csv_glob` with `usecols` parameter (#4405)
* FIX-#4394: Fix issue with multiindex metadata desync (#4395)
* FIX-#4438: Fix `reindex` function that doesn't preserve initial index metadata (#4442)
* FIX-#4425: Add parameters to groupby pct_change (#4429)
* FIX-#4457: Fix `loc` in case when need reindex item (#4457)
* FIX-#4414: Add missing f prefix on f-strings found at https://codereview.doctor (#4415)
* FIX-#4461: Fix S3 CSV data path (#4462)
* FIX-#4467: `drop_duplicates` no longer removes items based on index values (#4468)
* FIX-#4449: Drain the call queue before waiting on result in benchmark mode (#4472)
* FIX-#4518: Fix Modin Logging to report specific Modin warnings/errors (#4519)
* FIX-#4481: Allow clipping with a Modin Series of bounds (#4486)
* FIX-#4504: Support na_action in applymap (#4505)
* FIX-#4503: Stop the memory logging thread after session exit (#4515)
* FIX-#4531: Fix a makedirs race condition in to_parquet (#4533)
* FIX-#4464: Refactor Ray utils and quick fix groupby.count failing on virtual partitions (#4490)
* FIX-#4436: Fix to_pydatetime dtype for timezone None (#4437)
* FIX-#4541: Fix merge_asof with non-unique right index (#4542)
* Performance enhancements
* FEAT-#4320: Add connectorx as an alternative engine for read_sql (#4346)
* PERF-#4493: Use partition size caches more in Modin dataframe (#4495)
* Benchmarking enhancements
* FEAT-#4371: Add logging to Modin (#4372)
* FEAT-#4501: Add RSS Memory Profiling to Modin Logging (#4502)
* FEAT-#4524: Split Modin API and Memory log files (#4526)
* Refactor Codebase
* REFACTOR-#4284: use variable length unpacking when getting results from `deploy` function (#4285)
* REFACTOR-#3642: Move PyArrow storage format usage from main feature to experimental ones (#4374)
* REFACTOR-#4003: Delete the deprecated cloud mortgage example (#4406)
* REFACTOR-#4513: Fix spelling mistakes in docs and docstrings (#4514)
* REFACTOR-#4510: Align experimental and regular IO modules initializations (#4511)
* Pandas API implementations and improvements
*
* OmniSci enhancements
*
* XGBoost enhancements
*
* Developer API enhancements
* FEAT-#4359: Add __dataframe__ method to the protocol dataframe (#4360)
* Update testing suite
* TEST-#4363: Use Ray from pypi in CI (#4364)
* FIX-#4422: get rid of case sensitivity for `warns_that_defaulting_to_pandas` (#4423)
* TEST-#4426: Stop passing is_default kwarg to Modin and pandas (#4428)
* FIX-#4439: Fix flake8 CI fail (#4440)
* FIX-#4409: Fix `eval_insert` utility that doesn't actually check results of `insert` function (#4410)
* TEST-#4482: Fix getitem and loc with series of bools (#4483).
* Documentation improvements
* DOCS-#4296: Fix docs warnings (#4297)
* DOCS-#4388: Turn off fail_on_warning option for docs build (#4389)
* DOCS-#4469: Say that commit messages can start with PERF (#4470).
* DOCS-#4466: Recommend GitHub issues over bug_reports@modin.org (#4474).
* DOCS-#4487: Recommend GitHub issues over feature_requests@modin.org (#4489).
* DOCS-#4545: Add socials to README (#4555).
* Dependencies
* FIX-#4327: Update min pin for xgboost version (#4328)
* FIX-#4383: Remove `pathlib` from deps (#4384)
* FIX-#4390: Add `redis` to Modin dependencies (#4396)
* FIX-#3689: Add black and flake8 into development environment files (#4480)
* TEST-#4516: Add numpydoc to developer requirements (#4517)
* New Features
* FEAT-#4412: Add Batch Pipeline API to Modin (#4452)
Contributors
------------
@YarShev
@Garra1980
@prutskov
@alexander3774
@amyskov
@wangxiaoying
@jeffreykennethli
@mvashishtha
@anmyachev
@dchigarev
@devin-petersohn
@jrsacher
@orcahmlee
@naren-ponder
@RehanSD
================================================
FILE: docs/release_notes/release_notes-0.16.0.rst
================================================
:orphan:
Modin 0.16.0
Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-#4570: Replace ``np.bool`` -> ``np.bool_`` (#4571)
* FIX-#4543: Fix `read_csv` in case skiprows=<0, []> (#4544)
* FIX-#4059: Add cell-wise execution for binary ops, fix bin ops for empty dataframes (#4391)
* FIX-#4589: Pin protobuf<4.0.0 to fix ray (#4590)
* FIX-#4577: Set attribute of Modin dataframe to updated value (#4588)
* FIX-#4411: Fix binary_op between datetime64 Series and pandas timedelta (#4592)
* FIX-#4604: Fix `groupby` + `agg` in case when multicolumn can arise (#4642)
* FIX-#4582: Inherit custom log layer (#4583)
* FIX-#4639: Fix `storage_options` usage for `read_csv` and `read_csv_glob` (#4644)
* FIX-#4593: Ensure Modin warns when setting columns via attributes (#4621)
* FIX-#4584: Enable pdb debug when running cloud tests (#4585)
* FIX-#4564: Workaround import issues in Ray: auto-import pandas on python start if env var is set (#4603)
* FIX-#4641: Reindex pandas partitions in `df.describe()` (#4651)
* FIX-#2064: Fix `iloc`/`loc` assignment when dataframe is empty (#4677)
* FIX-#4634: Check for FrozenList as `by` in `df.groupby()` (#4667)
* FIX-#4680: Fix `read_csv` that started defaulting to pandas again in case of reading from a buffer and when a buffer has a non-zero starting position (#4681)
* FIX-#4491: Wait for all partitions in parallel in benchmark mode (#4656)
* FIX-#4358: MultiIndex `loc` shouldn't drop levels for full-key lookups (#4608)
* FIX-#4658: Expand exception handling for `read_*` functions from s3 storages (#4659)
* FIX-#4672: Fix incorrect warning when setting `frame.index` or `frame.columns` (#4721)
* FIX-#4686: Propagate metadata and drain call queue in unwrap_partitions (#4697)
* FIX-#4652: Support categorical data in `from_dataframe` (#4737)
* FIX-#4756: Correctly propagate `storage_options` in `read_parquet` (#4764)
* FIX-#4657: Use `fsspec` for handling s3/http-like paths instead of `s3fs` (#4710)
* FIX-#4676: drain sub-virtual-partition call queues (#4695)
* FIX-#4782: Exclude certain non-parquet files in `read_parquet` (#4783)
* FIX-#4808: Set dtypes correctly after column rename (#4809)
* FIX-#4811: Apply dataframe -> not_dataframe functions to virtual partitions (#4812)
* FIX-#4099: Use mangled column names but keep the original when building frames from arrow (#4767)
* FIX-#4838: Bump up modin-spreadsheet to latest master (#4839)
* FIX-#4840: Change modin-spreadsheet version for notebook requirements (#4841)
* FIX-#4835: Handle Pathlike paths in `read_parquet` (#4837)
* FIX-#4872: Stop checking the private ray mac memory limit (#4873)
* FIX-#4914: `base_lengths` should be computed from `base_frame` instead of `self` in `copartition` (#4915)
* FIX-#4848: Fix rebalancing partitions when NPartitions == 1 (#4874)
* FIX-#4927: Fix `dtypes` computation in `dataframe.filter` (#4928)
* FIX-#4907: Implement `radd` for Series and DataFrame (#4908)
* FIZ-#4945: Fix `_take_2d_positional` that loses indexes due to filtering empty dataframes (#4951)
* FIX-#4818, PERF-#4825: Fix where by using the new n-ary operator (#4820)
* FIX-#3983: FIX-#4107: Materialize 'rowid' columns when selecting rows by position (#4834)
* FIX-#4845: Fix KeyError from `__getitem_bool` for single row dataframes (#4845)
* FIX-#4734: Handle Series.apply when return type is a DataFrame (#4830)
* FIX-#4983: Set `frac` to `None` in _sample when `n=0` (#4984)
* FIX-#4993: Return `_default_to_pandas` in `df.attrs` (#4995)
* FIX-#5043: Fix `execute` function in ASV utils failed if `len(partitions) == 0` (#5044)
* FIX-#4597: Refactor Partition handling of func, args, kwargs (#4715)
* FIX-#4996: Evaluate BenchmarkMode at each function call (#4997)
* FIX-#4022: Fixed empty data frame with index (#4910)
* FIX-#4090: Fixed check if the index is trivial (#4936)
* FIX-#4966: Fix `to_timedelta` to return Series instead of TimedeltaIndex (#5028)
* FIX-#5042: Fix series __getitem__ with invalid strings (#5048)
* FIX-#4691: Fix binary operations between virtual partitions (#5049)
* FIX-#5045: Fix ray virtual_partition.wait with duplicate object refs (#5058)
* Performance enhancements
* PERF-#4182: Add cell-wise execution for binary ops, fix bin ops for empty dataframes (#4391)
* PERF-#4288: Improve perf of `groupby.mean` for narrow data (#4591)
* PERF-#4772: Remove `df.copy` call from `from_pandas` since it is not needed for Ray and Dask (#4781)
* PERF-#4325: Improve perf of multi-column assignment in `__setitem__` when no new column names are assigning (#4455)
* PERF-#3844: Improve perf of `drop` operation (#4694)
* PERF-#4727: Improve perf of `concat` operation (#4728)
* PERF-#4705: Improve perf of arithmetic operations between `Series` objects with shared `.index` (#4689)
* PERF-#4703: Improve performance in accessing `ser.cat.categories`, `ser.cat.ordered`, and `ser.__array_priority__` (#4704)
* PERF-#4305: Parallelize `read_parquet` over row groups (#4700)
* PERF-#4773: Compute `lengths` and `widths` in `put` method of Dask partition like Ray do (#4780)
* PERF-#4732: Avoid overwriting already-evaluated `PandasOnRayDataframePartition._length_cache` and `PandasOnRayDataframePartition._width_cache` (#4754)
* PERF-#4862: Don't call `compute_sliced_len.remote` when `row_labels/col_labels == slice(None)` (#4863)
* PERF-#4713: Stop overriding the ray MacOS object store size limit (#4792)
* PERF-#4851: Compute `dtypes` for binary operations that can only return bool type and the right operand is not a Modin object (#4852)
* PERF-#4842: `copy` should not trigger any previous computations (#4843)
* PERF-#4849: Compute `dtypes` in `concat` also for ROW_WISE case when possible (#4850)
* PERF-#4929: Compute `dtype` when using `Series.dt` accessor (#4930)
* PERF-#4892: Compute `lengths` in `rebalance_partitions` when possible (#4893)
* PERF-#4794: Compute caches in `_propagate_index_objs` (#4888)
* PERF-#4860: `PandasDataframeAxisPartition.deploy_axis_func` should be serialized only once (#4861)
* PERF-#4890: `PandasDataframeAxisPartition.drain` should be serialized only once (#4891)
* PERF-#4870: Avoid index materialization in `__getattribute__` and `__getitem__` (4911)
* PERF-#4886: Use lazy index and columns evaluation in `query` method (#4887)
* PERF-#4866: `iloc` function that used in `partition.mask` should be serialized only once (#4901)
* PERF-#4920: Avoid index and cache computations in `take_2d_labels_or_positional` unless they are needed (#4921)
* PERF-#4999: don't call `apply` in virtual partition' `drain_call_queue` if `call_queue` is empty (#4975)
* PERF-#4268: Implement partition-parallel __getitem__ for bool Series masks (#4753)
* PERF-#5017: `reset_index` shouldn't trigger index materialization if possible (#5018)
* PERF-#4963: Use partition `width/length` methods instead of `_compute_axis_labels_and_lengths` if index is already known (#4964)
* PERF-#4940: Optimize categorical dtype check in `concatenate` (#4953)
* Benchmarking enhancements
* TEST-#5066: Add outer join case for `TimeConcat` benchmark (#5067)
* TEST-#5083: Add `merge` op with categorical data (#5084)
* FEAT-#4706: Add Modin ClassLogger to PandasDataframePartitionManager (#4707)
* TEST-#5014: Simplify adding new ASV benchmarks (#5015)
* TEST-#5064: Update `TimeConcat` benchmark with new parameter `ignore_index` (#5065)
* PERF-#4944: Avoid default_to_pandas in ``Series.cat.codes``, ``Series.dt.tz``, and ``Series.dt.to_pytimedelta`` (#4833)
* TEST-#5068: Add binary op benchmark for Series (#5069)
* Refactor Codebase
* REFACTOR-#4530: Standardize access to physical data in partitions (#4563)
* REFACTOR-#4534: Replace logging meta class with class decorator (#4535)
* REFACTOR-#4708: Delete combine dtypes (#4709)
* REFACTOR-#4629: Add type annotations to modin/config (#4685)
* REFACTOR-#4717: Improve PartitionMgr.get_indices() usage (#4718)
* REFACTOR-#4730: make Indexer immutable (#4731)
* REFACTOR-#4774: remove `_build_treereduce_func` call from `_compute_dtypes` (#4775)
* REFACTOR-#4750: Delete BaseDataframeAxisPartition.shuffle (#4751)
* REFACTOR-#4722: Stop suppressing undefined name lint (#4723)
* REFACTOR-#4832: unify `split_result_of_axis_func_pandas` (#4831)
* REFACTOR-#4796: Introduce constant for __reduced__ column name (#4799)
* REFACTOR-#4000: Remove code duplication for `PandasOnRayDataframePartitionManager` (#4895)
* REFACTOR-#3780: Remove code duplication for `PandasOnDaskDataframe` (#3781)
* REFACTOR-#4530: Unify access to physical data for any partition type (#4829)
* REFACTOR-#4978: Align `modin/core/execution/dask/common/__init__.py` with `modin/core/execution/ray/common/__init__.py` (#4979)
* REFACTOR-#4949: Remove code duplication in `default2pandas/dataframe.py` and `default2pandas/any.py` (#4950)
* REFACTOR-#4976: Rename `RayTask` to `RayWrapper` in accordance with Dask (#4977)
* REFACTOR-#4885: De-duplicated take_2d_labels_or_positional methods (#4883)
* REFACTOR-#5005: Use `finalize` method instead of list comprehension + `drain_call_queue` (#5006)
* REFACTOR-#5001: Remove `jenkins` stuff (#5002)
* REFACTOR-#5026: Change exception names to simplify grepping (#5027)
* REFACTOR-#4970: Rewrite base implementations of a partition' `width/length` (#4971)
* REFACTOR-#4942: Remove `call` method in favor of `register` due to duplication (4943)
* REFACTOR-#4922: Helpers for take_2d_labels_or_positional (#4865)
* REFACTOR-#5024: Make `_row_lengths` and `_column_widths` public (#5025)
* REFACTOR-#5009: Use `RayWrapper.materialize` instead of `ray.get` (#5010)
* REFACTOR-#4755: Rewrite Pandas version mismatch warning (#4965)
* REFACTOR-#5012: Add mypy checks for singleton files in base modin directory (#5013)
* REFACTOR-#5038: Remove unnecessary _method argument from resamplers (#5039)
* REFACTOR-#5081: Remove `c323f7fe385011ed849300155de07645.db` file (#5082)
* Pandas API implementations and improvements
* FEAT-#4670: Implement convert_dtypes by mapping across partitions (#4671)
* OmniSci enhancements
* FEAT-#4913: Enabling pyhdk
* XGBoost enhancements
*
* Developer API enhancements
*
* Update testing suite
* TEST-#4508: Reduce test_partition_api pytest threads to deflake it (#4551)
* TEST-#4550: Use much less data in test_partition_api (#4554)
* TEST-#4610: Remove explicit installation of `black`/`flake8` for omnisci ci-notebooks (#4609)
* TEST-#2564: Add caching and use mamba for conda setups in GH (#4607)
* TEST-#4557: Delete multiindex sorts instead of xfailing (#4559)
* TEST-#4698: Stop passing invalid storage_options param (#4699)
* TEST-#4745: Pin flake8 to <5 to workaround installation conflict (#4752)
* TEST-#4875: XFail tests failing due to file gone missing (#4876)
* TEST-#4879: Use pandas `ensure_clean()` in place of `io_tests_data` (#4881)
* TEST-#4562: Use local Ray cluster in CI to resolve flaky `test-compat-win` (#5007)
* TEST-#5040: Rework test_series using eval_general() (#5041)
* TEST-#5050: Add black to pre-commit hook (#5051)
* Documentation improvements
* DOCS-#4552: Change default sphinx language to en to fix sphinx >= 5.0.0 build (#4553)
* DOCS-#4628: Add to_parquet partial support notes (#4648)
* DOCS-#4668: Set light theme for readthedocs page, remove theme switcher (#4669)
* DOCS-#4748: Apply the Triage label to new issues (#4749)
* DOCS-#4790: Give all templates issue type and triage labels (#4791)
* DOCS-#4521: Document how to benchmark modin (#5020)
* Dependencies
* FEAT-#4598: Add support for pandas 1.4.3 (#4599)
* FEAT-#4619: Integrate mypy static type checking (#4620)
* FEAT-#4202: Allow dask past 2022.2.0 (#4769)
* FEAT-#4925: Upgrade pandas to 1.4.4 (#4926)
* TEST-#4998: Add flake8 plugins to dev requirements (#5000)
* New Features
* FEAT-4463: Add experimental fuzzydata integration for testing against a randomized dataframe workflow (#4556)
* FEAT-#4419: Extend virtual partitioning API to pandas on Dask (#4420)
* FEAT-#4147: Add partial compatibility with Python 3.6 and pandas 1.1 (#4301)
* FEAT-#4569: Add error message when `read_` function defaults to pandas (#4647)
* FEAT-#4725: Make index and columns lazy in Modin DataFrame (#4726)
* FEAT-#4664: Finalize compatibility support for Python 3.6 (#4800)
* FEAT-#4746: Sync interchange protocol with recent API changes (#4763)
* FEAT-#4733: Support fastparquet as engine for `read_parquet` (#4807)
* FEAT-#4766: Support fsspec URLs in `read_csv` and `read_csv_glob` (#4898)
* FEAT-#4827: Implement `infer_types` dataframe algebra operator (#4871)
* FEAT-#4989: Switch pandas version to 1.5 (#5037)
Contributors
------------
@mvashishtha
@NickCrews
@prutskov
@vnlitvinov
@pyrito
@suhailrehman
@RehanSD
@helmeleegy
@anmyachev
@d33bs
@noloerino
@devin-petersohn
@YarShev
@naren-ponder
@jbrockmendel
@ienkovich
@Garra1980
@Billy2551
================================================
FILE: docs/release_notes/release_notes-template.rst
================================================
:orphan:
Modin X.X.X
Key Features and Updates
------------------------
* Stability and Bugfixes
*
* Performance enhancements
*
* Benchmarking enhancements
*
* Refactor Codebase
*
* Pandas API implementations and improvements
*
* HDK enhancements
*
* XGBoost enhancements
*
* Developer API enhancements
*
* Update testing suite
*
* Documentation improvements
*
* Dependencies
*
* New Features
Contributors
------------
================================================
FILE: docs/requirements-doc.txt
================================================
# install current modin checkout to bring all required dependencies
.[all]
# now install some more optional dependencies
colorama
click
flatbuffers
funcsigs
mock
opencv-python
pydata_sphinx_theme
pyyaml
recommonmark
sphinx<6.0.0
sphinx-click
ray>=2.10.0,<3
# Override to latest version of modin-spreadsheet
git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5
sphinxcontrib_plantuml
sphinx-issues
================================================
FILE: docs/supported_apis/dataframe_supported.rst
================================================
``pd.DataFrame`` supported APIs
===================================
The following table lists both implemented and not implemented methods. If you have need
of an operation that is listed as not implemented, feel free to open an issue on the
`GitHub repository`_, or give a thumbs up to already created issues. Contributions are
also welcome!
The following table is structured as follows: The first column contains the method name.
The second column contains link to a description of corresponding pandas method.
The third column is a flag for whether or not there is an implementation in Modin for
the method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands
for partial (meaning some parameters may not be supported yet), and ``D`` stands for
default to pandas.
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| DataFrame method | pandas Doc link | Implemented? (Y/N/P/D) | Notes for Current implementation |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``T`` | `T`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``abs`` | `abs`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``add`` | `add`_ | Y | **Ray** and **Dask**: Shuffles data in operations |
| | | | between DataFrames. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``add_prefix`` | `add_prefix`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``add_suffix`` | `add_suffix`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``agg`` / ``aggregate`` | `agg`_ / `aggregate`_ | P | - Dictionary ``func`` parameter defaults to pandas |
| | | | - Numpy operations default to pandas |
| | | | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``align`` | `align`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``all`` | `all`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``any`` | `any`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``apply`` | `apply`_ | Y | See ``agg`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``applymap`` | `applymap`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``asfreq`` | `asfreq`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``asof`` | `asof`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``assign`` | `assign`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``astype`` | `astype`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``at`` | `at`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``at_time`` | `at_time`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``axes`` | `axes`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``between_time`` | `between_time`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``bfill`` | `bfill`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``bool`` | `bool`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``boxplot`` | `boxplot`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``clip`` | `clip`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``combine`` | `combine`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``combine_first`` | `combine_first`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``compare`` | `compare`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``copy`` | `copy`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``corr`` | `corr`_ | P | Correlation floating point precision may slightly |
| | | | differ from pandas. For now pearson method is |
| | | | available only. For other methods and for |
| | | | ``numeric_only`` defaults to pandas. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``corrwith`` | `corrwith`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``count`` | `count`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``cov`` | `cov`_ | P | Covariance floating point precision may slightly |
| | | | differ from pandas. For ``numeric_only`` |
| | | | defaults to pandas. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``cummax`` | `cummax`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``cummin`` | `cummin`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``cumprod`` | `cumprod`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``cumsum`` | `cumsum`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``describe`` | `describe`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``diff`` | `diff`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``div`` | `div`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``divide`` | `divide`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``dot`` | `dot`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``drop`` | `drop`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``droplevel`` | `droplevel`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``drop_duplicates`` | `drop_duplicates`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``dropna`` | `dropna`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``dtypes`` | `dtypes`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``duplicated`` | `duplicated`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``empty`` | `empty`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``eq`` | `eq`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``equals`` | `equals`_ | Y | Requires shuffle, can be further optimized |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``eval`` | `eval`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``ewm`` | `ewm`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``expanding`` | `expanding`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``explode`` | `explode`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``ffill`` | `ffill`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``fillna`` | `fillna`_ | P | ``value`` parameter of type DataFrame defaults to |
| | | | pandas. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``filter`` | `filter`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``first`` | `first`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``first_valid_index`` | `first_valid_index`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``floordiv`` | `floordiv`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``from_dict`` | `from_dict`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``from_records`` | `from_records`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``ge`` | `ge`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``get`` | `get`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``groupby`` | `groupby`_ | Y | Not yet optimized for all operations. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``gt`` | `gt`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``head`` | `head`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``hist`` | `hist`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``iat`` | `iat`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``idxmax`` | `idxmax`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``idxmin`` | `idxmin`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``iloc`` | `iloc`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``infer_objects`` | `infer_objects`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``info`` | `info`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``insert`` | `insert`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``interpolate`` | `interpolate`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``isetitem`` | `isetitem`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``isin`` | `isin`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``isna`` | `isna`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``isnull`` | `isnull`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``items`` | `items`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``iterrows`` | `iterrows`_ | P | Modin does not parallelize iteration in Python |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``itertuples`` | `itertuples`_ | P | Modin does not parallelize iteration in Python |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``join`` | `join`_ | P | When ``on`` is set to ``right`` or ``outer`` or |
| | | | when ``validate`` is given defaults to pandas |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``keys`` | `keys`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``kurt`` | `kurt`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``kurtosis`` | `kurtosis`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``last`` | `last`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``last_valid_index`` | `last_valid_index`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``le`` | `le`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``loc`` | `loc`_ | P | We do not support: boolean array, callable. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``lt`` | `lt`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``mask`` | `mask`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``max`` | `max`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``mean`` | `mean`_ | P | Modin defaults to pandas if given the ``level`` |
| | | | param. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``median`` | `median`_ | P | Modin defaults to pandas if given the ``level`` |
| | | | param. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``melt`` | `melt`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``memory_usage`` | `memory_usage`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| | | | Implemented the following cases: |
| | | | ``left_index=True`` and ``right_index=True``, |
| | | | ``how=left`` and ``how=inner`` for all values |
| ``merge`` | `merge`_ | P | of parameters except ``left_index=True`` and |
| | | | ``right_index=False`` or ``left_index=False`` |
| | | | and ``right_index=True``. |
| | | | Defaults to pandas otherwise. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``min`` | `min`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``mod`` | `mod`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``mode`` | `mode`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``mul`` | `mul`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``multiply`` | `multiply`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``ndim`` | `ndim`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``ne`` | `ne`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``nlargest`` | `nlargest`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``notna`` | `notna`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``notnull`` | `notnull`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``nsmallest`` | `nsmallest`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``nunique`` | `nunique`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``pct_change`` | `pct_change`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``pipe`` | `pipe`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``pivot`` | `pivot`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``pivot_table`` | `pivot_table`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``plot`` | `plot`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``pop`` | `pop`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``pow`` | `pow`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``prod`` | `prod`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``product`` | `product`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``quantile`` | `quantile`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``query`` | `query`_ | P | Local variables not yet supported |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``radd`` | `radd`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rank`` | `rank`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rdiv`` | `rdiv`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``reindex`` | `reindex`_ | Y | Shuffles data |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``reindex_like`` | `reindex_like`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rename`` | `rename`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rename_axis`` | `rename_axis`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``reorder_levels`` | `reorder_levels`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``replace`` | `replace`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``resample`` | `resample`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``reset_index`` | `reset_index`_ | P | **Ray** and **Dask**: ``D`` when ``names`` or |
| | | | ``allow_duplicates`` is non-default |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rfloordiv`` | `rfloordiv`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rmod`` | `rmod`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rmul`` | `rmul`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rolling`` | `rolling`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``round`` | `round`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rpow`` | `rpow`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rsub`` | `rsub`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``rtruediv`` | `rtruediv`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sample`` | `sample`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``select_dtypes`` | `select_dtypes`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sem`` | `sem`_ | P | Modin defaults to pandas if given the ``level`` |
| | | | param. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``set_axis`` | `set_axis`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``set_index`` | `set_index`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``shape`` | `shape`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``shift`` | `shift`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``size`` | `size`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``skew`` | `skew`_ | P | Modin defaults to pandas if given the ``level`` |
| | | | param |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sort_index`` | `sort_index`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sort_values`` | `sort_values`_ | Y | Shuffles data. Order of indexes that have the |
| | | | same sort key is not guaranteed to be the same |
| | | | across sorts |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sparse`` | `sparse`_ | N | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``squeeze`` | `squeeze`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``stack`` | `stack`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``std`` | `std`_ | P | Modin defaults to pandas if given the ``level`` |
| | | | param. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``style`` | `style`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sub`` | `sub`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``subtract`` | `subtract`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``sum`` | `sum`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``swapaxes`` | `swapaxes`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``swaplevel`` | `swaplevel`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``tail`` | `tail`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``take`` | `take`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_clipboard`` | `to_clipboard`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_csv`` | `to_csv`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_dict`` | `to_dict`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_excel`` | `to_excel`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_feather`` | `to_feather`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_gbq`` | `to_gbq`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_hdf`` | `to_hdf`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_html`` | `to_html`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_json`` | `to_json`_ | D | |
| | | | Experimental implementation: |
| | | | DataFrame.modin.to_json_glob |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_xml`` | `to_xml`_ | D | |
| | | | Experimental implementation: |
| | | | DataFrame.modin.to_xml_glob |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_latex`` | `to_latex`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_orc`` | `to_orc`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_parquet`` | `to_parquet`_ | P | **Ray/Dask/Unidist**: Parallel implementation only |
| | | | if path parameter is a string. In that case, the |
| | | | ``path`` parameter specifies a directory where one |
| | | | file is written per row partition of the Modin |
| | | | dataframe. |
| | | | Experimental implementation: |
| | | | DataFrame.modin.to_parquet_glob |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_period`` | `to_period`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_pickle`` | `to_pickle`_ | D | Experimental implementation: |
| | | | DataFrame.modin.to_pickle_glob |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_records`` | `to_records`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_sql`` | `to_sql`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_stata`` | `to_stata`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_string`` | `to_string`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_timestamp`` | `to_timestamp`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``to_xarray`` | `to_xarray`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``transform`` | `transform`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``transpose`` | `transpose`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``truediv`` | `truediv`_ | Y | See ``add`` |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``truncate`` | `truncate`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``tz_convert`` | `tz_convert`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``tz_localize`` | `tz_localize`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``unstack`` | `unstack`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``update`` | `update`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``values`` | `values`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``value_counts`` | `value_counts`_ | D | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``var`` | `var`_ | P | Modin defaults to pandas if given the ``level`` |
| | | | param. |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
| ``where`` | `where`_ | Y | |
+----------------------------+---------------------------+------------------------+----------------------------------------------------+
.. _`GitHub repository`: https://github.com/modin-project/modin/issues
.. _`T`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.T.html#pandas.DataFrame.T
.. _`abs`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.abs.html#pandas.DataFrame.abs
.. _`add`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.add.html#pandas.DataFrame.add
.. _`add_prefix`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.add_prefix.html#pandas.DataFrame.add_prefix
.. _`add_suffix`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.add_suffix.html#pandas.DataFrame.add_suffix
.. _`agg`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.agg.html#pandas.DataFrame.agg
.. _`aggregate`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.aggregate.html#pandas.DataFrame.aggregate
.. _`align`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.align.html#pandas.DataFrame.align
.. _`all`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.all.html#pandas.DataFrame.all
.. _`any`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.any.html#pandas.DataFrame.any
.. _`apply`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html#pandas.DataFrame.apply
.. _`applymap`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.applymap.html#pandas.DataFrame.applymap
.. _`asfreq`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.asfreq.html#pandas.DataFrame.asfreq
.. _`asof`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.asof.html#pandas.DataFrame.asof
.. _`assign`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.assign.html#pandas.DataFrame.assign
.. _`astype`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.astype.html#pandas.DataFrame.astype
.. _`at`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.at.html#pandas.DataFrame.at
.. _`at_time`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.at_time.html#pandas.DataFrame.at_time
.. _`axes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.axes.html#pandas.DataFrame.axes
.. _`between_time`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.between_time.html#pandas.DataFrame.between_time
.. _`bfill`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.bfill.html#pandas.DataFrame.bfill
.. _`bool`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.bool.html#pandas.DataFrame.bool
.. _`boxplot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.boxplot.html#pandas.DataFrame.boxplot
.. _`clip`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.clip.html#pandas.DataFrame.clip
.. _`combine`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.combine.html#pandas.DataFrame.combine
.. _`combine_first`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.combine_first.html#pandas.DataFrame.combine_first
.. _`compare`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.compare.html#pandas.DataFrame.compare
.. _`compound`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.compound.html#pandas.DataFrame.compound
.. _`copy`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.copy.html#pandas.DataFrame.copy
.. _`corr`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html#pandas.DataFrame.corr
.. _`corrwith`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corrwith.html#pandas.DataFrame.corrwith
.. _`count`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.count.html#pandas.DataFrame.count
.. _`cov`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cov.html#pandas.DataFrame.cov
.. _`cummax`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cummax.html#pandas.DataFrame.cummax
.. _`cummin`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cummin.html#pandas.DataFrame.cummin
.. _`cumprod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cumprod.html#pandas.DataFrame.cumprod
.. _`cumsum`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.cumsum.html#pandas.DataFrame.cumsum
.. _`describe`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html#pandas.DataFrame.describe
.. _`diff`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.diff.html#pandas.DataFrame.diff
.. _`div`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.div.html#pandas.DataFrame.div
.. _`divide`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.divide.html#pandas.DataFrame.divide
.. _`dot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dot.html#pandas.DataFrame.dot
.. _`drop`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html#pandas.DataFrame.drop
.. _`droplevel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.droplevel.html
.. _`drop_duplicates`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html#pandas.DataFrame.drop_duplicates
.. _`dropna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html#pandas.DataFrame.dropna
.. _`dtypes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dtypes.html#pandas.DataFrame.dtypes
.. _`duplicated`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.duplicated.html#pandas.DataFrame.duplicated
.. _`empty`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.empty.html#pandas.DataFrame.empty
.. _`eq`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.eq.html#pandas.DataFrame.eq
.. _`equals`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.equals.html#pandas.DataFrame.equals
.. _`eval`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.eval.html#pandas.DataFrame.eval
.. _`ewm`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ewm.html#pandas.DataFrame.ewm
.. _`expanding`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.expanding.html#pandas.DataFrame.expanding
.. _`explode`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.explode.html#pandas-dataframe-explode
.. _`ffill`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ffill.html#pandas.DataFrame.ffill
.. _`fillna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html#pandas.DataFrame.fillna
.. _`filter`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.filter.html#pandas.DataFrame.filter
.. _`first`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.first.html#pandas.DataFrame.first
.. _`first_valid_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.first_valid_index.html#pandas.DataFrame.first_valid_index
.. _`floordiv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.floordiv.html#pandas.DataFrame.floordiv
.. _`from_dict`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict
.. _`from_records`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_records.html#pandas.DataFrame.from_records
.. _`ge`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ge.html#pandas.DataFrame.ge
.. _`get`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get.html#pandas.DataFrame.get
.. _`get_dtype_counts`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_dtype_counts.html#pandas.DataFrame.get_dtype_counts
.. _`get_ftype_counts`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_ftype_counts.html#pandas.DataFrame.get_ftype_counts
.. _`get_value`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_value.html#pandas.DataFrame.get_value
.. _`get_values`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.get_values.html#pandas.DataFrame.get_values
.. _`groupby`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html#pandas.DataFrame.groupby
.. _`gt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.gt.html#pandas.DataFrame.gt
.. _`head`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html#pandas.DataFrame.head
.. _`hist`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.hist.html#pandas.DataFrame.hist
.. _`iat`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iat.html#pandas.DataFrame.iat
.. _`idxmax`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.idxmax.html#pandas.DataFrame.idxmax
.. _`idxmin`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.idxmin.html#pandas.DataFrame.idxmin
.. _`iloc`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html#pandas.DataFrame.iloc
.. _`infer_objects`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.infer_objects.html#pandas.DataFrame.infer_objects
.. _`info`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.info.html#pandas.DataFrame.info
.. _`insert`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.insert.html#pandas.DataFrame.insert
.. _`interpolate`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.interpolate.html#pandas.DataFrame.interpolate
.. _`is_copy`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.is_copy.html#pandas.DataFrame.is_copy
.. _`isetitem`: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isetitem.html?#pandas-dataframe-isetitem
.. _`isin`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isin.html#pandas.DataFrame.isin
.. _`isna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isna.html#pandas.DataFrame.isna
.. _`isnull`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isnull.html#pandas.DataFrame.isnull
.. _`items`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.items.html#pandas.DataFrame.items
.. _`iterrows`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iterrows.html#pandas.DataFrame.iterrows
.. _`itertuples`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.itertuples.html#pandas.DataFrame.itertuples
.. _`ix`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ix.html#pandas.DataFrame.ix
.. _`join`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html#pandas.DataFrame.join
.. _`keys`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.keys.html#pandas.DataFrame.keys
.. _`kurt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurt.html#pandas.DataFrame.kurt
.. _`kurtosis`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html#pandas.DataFrame.kurtosis
.. _`last`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.last.html#pandas.DataFrame.last
.. _`last_valid_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.last_valid_index.html#pandas.DataFrame.last_valid_index
.. _`le`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.le.html#pandas.DataFrame.le
.. _`loc`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html#pandas.DataFrame.loc
.. _`lt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.lt.html#pandas.DataFrame.lt
.. _`mask`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mask.html#pandas.DataFrame.mask
.. _`max`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.max.html#pandas.DataFrame.max
.. _`mean`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html#pandas.DataFrame.mean
.. _`median`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.median.html#pandas.DataFrame.median
.. _`melt`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.melt.html#pandas.DataFrame.melt
.. _`memory_usage`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.memory_usage.html#pandas.DataFrame.memory_usage
.. _`merge`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html#pandas.DataFrame.merge
.. _`min`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.min.html#pandas.DataFrame.min
.. _`mod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mod.html#pandas.DataFrame.mod
.. _`mode`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mode.html#pandas.DataFrame.mode
.. _`mul`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mul.html#pandas.DataFrame.mul
.. _`multiply`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.multiply.html#pandas.DataFrame.multiply
.. _`ndim`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ndim.html#pandas.DataFrame.ndim
.. _`ne`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ne.html#pandas.DataFrame.ne
.. _`nlargest`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nlargest.html#pandas.DataFrame.nlargest
.. _`notna`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.notna.html#pandas.DataFrame.notna
.. _`notnull`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.notnull.html#pandas.DataFrame.notnull
.. _`nsmallest`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nsmallest.html#pandas.DataFrame.nsmallest
.. _`nunique`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.nunique.html#pandas.DataFrame.nunique
.. _`pct_change`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pct_change.html#pandas.DataFrame.pct_change
.. _`pipe`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pipe.html#pandas.DataFrame.pipe
.. _`pivot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot.html#pandas.DataFrame.pivot
.. _`pivot_table`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot_table.html#pandas.DataFrame.pivot_table
.. _`plot`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html#pandas.DataFrame.plot
.. _`pop`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pop.html#pandas.DataFrame.pop
.. _`pow`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pow.html#pandas.DataFrame.pow
.. _`prod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.prod.html#pandas.DataFrame.prod
.. _`product`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.product.html#pandas.DataFrame.product
.. _`quantile`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.quantile.html#pandas.DataFrame.quantile
.. _`query`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html#pandas.DataFrame.query
.. _`radd`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.radd.html#pandas.DataFrame.radd
.. _`rank`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rank.html#pandas.DataFrame.rank
.. _`rdiv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rdiv.html#pandas.DataFrame.rdiv
.. _`reindex`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reindex.html#pandas.DataFrame.reindex
.. _`reindex_like`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reindex_like.html#pandas.DataFrame.reindex_like
.. _`rename`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html#pandas.DataFrame.rename
.. _`rename_axis`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename_axis.html#pandas.DataFrame.rename_axis
.. _`reorder_levels`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reorder_levels.html#pandas.DataFrame.reorder_levels
.. _`replace`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.replace.html#pandas.DataFrame.replace
.. _`resample`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html#pandas.DataFrame.resample
.. _`reset_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html#pandas.DataFrame.reset_index
.. _`rfloordiv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rfloordiv.html#pandas.DataFrame.rfloordiv
.. _`rmod`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rmod.html#pandas.DataFrame.rmod
.. _`rmul`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rmul.html#pandas.DataFrame.rmul
.. _`rolling`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rolling.html#pandas.DataFrame.rolling
.. _`round`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.round.html#pandas.DataFrame.round
.. _`rpow`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rpow.html#pandas.DataFrame.rpow
.. _`rsub`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rsub.html#pandas.DataFrame.rsub
.. _`rtruediv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rtruediv.html#pandas.DataFrame.rtruediv
.. _`sample`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html#pandas.DataFrame.sample
.. _`select_dtypes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.select_dtypes.html#pandas.DataFrame.select_dtypes
.. _`sem`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sem.html#pandas.DataFrame.sem
.. _`set_axis`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_axis.html#pandas.DataFrame.set_axis
.. _`set_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_index.html#pandas.DataFrame.set_index
.. _`set_value`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.set_value.html#pandas.DataFrame.set_value
.. _`shape`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html#pandas.DataFrame.shape
.. _`shift`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shift.html#pandas.DataFrame.shift
.. _`size`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.size.html#pandas.DataFrame.size
.. _`skew`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.skew.html#pandas.DataFrame.skew
.. _`sort_index`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_index.html#pandas.DataFrame.sort_index
.. _`sort_values`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_values.html#pandas.DataFrame.sort_values
.. _`sparse`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sparse.html#pandas-dataframe-sparse
.. _`squeeze`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.squeeze.html#pandas.DataFrame.squeeze
.. _`stack`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.stack.html#pandas.DataFrame.stack
.. _`std`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.std.html#pandas.DataFrame.std
.. _`style`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.style.html#pandas.DataFrame.style
.. _`sub`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sub.html#pandas.DataFrame.sub
.. _`subtract`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.subtract.html#pandas.DataFrame.subtract
.. _`sum`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sum.html#pandas.DataFrame.sum
.. _`swapaxes`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.swapaxes.html#pandas.DataFrame.swapaxes
.. _`swaplevel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.swaplevel.html#pandas.DataFrame.swaplevel
.. _`tail`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html#pandas.DataFrame.tail
.. _`take`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.take.html#pandas.DataFrame.take
.. _`to_clipboard`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_clipboard.html#pandas.DataFrame.to_clipboard
.. _`to_csv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html#pandas.DataFrame.to_csv
.. _`to_dict`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_dict.html#pandas.DataFrame.to_dict
.. _`to_excel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html#pandas.DataFrame.to_excel
.. _`to_feather`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_feather.html#pandas.DataFrame.to_feather
.. _`to_gbq`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_gbq.html#pandas.DataFrame.to_gbq
.. _`to_hdf`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_hdf.html#pandas.DataFrame.to_hdf
.. _`to_html`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_html.html#pandas.DataFrame.to_html
.. _`to_json`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json
.. _`to_xml`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xml.html#pandas.DataFrame.to_xml
.. _`to_latex`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_latex.html#pandas.DataFrame.to_latex
.. _`to_orc`: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_orc.html#pandas.DataFrame.to_orc
.. _`to_parquet`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet
.. _`to_period`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_period.html#pandas.DataFrame.to_period
.. _`to_pickle`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_pickle.html#pandas.DataFrame.to_pickle
.. _`to_records`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_records.html#pandas.DataFrame.to_records
.. _`to_sql`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html#pandas.DataFrame.to_sql
.. _`to_stata`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_stata.html#pandas.DataFrame.to_stata
.. _`to_string`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_string.html#pandas.DataFrame.to_string
.. _`to_timestamp`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_timestamp.html#pandas.DataFrame.to_timestamp
.. _`to_xarray`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xarray.html#pandas.DataFrame.to_xarray
.. _`transform`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transform.html#pandas.DataFrame.transform
.. _`transpose`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transpose.html#pandas.DataFrame.transpose
.. _`truediv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.truediv.html#pandas.DataFrame.truediv
.. _`truncate`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.truncate.html#pandas.DataFrame.truncate
.. _`tz_convert`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tz_convert.html#pandas.DataFrame.tz_convert
.. _`tz_localize`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tz_localize.html#pandas.DataFrame.tz_localize
.. _`unstack`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.unstack.html#pandas.DataFrame.unstack
.. _`update`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.update.html#pandas.DataFrame.update
.. _`value_counts`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.value_counts.html#pandas.DataFrame.value_counts
.. _`values`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.values.html#pandas.DataFrame.values
.. _`var`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.var.html#pandas.DataFrame.var
.. _`where`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.where.html#pandas.DataFrame.where
.. _`xs`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.xs.html#pandas.DataFrame.xs
================================================
FILE: docs/supported_apis/defaulting_to_pandas.rst
================================================
Defaulting to pandas
====================
Currently Modin does not support distributed execution for all methods from pandas API.
The remaining unimplemented methods are being executed in a mode called "default to pandas".
This allows users to continue using Modin even though their workloads contain functions not yet implemented in Modin.
Here is a diagram of how we convert to pandas and perform the operation:
.. image:: /img/convert_to_pandas.png
:align: center
We first convert to a pandas DataFrame, then perform the operation. **There is a
performance penalty for going from a partitioned Modin DataFrame to pandas because of
the communication cost and single-threaded nature of pandas.** Once the pandas operation
has completed, we convert the DataFrame back into a partitioned Modin DataFrame. This
way, operations performed after something defaults to pandas will be optimized with
Modin.
The exact methods we have implemented are listed in the respective subsections:
* :doc:`DataFrame `
* :doc:`Series `
* :doc:`utilities `
* :doc:`I/O `
We have taken a community-driven approach to implementing new methods. We did a `study
on pandas usage`_ to learn what the most-used APIs are. Modin currently supports **93%**
of the pandas API based on our study of pandas usage, and we are actively expanding the
API.
**To request implementation, file an issue at https://github.com/modin-project/modin/issues
or send an email to feature_requests@modin.org.**
.. _`study on pandas usage`: https://github.com/modin-project/study_kaggle_usage
================================================
FILE: docs/supported_apis/index.rst
================================================
Supported APIs
==============
For your convenience, we have compiled a list of currently implemented APIs and methods
available in Modin. This documentation is updated as new methods and APIs are merged
into the main branch, and not necessarily correct as of the most recent release.
To view the docs for the most recent release, check that you're viewing the
`stable version`_ of the docs.
In order to install the latest version of Modin, follow the directions found on the
:doc:`installation page `.
Questions on implementation details
-----------------------------------
If you have a question about the implementation details or would like more information
about an API or method in Modin, please contact the Modin `developer mailing list`_.
.. toctree::
:titlesonly:
:hidden:
defaulting_to_pandas
dataframe_supported
series_supported
utilities_supported
io_supported
older_pandas_compat
.. meta::
:description lang=en:
Compilation of implemented pandas APIs in Modin.
.. _developer mailing list: https://groups.google.com/forum/#!forum/modin-dev
.. _stable version: https://modin.readthedocs.io/en/stable/supported_apis/index.html
================================================
FILE: docs/supported_apis/io_supported.rst
================================================
``pd.read_`` and I/O APIs
=================================
A number of IO methods default to pandas. We have parallelized ``read_csv``,
``read_parquet`` and some more (see table), though many of the remaining methods
can be relatively easily parallelized. Some of the operations default to the
pandas implementation, meaning it will read in serially as a single, non-distributed
DataFrame and distribute it. Performance will be affected by this.
The following table is structured as follows: The first column contains the method name.
The second column is a flag for whether or not there is an implementation in Modin for
the method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands
for partial (meaning some parameters may not be supported yet), and ``D`` stands for
default to pandas.
.. note::
Support for fully asynchronous reading has been added for the following functions:
``read_csv``, ``read_fwf``, ``read_table``, ``read_custom_text``.
This mode is disabled by default, one can enable it using ``MODIN_ASYNC_READ_MODE=True``
environment variable. Some parameter combinations are not supported and the function
will be executed in synchronous mode.
+-------------------+---------------------------------+--------------------------------------------------------+
| IO method | Modin Implementation? (Y/N/P/D) | Notes for Current implementation |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_csv`_ | Y | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_fwf`_ | Y | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_table`_ | Y | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_parquet`_ | P | Parameters besides ``filters`` and ``storage_options`` |
| | | passed via ``**kwargs`` are not supported. |
| | | ``use_nullable_dtypes`` == True is not supported. |
| | | |
| | | Experimental implementation: read_parquet_glob |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_json`_ | P | Implemented for ``lines=True`` |
| | | Experimental implementation: read_json_glob |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_xml` | D | Experimental implementation: read_xml_glob |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_html`_ | D | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_clipboard`_ | D | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_excel`_ | D | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_hdf`_ | D | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_feather`_ | Y | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_stata`_ | D | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_sas`_ | D | |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_pickle`_ | D | Experimental implementation: |
| | | read_pickle_glob |
+-------------------+---------------------------------+--------------------------------------------------------+
| `read_sql`_ | Y | |
+-------------------+---------------------------------+--------------------------------------------------------+
.. _`read_csv`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv
.. _`read_fwf`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html#pandas.read_fwf
.. _`read_table`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html#pandas.read_table
.. _`read_parquet`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_parquet.html#pandas.read_parquet
.. _`read_json`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html#pandas.read_json
.. _`read_html`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html#pandas.read_html
.. _`read_clipboard`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_clipboard.html#pandas.read_clipboard
.. _`read_excel`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html#pandas.read_excel
.. _`read_hdf`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_hdf.html#pandas.read_hdf
.. _`read_feather`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_feather.html#pandas.read_feather
.. _`read_stata`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_stata.html#pandas.read_stata
.. _`read_sas`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sas.html#pandas.read_sas
.. _`read_pickle`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_pickle.html#pandas.read_pickle
.. _`read_sql`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql.html#pandas.read_sql
================================================
FILE: docs/supported_apis/older_pandas_compat.rst
================================================
===================================
Pandas backwards compatibility mode
===================================
Modin verions 0.16 and 0.17, but no later minor versions, had limited support
for running with legacy pandas versions. The latest version of Modin no longer
has such support.
Motivation for compatibility mode
---------------------------------
Modin aims to keep compatibility with latest pandas release, hopefully catching up each release
within a few days.
However, due to certain restrictions like need to use Python 3.6 it forces some users to
use older pandas (1.1.x for Python 3.6, specifically), which normally would mean they're
bound to be using ancient Modin as well.
To overcome this, Modin has special "compatibility mode" where some basic functionality
works, but please note that the support is "best possible effort" (e.g. not all older bugs
are worth fixing).
Known issues with pandas 1.1.x
------------------------------
* ``pd.append()`` does not preserve the order of columns in older pandas while Modin does
* ``.astype()`` produces different error type on incompatible dtypes
* ``read_csv()`` does not support reading from ZIP file *with compression* in parallel mode
* ``read_*`` do not support ``storage_option`` named argument
* ``to_csv()`` does not support binary mode for output file
* ``read_excel()`` does not support ``.xlsx`` files
* ``read_fwf()`` has a bug with list of skiprows and non-None nrows: `pandas-dev#10261`_
* ``.agg(int-value)`` produces TypeError in older pandas but Modin raises AssertionError
* ``Series.reset_index(drop=True)`` does not ignore ``name`` in older pandas while Modin ignores it
* ``.sort_index(ascending=None)`` does not raise ValueError in older pandas while Modin raises it
Please keep in mind that there are probably more issues which are not yet uncovered!
.. _`pandas-dev#10261`: https://github.com/pandas-dev/pandas/issues/10261
================================================
FILE: docs/supported_apis/series_supported.rst
================================================
``pd.Series`` supported APIs
============================
The following table lists both implemented and not implemented methods. If you have need
of an operation that is listed as not implemented, feel free to open an issue on the
`GitHub repository`_, or give a thumbs up to already created issues. Contributions are
also welcome!
The following table is structured as follows: The first column contains the method name.
The second column is a flag for whether or not there is an implementation in Modin for
the method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands
for partial (meaning some parameters may not be supported yet), and ``D`` stands for
default to pandas. To learn more about the implementations that default to pandas, see
the related section on :doc:`Defaulting to pandas `.
+-----------------------------+---------------------------------+----------------------------------------------------+
| Series method | Modin Implementation? (Y/N/P/D) | Notes for Current implementation |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``abs`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``add`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``add_prefix`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``add_suffix`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``agg`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``aggregate`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``align`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``all`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``any`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``apply`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``argmax`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``argmin`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``argsort`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``array`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``asfreq`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``asobject`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``asof`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``astype`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``at`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``at_time`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``autocorr`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``axes`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``base`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``between`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``between_time`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``bfill`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``bool`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``cat`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``clip`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``combine`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``combine_first`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``compare`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``compress`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``copy`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``corr`` | Y | Correlation floating point precision may slightly |
| | | differ from pandas. For now pearson method is |
| | | available only. For other methods defaults to |
| | | pandas. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``count`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``cov`` | Y | Covariance floating point precision may slightly |
| | | differ from pandas. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``cummax`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``cummin`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``cumprod`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``cumsum`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``data`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``describe`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``diff`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``div`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``divide`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``divmod`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``dot`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``drop`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``drop_duplicates`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``droplevel`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``dropna`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``dt`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``dtype`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``dtypes`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``duplicated`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``empty`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``eq`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``equals`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ewm`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``expanding`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``explode`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``factorize`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ffill`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``fillna`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``filter`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``first`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``first_valid_index`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``flags`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``floordiv`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``from_array`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ftype`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ge`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``get`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``get_dtype_counts`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``get_ftype_counts`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``get_value`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``get_values`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``groupby`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``gt`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``hasnans`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``head`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``hist`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``iat`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``idxmax`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``idxmin`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``iloc`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``imag`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``index`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``infer_objects`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``interpolate`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``is_monotonic_decreasing`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``is_monotonic_increasing`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``is_unique`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``isin`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``isna`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``isnull`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``item`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``items`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``itemsize`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``keys`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``kurt`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``kurtosis`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``last`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``last_valid_index`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``le`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``loc`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``lt`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``map`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``mask`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``max`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``mean`` | P | Modin defaults to pandas if given the ``level`` |
| | | param. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``median`` | P | Modin defaults to pandas if given the ``level`` |
| | | param. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``memory_usage`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``min`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``mod`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``mode`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``mul`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``multiply`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``name`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``nbytes`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ndim`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ne`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``nlargest`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``nonzero`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``notna`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``notnull`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``nsmallest`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``nunique`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``pct_change`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``pipe`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``plot`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``pop`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``pow`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``prod`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``product`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ptp`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``put`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``quantile`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``radd`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rank`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ravel`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rdiv`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rdivmod`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``real`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``reindex`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``reindex_like`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rename`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rename_axis`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``reorder_levels`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``repeat`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``replace`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``resample`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``reset_index`` | P | **Ray** and **Dask**: ``D`` when ``names`` or |
| | | ``allow_duplicates`` is non-default |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rfloordiv`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rmod`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rmul`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rolling`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``round`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rpow`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rsub`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``rtruediv`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sample`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``searchsorted`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sem`` | P | Modin defaults to pandas if given the ``level`` |
| | | param. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``set_axis`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``set_value`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``shape`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``shift`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``size`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``skew`` | P | Modin defaults to pandas if given the ``level`` |
| | | param. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sort_index`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sort_values`` | Y | Order of indexes that have the same sort key |
| | | is not guaranteed to be the same across sorts; |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sparse`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``squeeze`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``std`` | P | Modin defaults to pandas if given the ``level`` |
| | | param. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``str`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``strides`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sub`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``subtract`` | Y | See ``add``; |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``sum`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``swapaxes`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``swaplevel`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``tail`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``take`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_clipboard`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_csv`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_dict`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_excel`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_frame`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_hdf`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_json`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_latex`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_list`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_numpy`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_period`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_pickle`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_sql`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_string`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_timestamp`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``to_xarray`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``tolist`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``transform`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``transpose`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``truediv`` | Y | See ``add`` |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``truncate`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``tz_convert`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``tz_localize`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``unique`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``unstack`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``update`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``valid`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``value_counts`` | Y | The indices order of resulting object may differ |
| | | from pandas. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``values`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``var`` | P | Modin defaults to pandas if given the ``level`` |
| | | param. |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``view`` | D | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``where`` | Y | |
+-----------------------------+---------------------------------+----------------------------------------------------+
.. _`GitHub repository`: https://github.com/modin-project/modin/issues
================================================
FILE: docs/supported_apis/utilities_supported.rst
================================================
pandas Utilities Supported
==========================
If you run ``import modin.pandas as pd``, the following operations are available from
``pd.``, e.g. ``pd.concat``. If you do not see an operation that pandas enables and
would like to request it, feel free to `open an issue`_. Make sure you tell us your
primary use-case so we can make it happen faster!
The following table is structured as follows: The first column contains the method name.
The second column is a flag for whether or not there is an implementation in Modin for
the method in the left column. ``Y`` stands for yes, ``N`` stands for no, ``P`` stands
for partial (meaning some parameters may not be supported yet), and ``D`` stands for
default to pandas.
+---------------------------+---------------------------------+----------------------------------------------------+
| Utility method | Modin Implementation? (Y/N/P/D) | Notes for Current implementation |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.concat`_ | Y | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.eval`_ | Y | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.unique`_ | Y | |
+---------------------------+---------------------------------+----------------------------------------------------+
| ``pd.value_counts`` | Y | The indices order of resulting object may differ |
| | | from pandas. |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.cut`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.to_numeric`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.factorize`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.from_dummies`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.qcut`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| ``pd.match`` | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.to_datetime`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.get_dummies`_ | Y | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.date_range`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.bdate_range`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| `pd.to_timedelta`_ | D | |
+---------------------------+---------------------------------+----------------------------------------------------+
| ``pd.options`` | Y | |
+---------------------------+---------------------------------+----------------------------------------------------+
Other objects & structures
--------------------------
This list is a list of objects not currently distributed by Modin. All of these objects
are compatible with the distributed components of Modin. If you are interested in
contributing a distributed version of any of these objects, feel free to open a
`pull request`_.
* Panel
* Index
* MultiIndex
* CategoricalIndex
* DatetimeIndex
* Timedelta
* Timestamp
* NaT
* PeriodIndex
* Categorical
* Interval
* UInt8Dtype
* UInt16Dtype
* UInt32Dtype
* UInt64Dtype
* SparseDtype
* Int8Dtype
* Int16Dtype
* Int32Dtype
* Int64Dtype
* CategoricalDtype
* DatetimeTZDtype
* IntervalDtype
* PeriodDtype
* RangeIndex
* TimedeltaIndex
* IntervalIndex
* IndexSlice
* TimeGrouper
* Grouper
* array
* Period
* DateOffset
* ExcelWriter
* SparseArray
.. _open an issue: https://github.com/modin-project/modin/issues
.. _pull request: https://github.com/modin-project/modin/pulls
.. _`pd.concat`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html#pandas.concat
.. _`pd.eval`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.eval.html#pandas.eval
.. _`pd.unique`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.unique.html#pandas.unique
.. _`pd.cut`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html#pandas.cut
.. _`pd.to_numeric`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_numeric.html#pandas.to_numeric
.. _`pd.factorize`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.factorize.html#pandas.factorize
.. _`pd.from_dummies`: https://pandas.pydata.org/docs/reference/api/pandas.from_dummies.html#pandas-from-dummies
.. _`pd.qcut`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html#pandas.qcut
.. _`pd.to_datetime`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html#pandas.to_datetime
.. _`pd.get_dummies`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html#pandas.get_dummies
.. _`pd.date_range`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.date_range.html#pandas.date_range
.. _`pd.bdate_range`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.bdate_range.html#pandas.bdate_range
.. _`pd.to_timedelta`: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_timedelta.html#pandas.to_timedelta
================================================
FILE: docs/usage_guide/advanced_usage/batch.rst
================================================
Batch Pipline API Usage Guide
=============================
Modin provides an experimental batching feature that pipelines row-parallel queries. This feature
is currently only supported for the ``PandasOnRay`` engine. Please note that this feature is experimental
and behavior or interfaces could be changed.
Usage examples
--------------
In examples below we build and run some pipelines. It is important to note that the queries passed to
the pipeline operate on Modin DataFrame partitions, which are backed by ``pandas``. When using ``pandas``-
module level functions, please make sure to import and use ``pandas`` rather than ``modin.pandas``.
Simple Batch Pipelining
^^^^^^^^^^^^^^^^^^^^^^^
This example walks through a simple batch pipeline in order to familiarize the user with the API.
.. code-block:: python
from modin.experimental.batch import PandasQueryPipeline
import modin.pandas as pd
import numpy as np
df = pd.DataFrame(
np.random.randint(0, 100, (100, 100)),
columns=[f"col {i}" for i in range(1, 101)],
) # Build the dataframe we will pipeline.
pipeline = PandasQueryPipeline(df) # Build the pipeline.
pipeline.add_query(lambda df: df + 1, is_output=True) # Add the first query and specify that
# it is an output query.
pipeline.add_query(
lambda df: df.rename(columns={f"col {i}":f"col {i-1}" for i in range(1, 101)})
) # Add a second query.
pipeline.add_query(
lambda df: df.drop(columns=['col 99']),
is_output=True,
) # Add a third query and specify that it is an output query.
new_df = pd.DataFrame(
np.ones((100, 100)),
columns=[f"col {i}" for i in range(1, 101)],
) # Build a second dataframe that we will pipeline now instead.
pipeline.update_df(new_df) # Update the dataframe that we will pipeline to be `new_df`
# instead of `df`.
result_dfs = pipeline.compute_batch() # Begin batch processing.
# Print pipeline results
print(f"Result of Query 1:\n{result_dfs[0]}")
print(f"Result of Query 2:\n{result_dfs[1]}")
# Output IDs can also be specified
pipeline = PandasQueryPipeline(df) # Build the pipeline.
pipeline.add_query(
lambda df: df + 1,
is_output=True,
output_id=1,
) # Add the first query, specify that it is an output query, as well as specify an output id.
pipeline.add_query(
lambda df: df.rename(columns={f"col {i}":f"col {i-1}" for i in range(1, 101)})
) # Add a second query.
pipeline.add_query(
lambda df: df.drop(columns=['col 99']),
is_output=True,
output_id=2,
) # Add a third query, specify that it is an output query, and specify an output_id.
result_dfs = pipeline.compute_batch() # Begin batch processing.
# Print pipeline results - should be a dictionary mapping Output IDs to resulting dataframes:
print(f"Mapping of Output ID to dataframe:\n{result_dfs}")
# Print results
for query_id, res_df in result_dfs.items():
print(f"Query {query_id} resulted in\n{res_df}")
Batch Pipelining with Postprocessing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A postprocessing function can also be provided when calling ``pipeline.compute_batch``. The example
below runs a similar pipeline as above, but the postprocessing function writes the output dfs to
a parquet file.
.. code-block:: python
from modin.experimental.batch import PandasQueryPipeline
import modin.pandas as pd
import numpy as np
import os
import shutil
df = pd.DataFrame(
np.random.randint(0, 100, (100, 100)),
columns=[f"col {i}" for i in range(1, 101)],
) # Build the dataframe we will pipeline.
pipeline = PandasQueryPipeline(df) # Build the pipeline.
pipeline.add_query(
lambda df: df + 1,
is_output=True,
output_id=1,
) # Add the first query, specify that it is an output query, as well as specify an output id.
pipeline.add_query(
lambda df: df.rename(columns={f"col {i}":f"col {i-1}" for i in range(1, 101)})
) # Add a second query.
pipeline.add_query(
lambda df: df.drop(columns=['col 99']),
is_output=True,
output_id=2,
) # Add a third query, specify that it is an output query, and specify an output_id.
def postprocessing_func(df, output_id, partition_id):
filepath = f"query_{output_id}/"
os.makedirs(filepath, exist_ok=True)
filepath += f"part-{partition_id:04d}.parquet"
df.to_parquet(filepath)
return df
result_dfs = pipeline.compute_batch(
postprocessor=postprocessing_func,
pass_partition_id=True,
pass_output_id=True,
) # Begin computation, pass in a postprocessing function, and specify that partition ID and
# output ID should be passed to that postprocessing function.
print(os.system("ls query_1/")) # Should show `NPartitions.get()` parquet files - which
# correspond to partitions of the output of query 1.
print(os.system("ls query_2/")) # Should show `NPartitions.get()` parquet files - which
# correspond to partitions of the output of query 2.
for query_id, res_df in result_dfs.items():
written_df = pd.read_parquet(f"query_{query_id}/")
shutil.rmtree(f"query_{query_id}/") # Clean up
print(f"Written and Computed DF are " +
f"{'equal' if res_df.equals(written_df) else 'not equal'} for query {query_id}")
Batch Pipelining with Fan Out
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If the input dataframe to a query is small (consisting of only one partition), it is possible to
induce additional parallelism using the ``fan_out`` argument. The ``fan_out`` argument replicates
the input partition, applies the query to each replica, and then coalesces all of the replicas back
to one partition using the ``reduce_fn`` that must also be specified when ``fan_out`` is ``True``.
It is possible to control the parallelism via the ``num_partitions`` parameter passed to the
constructor of the ``PandasQueryPipeline``. This parameter designates the desired number of partitions,
and defaults to ``NPartitions.get()`` when not specified. During fan out, the input partition is replicated
``num_partitions`` times. In the previous examples, ``num_partitions`` was not specified, and so defaulted
to ``NPartitions.get()``.
The example below demonstrates the usage of ``fan_out`` and ``num_partitions``. We first demonstrate
an example of a function that would benefit from this computation pattern:
.. code-block:: python
import glob
from PIL import Image
import torchvision.transforms as T
import torchvision
transforms = T.Compose([T.ToTensor()])
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
COCO_INSTANCE_CATEGORY_NAMES = [
'__background__', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus',
'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'N/A', 'stop sign',
'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
'elephant', 'bear', 'zebra', 'giraffe', 'N/A', 'backpack', 'umbrella', 'N/A', 'N/A',
'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball',
'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket',
'bottle', 'N/A', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl',
'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza',
'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'N/A', 'dining table',
'N/A', 'N/A', 'toilet', 'N/A', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',
'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'N/A', 'book',
'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush'
]
def contains_cat(image, model):
image = transforms(image)
labels = [COCO_INSTANCE_CATEGORY_NAMES[i] for i in model([image])[0]['labels']]
return 'cat' in labels
def serial_query(df):
"""
This function takes as input a dataframe with a single row corresponding to a folder
containing images to parse. Each image in the folder is passed through a neural network
that detects whether it contains a cat, in serial, and a new column is computed for the
dataframe that counts the number of images containing cats.
Parameters
----------
df : a dataframe
The dataframe to process
Returns
-------
The same dataframe as before, with an additional column containing the count of images
containing cats.
"""
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
img_folder = df['images'][0]
images = sorted(glob.glob(f"{img_folder}/*.jpg"))
cats = 0
for img in images:
cats = cats + 1 if contains_cat(Image.open(img), model) else cats
df['cat_count'] = cats
return df
To download the image files to test out this code, run the following bash script, which downloads
the images from the fast-ai-coco S3 bucket to a folder called ``images`` in your current working
directory:
.. code-block:: shell
aws s3 cp s3://fast-ai-coco/coco_tiny.tgz . --no-sign-request; tar -xf coco_tiny.tgz; mkdir \
images; mv coco_tiny/train/* images/; rm -rf coco_tiny; rm -rf coco_tiny.tgz
We can pipeline that code like so:
.. code-block:: python
import modin.pandas as pd
from modin.experimental.batch import PandasQueryPipeline
from time import time
df = pd.DataFrame([['images']], columns=['images'])
pipeline = PandasQueryPipeline(df)
pipeline.add_query(serial_query, is_output=True)
serial_start = time()
df_with_cat_count = pipeline.compute_batch()[0]
serial_end = time()
print(f"Result of pipeline:\n{df_with_cat_count}")
We can induce `8x` parallelism into the pipeline above by combining the ``fan_out`` and ``num_partitions`` parameters like so:
.. code-block:: python
import modin.pandas as pd
from modin.experimental.batch import PandasQueryPipeline
import shutil
from time import time
df = pd.DataFrame([['images']], columns=['images'])
desired_num_partitions = 8
def parallel_query(df, partition_id):
"""
This function takes as input a dataframe with a single row corresponding to a folder
containing images to parse. It parses `total_images/desired_num_partitions` images every
time it is called. A new column is computed for the dataframe that counts the number of
images containing cats.
Parameters
----------
df : a dataframe
The dataframe to process
partition_id : int
The partition id of the dataframe that this function runs on.
Returns
-------
The same dataframe as before, with an additional column containing the count of images
containing cats.
"""
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
img_folder = df['images'][0]
images = sorted(glob.glob(f"{img_folder}/*.jpg"))
total_images = len(images)
cats = 0
start_index = partition_id * (total_images // desired_num_partitions)
if partition_id == desired_num_partitions - 1: # Last partition must parse to end of list
images = images[start_index:]
else:
end_index = (partition_id + 1) * (total_images // desired_num_partitions)
images = images[start_index:end_index]
for img in images:
cats = cats + 1 if contains_cat(Image.open(img), model) else cats
df['cat_count'] = cats
return df
def reduce_fn(dfs):
"""
Coalesce the results of fanning out the `parallel_query` query.
Parameters
----------
dfs : a list of dataframes
The resulting dataframes from fanning out `parallel_query`
Returns
-------
A new dataframe whose `cat_count` column is the sum of the `cat_count` column of all
dataframes in `dfs`
"""
df = dfs[0]
cat_count = df['cat_count'][0]
for dataframe in dfs[1:]:
cat_count += dataframe['cat_count'][0]
df['cat_count'] = cat_count
return df
pipeline = PandasQueryPipeline(df, desired_num_partitions)
pipeline.add_query(
parallel_query,
fan_out=True,
reduce_fn=reduce_fn,
is_output=True,
pass_partition_id=True
)
parallel_start = time()
df_with_cat_count = pipeline.compute_batch()[0]
parallel_end = time()
print(f"Result of pipeline:\n{df_with_cat_count}")
print(f"Total Time in Serial: {serial_end - serial_start}")
print(f"Total time with induced parallelism: {parallel_end - parallel_start}")
shutil.rmtree("images/") # Clean up
Batch Pipelining with Dynamic Repartitioning
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Similarly, it is also possible to hint to the Pipeline API to repartition after a node completes
computation. This is currently only supported if the input dataframe consists of only one partition.
The number of partitions after repartitioning is controlled by the ``num_partitions`` parameter
passed to the constructor of the ``PandasQueryPipeline``.
The following example demonstrates how to use the ``repartition_after`` parameter.
.. code-block:: python
import modin.pandas as pd
from modin.experimental.batch import PandasQueryPipeline
import numpy as np
small_df = pd.DataFrame([[1, 2, 3]]) # Create a small dataframe
def increase_dataframe_size(df):
import pandas
new_df = pandas.concat([df] * 1000)
new_df = new_df.reset_index(drop=True) # Get a new range index that isn't duplicated
return new_df
desired_num_partitions = 24 # We will repartition to 24 partitions
def add_partition_id_to_df(df, partition_id):
import pandas
new_col = pandas.Series([partition_id]*len(df), name="partition_id", index=df.index)
return pandas.concat([df, new_col], axis=1)
pipeline = PandasQueryPipeline(small_df, desired_num_partitions)
pipeline.add_query(increase_dataframe_size, repartition_after=True)
pipeline.add_query(add_partition_id_to_df, pass_partition_id=True, is_output=True)
result_df = pipeline.compute_batch()[0]
print(f"Number of partitions passed to second query: " +
f"{len(np.unique(result_df['partition_id'].values))}")
print(f"Result of pipeline:\n{result_df}")
================================================
FILE: docs/usage_guide/advanced_usage/index.rst
================================================
Advanced Usage
==============
.. toctree::
:titlesonly:
:hidden:
/flow/modin/distributed/dataframe/pandas
spreadsheets_api
progress_bar
modin_xgboost
modin_logging
modin_metrics
batch
modin_engines
.. meta::
:description lang=en:
Description of Modin's advanced features.
Modin aims to not only optimize pandas, but also provide a comprehensive,
integrated toolkit for data scientists. We are actively developing data science tools
such as DataFrame spreadsheet integration, DataFrame algebra, progress bars, SQL queries
on DataFrames, and more. Join us on `Slack`_ for the latest updates!
Modin engines
-------------
Modin supports a series of execution engines such as Ray_, Dask_, `MPI through unidist`_,
each of which might be a more beneficial choice for a specific scenario. When doing the first operation
with Modin it automatically initializes one of the engines to further perform distributed/parallel computation.
If you are familiar with a concrete execution engine, it is possible to initialize the engine on your own and
Modin will automatically attach to it. Refer to :doc:`Modin engines ` page
for more details.
Additional APIs
---------------
Modin also supports these additional APIs on top of pandas to improve user experience.
- :py:meth:`~modin.pandas.DataFrame.modin.to_pandas` -- convert a Modin DataFrame/Series to a pandas DataFrame/Series.
- :py:meth:`~modin.pandas.DataFrame.get_backend` -- Get the ``Backend`` :doc:`configuration variable ` of this ``DataFrame``.
- :py:meth:`~modin.pandas.DataFrame.move_to` -- Move data and execution for this ``DataFrame`` to the given ``Backend`` :doc:`configuration variable `. This method is an alias for ``DataFrame.set_backend``.
- :py:meth:`~modin.pandas.DataFrame.set_backend` -- Move data and execution for this ``DataFrame`` to the given ``Backend`` :doc:`configuration variable `. This method is an alias for ``DatFrame.move_to``.
- :py:func:`~modin.pandas.io.from_pandas` -- convert a pandas DataFrame to a Modin DataFrame.
- :py:meth:`~modin.pandas.DataFrame.modin.to_ray` -- convert a Modin DataFrame/Series to a Ray Dataset.
- :py:func:`~modin.pandas.io.from_ray` -- convert a Ray Dataset to a Modin DataFrame.
- :py:meth:`~modin.pandas.DataFrame.modin.to_dask` -- convert a Modin DataFrame/Series to a Ray Dataset.
- :py:func:`~modin.pandas.io.from_dask` -- convert a Modin DataFrame/Series to a Dask DataFrame/Series.
- :py:func:`~modin.pandas.io.from_map` -- create a Modin DataFrame from map function applied to an iterable object.
- :py:func:`~modin.pandas.io.from_arrow` -- convert an Arrow Table to a Modin DataFrame.
- :py:func:`~modin.experimental.pandas.read_csv_glob` -- read multiple files in a directory.
- :py:func:`~modin.experimental.pandas.read_sql` -- add optional parameters for the database connection.
- :py:func:`~modin.experimental.pandas.read_custom_text` -- read custom text data from file.
- :py:func:`~modin.experimental.pandas.read_pickle_glob` -- read multiple pickle files in a directory.
- :py:func:`~modin.experimental.pandas.read_parquet_glob` -- read multiple parquet files in a directory.
- :py:func:`~modin.experimental.pandas.read_json_glob` -- read multiple json files in a directory.
- :py:func:`~modin.experimental.pandas.read_xml_glob` -- read multiple xml files in a directory.
- :py:meth:`~modin.pandas.DataFrame.modin.to_pickle_glob` -- write to multiple pickle files in a directory.
- :py:meth:`~modin.pandas.DataFrame.modin.to_parquet_glob` -- write to multiple parquet files in a directory.
- :py:meth:`~modin.pandas.DataFrame.modin.to_json_glob` -- write to multiple json files in a directory.
- :py:meth:`~modin.pandas.DataFrame.modin.to_xml_glob` -- write to multiple xml files in a directory.
DataFrame partitioning API
--------------------------
Modin DataFrame provides an API to directly access partitions: you can extract physical partitions from
a :py:class:`~modin.pandas.dataframe.DataFrame`, modify their structure by reshuffling or applying some
functions, and create a DataFrame from those modified partitions. Visit
:doc:`pandas partitioning API ` documentation to learn more.
Modin Spreadsheet API
---------------------
The Spreadsheet API for Modin allows you to render the dataframe as a spreadsheet to easily explore
your data and perform operations on a graphical user interface. The API also includes features for recording
the changes made to the dataframe and exporting them as reproducible code. Built on top of Modin and SlickGrid,
the spreadsheet interface is able to provide interactive response times even at a scale of billions of rows.
See our `Modin Spreadsheet API documentation`_ for more details.
.. figure:: /img/modin_spreadsheet_mini_demo.gif
:align: center
:width: 650px
:height: 350px
Progress Bar
------------
Visual progress bar for Dataframe operations such as groupby and fillna, as well as for file reading operations such as
read_csv. Built using the `tqdm`_ library and Ray execution engine. See `Progress Bar documentation`_ for more details.
.. figure:: /img/progress_bar_example.png
:align: center
Dataframe Algebra
-----------------
A minimal set of operators that can be composed to express any dataframe query for use in query planning and optimization.
See our `paper`_ for more information, and full documentation is coming soon!
Distributed XGBoost on Modin
----------------------------
Modin provides an implementation of `distributed XGBoost`_ machine learning algorithm on Modin DataFrames. See our
:doc:`Distributed XGBoost on Modin documentation ` for details about installation and usage, as well as
:doc:`Modin XGBoost architecture documentation ` for information about implementation and
internal execution flow.
Logging with Modin
------------------
Modin logging offers users greater insight into their queries by logging internal Modin API calls, partition metadata,
and system memory. Logging is disabled by default, but when it is enabled, log files are written to a local `.modin` directory
at the same directory level as the notebook/script used to run Modin. See our :doc:`Logging with Modin documentation `
for usage information.
Batch Pipeline API
------------------
Modin provides an experimental batched API that pipelines row parallel queries. See our :doc:`Batch Pipline API Usage Guide `
for a walkthrough on how to use this feature, as well as :doc:`Batch Pipeline API documentation `
for more information about the API.
Fuzzydata Testing
-----------------
An experimental GitHub Action on pull request has been added to Modin, which automatically runs the Modin codebase against
`fuzzydata`, a random dataframe workflow generator. The resulting workflow that was used to test Modin codebase can be
downloaded as an artifact from the GitHub Actions tab for further inspection. See `fuzzydata`_ for more details.
.. _`Modin Spreadsheet API documentation`: spreadsheets_api.html
.. _`Progress Bar documentation`: progress_bar.html
.. _`Paper`: https://arxiv.org/pdf/2001.00888.pdf
.. _`Slack`: https://modin.org/slack.html
.. _`tqdm`: https://github.com/tqdm/tqdm
.. _`distributed XGBoost`: https://medium.com/intel-analytics-software/distributed-xgboost-with-modin-on-ray-fc17edef7720
.. _`fuzzydata`: https://github.com/suhailrehman/fuzzydata
.. _Ray: https://github.com/ray-project/ray
.. _Dask: https://github.com/dask/distributed
.. _`MPI through unidist`: https://github.com/modin-project/unidist
================================================
FILE: docs/usage_guide/advanced_usage/modin_engines.rst
================================================
Modin engines
=============
As a rule, you don't have to worry about initialization of an execution engine as
Modin itself automatically initializes one when performing the first operation.
Also, Modin has a broad range of :doc:`configuration settings `, which
you can use to configure an execution engine. If there is a reason to initialize an execution engine
on your own and you are sure what to do, Modin will automatically attach to whichever engine is available.
Below, you can find some examples on how to initialize a specific execution engine on your own.
Ray
---
You can initialize Ray engine with a specific number of CPUs (worker processes) to perform computation.
.. code-block:: python
import ray
import modin.config as modin_cfg
ray.init(num_cpus=)
modin_cfg.Engine.put("ray") # Modin will use Ray engine
modin_cfg.CpuCount.put()
To get more details on all possible parameters for initialization refer to `Ray documentation`_.
Dask
----
You can initialize Dask engine with a specific number of worker processes and threads per worker to perform computation.
.. code-block:: python
from distributed import Client
import modin.config as modin_cfg
client = Client(n_workers=, threads_per_worker=)
modin_cfg.Engine.put("dask") # # Modin will use Dask engine
modin_cfg.CpuCount.put()
To get more details on all possible parameters for initialization refer to `Dask Distributed documentation`_.
MPI through unidist
-------------------
You can initialize MPI through unidist engine with a specific number of CPUs (worker processes) to perform computation.
.. code-block:: python
import unidist
import unidist.config as unidist_cfg
import modin.config as modin_cfg
unidist_cfg.Backend.put("mpi")
unidist_cfg.CpuCount.put()
unidist.init()
modin_cfg.Engine.put("unidist") # # Modin will use MPI through unidist engine
modin_cfg.CpuCount.put()
To get more details on all possible parameters for initialization refer to `unidist documentation`_.
.. _`Ray documentation`: https://docs.ray.io/en/latest
.. _Dask Distributed documentation: https://distributed.dask.org/en/latest
.. _`unidist documentation`: https://unidist.readthedocs.io/en/latest
================================================
FILE: docs/usage_guide/advanced_usage/modin_logging.rst
================================================
Modin Logging
=============
Modin logging offers users greater insight into their queries by logging internal Modin API calls, partition metadata,
and profiling system memory. When Modin logging is enabled (default disabled), log files are written to a local ``.modin`` directory at the same
directory level as the notebook/script used to run Modin.
The logs generated by Modin Logging will be written to a ``.modin/logs/job_`` directory, uniquely named after the job uuid.
The logs that contain the Modin API stack traces are named ``trace.log``. The logs that contain the memory utilization metrics are
named ``memory.log``. By default, if any log file exceeds 10MB (configurable with ``LogFileSize``), that file will be saved and a
separate log file will be created. For instance, if users have 20MB worth of Modin API logs, they can expect to find ``trace.log.1``
and ``trace.log.2`` in the ``.modin/logs/job_`` directory. After ``10 * LogFileSize`` MB or by default 100MB of logs, the logs will
rollover and the original log files beginning with ``trace.log.1`` will be overwritten with the new log lines.
**Developer Warning:** In some cases, running services like JupyterLab in the ``modin/modin`` directory may result in circular dependency issues.
This is due to a naming conflict between the ``modin/logging`` directory and the Python ``logging`` module, which may be used as a default in
such environments. To resolve this, please run Jupyterlab or other similar services from directories other than ``modin/modin``.
Usage examples
--------------
In the example below, we enable logging for internal Modin API calls, partition metadata and memory profiling.
We can set the granularity (in seconds) at which the system memory utilization is logged using ``LogMemoryInterval``.
We can also set the maximum size of the logs (in MBs) using ``LogFileSize``.
.. code-block:: python
import modin.pandas as pd
from modin.config import LogMode, LogMemoryInterval, LogFileSize
LogMode.enable()
LogMemoryInterval.put(2) # Defaults to 5 seconds, new interval is 2 seconds
LogFileSize.put(5) # Defaults to 10 MB per log file, new size is 5 MB
# User code goes here
Disable Modin logging like so:
.. code-block:: python
import modin.pandas as pd
from modin.config import LogMode
LogMode.disable()
# User code goes here
In Modin the lower-level functionality is logged in debug level, and higher level functionality in info level.
By default when logging is enabled in Modin, both high level and low level functionality are logged.
The below example script could be used to switch between logging all functions vs only logging higher level functions.
Setting logger level to ``logging.INFO`` logs only higher level functions.
.. code-block:: python
import modin.pandas as pd
from modin.logging.config import get_logger
from modin.config import LogMode
import logging
LogMode.enable()
logger = get_logger()
logger.setLevel(logging.INFO) # Replace with logger.setLevel(logging.DEBUG) for lower level logs
df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
df = pd.concat([df, df])
Debugging from user defined functions:
.. warning::
When attempting to use Modin logging in user defined functions that execute in workers for logging lower-level operators
as in example below, multiple log directories ``.modin/logs/job_**`` would be created for each worker executing the UDF.
.. code-block:: python
import modin.pandas as pd
def udf(x):
from modin.config import LogMode
LogMode.enable()
return x + 1
modin_df = pd.DataFrame([0, 1, 2, 3])
print(modin_df.map(udf))
So the **recommended** approach would be to use a different logger as in the below snipet
to log from user defined functions that execute on workers.
Below is an an example to log from UDF. For this the logger config has to be specified inside the UDF that would execute on a remote worker.
.. code-block:: python
import logging
import modin.pandas as pd
def udf(x):
logging.basicConfig(filename='modin_udf.log', level=logging.INFO)
logging.info("This log message will be written to modin_udf.log ")
# User code goes here
return x + 1
modin_df = pd.DataFrame([0, 1, 2, 3])
print(modin_df.map(udf))
================================================
FILE: docs/usage_guide/advanced_usage/modin_metrics.rst
================================================
Modin Metrics
=============
Modin allows for third-party systems to register a metrics handler to collect specific API statistics.
Metrics have a name and a value, can be aggregated, discarded, or emitted without impact to the program.
CPU load, memory usage, and disk usage are all typical metrics; but modin currently only emits metrics on API timings which can be used to optimize end-user interactive performance. New metrics may
be added in the future.
It is the responsibility of the handler to process or forward these metrics. The name of the metric will
be in "dot format" and all lowercase, similar to graphite or rrd. The value is an integer or float.
Example metric names include:
* 'modin.core-dataframe.pandasdataframe.copy_index_cache'
* 'modin.core-dataframe.pandasdataframe.transpose'
* 'modin.query-compiler.pandasquerycompiler.transpose'
* 'modin.query-compiler.basequerycompiler.columnarize'
* 'modin.pandas-api.series.__init__'
* 'modin.pandas-api.dataframe._reduce_dimension'
* 'modin.pandas-api.dataframe.sum'
Handlers are functions of the form: `fn(str, int|float)` and can be registered with:
.. code-block:: python
import modin.pandas as pd
from modin.logging.metrics import add_metric_handler
def func(name: str, value: int | float):
print(f"Got metric {name} value {value}")
add_metric_handler(func)
.. warning::
A metric handler should be non-blocking, returning within 100ms, although this is not enforced. It must not throw exceptions or it will
be deregistered. These restrictions are to help guard against the implementation of a metrics collector which would impact
interactice performance significantly. The data from metrics should generally be offloaded to another system for processing
and not involve any blocking network calls.
Metrics are enabled by default. Modin metrics can be disabled like so:
.. code-block:: python
import modin.pandas as pd
from modin.config import MetricsMode
MetricsMode.disable()
================================================
FILE: docs/usage_guide/advanced_usage/modin_xgboost.rst
================================================
Distributed XGBoost on Modin
============================
Modin provides an implementation of `distributed XGBoost`_ machine learning
algorithm on Modin DataFrames. Please note that this feature is experimental and behavior or
interfaces could be changed.
Install XGBoost on Modin
------------------------
Modin comes with all the dependencies except ``xgboost`` package by default.
Currently, distributed XGBoost on Modin is only supported on the Ray execution engine, therefore, see
the :doc:`installation page ` for more information on installing Modin with the Ray engine.
To install ``xgboost`` package you can use ``pip``:
.. code-block:: bash
pip install xgboost
XGBoost Train and Predict
-------------------------
Distributed XGBoost functionality is placed in ``modin.experimental.xgboost`` module.
``modin.experimental.xgboost`` provides a drop-in replacement API for ``train`` and ``Booster.predict`` xgboost functions.
.. automodule:: modin.experimental.xgboost
:noindex:
:members: train
.. autoclass:: modin.experimental.xgboost.Booster
:noindex:
:members: predict
ModinDMatrix
------------
Data is passed to ``modin.experimental.xgboost`` functions via a Modin ``DMatrix`` object.
.. automodule:: modin.experimental.xgboost
:noindex:
:members: DMatrix
Currently, the Modin ``DMatrix`` supports ``modin.pandas.DataFrame`` only as an input.
A Single Node / Cluster setup
-----------------------------
The XGBoost part of Modin uses a Ray resources by similar way as all Modin functions.
To start the Ray runtime on a single node:
.. code-block:: python
import ray
# Look at the Ray documentation with respect to the Ray configuration suited to you most.
ray.init()
If you already had the Ray cluster you can connect to it by next way:
.. code-block:: python
import ray
ray.init(address='auto')
A detailed information about initializing the Ray runtime you can find in `starting ray`_ page.
Usage example
-------------
In example below we train XGBoost model using `the Iris Dataset`_ and get prediction on the same data.
All processing will be in a `single node` mode.
.. code-block:: python
from sklearn import datasets
import ray
# Look at the Ray documentation with respect to the Ray configuration suited to you most.
ray.init() # Start the Ray runtime for single-node
import modin.pandas as pd
import modin.experimental.xgboost as xgb
# Load iris dataset from sklearn
iris = datasets.load_iris()
# Create Modin DataFrames
X = pd.DataFrame(iris.data)
y = pd.DataFrame(iris.target)
# Create DMatrix
dtrain = xgb.DMatrix(X, y)
dtest = xgb.DMatrix(X, y)
# Set training parameters
xgb_params = {
"eta": 0.3,
"max_depth": 3,
"objective": "multi:softprob",
"num_class": 3,
"eval_metric": "mlogloss",
}
steps = 20
# Create dict for evaluation results
evals_result = dict()
# Run training
model = xgb.train(
xgb_params,
dtrain,
steps,
evals=[(dtrain, "train")],
evals_result=evals_result
)
# Print evaluation results
print(f'Evals results:\n{evals_result}')
# Predict results
prediction = model.predict(dtest)
# Print prediction results
print(f'Prediction results:\n{prediction}')
.. _Dataframe: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html
.. _`starting ray`: https://docs.ray.io/en/master/starting-ray.html
.. _`the Iris Dataset`: https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html
.. _`distributed XGBoost`: https://medium.com/intel-analytics-software/distributed-xgboost-with-modin-on-ray-fc17edef7720
================================================
FILE: docs/usage_guide/advanced_usage/progress_bar.rst
================================================
Progress Bar
============
The progress bar allows users to see the estimated progress and completion time of each line they run,
in environments such as a shell or Jupyter notebook.
.. figure:: /img/progress_bar.gif
:align: center
Quickstart
""""""""""
The progress bar uses the `tqdm` library to visualize displays:
.. code-block:: bash
pip install tqdm
Import the progress bar into your notebook by running the following:
.. code-block:: python
from modin.config import ProgressBar
ProgressBar.enable()
================================================
FILE: docs/usage_guide/advanced_usage/spreadsheets_api.rst
================================================
Modin Spreadsheets API
======================
Getting started
---------------
Install Modin-spreadsheet using pip:
.. code-block:: bash
pip install "modin[spreadsheet]"
The following code snippet creates a spreadsheet using the FiveThirtyEight dataset on labor force information by college majors (licensed under CC BY 4.0):
.. code-block:: python
import modin.pandas as pd
import modin.experimental.spreadsheet as mss
df = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/college-majors/all-ages.csv')
spreadsheet = mss.from_dataframe(df)
spreadsheet
.. figure:: /img/modin_spreadsheets_installation.png
:align: center
Basic Manipulations through User Interface
------------------------------------------
The Spreadsheet API allows users to manipulate the DataFrame with simple graphical controls for sorting, filtering, and editing.
Here are the instructions for each operation:
* **Sort**: Click on the column header of the column to sort on.
* **Filter**: Click on the filter button on the column header and apply the desired filter to the column. The filter dropdown changes depending on the type of the column. Multiple filters are automatically combined.
* **Edit Cell**: Double click on a cell and enter the new value.
* **Add Rows**: Click on the “Add Row” button in the toolbar to duplicate the last row in the DataFrame. The duplicated values provide a convenient default and can be edited as necessary.
* **Remove Rows**: Select row(s) and click the “Remove Row” button. Select a single row by clicking on it. Multiple rows can be selected with Cmd+Click (Windows: Ctrl+Click) on the desired rows or with Shift+Click to specify a range of rows.
Some of these operations can also be done through the spreadsheet’s programmatic interface. Sorts and filters can be reset using the toolbar buttons. Edits and adding/removing rows can only be undone manually.
Virtual Rendering
-----------------
The spreadsheet will only render data based on the user’s viewport. This allows for quick rendering
even on very large DataFrames because only a handful of rows are loaded at any given time. As a result, scrolling and viewing your data is smooth and responsive.
Transformation History and Exporting Code
-----------------------------------------
All operations on the spreadsheet are recorded and are easily exported as code for sharing or reproducibility.
This history is automatically displayed in the history cell, which is generated below the spreadsheet whenever the spreadsheet widget is displayed.
The history cell is displayed on default, but this can be turned off. Modin Spreadsheet API provides a few methods for interacting with the history:
* `SpreadsheetWidget.get_history()`: Retrieves the transformation history in the form of reproducible code.
* `SpreadsheetWidget.filter_relevant_history(persist=True)`: Returns the transformation history that contains only code relevant to the final state of the spreadsheet. The `persist` parameter determines whether the internal state and the displayed history is also filtered.
* `SpreadsheetWidget.reset_history()`: Clears the history of transformation.
Customizable Interface
----------------------
The spreadsheet widget provides a number of options that allows the user to change the appearance and the interactivity of the spreadsheet. Options include:
* Row height/Column width
* Preventing edits, sorts, or filters on the whole spreadsheet or on a per-column basis
* Hiding the toolbar and history cell
* Float precision
* Highlighting of cells and rows
* Viewport size
Converting Spreadsheets To and From Dataframes
----------------------------------------------
.. automodule:: modin.experimental.spreadsheet.general
:noindex:
:members: from_dataframe
.. automodule:: modin.experimental.spreadsheet.general
:noindex:
:members: to_dataframe
Further API Documentation
-------------------------
.. automodule:: modin_spreadsheet.grid
:noindex:
:members: SpreadsheetWidget
================================================
FILE: docs/usage_guide/benchmarking.rst
================================================
Benchmarking Modin
==================
Summary
-------
To benchmark a single Modin function, often turning on the
:doc:`configuration variable ` variable
:code:`BenchmarkMode` will suffice.
There is no simple way to benchmark more complex Modin workflows, though
benchmark mode or calling ``modin.utils.execute`` on Modin objects may be useful.
The :doc:`Modin logs ` may help you
identify bottlenecks in your code, and they may also help profile the execution
of each Modin function.
Modin's execution and benchmark mode
------------------------------------
Most of Modin's execution happens asynchronously, i.e. in separate processes that run
independently of the main program flow. Some execution is also lazy, meaning that it
doesn't start immediately once the user calls a Modin function. While Modin provides
the same API as pandas, lazy and asynchronous execution can often make it hard to
tell how much time each Modin function call takes, as well as to compare Modin's
performance to pandas and other similar libraries.
.. note::
All examples in this doc use the system specified at the bottom of this page.
Consider the following ipython script:
.. code-block:: python
import modin.pandas as pd
from modin.config import MinRowPartitionSize
import time
import ray
# Look at the Ray documentation with respect to the Ray configuration suited to you most.
ray.init()
df = pd.DataFrame(list(range(MinRowPartitionSize.get() * 2)))
%time result = df.map(lambda x: time.sleep(0.1) or x)
%time print(result)
Modin takes just 2.68 milliseconds for the ``map``, and 3.78 seconds to print
the result. However, if we run this script in pandas by replacing
:code:`import modin.pandas as pd` with :code:`import pandas as pd`, the ``map``
takes 6.63 seconds, and printing the result takes just 5.53 milliseconds.
Both pandas and Modin start executing the ``map`` as soon as the interpreter
evalutes it. While pandas blocks until the ``map`` has finished, Modin just kicks
off asynchronous functions in remote ray processes. Printing the function result
is fairly fast in pandas and Modin, but before Modin can print the data, it has to
wait until all the remote functions complete.
To time how long Modin takes for a single operation, you should typically use
benchmark mode. Benchmark mode will wait for all asynchronous remote execution
to complete. You can turn on benchmark mode on at any point as follows:
.. code-block:: python
from modin.config import BenchmarkMode
BenchmarkMode.put(True)
Rerunning the script above with benchmark mode on, the Modin ``map`` takes
3.59 seconds, and the ``print`` takes 183 milliseconds. These timings better
reflect where Modin is spending its execution time.
A caveat about benchmark mode
-----------------------------
While benchmark code is often good for measuring the performance of a single
Modin function call, it can underestimate Modin's performance in cases where
Modin's asynchronous execution improves Modin's performance. Consider the
following script with benchmark mode on:
.. code-block:: python
import numpy as np
import time
import ray
from io import BytesIO
import modin.pandas as pd
from modin.config import BenchmarkMode, MinRowPartitionSize
BenchmarkMode.put(True)
start = time.time()
df = pd.DataFrame(list(range(MinRowPartitionSize.get())), columns=['A'])
result1 = df.map(lambda x: time.sleep(0.2) or x + 1)
result2 = df.map(lambda x: time.sleep(0.2) or x + 2)
result1.to_parquet(BytesIO())
result2.to_parquet(BytesIO())
end = time.time()
print(f'map and write to parquet took {end - start} seconds.')
.. code-block::python
The script does two slow ``map`` on a dataframe and then writes each result
to a buffer. The whole script takes 13 seconds with benchmark mode on, but
just 7 seconds with benchmark mode off. Because Modin can run the ``map``
asynchronously, it can start writing the first result to its buffer while
it's still computing the second result. With benchmark mode on, Modin has to
execute every function synchronously instead.
How to benchmark complex workflows
----------------------------------
Typically, to benchmark Modin's overall performance on your workflow, you
should start by looking at end-to-end performance with benchmark mode off.
It's common for Modin worfklows to end with writing results to one or more
files, or with printing some Modin objects to an interactive console. Such
end points are natural ways to make sure that all of the Modin execution that
you require is complete.
To measure more fine-grained performance, it can be helpful to turn
benchmark mode on, but beware that doing so may reduce your script's overall
performance and thus may not reflect where Modin is normally spending execution
time, as pointed out above.
Turning on :doc:`Modin logging ` and
using the Modin logs can also help you profile your workflow. The Modin logs
can also give a detailed break down of the performance of each Modin function
at each Modin :doc:`layer `. Log mode is more
useful when used in conjuction with benchmark mode.
Sometimes, if you don't have a natural end-point to your workflow, you can
just call ``modin.utils.execute`` on the workflow's final Modin objects.
That will typically block on any asynchronous computation:
.. code-block:: python
import time
import ray
from io import BytesIO
import modin.pandas as pd
from modin.config import MinRowPartitionSize, NPartitions
import modin.utils
MinRowPartitionSize.put(32)
NPartitions.put(16)
def slow_add_one(x):
if x == 5000:
time.sleep(10)
return x + 1
# Look at the Ray documentation with respect to the Ray configuration suited to you most.
ray.init()
df1 = pd.DataFrame(list(range(10_000)), columns=['A'])
result = df1.map(slow_add_one)
# %time modin.utils.execute(result)
%time result.to_parquet(BytesIO())
.. code-block::python
Writing the result to a buffer takes 9.84 seconds. However, if you uncomment
the :code:`%time modin.utils.execute(result)` before the :code:`to_parquet`
call, the :code:`to_parquet` takes just 23.8 milliseconds!
.. note::
If you see any Modin documentation touting Modin's speed without using
benchmark mode or otherwise guaranteeing that Modin is finishing all asynchronous
and deferred computation, you should file an issue on the Modin GitHub. It's
not fair to compare the speed of an async Modin function call to an equivalent
synchronous call using another library.
Appendix: System Information
----------------------------
The example scripts here were run on the following system:
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS Monterey 12.4
- **Modin version**: d6d503ac7c3028d871c34d9e99e925ddb0746df6
- **Ray version**: 2.0.0
- **Python version**: 3.10.4
- **Machine**: MacBook Pro (16-inch, 2019)
- **Processor**: 2.3 GHz 8-core Intel Core i9 processor
- **Memory**: 16 GB 2667 MHz DDR4
================================================
FILE: docs/usage_guide/examples/index.rst
================================================
Modin Usage Examples
====================
This section shows Modin usage examples in different scenarios like Modin on a local/remote cluster,
the use of Modin spreadsheet.
Tutorials
'''''''''
The following tutorials cover the basic usage of Modin. `Here `_ is a one hour video tutorial that walks through these basic exercises.
- Exercise 1: Introduction to Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]
- Exercise 2: Speed Improvements with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]
- Exercise 3: Defaulting to pandas with Modin [`Source PandasOnRay `__, `Source PandasOnDask `__]
The following tutorials covers more advanced features in Modin:
- Exercise 4: Experimental Features in Modin (Spreadsheet, Progress Bar) [`Source PandasOnRay `__, `Source PandasOnDask `__]
- Exercise 5: Setting up Modin in a Cluster Environment [`Source PandasOnRay `__]
How to get required dependencies for the tutorial notebooks and to run them please refer to the respective `README.md `__ file.
Data Science Benchmarks
'''''''''''''''''''''''
- Using Modin with the NYC Taxi Dataset [`Source `__]
- Using Modin with the Census Dataset (coming soon...)
- Using Modin with the Plasticc Dataset (coming soon...)
Modin Spreadsheets
''''''''''''''''''
- Using Modin along with the Spreadsheets API [`Source `__]
Modin with scikit-learn
'''''''''''''''''''''''
- Modin for Machine Learning with scikit-learn [`Source `__]
================================================
FILE: docs/usage_guide/index.rst
================================================
Usage Guide
===========
This guide describes both basic and advanced Modin usage, including usage examples,
details regarding Modin configuration settings, as well as tips and tricks on
how you can further optimize the performance of your workload with Modin.
.. toctree::
:maxdepth: 4
/flow/modin/config
examples/index
advanced_usage/index
optimization_notes/index
benchmarking
integrations
.. meta::
:description lang=en:
Usage-specific documentation.
================================================
FILE: docs/usage_guide/integrations.rst
================================================
Third Party Library Integrations
================================
Modin is a drop-in replacement for Pandas, so we want it to interoperate with third-party libraries just as Pandas does.
To see where Modin performs well and where it needs to improve, we've selected a number of important machine learning + visualization + statistics libraries,
and then looked at examples (from their documentation, if possible) about how they work with Pandas. Then we ran those same workflows with Modin, and
tracked what worked, and what failed.
In the table below, you'll see, for each third-party library we tested, the number of successful test calls / total test calls, and a qualitative description of how both Pandas and Modin integrate with that library.
In the deeper dive, you can view the Jupyter notebook we have used to test API calls and the corresponding Github issues filed. If you come across other issues/ examples
in your own workflows we encourage you to file an `issue `_ or contribute a `PR `_!
.. note::
These interoperability metrics are preliminary and not all APIs for each library have been tested. Feel free to add more!
Modin Interoperability by Library
'''''''''''''''''''''''''''''''''
.. list-table::
:widths: 5 5 20
:header-rows: 1
* - Library
- API successes / calls
- Interoperability
* - seaborn
- 73% (11/15)
- **Pandas**: Accepts Pandas DataFrames as inputs for producing plot |br|
**Modin**: Mostly accepts Modin DataFrames as inputs for producing plots, but fails completely in some cases (pairplot, lmplot), and in others (catplot, objects.Plot) only works for some parameter combinations
* - plotly
- 78% (7 / 9)
- **Pandas**: Accepts Pandas DataFrames as inputs for producing plots, including specifying X and Y parameters as df columns |br|
**Modin**: Mostly accepts Modin DataFrames as inputs for producing plots (the exception is choropleth), but fails when specifying X and Y parameters as df columns
* - matplotlib
- 100% (5 / 5)
- **Pandas**: Accepts Pandas DataFrames as inputs for producing plots like scatter, barh, etc. |br|
**Modin**: Accepts Modin DataFrames as inputs for producing plots like scatter, barh, etc.
* - altair
- 0% (0 / 1)
- **Pandas**: Accepts Pandas DataFrames as inputs for producing charts through Chart |br|
**Modin**: Does not accept Modin DataFrames as inputs for producing charts through Chart
* - bokeh
- 0% (0 / 1)
- **Pandas**: Loads Pandas DataFrames through ColumnDataSource |br|
**Modin**: Does not load Modin DataFrames through ColumnDataSource
* - sklearn
- 100% (6 / 6)
- **Pandas**: Many functions take Pandas DataFrames as inputs |br|
**Modin**: Many functions take Modin DataFrames as inputs
* - Hugging Face (Transformers, Datasets)
- 100% (2 / 2)
- **Pandas**: Loads Pandas DataFrames into Datasets, and processes Pandas DataFrame rows as inputs using Transformers.InputExample (deprecated) |br|
**Modin**: Loads Modin DataFrames into Datasets (though slowly), and processes Modin DataFrame rows as inputs through Transformers.InputExample (deprecated)
* - Tensorflow
- 75% (3 / 4)
- **Pandas**: Converts Pandas dataframes to tensors |br|
**Modin**: Converts Modin DataFrames to tensors, but specialized APIs like Keras might not work yet
* - NLTK
- 100% (1 / 1)
- **Pandas**: Performs transformations like tokenization on Pandas DataFrames |br|
**Modin**: Performs transformations like tokenization on Modin DataFrames
* - XGBoost
- 100% (1 / 1)
- **Pandas**: Loads Pandas DataFrames through the DMatrix function |br|
**Modin**: Loads Modin DataFrames through the DMatrix function
* - statsmodels
- 50% (1 / 2)
- **Pandas**: Can accept Pandas DataFrames when fitting models |br|
**Modin**: Sometimes accepts Modin DataFrames when fitting models (e.g., formula.api.ols), but does not in others (e.g., api.OLS)
.. |br| raw:: html
A Deeper Dive
''''''''''''''
**seaborn**
-----------
`Jupyter Notebook `__
Github Issues
* https://github.com/modin-project/modin/issues/5435
* https://github.com/modin-project/modin/issues/5433
**plotly**
----------
`Jupyter Notebook `__
Github Issues
* https://github.com/modin-project/modin/issues/5447
* https://github.com/modin-project/modin/issues/5445
**matplotlib**
--------------
`Jupyter Notebook `__
**altair**
----------
`Jupyter Notebook `__
Github Issues
* https://github.com/modin-project/modin/issues/5438
**bokeh**
---------
`Jupyter Notebook `__
Github Issues
* https://github.com/modin-project/modin/issues/5437
**sklearn**
-----------
`Jupyter Notebook `__
**Hugging Face**
----------------
`Jupyter Notebook `__
**Tensorflow**
--------------
`Jupyter Notebook `__
Github Issues
* https://github.com/modin-project/modin/issues/5439
**NLTK**
---------
`Jupyter Notebook `__
**XGBoost**
-----------
`Jupyter Notebook `__
**statsmodels**
---------------
`Jupyter Notebook `__
Github Issues
* https://github.com/modin-project/modin/issues/5440
Appendix: System Information
'''''''''''''''''''''''''''''
The example scripts here were run on the following system:
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS Big Sur 11.5.2
- **Modin version**: 0.18.0+3.g4114183f
- **Ray version**: 2.0.1
- **Python version**: 3.9.7.final.0
- **Machine**: MacBook Pro (16-inch, 2019)
- **Processor**: 2.3 GHz 8-core Intel Core i9 processor
- **Memory**: 16 GB 2667 MHz DDR4
================================================
FILE: docs/usage_guide/optimization_notes/index.rst
================================================
Optimization Notes
==================
Modin has chosen default values for a lot of the configurations here that provide excellent performance in most
cases. This page is for those who love to optimize their code and those who are curious about existing optimizations
within Modin. Here you can find more information about Modin's optimizations both for a pipeline of operations as
well as for specific operations. If you want to go ahead and tune the Modin behavior on your own, refer to
:doc:`Modin Configuration Settings ` page for the full set of configurations available in Modin.
Range-partitioning in Modin
"""""""""""""""""""""""""""
Modin utilizes a range-partitioning approach for specific operations, significantly enhancing
parallelism and reducing memory consumption in certain scenarios. Range-partitioning is typically
engaged for operations that has key columns (to group on, to merge on, etc).
You can enable `range-partitioning`_ by specifying ``cfg.RangePartitioning`` :doc:`configuration variable: `
.. code-block:: python
import modin.pandas as pd
import modin.config as cfg
cfg.RangePartitioning.put(True) # past this point methods that support range-partitioning
# will use it
pd.DataFrame(...).groupby(...).mean() # use range-partitioning for groupby.mean()
cfg.Range-partitioning.put(False)
pd.DataFrame(...).groupby(...).mean() # use MapReduce implementation for groupby.mean()
Building range-partitioning assumes data reshuffling, which may result into breaking the original
order of rows, for some operation, it will mean that the result will be different from Pandas.
Range-partitioning is not a silver bullet, meaning that enabling it is not always beneficial. Below you find
a link to the list of operations that have support for range-partitioning and practical advices on when one should
enable it: :doc:`operations that support range-partitioning `.
Dynamic-partitioning in Modin
"""""""""""""""""""""""""""""
Ray engine experiences slowdowns when running a large number of small remote tasks at the same time. Ray Core recommends to `avoid tiny task`_.
When modin DataFrame has a large number of partitions, some functions produce a large number of remote tasks, which can cause slowdowns.
To solve this problem, Modin suggests using dynamic partitioning. This approach reduces the number of remote tasks
by combining multiple partitions into a single virtual partition and perform a common remote task on them.
Dynamic partitioning is typically used for operations that are fully or partially executed on all partitions separately.
.. code-block:: python
import modin.pandas as pd
from modin.config import context
df = pd.DataFrame(...)
with context(DynamicPartitioning=True):
df.abs()
Dynamic partitioning is also not always useful, and this approach is usually used for medium-sized DataFrames with a large number of columns.
If the number of columns is small, the number of partitions will be close to the number of CPUs, and Ray will not have this problem.
If the DataFrame has too many rows, this is also not a good case for using Dynamic-partitioning, since each task is no longer tiny and performing
the combined tasks carries more overhead than assigning them separately.
Unfortunately, the use of Dynamic-partitioning depends on various factors such as data size, number of CPUs, operations performed,
and it is up to the user to determine whether Dynamic-partitioning will give a boost in his case or not.
..
TODO: Define heuristics to automatically enable dynamic partitioning without performance penalty.
`Issue #7370 `_
Understanding Modin's partitioning mechanism
""""""""""""""""""""""""""""""""""""""""""""
Modin's partitioning is crucial for performance; so we recommend expert users to understand Modin's
partitioning mechanism and how to tune it in order to achieve better performance.
How Modin partitions a dataframe
--------------------------------
Modin uses a partitioning scheme that partitions a dataframe along both axes, resulting in a matrix
of partitions. The row and column chunk sizes are computed independently based
on the length of the appropriate axis and Modin's special :doc:`configuration variables `
(``NPartitions``, ``MinRowPartitionSize`` and ``MinColumnPartitionSize``):
- ``NPartitions`` is the maximum number of splits along an axis; by default, it equals to the number of cores
on your local machine or cluster of nodes.
- ``MinRowPartitionSize`` is the minimum number of rows to do a split. For instance, if ``MinRowPartitionSize``
is 32, the row axis will not be split unless the amount of rows is greater than 32. If it is is greater, for example, 34,
then the row axis is sliced into two partitions: containing 32 and 2 rows accordingly.
- ``MinColumnPartitionSize`` is the minimum number of columns to do a split. For instance, if ``MinColumnPartitionSize``
is 32, the column axis will not be split unless the amount of columns is greater than 32. If it is is greater, for example, 34,
then the column axis is sliced into two partitions: containing 32 and 2 columns accordingly.
Beware that ``NPartitions`` specifies a limit for the number of partitions `along a single axis`, which means, that
the actual limit for the entire dataframe itself is the square of ``NPartitions``.
.. figure:: /img/partitioning_mechanism/partitioning_examples.svg
:align: center
Full-axis functions
-------------------
Some of the aggregation functions require knowledge about the entire axis, for example at ``.apply(foo, axis=0)``
the passed function ``foo`` expects to receive data for the whole column at once.
When a full-axis function is applied, the partitions along this axis are collected at a single worker
that processes the function. After the function is done, the partitioning of the data is back to normal.
.. figure:: /img/partitioning_mechanism/full_axis_function.svg
:align: center
Note that the amount of remote calls is equal to the number of partitions, which means that since the number
of partitions is decreased for full-axis functions it also decreases the potential for parallelism.
Also note, that reduce functions such as ``.sum()``, ``.mean()``, ``.max()``, etc, are not considered
to be full-axis, so they do not suffer from the decreasing level of parallelism.
How to tune partitioning
------------------------
Configure Modin's default partitioning scheme
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As you can see from the examples above, the more the dataframe's shape is closer to a square, the closer the number of
partitions to the square of ``NPartitions``. In the case of ``NPartitions`` equals to the number of workers,
that means that a single worker is going to process multiple partitions at once, which slows down overall performance.
If your workflow mainly operates with wide dataframes and non-full-axis functions, it makes sense to reduce the
``NPartitions`` value so a single worker would process a single partition.
.. figure:: /img/partitioning_mechanism/repartition_square_frames.svg
:align: center
Copy-pastable example, showing how tuning ``NPartitions`` value for wide frames may improve performance on your machine:
.. code-block:: python
from multiprocessing import cpu_count
from modin.distributed.dataframe.pandas import unwrap_partitions
import modin.config as cfg
import modin.pandas as pd
import numpy as np
import timeit
# Generating data for a square-like dataframe
data = np.random.randint(0, 100, size=(5000, 5000))
# Explicitly setting `NPartitions` to its default value
cfg.NPartitions.put(cpu_count())
# Each worker processes `cpu_count()` amount of partitions
df = pd.DataFrame(data)
print(f"NPartitions: {cfg.NPartitions.get()}")
# Getting raw partitions to count them
partitions_shape = np.array(unwrap_partitions(df)).shape
print(
f"The frame has {partitions_shape[0]}x{partitions_shape[1]}={np.prod(partitions_shape)} partitions "
f"when the CPU has only {cpu_count()} cores."
)
print(f"10 times of .abs(): {timeit.timeit(lambda: df.abs(), number=10)}s.")
# Possible output:
# NPartitions: 112
# The frame has 112x112=12544 partitions when the CPU has only 112 cores.
# 10 times of .abs(): 23.64s.
# Taking a square root of the the current `cpu_count` to make more even partitioning
cfg.NPartitions.put(int(cpu_count() ** 0.5))
# Each worker processes a single partition
df = pd.DataFrame(data)
print(f"NPartitions: {cfg.NPartitions.get()}")
# Getting raw partitions to count them
partitions_shape = np.array(unwrap_partitions(df)).shape
print(
f"The frame has {partitions_shape[0]}x{partitions_shape[1]}={np.prod(partitions_shape)} "
f"when the CPU has {cpu_count()} cores."
)
print(f"10 times of .abs(): {timeit.timeit(lambda: df.abs(), number=10)}s.")
# Possible output:
# NPartitions: 10
# The frame has 10x10=100 partitions when the CPU has 112 cores.
# 10 times of .abs(): 0.25s.
Manually trigger repartitioning
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you're getting unexpectedly poor performance, although you configured ``MODIN_NPARTITIONS``
correctly, then this might be caused by unbalanced partitioning that occurred during the
workflow's execution.
Modin's idealogy is to handle partitioning internally and not let users worry about the possible
consequences of applying a lot of "bad" operations that may affect DataFrame's partitioning.
We're constantly making efforts to find and fix cases where partitioning may cause a headache
for users.
However, if you feel that you're dealing with unbalanced partitioning you may try to call an
internal :py:meth:`modin.pandas.dataframe.DataFrame._repartition` method on your :py:class:`~modin.pandas.dataframe.DataFrame` in order to manually
trigger partitions rebalancing and see whether it improves performance for your case.
.. automethod:: modin.pandas.dataframe.DataFrame._repartition
An actual use-case for this method may be the following:
.. code-block:: python
import modin.pandas as pd
import timeit
df = pd.DataFrame({"col0": [1, 2, 3, 4]})
# Appending a lot of columns may result into unbalanced partitioning
for i in range(1, 128):
df[f"col{i}"] = pd.Series([1, 2, 3, 4])
print(
"DataFrame with unbalanced partitioning:",
timeit.timeit(lambda: df.sum(), number=10)
) # 1.44s
df = df._repartition()
print(
"DataFrame after '._repartition()':",
timeit.timeit(lambda: df.sum(), number=10)
) # 0.21s.
Avoid iterating over Modin DataFrame
""""""""""""""""""""""""""""""""""""
Use ``df.apply()`` or other aggregation methods when possible instead of iterating over a dataframe.
For-loops don't scale and forces the distributed data to be collected back at the driver.
Copy-pastable example, showing how replacing a for-loop to the equivalent ``.apply()`` may improve performance:
.. code-block:: python
import modin.pandas as pd
import numpy as np
from timeit import default_timer as timer
data = np.random.randint(1, 100, (2 ** 10, 2 ** 2))
md_df = pd.DataFrame(data)
result = []
t1 = timer()
# Iterating over a dataframe forces to collect distributed data to the driver and doesn't scale
for idx, row in md_df.iterrows():
result.append((row[1] + row[2]) / row[3])
print(f"Filling a list by iterating a Modin frame: {timer() - t1:.2f}s.")
# Possible output: 36.15s.
t1 = timer()
# Using `.apply()` perfectly scales to all axis-partitions
result = md_df.apply(lambda row: (row[1] + row[2]) / row[3], axis=1).to_numpy().tolist()
print(f"Filling a list by using '.apply()' and converting the result to a list: {timer() - t1:.2f}s.")
# Possible output: 0.22s.
Use Modin's Dataframe Algebra API to implement custom parallel functions
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Modin provides a set of low-level parallel-implemented operators which can be used to build most of the
aggregation functions. These operators are present in the :doc:`algebra module `.
Modin DataFrame allows users to use their own aggregations built with this module. Visit the
:doc:`DataFrame's algebra ` page of the documentation for the steps to do it.
Avoid mixing pandas and Modin DataFrames
""""""""""""""""""""""""""""""""""""""""
Although Modin is considered to be a drop-in replacement for pandas, Modin and pandas are not intended to be used together
in a single flow. Passing a pandas DataFrame as an argument for a Modin's DataFrame method may either slowdown
the function (because it has to process non-distributed object) or raise an error. You would also get an undefined
behavior if you pass a Modin DataFrame as an input to pandas methods, since pandas identifies Modin's objects as a simple iterable,
and so can't leverage its benefits as a distributed dataframe.
Copy-pastable example, showing how mixing pandas and Modin DataFrames in a single flow may bottleneck performance:
.. code-block:: python
import modin.pandas as pd
import numpy as np
import timeit
import pandas
data = np.random.randint(0, 100, (2 ** 20, 2 ** 2))
md_df, md_df_copy = pd.DataFrame(data), pd.DataFrame(data)
pd_df, pd_df_copy = pandas.DataFrame(data), pandas.DataFrame(data)
print("concat modin frame + pandas frame:")
# Concatenating modin frame + pandas frame using modin '.concat()'
# This case is bad because Modin have to process non-distributed pandas object
time = timeit.timeit(lambda: pd.concat([md_df, pd_df]), number=10)
print(f"\t{time}s.\n")
# Possible output: 0.44s.
print("concat modin frame + modin frame:")
# Concatenating modin frame + modin frame using modin '.concat()'
# This is an ideal case, Modin is being used as intended
time = timeit.timeit(lambda: pd.concat([md_df, md_df_copy]), number=10)
print(f"\t{time}s.\n")
# Possible output: 0.05s.
print("concat pandas frame + pandas frame:")
# Concatenating pandas frame + pandas frame using pandas '.concat()'
time = timeit.timeit(lambda: pandas.concat([pd_df, pd_df_copy]), number=10)
print(f"\t{time}s.\n")
# Possible output: 0.31s.
print("concat pandas frame + modin frame:")
# Concatenating pandas frame + modin frame using pandas '.concat()'
time = timeit.timeit(lambda: pandas.concat([pd_df, md_df]), number=10)
print(f"\t{time}s.\n")
# Possible output: TypeError
Using pandas to execute queries with Modin's ``"Pandas"`` backend
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
By default, Modin distributes the data in a dataframe (or series) and attempts
to process data for different partitions in parallel.
However, for certain scenarios, such as handling small datasets, Modin's
parallel execution may introduce unnecessary overhead. In such cases, it's more
efficient to use serial execution with a single, unpartitioned pandas dataframe.
You can enable this kind of local pandas execution by setting Modin's
``Backend``
:doc:`configuration variable ` to ``"Pandas"``.
DataFrames created while Modin's global backend is set to ``"Pandas"``
will continue to use native execution even if you switch the global backend
later. Modin supports interoperability between distributed Modin DataFrames
and those using the pandas backend.
Here is an example of using the pandas backend.
.. code-block:: python
import modin.pandas as pd
from modin.config import Backend
# This dataframe will use Modin's default, distributed execution.
original_backend = Backend.get()
assert original_backend != "Pandas"
distributed_df_1 = pd.DataFrame([0])
# Set backend to "Pandas" for local pandas execution.
Backend.put("Pandas")
modin_on_pandas_df = pd.DataFrame([1])
assert modin_on_pandas_df.get_backend() == "Pandas"
# Revert to default settings for distributed execution
Backend.put(original_backend)
distributed_df_2 = pd.DataFrame([2])
assert distributed_df_2.get_backend() == original_backend
You can also use the pandas backend for some dataframes while using different
backends for other dataframes. You can switch the backend of an individual
dataframe or series with ``set_backend()`` or its synonym ``move_to()``.
Here's an example of switching the backend for an individual dataframe.
.. code-block:: python
import modin.pandas as pd
# This dataframe will use Modin's default, distributed execution.
original_backend = Backend.get()
assert original_backend != "Pandas"
distributed_df_1 = pd.DataFrame([0])
pandas_df_1 = distributed_df_1.move_to("Pandas")
assert pandas_df_1.get_backend() == "Pandas"
pandas_df_1 = pandas_df_1.sort_values(0)
assert pandas_df_1.get_backend() == "Pandas"
new_df = pandas_df_1.move_to(original_backend)
assert new_df.get_backend() == original_backend
new_df.set_backend("Pandas", inplace=True)
assert new_df.get_backend() == "Pandas"
Automatic backend switching
"""""""""""""""""""""""""""
*This feature is under active development, and the API is subject to change.*
Modin's backends may define heuristics for whether to automatically move data to another backend
for more efficient computation of certain operations. Modin does not currently define these heuristics
for any of its default backends, but any backends that wish to do so should implement the query
compiler methods discussed in
:ref:`the architecture document`.
After implementing the relevant query compiler methods, the following APIs can be used to control
when automatic switching occurs:
.. code-block:: python
import modin.pandas as pd
from modin.core.storage_formats.pandas.query_compiler_caster import (
register_function_for_post_op_switch,
register_function_for_pre_op_switch,
)
from modin.config import AutoSwitchBackend
# Enable automatic switching BEFORE computation for DataFrame.apply
# when the DataFrame's backend is Pandas
register_function_for_pre_op_switch(
class_name="DataFrame",
method="apply",
backend="Pandas",
)
# Enable automatic switching AFTER computation for Series.max
# when the Series's backend is Pandas
register_function_for_post_op_switch(
class_name="Series",
method="max",
backend="Pandas",
)
# Enable automatic switching globally (use .disable() to turn off)
AutoSwitchBackend.enable()
df = pd.DataFrame([[1, 2, 3]])
# "pin" a single DataFrame/Series, preventing it from
# automatically switching backends
df.pin_backend(inplace=True)
# "unpin" it to re-enable automatic switching
df.unpin_backend(inplace=True)
Operation-specific optimizations
""""""""""""""""""""""""""""""""
merge
-----
``merge`` operation in Modin uses the broadcast join algorithm: combining a right Modin DataFrame into a pandas DataFrame and
broadcasting it to the row partitions of the left Modin DataFrame. In order to minimize interprocess communication cost when doing
an inner join you may want to swap left and right DataFrames.
.. code-block:: python
import modin.pandas as pd
import numpy as np
left_data = np.random.randint(0, 100, size=(2**8, 2**8))
right_data = np.random.randint(0, 100, size=(2**12, 2**12))
left_df = pd.DataFrame(left_data)
right_df = pd.DataFrame(right_data)
%timeit left_df.merge(right_df, how="inner", on=10)
3.59 s 107 ms per loop (mean std. dev. of 7 runs, 1 loop each)
%timeit right_df.merge(left_df, how="inner", on=10)
1.22 s 40.1 ms per loop (mean std. dev. of 7 runs, 1 loop each)
Note that result columns order may differ for first and second ``merge``.
.. _range-partitioning: https://www.techopedia.com/definition/31994/range-partitioning
.. _`avoid tiny task`: https://docs.ray.io/en/latest/ray-core/tips-for-first-time.html#tip-2-avoid-tiny-tasks
================================================
FILE: docs/usage_guide/optimization_notes/range_partitioning_ops.rst
================================================
:orphan:
Operations that support range-partitioning in Modin
###################################################
The following operations change their behavior once ``cfg.RangePartitioning`` variable is set to ``True``.
Go through the list find out when it could be beneficial to engage range-partitioning for a certain method.
GroupBy
=======
.. note::
When grouping on multiple columns using range-partitioning implementation, the result
may not be sorted even if ``groupby(sort=True, ...)`` was passed: https://github.com/modin-project/modin/issues/6875.
Range-partitioning groupby implementation is automatically engaged for ``groupby.apply()``, ``groupby.transform()``,
``groupby.rolling()``. For groupby aggregations from `this list`_, MapReduce implementation is used by default.
MapReduce tends to show better performance for groupby with low-cardinality. If the cardinality of your columns
to group is expected to be high, it's recommended to engage range-partitioning implementation.
Merge
=====
.. note::
Range-partitioning approach is implemented only for "left" and "inner" merge and only
when merging on a single column using `on` argument.
Range-partitioning merge replaces broadcast merge. It is recommended to use range-partitioning implementation
if the right dataframe in merge is as big as the left dataframe. In this case, range-partitioning
implementation works faster and consumes less RAM.
Under the spoiler you can find performance comparison of range-partitioning and broadcast merge in different
scenarios:
.. raw:: html
Performance measurements for merge
The performance was measured on `h2o join queries`_ using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores),
with the number of cores allocated for Modin limited by 44 (``MODIN_CPUS=44``).
Measurements for small 500mb data:
.. image:: /img/range_partitioning_measurements/merge_h2o_500mb.jpg
:align: center
Measurements for medium 5gb data:
.. image:: /img/range_partitioning_measurements/merge_h2o_5gb.png
:align: center
.. raw:: html
``.unique()`` and ``.drop_duplicates()``
========================================
.. note::
When range-partitioning is enabled, both ``.unique()`` and ``.drop_duplicates()`` will
yield results that are sorted along rows. If range-partitioning is disabled,
the original order will be maintained.
Range-partitioning implementation of ``.unique()`` / ``.drop_duplicates()`` works best when the input data size is big (more than
5_000_000 rows) and when the output size is also expected to be big (no more than 80% values are duplicates).
Under the spoiler you can find performance comparisons in different scenarios:
.. raw:: html
Performance measurements for ``.unique()``
The performance was measured on randomly generated data using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).
The `duplicate rate` shows the procentage of duplicated rows in the dataset. You can learn more about this micro-benchmark
by reading its source code:
.. raw:: html
Micro-benchmark's source code
.. code-block:: python
import modin.pandas as pd
import numpy as np
import modin.config as cfg
from modin.utils import execute
from timeit import default_timer as timer
import pandas
cfg.CpuCount.put(16)
def get_data(nrows, dtype):
if dtype == int:
return np.arange(nrows)
elif dtype == float:
return np.arange(nrows).astype(float)
elif dtype == str:
return np.array([f"value{i}" for i in range(nrows)])
else:
raise NotImplementedError(dtype)
pd.DataFrame(np.arange(cfg.NPartitions.get() * cfg.MinRowPartitionSize.get())).to_numpy()
nrows = [1_000_000, 5_000_000, 10_000_000, 25_000_000, 50_000_000, 100_000_000]
duplicate_rate = [0, 0.1, 0.5, 0.95]
dtypes = [int, str]
use_range_part = [True, False]
columns = pandas.MultiIndex.from_product([dtypes, duplicate_rate, use_range_part], names=["dtype", "duplicate rate", "use range-part"])
result = pandas.DataFrame(index=nrows, columns=columns)
i = 0
total_its = len(nrows) * len(duplicate_rate) * len(dtypes) * len(use_range_part)
for dt in dtypes:
for nrow in nrows:
data = get_data(nrow, dt)
np.random.shuffle(data)
for dpr in duplicate_rate:
data_c = data.copy()
dupl_val = data_c[0]
num_duplicates = int(dpr * nrow)
dupl_indices = np.random.choice(np.arange(nrow), num_duplicates, replace=False)
data_c[dupl_indices] = dupl_val
for impl in use_range_part:
print(f"{round((i / total_its) * 100, 2)}%")
i += 1
cfg.RangePartitioning.put(impl)
sr = pd.Series(data_c)
execute(sr)
t1 = timer()
# returns a list, so no need for materialization
sr.unique()
tm = timer() - t1
print(nrow, dpr, dt, impl, tm)
result.loc[nrow, (dt, dpr, impl)] = tm
result.to_excel("unique.xlsx")
.. raw:: html
Measurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):
.. image:: /img/range_partitioning_measurements/unique_16cpus.jpg
:align: center
Measurements with 44 cores being allocated for Modin (``MODIN_CPUS=44``):
.. image:: /img/range_partitioning_measurements/unique_44cpus.jpg
:align: center
.. raw:: html
.. raw:: html
Performance measurements for ``.drop_duplicates()``
The performance was measured on randomly generated data using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).
The `duplicate rate` shows the procentage of duplicated rows in the dataset. The `subset size` shows the number of
columns being specified as a ``subset`` parameter for ``df.drop_duplicates()``. You can learn more about this
micro-benchmark by reading its source code:
.. raw:: html
Micro-benchmark's source code
.. code-block:: python
import modin.pandas as pd
import numpy as np
import modin.config as cfg
from modin.utils import execute
from timeit import default_timer as timer
import pandas
cfg.CpuCount.put(16)
pd.DataFrame(np.arange(cfg.NPartitions.get() * cfg.MinRowPartitionSize.get())).to_numpy()
nrows = [1_000_000, 5_000_000, 10_000_000, 25_000_000]
duplicate_rate = [0, 0.1, 0.5, 0.95]
subset = [["col0"], ["col1", "col2", "col3", "col4"], None]
ncols = 15
use_range_part = [True, False]
columns = pandas.MultiIndex.from_product(
[
[len(sbs) if sbs is not None else ncols for sbs in subset],
duplicate_rate,
use_range_part
],
names=["subset size", "duplicate rate", "use range-part"]
)
result = pandas.DataFrame(index=nrows, columns=columns)
i = 0
total_its = len(nrows) * len(duplicate_rate) * len(subset) * len(use_range_part)
for sbs in subset:
for nrow in nrows:
data = {f"col{i}": np.arange(nrow) for i in range(ncols)}
pandas_df = pandas.DataFrame(data)
for dpr in duplicate_rate:
pandas_df_c = pandas_df.copy()
dupl_val = pandas_df_c.iloc[0]
num_duplicates = int(dpr * nrow)
dupl_indices = np.random.choice(np.arange(nrow), num_duplicates, replace=False)
pandas_df_c.iloc[dupl_indices] = dupl_val
for impl in use_range_part:
print(f"{round((i / total_its) * 100, 2)}%")
i += 1
cfg.RangePartitioning.put(impl)
md_df = pd.DataFrame(pandas_df_c)
execute(md_df)
t1 = timer()
res = md_df.drop_duplicates(subset=sbs)
execute(res)
tm = timer() - t1
sbs_s = len(sbs) if sbs is not None else ncols
print("len()", res.shape, nrow, dpr, sbs_s, impl, tm)
result.loc[nrow, (sbs_s, dpr, impl)] = tm
result.to_excel("drop_dupl.xlsx")
.. raw:: html
Measurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):
.. image:: /img/range_partitioning_measurements/drop_duplicates_16cpus.jpg
:align: center
Measurements with 44 cores being allocated for Modin (``MODIN_CPUS=44``):
.. image:: /img/range_partitioning_measurements/drop_duplicates_44cpus.jpg
:align: center
.. raw:: html
'.nunique()'
============
.. note::
Range-partitioning approach is implemented only for ``pd.Series.nunique()`` and 1-column dataframes.
For multi-column dataframes ``.nunique()`` can only use full-axis reduce implementation.
Range-partitioning implementation of '.nunique()'' works best when the input data size is big (more than
5_000_000 rows) and when the output size is also expected to be big (no more than 80% values are duplicates).
Under the spoiler you can find performance comparisons in different scenarios:
.. raw:: html
Performance measurements for ``.nunique()``
The performance was measured on randomly generated data using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).
The `duplicate rate` shows the procentage of duplicated rows in the dataset. You can learn more about this micro-benchmark
by reading its source code:
.. raw:: html
Micro-benchmark's source code
.. code-block:: python
import modin.pandas as pd
import numpy as np
import modin.config as cfg
from modin.utils import execute
from timeit import default_timer as timer
import pandas
cfg.CpuCount.put(16)
def get_data(nrows, dtype):
if dtype == int:
return np.arange(nrows)
elif dtype == float:
return np.arange(nrows).astype(float)
elif dtype == str:
return np.array([f"value{i}" for i in range(nrows)])
else:
raise NotImplementedError(dtype)
pd.DataFrame(np.arange(cfg.NPartitions.get() * cfg.MinRowPartitionSize.get())).to_numpy()
nrows = [1_000_000, 5_000_000, 10_000_000, 25_000_000, 50_000_000, 100_000_000]
duplicate_rate = [0, 0.1, 0.5, 0.95]
dtypes = [int, str]
use_range_part = [True, False]
columns = pandas.MultiIndex.from_product([dtypes, duplicate_rate, use_range_part], names=["dtype", "duplicate rate", "use range-part"])
result = pandas.DataFrame(index=nrows, columns=columns)
i = 0
total_its = len(nrows) * len(duplicate_rate) * len(dtypes) * len(use_range_part)
for dt in dtypes:
for nrow in nrows:
data = get_data(nrow, dt)
np.random.shuffle(data)
for dpr in duplicate_rate:
data_c = data.copy()
dupl_val = data_c[0]
num_duplicates = int(dpr * nrow)
dupl_indices = np.random.choice(np.arange(nrow), num_duplicates, replace=False)
data_c[dupl_indices] = dupl_val
for impl in use_range_part:
print(f"{round((i / total_its) * 100, 2)}%")
i += 1
cfg.RangePartitioning.put(impl)
sr = pd.Series(data_c)
execute(sr)
t1 = timer()
# returns a scalar, so no need for materialization
res = sr.nunique()
tm = timer() - t1
print(nrow, dpr, dt, impl, tm)
result.loc[nrow, (dt, dpr, impl)] = tm
result.to_excel("nunique.xlsx")
.. raw:: html
Measurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):
.. image:: /img/range_partitioning_measurements/nunique_16cpus.jpg
:align: center
.. raw:: html
Resample
========
.. note::
Range-partitioning approach doesn't support transform-like functions (like `.interpolate()`, `.ffill()`, `.bfill()`, ...)
It is recommended to use range-partitioning for resampling if you're dealing with a dataframe that has more than
5_000_000 rows and the expected output is also expected to be big (more than 500_000 rows).
Under the spoiler you can find performance comparisons in different scenarios:
.. raw:: html
Performance measurements for ``.resample()``
The script below measures performance of ``df.resample(rule).sum()`` using Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz (56 cores).
You can learn more about this micro-benchmark by reading its source code:
.. raw:: html
Micro-benchmark's source code
.. code-block:: python
import pandas
import numpy as np
import modin.pandas as pd
import modin.config as cfg
from timeit import default_timer as timer
from modin.utils import execute
cfg.CpuCount.put(16)
nrows = [1_000_000, 5_000_000, 10_000_000]
ncols = [5, 33]
rules = [
"500ms", # doubles nrows
"30s", # decreases nrows in 30 times
"5min", # decreases nrows in 300
]
use_rparts = [True, False]
cols = pandas.MultiIndex.from_product([rules, ncols, use_rparts], names=["rule", "ncols", "USE RANGE PART"])
rres = pandas.DataFrame(index=nrows, columns=cols)
total_nits = len(nrows) * len(ncols) * len(rules) * len(use_rparts)
i = 0
for nrow in nrows:
for ncol in ncols:
index = pandas.date_range("31/12/2000", periods=nrow, freq="s")
data = {f"col{i}": np.arange(nrow) for i in range(ncol)}
pd_df = pandas.DataFrame(data, index=index)
for rule in rules:
for rparts in use_rparts:
print(f"{round((i / total_nits) * 100, 2)}%")
i += 1
cfg.RangePartitioning.put(rparts)
df = pd.DataFrame(data, index=index)
execute(df)
t1 = timer()
res = df.resample(rule).sum()
execute(res)
ts = timer() - t1
print(nrow, ncol, rule, rparts, ts)
rres.loc[nrow, (rule, ncol, rparts)] = ts
rres.to_excel("resample.xlsx")
.. raw:: html
Measurements with 16 cores being allocated for Modin (``MODIN_CPUS=16``):
.. image:: /img/range_partitioning_measurements/resample_16cpus.jpg
:align: center
.. raw:: html
pivot_table
===========
Range-partitioning implementation is automatically applied for ``df.pivot_table``
whenever possible, users can't control this.
sort_values
===========
Range-partitioning implementation is automatically applied for ``df.sort_values``
whenever possible, users can't control this.
.. _h2o join queries: https://h2oai.github.io/db-benchmark/
.. _this list: https://github.com/modin-project/modin/blob/7b233e4a920d5f03dce7a82847847b92ae7ad617/modin/core/storage_formats/pandas/groupby.py#L236-L247
================================================
FILE: environment-dev.yml
================================================
name: modin
channels:
- conda-forge
dependencies:
- pip
# required dependencies
- pandas>=2.2,<2.4
- numpy>=1.22.4
- fsspec>=2022.11.0
- packaging>=21.0
- psutil>=5.8.0
# optional dependencies
# NOTE Keep the ray and dask dependencies in sync with the Linux and Windows
# Unidist environment dependencies.
- ray-core>=2.10.0,<3
- pyarrow>=10.0.1
# workaround for https://github.com/conda/conda/issues/11744
- grpcio!=1.45.*
- grpcio!=1.46.*
- dask>=2.22.0
- distributed>=2.22.0
- xarray>=2022.12.0
- jinja2>=3.1.2
- scipy>=1.10.0
- s3fs>=2022.11.0
- lxml>=4.9.2
- openpyxl>=3.1.0
- xlrd>=2.0.1
- matplotlib>=3.6.3
- sqlalchemy>=2.0.0
- pandas-gbq>=0.19.0
- pytables>=3.8.0
# pymssql==2.2.8 broken: https://github.com/modin-project/modin/issues/6429
- pymssql>=2.1.5,!=2.2.8
- psycopg2>=2.9.6
- fastparquet>=2022.12.0
- tqdm>=4.60.0
- numexpr>=2.8.4
# dependencies for making release
- pygithub>=v1.58.0
- pygit2>=1.9.2
# test dependencies
- coverage>=7.1.0
- moto>=4.1.0
- pytest>=7.3.2
- pytest-benchmark>=4.0.0
- pytest-cov>=4.0.0
- pytest-xdist>=3.2.0
- typing_extensions
# code linters
- black>=24.1.0
- flake8>=6.0.0
- flake8-no-implicit-concat>=0.3.4
- flake8-print>=5.0.0
- mypy>=1.0.0
- pandas-stubs>=2.0.0
- isort>=5.12
- pip:
- dataframe-api-compat>=0.2.7
- asv==0.5.1
# no conda package for windows so we install it with pip
- connectorx>=0.2.6a4
- fuzzydata>=0.0.11
# Fixes breaking ipywidgets changes, but didn't release yet.
- git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5
# The `numpydoc` version should match the version installed in the `lint-pydocstyle` job of the CI.
- numpydoc==1.6.0
- polars
================================================
FILE: examples/data/boston_housing.csv
================================================
,CRIM,ZN,INDUS,CHAS,NOX,RM,AGE,DIS,RAD,TAX,PTRATIO,B,LSTAT,PRICE
0,0.00632,18.0,2.31,0.0,0.538,6.575,65.2,4.09,1.0,296.0,15.3,396.9,4.98,24.0
1,0.02731,0.0,7.07,0.0,0.469,6.421,78.9,4.9671,2.0,242.0,17.8,396.9,9.14,21.6
2,0.02729,0.0,7.07,0.0,0.469,7.185,61.1,4.9671,2.0,242.0,17.8,392.83,4.03,34.7
3,0.03237,0.0,2.18,0.0,0.458,6.998,45.8,6.0622,3.0,222.0,18.7,394.63,2.94,33.4
4,0.06905,0.0,2.18,0.0,0.458,7.147,54.2,6.0622,3.0,222.0,18.7,396.9,5.33,36.2
5,0.02985,0.0,2.18,0.0,0.458,6.43,58.7,6.0622,3.0,222.0,18.7,394.12,5.21,28.7
6,0.08829,12.5,7.87,0.0,0.524,6.012,66.6,5.5605,5.0,311.0,15.2,395.6,12.43,22.9
7,0.14455,12.5,7.87,0.0,0.524,6.172,96.1,5.9505,5.0,311.0,15.2,396.9,19.15,27.1
8,0.21124,12.5,7.87,0.0,0.524,5.631,100.0,6.0821,5.0,311.0,15.2,386.63,29.93,16.5
9,0.17004,12.5,7.87,0.0,0.524,6.004,85.9,6.5921,5.0,311.0,15.2,386.71,17.1,18.9
10,0.22489,12.5,7.87,0.0,0.524,6.377,94.3,6.3467,5.0,311.0,15.2,392.52,20.45,15.0
11,0.11747,12.5,7.87,0.0,0.524,6.009,82.9,6.2267,5.0,311.0,15.2,396.9,13.27,18.9
12,0.09378,12.5,7.87,0.0,0.524,5.889,39.0,5.4509,5.0,311.0,15.2,390.5,15.71,21.7
13,0.62976,0.0,8.14,0.0,0.538,5.949,61.8,4.7075,4.0,307.0,21.0,396.9,8.26,20.4
14,0.63796,0.0,8.14,0.0,0.538,6.096,84.5,4.4619,4.0,307.0,21.0,380.02,10.26,18.2
15,0.62739,0.0,8.14,0.0,0.538,5.834,56.5,4.4986,4.0,307.0,21.0,395.62,8.47,19.9
16,1.05393,0.0,8.14,0.0,0.538,5.935,29.3,4.4986,4.0,307.0,21.0,386.85,6.58,23.1
17,0.7842,0.0,8.14,0.0,0.538,5.99,81.7,4.2579,4.0,307.0,21.0,386.75,14.67,17.5
18,0.80271,0.0,8.14,0.0,0.538,5.456,36.6,3.7965,4.0,307.0,21.0,288.99,11.69,20.2
19,0.7258,0.0,8.14,0.0,0.538,5.727,69.5,3.7965,4.0,307.0,21.0,390.95,11.28,18.2
20,1.25179,0.0,8.14,0.0,0.538,5.57,98.1,3.7979,4.0,307.0,21.0,376.57,21.02,13.6
21,0.85204,0.0,8.14,0.0,0.538,5.965,89.2,4.0123,4.0,307.0,21.0,392.53,13.83,19.6
22,1.23247,0.0,8.14,0.0,0.538,6.142,91.7,3.9769,4.0,307.0,21.0,396.9,18.72,15.2
23,0.98843,0.0,8.14,0.0,0.538,5.813,100.0,4.0952,4.0,307.0,21.0,394.54,19.88,14.5
24,0.75026,0.0,8.14,0.0,0.538,5.924,94.1,4.3996,4.0,307.0,21.0,394.33,16.3,15.6
25,0.84054,0.0,8.14,0.0,0.538,5.599,85.7,4.4546,4.0,307.0,21.0,303.42,16.51,13.9
26,0.67191,0.0,8.14,0.0,0.538,5.813,90.3,4.682,4.0,307.0,21.0,376.88,14.81,16.6
27,0.95577,0.0,8.14,0.0,0.538,6.047,88.8,4.4534,4.0,307.0,21.0,306.38,17.28,14.8
28,0.77299,0.0,8.14,0.0,0.538,6.495,94.4,4.4547,4.0,307.0,21.0,387.94,12.8,18.4
29,1.00245,0.0,8.14,0.0,0.538,6.674,87.3,4.239,4.0,307.0,21.0,380.23,11.98,21.0
30,1.13081,0.0,8.14,0.0,0.538,5.713,94.1,4.233,4.0,307.0,21.0,360.17,22.6,12.7
31,1.35472,0.0,8.14,0.0,0.538,6.072,100.0,4.175,4.0,307.0,21.0,376.73,13.04,14.5
32,1.38799,0.0,8.14,0.0,0.538,5.95,82.0,3.99,4.0,307.0,21.0,232.6,27.71,13.2
33,1.15172,0.0,8.14,0.0,0.538,5.701,95.0,3.7872,4.0,307.0,21.0,358.77,18.35,13.1
34,1.61282,0.0,8.14,0.0,0.538,6.096,96.9,3.7598,4.0,307.0,21.0,248.31,20.34,13.5
35,0.06417,0.0,5.96,0.0,0.499,5.933,68.2,3.3603,5.0,279.0,19.2,396.9,9.68,18.9
36,0.09744,0.0,5.96,0.0,0.499,5.841,61.4,3.3779,5.0,279.0,19.2,377.56,11.41,20.0
37,0.08014,0.0,5.96,0.0,0.499,5.85,41.5,3.9342,5.0,279.0,19.2,396.9,8.77,21.0
38,0.17505,0.0,5.96,0.0,0.499,5.966,30.2,3.8473,5.0,279.0,19.2,393.43,10.13,24.7
39,0.02763,75.0,2.95,0.0,0.428,6.595,21.8,5.4011,3.0,252.0,18.3,395.63,4.32,30.8
40,0.03359,75.0,2.95,0.0,0.428,7.024,15.8,5.4011,3.0,252.0,18.3,395.62,1.98,34.9
41,0.12744,0.0,6.91,0.0,0.448,6.77,2.9,5.7209,3.0,233.0,17.9,385.41,4.84,26.6
42,0.1415,0.0,6.91,0.0,0.448,6.169,6.6,5.7209,3.0,233.0,17.9,383.37,5.81,25.3
43,0.15936,0.0,6.91,0.0,0.448,6.211,6.5,5.7209,3.0,233.0,17.9,394.46,7.44,24.7
44,0.12269,0.0,6.91,0.0,0.448,6.069,40.0,5.7209,3.0,233.0,17.9,389.39,9.55,21.2
45,0.17142,0.0,6.91,0.0,0.448,5.682,33.8,5.1004,3.0,233.0,17.9,396.9,10.21,19.3
46,0.18836,0.0,6.91,0.0,0.448,5.786,33.3,5.1004,3.0,233.0,17.9,396.9,14.15,20.0
47,0.22927,0.0,6.91,0.0,0.448,6.03,85.5,5.6894,3.0,233.0,17.9,392.74,18.8,16.6
48,0.25387,0.0,6.91,0.0,0.448,5.399,95.3,5.87,3.0,233.0,17.9,396.9,30.81,14.4
49,0.21977,0.0,6.91,0.0,0.448,5.602,62.0,6.0877,3.0,233.0,17.9,396.9,16.2,19.4
50,0.08873,21.0,5.64,0.0,0.439,5.963,45.7,6.8147,4.0,243.0,16.8,395.56,13.45,19.7
51,0.04337,21.0,5.64,0.0,0.439,6.115,63.0,6.8147,4.0,243.0,16.8,393.97,9.43,20.5
52,0.0536,21.0,5.64,0.0,0.439,6.511,21.1,6.8147,4.0,243.0,16.8,396.9,5.28,25.0
53,0.04981,21.0,5.64,0.0,0.439,5.998,21.4,6.8147,4.0,243.0,16.8,396.9,8.43,23.4
54,0.0136,75.0,4.0,0.0,0.41,5.888,47.6,7.3197,3.0,469.0,21.1,396.9,14.8,18.9
55,0.01311,90.0,1.22,0.0,0.403,7.249,21.9,8.6966,5.0,226.0,17.9,395.93,4.81,35.4
56,0.02055,85.0,0.74,0.0,0.41,6.383,35.7,9.1876,2.0,313.0,17.3,396.9,5.77,24.7
57,0.01432,100.0,1.32,0.0,0.411,6.816,40.5,8.3248,5.0,256.0,15.1,392.9,3.95,31.6
58,0.15445,25.0,5.13,0.0,0.453,6.145,29.2,7.8148,8.0,284.0,19.7,390.68,6.86,23.3
59,0.10328,25.0,5.13,0.0,0.453,5.927,47.2,6.932,8.0,284.0,19.7,396.9,9.22,19.6
60,0.14932,25.0,5.13,0.0,0.453,5.741,66.2,7.2254,8.0,284.0,19.7,395.11,13.15,18.7
61,0.17171,25.0,5.13,0.0,0.453,5.966,93.4,6.8185,8.0,284.0,19.7,378.08,14.44,16.0
62,0.11027,25.0,5.13,0.0,0.453,6.456,67.8,7.2255,8.0,284.0,19.7,396.9,6.73,22.2
63,0.1265,25.0,5.13,0.0,0.453,6.762,43.4,7.9809,8.0,284.0,19.7,395.58,9.5,25.0
64,0.01951,17.5,1.38,0.0,0.4161,7.104,59.5,9.2229,3.0,216.0,18.6,393.24,8.05,33.0
65,0.03584,80.0,3.37,0.0,0.398,6.29,17.8,6.6115,4.0,337.0,16.1,396.9,4.67,23.5
66,0.04379,80.0,3.37,0.0,0.398,5.787,31.1,6.6115,4.0,337.0,16.1,396.9,10.24,19.4
67,0.05789,12.5,6.07,0.0,0.409,5.878,21.4,6.498,4.0,345.0,18.9,396.21,8.1,22.0
68,0.13554,12.5,6.07,0.0,0.409,5.594,36.8,6.498,4.0,345.0,18.9,396.9,13.09,17.4
69,0.12816,12.5,6.07,0.0,0.409,5.885,33.0,6.498,4.0,345.0,18.9,396.9,8.79,20.9
70,0.08826,0.0,10.81,0.0,0.413,6.417,6.6,5.2873,4.0,305.0,19.2,383.73,6.72,24.2
71,0.15876,0.0,10.81,0.0,0.413,5.961,17.5,5.2873,4.0,305.0,19.2,376.94,9.88,21.7
72,0.09164,0.0,10.81,0.0,0.413,6.065,7.8,5.2873,4.0,305.0,19.2,390.91,5.52,22.8
73,0.19539,0.0,10.81,0.0,0.413,6.245,6.2,5.2873,4.0,305.0,19.2,377.17,7.54,23.4
74,0.07896,0.0,12.83,0.0,0.437,6.273,6.0,4.2515,5.0,398.0,18.7,394.92,6.78,24.1
75,0.09512,0.0,12.83,0.0,0.437,6.286,45.0,4.5026,5.0,398.0,18.7,383.23,8.94,21.4
76,0.10153,0.0,12.83,0.0,0.437,6.279,74.5,4.0522,5.0,398.0,18.7,373.66,11.97,20.0
77,0.08707,0.0,12.83,0.0,0.437,6.14,45.8,4.0905,5.0,398.0,18.7,386.96,10.27,20.8
78,0.05646,0.0,12.83,0.0,0.437,6.232,53.7,5.0141,5.0,398.0,18.7,386.4,12.34,21.2
79,0.08387,0.0,12.83,0.0,0.437,5.874,36.6,4.5026,5.0,398.0,18.7,396.06,9.1,20.3
80,0.04113,25.0,4.86,0.0,0.426,6.727,33.5,5.4007,4.0,281.0,19.0,396.9,5.29,28.0
81,0.04462,25.0,4.86,0.0,0.426,6.619,70.4,5.4007,4.0,281.0,19.0,395.63,7.22,23.9
82,0.03659,25.0,4.86,0.0,0.426,6.302,32.2,5.4007,4.0,281.0,19.0,396.9,6.72,24.8
83,0.03551,25.0,4.86,0.0,0.426,6.167,46.7,5.4007,4.0,281.0,19.0,390.64,7.51,22.9
84,0.05059,0.0,4.49,0.0,0.449,6.389,48.0,4.7794,3.0,247.0,18.5,396.9,9.62,23.9
85,0.05735,0.0,4.49,0.0,0.449,6.63,56.1,4.4377,3.0,247.0,18.5,392.3,6.53,26.6
86,0.05188,0.0,4.49,0.0,0.449,6.015,45.1,4.4272,3.0,247.0,18.5,395.99,12.86,22.5
87,0.07151,0.0,4.49,0.0,0.449,6.121,56.8,3.7476,3.0,247.0,18.5,395.15,8.44,22.2
88,0.0566,0.0,3.41,0.0,0.489,7.007,86.3,3.4217,2.0,270.0,17.8,396.9,5.5,23.6
89,0.05302,0.0,3.41,0.0,0.489,7.079,63.1,3.4145,2.0,270.0,17.8,396.06,5.7,28.7
90,0.04684,0.0,3.41,0.0,0.489,6.417,66.1,3.0923,2.0,270.0,17.8,392.18,8.81,22.6
91,0.03932,0.0,3.41,0.0,0.489,6.405,73.9,3.0921,2.0,270.0,17.8,393.55,8.2,22.0
92,0.04203,28.0,15.04,0.0,0.464,6.442,53.6,3.6659,4.0,270.0,18.2,395.01,8.16,22.9
93,0.02875,28.0,15.04,0.0,0.464,6.211,28.9,3.6659,4.0,270.0,18.2,396.33,6.21,25.0
94,0.04294,28.0,15.04,0.0,0.464,6.249,77.3,3.615,4.0,270.0,18.2,396.9,10.59,20.6
95,0.12204,0.0,2.89,0.0,0.445,6.625,57.8,3.4952,2.0,276.0,18.0,357.98,6.65,28.4
96,0.11504,0.0,2.89,0.0,0.445,6.163,69.6,3.4952,2.0,276.0,18.0,391.83,11.34,21.4
97,0.12083,0.0,2.89,0.0,0.445,8.069,76.0,3.4952,2.0,276.0,18.0,396.9,4.21,38.7
98,0.08187,0.0,2.89,0.0,0.445,7.82,36.9,3.4952,2.0,276.0,18.0,393.53,3.57,43.8
99,0.0686,0.0,2.89,0.0,0.445,7.416,62.5,3.4952,2.0,276.0,18.0,396.9,6.19,33.2
100,0.14866,0.0,8.56,0.0,0.52,6.727,79.9,2.7778,5.0,384.0,20.9,394.76,9.42,27.5
101,0.11432,0.0,8.56,0.0,0.52,6.781,71.3,2.8561,5.0,384.0,20.9,395.58,7.67,26.5
102,0.22876,0.0,8.56,0.0,0.52,6.405,85.4,2.7147,5.0,384.0,20.9,70.8,10.63,18.6
103,0.21161,0.0,8.56,0.0,0.52,6.137,87.4,2.7147,5.0,384.0,20.9,394.47,13.44,19.3
104,0.1396,0.0,8.56,0.0,0.52,6.167,90.0,2.421,5.0,384.0,20.9,392.69,12.33,20.1
105,0.13262,0.0,8.56,0.0,0.52,5.851,96.7,2.1069,5.0,384.0,20.9,394.05,16.47,19.5
106,0.1712,0.0,8.56,0.0,0.52,5.836,91.9,2.211,5.0,384.0,20.9,395.67,18.66,19.5
107,0.13117,0.0,8.56,0.0,0.52,6.127,85.2,2.1224,5.0,384.0,20.9,387.69,14.09,20.4
108,0.12802,0.0,8.56,0.0,0.52,6.474,97.1,2.4329,5.0,384.0,20.9,395.24,12.27,19.8
109,0.26363,0.0,8.56,0.0,0.52,6.229,91.2,2.5451,5.0,384.0,20.9,391.23,15.55,19.4
110,0.10793,0.0,8.56,0.0,0.52,6.195,54.4,2.7778,5.0,384.0,20.9,393.49,13.0,21.7
111,0.10084,0.0,10.01,0.0,0.547,6.715,81.6,2.6775,6.0,432.0,17.8,395.59,10.16,22.8
112,0.12329,0.0,10.01,0.0,0.547,5.913,92.9,2.3534,6.0,432.0,17.8,394.95,16.21,18.8
113,0.22212,0.0,10.01,0.0,0.547,6.092,95.4,2.548,6.0,432.0,17.8,396.9,17.09,18.7
114,0.14231,0.0,10.01,0.0,0.547,6.254,84.2,2.2565,6.0,432.0,17.8,388.74,10.45,18.5
115,0.17134,0.0,10.01,0.0,0.547,5.928,88.2,2.4631,6.0,432.0,17.8,344.91,15.76,18.3
116,0.13158,0.0,10.01,0.0,0.547,6.176,72.5,2.7301,6.0,432.0,17.8,393.3,12.04,21.2
117,0.15098,0.0,10.01,0.0,0.547,6.021,82.6,2.7474,6.0,432.0,17.8,394.51,10.3,19.2
118,0.13058,0.0,10.01,0.0,0.547,5.872,73.1,2.4775,6.0,432.0,17.8,338.63,15.37,20.4
119,0.14476,0.0,10.01,0.0,0.547,5.731,65.2,2.7592,6.0,432.0,17.8,391.5,13.61,19.3
120,0.06899,0.0,25.65,0.0,0.581,5.87,69.7,2.2577,2.0,188.0,19.1,389.15,14.37,22.0
121,0.07165,0.0,25.65,0.0,0.581,6.004,84.1,2.1974,2.0,188.0,19.1,377.67,14.27,20.3
122,0.09299,0.0,25.65,0.0,0.581,5.961,92.9,2.0869,2.0,188.0,19.1,378.09,17.93,20.5
123,0.15038,0.0,25.65,0.0,0.581,5.856,97.0,1.9444,2.0,188.0,19.1,370.31,25.41,17.3
124,0.09849,0.0,25.65,0.0,0.581,5.879,95.8,2.0063,2.0,188.0,19.1,379.38,17.58,18.8
125,0.16902,0.0,25.65,0.0,0.581,5.986,88.4,1.9929,2.0,188.0,19.1,385.02,14.81,21.4
126,0.38735,0.0,25.65,0.0,0.581,5.613,95.6,1.7572,2.0,188.0,19.1,359.29,27.26,15.7
127,0.25915,0.0,21.89,0.0,0.624,5.693,96.0,1.7883,4.0,437.0,21.2,392.11,17.19,16.2
128,0.32543,0.0,21.89,0.0,0.624,6.431,98.8,1.8125,4.0,437.0,21.2,396.9,15.39,18.0
129,0.88125,0.0,21.89,0.0,0.624,5.637,94.7,1.9799,4.0,437.0,21.2,396.9,18.34,14.3
130,0.34006,0.0,21.89,0.0,0.624,6.458,98.9,2.1185,4.0,437.0,21.2,395.04,12.6,19.2
131,1.19294,0.0,21.89,0.0,0.624,6.326,97.7,2.271,4.0,437.0,21.2,396.9,12.26,19.6
132,0.59005,0.0,21.89,0.0,0.624,6.372,97.9,2.3274,4.0,437.0,21.2,385.76,11.12,23.0
133,0.32982,0.0,21.89,0.0,0.624,5.822,95.4,2.4699,4.0,437.0,21.2,388.69,15.03,18.4
134,0.97617,0.0,21.89,0.0,0.624,5.757,98.4,2.346,4.0,437.0,21.2,262.76,17.31,15.6
135,0.55778,0.0,21.89,0.0,0.624,6.335,98.2,2.1107,4.0,437.0,21.2,394.67,16.96,18.1
136,0.32264,0.0,21.89,0.0,0.624,5.942,93.5,1.9669,4.0,437.0,21.2,378.25,16.9,17.4
137,0.35233,0.0,21.89,0.0,0.624,6.454,98.4,1.8498,4.0,437.0,21.2,394.08,14.59,17.1
138,0.2498,0.0,21.89,0.0,0.624,5.857,98.2,1.6686,4.0,437.0,21.2,392.04,21.32,13.3
139,0.54452,0.0,21.89,0.0,0.624,6.151,97.9,1.6687,4.0,437.0,21.2,396.9,18.46,17.8
140,0.2909,0.0,21.89,0.0,0.624,6.174,93.6,1.6119,4.0,437.0,21.2,388.08,24.16,14.0
141,1.62864,0.0,21.89,0.0,0.624,5.019,100.0,1.4394,4.0,437.0,21.2,396.9,34.41,14.4
142,3.32105,0.0,19.58,1.0,0.871,5.403,100.0,1.3216,5.0,403.0,14.7,396.9,26.82,13.4
143,4.0974,0.0,19.58,0.0,0.871,5.468,100.0,1.4118,5.0,403.0,14.7,396.9,26.42,15.6
144,2.77974,0.0,19.58,0.0,0.871,4.903,97.8,1.3459,5.0,403.0,14.7,396.9,29.29,11.8
145,2.37934,0.0,19.58,0.0,0.871,6.13,100.0,1.4191,5.0,403.0,14.7,172.91,27.8,13.8
146,2.15505,0.0,19.58,0.0,0.871,5.628,100.0,1.5166,5.0,403.0,14.7,169.27,16.65,15.6
147,2.36862,0.0,19.58,0.0,0.871,4.926,95.7,1.4608,5.0,403.0,14.7,391.71,29.53,14.6
148,2.33099,0.0,19.58,0.0,0.871,5.186,93.8,1.5296,5.0,403.0,14.7,356.99,28.32,17.8
149,2.73397,0.0,19.58,0.0,0.871,5.597,94.9,1.5257,5.0,403.0,14.7,351.85,21.45,15.4
150,1.6566,0.0,19.58,0.0,0.871,6.122,97.3,1.618,5.0,403.0,14.7,372.8,14.1,21.5
151,1.49632,0.0,19.58,0.0,0.871,5.404,100.0,1.5916,5.0,403.0,14.7,341.6,13.28,19.6
152,1.12658,0.0,19.58,1.0,0.871,5.012,88.0,1.6102,5.0,403.0,14.7,343.28,12.12,15.3
153,2.14918,0.0,19.58,0.0,0.871,5.709,98.5,1.6232,5.0,403.0,14.7,261.95,15.79,19.4
154,1.41385,0.0,19.58,1.0,0.871,6.129,96.0,1.7494,5.0,403.0,14.7,321.02,15.12,17.0
155,3.53501,0.0,19.58,1.0,0.871,6.152,82.6,1.7455,5.0,403.0,14.7,88.01,15.02,15.6
156,2.44668,0.0,19.58,0.0,0.871,5.272,94.0,1.7364,5.0,403.0,14.7,88.63,16.14,13.1
157,1.22358,0.0,19.58,0.0,0.605,6.943,97.4,1.8773,5.0,403.0,14.7,363.43,4.59,41.3
158,1.34284,0.0,19.58,0.0,0.605,6.066,100.0,1.7573,5.0,403.0,14.7,353.89,6.43,24.3
159,1.42502,0.0,19.58,0.0,0.871,6.51,100.0,1.7659,5.0,403.0,14.7,364.31,7.39,23.3
160,1.27346,0.0,19.58,1.0,0.605,6.25,92.6,1.7984,5.0,403.0,14.7,338.92,5.5,27.0
161,1.46336,0.0,19.58,0.0,0.605,7.489,90.8,1.9709,5.0,403.0,14.7,374.43,1.73,50.0
162,1.83377,0.0,19.58,1.0,0.605,7.802,98.2,2.0407,5.0,403.0,14.7,389.61,1.92,50.0
163,1.51902,0.0,19.58,1.0,0.605,8.375,93.9,2.162,5.0,403.0,14.7,388.45,3.32,50.0
164,2.24236,0.0,19.58,0.0,0.605,5.854,91.8,2.422,5.0,403.0,14.7,395.11,11.64,22.7
165,2.924,0.0,19.58,0.0,0.605,6.101,93.0,2.2834,5.0,403.0,14.7,240.16,9.81,25.0
166,2.01019,0.0,19.58,0.0,0.605,7.929,96.2,2.0459,5.0,403.0,14.7,369.3,3.7,50.0
167,1.80028,0.0,19.58,0.0,0.605,5.877,79.2,2.4259,5.0,403.0,14.7,227.61,12.14,23.8
168,2.3004,0.0,19.58,0.0,0.605,6.319,96.1,2.1,5.0,403.0,14.7,297.09,11.1,23.8
169,2.44953,0.0,19.58,0.0,0.605,6.402,95.2,2.2625,5.0,403.0,14.7,330.04,11.32,22.3
170,1.20742,0.0,19.58,0.0,0.605,5.875,94.6,2.4259,5.0,403.0,14.7,292.29,14.43,17.4
171,2.3139,0.0,19.58,0.0,0.605,5.88,97.3,2.3887,5.0,403.0,14.7,348.13,12.03,19.1
172,0.13914,0.0,4.05,0.0,0.51,5.572,88.5,2.5961,5.0,296.0,16.6,396.9,14.69,23.1
173,0.09178,0.0,4.05,0.0,0.51,6.416,84.1,2.6463,5.0,296.0,16.6,395.5,9.04,23.6
174,0.08447,0.0,4.05,0.0,0.51,5.859,68.7,2.7019,5.0,296.0,16.6,393.23,9.64,22.6
175,0.06664,0.0,4.05,0.0,0.51,6.546,33.1,3.1323,5.0,296.0,16.6,390.96,5.33,29.4
176,0.07022,0.0,4.05,0.0,0.51,6.02,47.2,3.5549,5.0,296.0,16.6,393.23,10.11,23.2
177,0.05425,0.0,4.05,0.0,0.51,6.315,73.4,3.3175,5.0,296.0,16.6,395.6,6.29,24.6
178,0.06642,0.0,4.05,0.0,0.51,6.86,74.4,2.9153,5.0,296.0,16.6,391.27,6.92,29.9
179,0.0578,0.0,2.46,0.0,0.488,6.98,58.4,2.829,3.0,193.0,17.8,396.9,5.04,37.2
180,0.06588,0.0,2.46,0.0,0.488,7.765,83.3,2.741,3.0,193.0,17.8,395.56,7.56,39.8
181,0.06888,0.0,2.46,0.0,0.488,6.144,62.2,2.5979,3.0,193.0,17.8,396.9,9.45,36.2
182,0.09103,0.0,2.46,0.0,0.488,7.155,92.2,2.7006,3.0,193.0,17.8,394.12,4.82,37.9
183,0.10008,0.0,2.46,0.0,0.488,6.563,95.6,2.847,3.0,193.0,17.8,396.9,5.68,32.5
184,0.08308,0.0,2.46,0.0,0.488,5.604,89.8,2.9879,3.0,193.0,17.8,391.0,13.98,26.4
185,0.06047,0.0,2.46,0.0,0.488,6.153,68.8,3.2797,3.0,193.0,17.8,387.11,13.15,29.6
186,0.05602,0.0,2.46,0.0,0.488,7.831,53.6,3.1992,3.0,193.0,17.8,392.63,4.45,50.0
187,0.07875,45.0,3.44,0.0,0.437,6.782,41.1,3.7886,5.0,398.0,15.2,393.87,6.68,32.0
188,0.12579,45.0,3.44,0.0,0.437,6.556,29.1,4.5667,5.0,398.0,15.2,382.84,4.56,29.8
189,0.0837,45.0,3.44,0.0,0.437,7.185,38.9,4.5667,5.0,398.0,15.2,396.9,5.39,34.9
190,0.09068,45.0,3.44,0.0,0.437,6.951,21.5,6.4798,5.0,398.0,15.2,377.68,5.1,37.0
191,0.06911,45.0,3.44,0.0,0.437,6.739,30.8,6.4798,5.0,398.0,15.2,389.71,4.69,30.5
192,0.08664,45.0,3.44,0.0,0.437,7.178,26.3,6.4798,5.0,398.0,15.2,390.49,2.87,36.4
193,0.02187,60.0,2.93,0.0,0.401,6.8,9.9,6.2196,1.0,265.0,15.6,393.37,5.03,31.1
194,0.01439,60.0,2.93,0.0,0.401,6.604,18.8,6.2196,1.0,265.0,15.6,376.7,4.38,29.1
195,0.01381,80.0,0.46,0.0,0.422,7.875,32.0,5.6484,4.0,255.0,14.4,394.23,2.97,50.0
196,0.04011,80.0,1.52,0.0,0.404,7.287,34.1,7.309,2.0,329.0,12.6,396.9,4.08,33.3
197,0.04666,80.0,1.52,0.0,0.404,7.107,36.6,7.309,2.0,329.0,12.6,354.31,8.61,30.3
198,0.03768,80.0,1.52,0.0,0.404,7.274,38.3,7.309,2.0,329.0,12.6,392.2,6.62,34.6
199,0.0315,95.0,1.47,0.0,0.403,6.975,15.3,7.6534,3.0,402.0,17.0,396.9,4.56,34.9
200,0.01778,95.0,1.47,0.0,0.403,7.135,13.9,7.6534,3.0,402.0,17.0,384.3,4.45,32.9
201,0.03445,82.5,2.03,0.0,0.415,6.162,38.4,6.27,2.0,348.0,14.7,393.77,7.43,24.1
202,0.02177,82.5,2.03,0.0,0.415,7.61,15.7,6.27,2.0,348.0,14.7,395.38,3.11,42.3
203,0.0351,95.0,2.68,0.0,0.4161,7.853,33.2,5.118,4.0,224.0,14.7,392.78,3.81,48.5
204,0.02009,95.0,2.68,0.0,0.4161,8.034,31.9,5.118,4.0,224.0,14.7,390.55,2.88,50.0
205,0.13642,0.0,10.59,0.0,0.489,5.891,22.3,3.9454,4.0,277.0,18.6,396.9,10.87,22.6
206,0.22969,0.0,10.59,0.0,0.489,6.326,52.5,4.3549,4.0,277.0,18.6,394.87,10.97,24.4
207,0.25199,0.0,10.59,0.0,0.489,5.783,72.7,4.3549,4.0,277.0,18.6,389.43,18.06,22.5
208,0.13587,0.0,10.59,1.0,0.489,6.064,59.1,4.2392,4.0,277.0,18.6,381.32,14.66,24.4
209,0.43571,0.0,10.59,1.0,0.489,5.344,100.0,3.875,4.0,277.0,18.6,396.9,23.09,20.0
210,0.17446,0.0,10.59,1.0,0.489,5.96,92.1,3.8771,4.0,277.0,18.6,393.25,17.27,21.7
211,0.37578,0.0,10.59,1.0,0.489,5.404,88.6,3.665,4.0,277.0,18.6,395.24,23.98,19.3
212,0.21719,0.0,10.59,1.0,0.489,5.807,53.8,3.6526,4.0,277.0,18.6,390.94,16.03,22.4
213,0.14052,0.0,10.59,0.0,0.489,6.375,32.3,3.9454,4.0,277.0,18.6,385.81,9.38,28.1
214,0.28955,0.0,10.59,0.0,0.489,5.412,9.8,3.5875,4.0,277.0,18.6,348.93,29.55,23.7
215,0.19802,0.0,10.59,0.0,0.489,6.182,42.4,3.9454,4.0,277.0,18.6,393.63,9.47,25.0
216,0.0456,0.0,13.89,1.0,0.55,5.888,56.0,3.1121,5.0,276.0,16.4,392.8,13.51,23.3
217,0.07013,0.0,13.89,0.0,0.55,6.642,85.1,3.4211,5.0,276.0,16.4,392.78,9.69,28.7
218,0.11069,0.0,13.89,1.0,0.55,5.951,93.8,2.8893,5.0,276.0,16.4,396.9,17.92,21.5
219,0.11425,0.0,13.89,1.0,0.55,6.373,92.4,3.3633,5.0,276.0,16.4,393.74,10.5,23.0
220,0.35809,0.0,6.2,1.0,0.507,6.951,88.5,2.8617,8.0,307.0,17.4,391.7,9.71,26.7
221,0.40771,0.0,6.2,1.0,0.507,6.164,91.3,3.048,8.0,307.0,17.4,395.24,21.46,21.7
222,0.62356,0.0,6.2,1.0,0.507,6.879,77.7,3.2721,8.0,307.0,17.4,390.39,9.93,27.5
223,0.6147,0.0,6.2,0.0,0.507,6.618,80.8,3.2721,8.0,307.0,17.4,396.9,7.6,30.1
224,0.31533,0.0,6.2,0.0,0.504,8.266,78.3,2.8944,8.0,307.0,17.4,385.05,4.14,44.8
225,0.52693,0.0,6.2,0.0,0.504,8.725,83.0,2.8944,8.0,307.0,17.4,382.0,4.63,50.0
226,0.38214,0.0,6.2,0.0,0.504,8.04,86.5,3.2157,8.0,307.0,17.4,387.38,3.13,37.6
227,0.41238,0.0,6.2,0.0,0.504,7.163,79.9,3.2157,8.0,307.0,17.4,372.08,6.36,31.6
228,0.29819,0.0,6.2,0.0,0.504,7.686,17.0,3.3751,8.0,307.0,17.4,377.51,3.92,46.7
229,0.44178,0.0,6.2,0.0,0.504,6.552,21.4,3.3751,8.0,307.0,17.4,380.34,3.76,31.5
230,0.537,0.0,6.2,0.0,0.504,5.981,68.1,3.6715,8.0,307.0,17.4,378.35,11.65,24.3
231,0.46296,0.0,6.2,0.0,0.504,7.412,76.9,3.6715,8.0,307.0,17.4,376.14,5.25,31.7
232,0.57529,0.0,6.2,0.0,0.507,8.337,73.3,3.8384,8.0,307.0,17.4,385.91,2.47,41.7
233,0.33147,0.0,6.2,0.0,0.507,8.247,70.4,3.6519,8.0,307.0,17.4,378.95,3.95,48.3
234,0.44791,0.0,6.2,1.0,0.507,6.726,66.5,3.6519,8.0,307.0,17.4,360.2,8.05,29.0
235,0.33045,0.0,6.2,0.0,0.507,6.086,61.5,3.6519,8.0,307.0,17.4,376.75,10.88,24.0
236,0.52058,0.0,6.2,1.0,0.507,6.631,76.5,4.148,8.0,307.0,17.4,388.45,9.54,25.1
237,0.51183,0.0,6.2,0.0,0.507,7.358,71.6,4.148,8.0,307.0,17.4,390.07,4.73,31.5
238,0.08244,30.0,4.93,0.0,0.428,6.481,18.5,6.1899,6.0,300.0,16.6,379.41,6.36,23.7
239,0.09252,30.0,4.93,0.0,0.428,6.606,42.2,6.1899,6.0,300.0,16.6,383.78,7.37,23.3
240,0.11329,30.0,4.93,0.0,0.428,6.897,54.3,6.3361,6.0,300.0,16.6,391.25,11.38,22.0
241,0.10612,30.0,4.93,0.0,0.428,6.095,65.1,6.3361,6.0,300.0,16.6,394.62,12.4,20.1
242,0.1029,30.0,4.93,0.0,0.428,6.358,52.9,7.0355,6.0,300.0,16.6,372.75,11.22,22.2
243,0.12757,30.0,4.93,0.0,0.428,6.393,7.8,7.0355,6.0,300.0,16.6,374.71,5.19,23.7
244,0.20608,22.0,5.86,0.0,0.431,5.593,76.5,7.9549,7.0,330.0,19.1,372.49,12.5,17.6
245,0.19133,22.0,5.86,0.0,0.431,5.605,70.2,7.9549,7.0,330.0,19.1,389.13,18.46,18.5
246,0.33983,22.0,5.86,0.0,0.431,6.108,34.9,8.0555,7.0,330.0,19.1,390.18,9.16,24.3
247,0.19657,22.0,5.86,0.0,0.431,6.226,79.2,8.0555,7.0,330.0,19.1,376.14,10.15,20.5
248,0.16439,22.0,5.86,0.0,0.431,6.433,49.1,7.8265,7.0,330.0,19.1,374.71,9.52,24.5
249,0.19073,22.0,5.86,0.0,0.431,6.718,17.5,7.8265,7.0,330.0,19.1,393.74,6.56,26.2
250,0.1403,22.0,5.86,0.0,0.431,6.487,13.0,7.3967,7.0,330.0,19.1,396.28,5.9,24.4
251,0.21409,22.0,5.86,0.0,0.431,6.438,8.9,7.3967,7.0,330.0,19.1,377.07,3.59,24.8
252,0.08221,22.0,5.86,0.0,0.431,6.957,6.8,8.9067,7.0,330.0,19.1,386.09,3.53,29.6
253,0.36894,22.0,5.86,0.0,0.431,8.259,8.4,8.9067,7.0,330.0,19.1,396.9,3.54,42.8
254,0.04819,80.0,3.64,0.0,0.392,6.108,32.0,9.2203,1.0,315.0,16.4,392.89,6.57,21.9
255,0.03548,80.0,3.64,0.0,0.392,5.876,19.1,9.2203,1.0,315.0,16.4,395.18,9.25,20.9
256,0.01538,90.0,3.75,0.0,0.394,7.454,34.2,6.3361,3.0,244.0,15.9,386.34,3.11,44.0
257,0.61154,20.0,3.97,0.0,0.647,8.704,86.9,1.801,5.0,264.0,13.0,389.7,5.12,50.0
258,0.66351,20.0,3.97,0.0,0.647,7.333,100.0,1.8946,5.0,264.0,13.0,383.29,7.79,36.0
259,0.65665,20.0,3.97,0.0,0.647,6.842,100.0,2.0107,5.0,264.0,13.0,391.93,6.9,30.1
260,0.54011,20.0,3.97,0.0,0.647,7.203,81.8,2.1121,5.0,264.0,13.0,392.8,9.59,33.8
261,0.53412,20.0,3.97,0.0,0.647,7.52,89.4,2.1398,5.0,264.0,13.0,388.37,7.26,43.1
262,0.52014,20.0,3.97,0.0,0.647,8.398,91.5,2.2885,5.0,264.0,13.0,386.86,5.91,48.8
263,0.82526,20.0,3.97,0.0,0.647,7.327,94.5,2.0788,5.0,264.0,13.0,393.42,11.25,31.0
264,0.55007,20.0,3.97,0.0,0.647,7.206,91.6,1.9301,5.0,264.0,13.0,387.89,8.1,36.5
265,0.76162,20.0,3.97,0.0,0.647,5.56,62.8,1.9865,5.0,264.0,13.0,392.4,10.45,22.8
266,0.7857,20.0,3.97,0.0,0.647,7.014,84.6,2.1329,5.0,264.0,13.0,384.07,14.79,30.7
267,0.57834,20.0,3.97,0.0,0.575,8.297,67.0,2.4216,5.0,264.0,13.0,384.54,7.44,50.0
268,0.5405,20.0,3.97,0.0,0.575,7.47,52.6,2.872,5.0,264.0,13.0,390.3,3.16,43.5
269,0.09065,20.0,6.96,1.0,0.464,5.92,61.5,3.9175,3.0,223.0,18.6,391.34,13.65,20.7
270,0.29916,20.0,6.96,0.0,0.464,5.856,42.1,4.429,3.0,223.0,18.6,388.65,13.0,21.1
271,0.16211,20.0,6.96,0.0,0.464,6.24,16.3,4.429,3.0,223.0,18.6,396.9,6.59,25.2
272,0.1146,20.0,6.96,0.0,0.464,6.538,58.7,3.9175,3.0,223.0,18.6,394.96,7.73,24.4
273,0.22188,20.0,6.96,1.0,0.464,7.691,51.8,4.3665,3.0,223.0,18.6,390.77,6.58,35.2
274,0.05644,40.0,6.41,1.0,0.447,6.758,32.9,4.0776,4.0,254.0,17.6,396.9,3.53,32.4
275,0.09604,40.0,6.41,0.0,0.447,6.854,42.8,4.2673,4.0,254.0,17.6,396.9,2.98,32.0
276,0.10469,40.0,6.41,1.0,0.447,7.267,49.0,4.7872,4.0,254.0,17.6,389.25,6.05,33.2
277,0.06127,40.0,6.41,1.0,0.447,6.826,27.6,4.8628,4.0,254.0,17.6,393.45,4.16,33.1
278,0.07978,40.0,6.41,0.0,0.447,6.482,32.1,4.1403,4.0,254.0,17.6,396.9,7.19,29.1
279,0.21038,20.0,3.33,0.0,0.4429,6.812,32.2,4.1007,5.0,216.0,14.9,396.9,4.85,35.1
280,0.03578,20.0,3.33,0.0,0.4429,7.82,64.5,4.6947,5.0,216.0,14.9,387.31,3.76,45.4
281,0.03705,20.0,3.33,0.0,0.4429,6.968,37.2,5.2447,5.0,216.0,14.9,392.23,4.59,35.4
282,0.06129,20.0,3.33,1.0,0.4429,7.645,49.7,5.2119,5.0,216.0,14.9,377.07,3.01,46.0
283,0.01501,90.0,1.21,1.0,0.401,7.923,24.8,5.885,1.0,198.0,13.6,395.52,3.16,50.0
284,0.00906,90.0,2.97,0.0,0.4,7.088,20.8,7.3073,1.0,285.0,15.3,394.72,7.85,32.2
285,0.01096,55.0,2.25,0.0,0.389,6.453,31.9,7.3073,1.0,300.0,15.3,394.72,8.23,22.0
286,0.01965,80.0,1.76,0.0,0.385,6.23,31.5,9.0892,1.0,241.0,18.2,341.6,12.93,20.1
287,0.03871,52.5,5.32,0.0,0.405,6.209,31.3,7.3172,6.0,293.0,16.6,396.9,7.14,23.2
288,0.0459,52.5,5.32,0.0,0.405,6.315,45.6,7.3172,6.0,293.0,16.6,396.9,7.6,22.3
289,0.04297,52.5,5.32,0.0,0.405,6.565,22.9,7.3172,6.0,293.0,16.6,371.72,9.51,24.8
290,0.03502,80.0,4.95,0.0,0.411,6.861,27.9,5.1167,4.0,245.0,19.2,396.9,3.33,28.5
291,0.07886,80.0,4.95,0.0,0.411,7.148,27.7,5.1167,4.0,245.0,19.2,396.9,3.56,37.3
292,0.03615,80.0,4.95,0.0,0.411,6.63,23.4,5.1167,4.0,245.0,19.2,396.9,4.7,27.9
293,0.08265,0.0,13.92,0.0,0.437,6.127,18.4,5.5027,4.0,289.0,16.0,396.9,8.58,23.9
294,0.08199,0.0,13.92,0.0,0.437,6.009,42.3,5.5027,4.0,289.0,16.0,396.9,10.4,21.7
295,0.12932,0.0,13.92,0.0,0.437,6.678,31.1,5.9604,4.0,289.0,16.0,396.9,6.27,28.6
296,0.05372,0.0,13.92,0.0,0.437,6.549,51.0,5.9604,4.0,289.0,16.0,392.85,7.39,27.1
297,0.14103,0.0,13.92,0.0,0.437,5.79,58.0,6.32,4.0,289.0,16.0,396.9,15.84,20.3
298,0.06466,70.0,2.24,0.0,0.4,6.345,20.1,7.8278,5.0,358.0,14.8,368.24,4.97,22.5
299,0.05561,70.0,2.24,0.0,0.4,7.041,10.0,7.8278,5.0,358.0,14.8,371.58,4.74,29.0
300,0.04417,70.0,2.24,0.0,0.4,6.871,47.4,7.8278,5.0,358.0,14.8,390.86,6.07,24.8
301,0.03537,34.0,6.09,0.0,0.433,6.59,40.4,5.4917,7.0,329.0,16.1,395.75,9.5,22.0
302,0.09266,34.0,6.09,0.0,0.433,6.495,18.4,5.4917,7.0,329.0,16.1,383.61,8.67,26.4
303,0.1,34.0,6.09,0.0,0.433,6.982,17.7,5.4917,7.0,329.0,16.1,390.43,4.86,33.1
304,0.05515,33.0,2.18,0.0,0.472,7.236,41.1,4.022,7.0,222.0,18.4,393.68,6.93,36.1
305,0.05479,33.0,2.18,0.0,0.472,6.616,58.1,3.37,7.0,222.0,18.4,393.36,8.93,28.4
306,0.07503,33.0,2.18,0.0,0.472,7.42,71.9,3.0992,7.0,222.0,18.4,396.9,6.47,33.4
307,0.04932,33.0,2.18,0.0,0.472,6.849,70.3,3.1827,7.0,222.0,18.4,396.9,7.53,28.2
308,0.49298,0.0,9.9,0.0,0.544,6.635,82.5,3.3175,4.0,304.0,18.4,396.9,4.54,22.8
309,0.3494,0.0,9.9,0.0,0.544,5.972,76.7,3.1025,4.0,304.0,18.4,396.24,9.97,20.3
310,2.63548,0.0,9.9,0.0,0.544,4.973,37.8,2.5194,4.0,304.0,18.4,350.45,12.64,16.1
311,0.79041,0.0,9.9,0.0,0.544,6.122,52.8,2.6403,4.0,304.0,18.4,396.9,5.98,22.1
312,0.26169,0.0,9.9,0.0,0.544,6.023,90.4,2.834,4.0,304.0,18.4,396.3,11.72,19.4
313,0.26938,0.0,9.9,0.0,0.544,6.266,82.8,3.2628,4.0,304.0,18.4,393.39,7.9,21.6
314,0.3692,0.0,9.9,0.0,0.544,6.567,87.3,3.6023,4.0,304.0,18.4,395.69,9.28,23.8
315,0.25356,0.0,9.9,0.0,0.544,5.705,77.7,3.945,4.0,304.0,18.4,396.42,11.5,16.2
316,0.31827,0.0,9.9,0.0,0.544,5.914,83.2,3.9986,4.0,304.0,18.4,390.7,18.33,17.8
317,0.24522,0.0,9.9,0.0,0.544,5.782,71.7,4.0317,4.0,304.0,18.4,396.9,15.94,19.8
318,0.40202,0.0,9.9,0.0,0.544,6.382,67.2,3.5325,4.0,304.0,18.4,395.21,10.36,23.1
319,0.47547,0.0,9.9,0.0,0.544,6.113,58.8,4.0019,4.0,304.0,18.4,396.23,12.73,21.0
320,0.1676,0.0,7.38,0.0,0.493,6.426,52.3,4.5404,5.0,287.0,19.6,396.9,7.2,23.8
321,0.18159,0.0,7.38,0.0,0.493,6.376,54.3,4.5404,5.0,287.0,19.6,396.9,6.87,23.1
322,0.35114,0.0,7.38,0.0,0.493,6.041,49.9,4.7211,5.0,287.0,19.6,396.9,7.7,20.4
323,0.28392,0.0,7.38,0.0,0.493,5.708,74.3,4.7211,5.0,287.0,19.6,391.13,11.74,18.5
324,0.34109,0.0,7.38,0.0,0.493,6.415,40.1,4.7211,5.0,287.0,19.6,396.9,6.12,25.0
325,0.19186,0.0,7.38,0.0,0.493,6.431,14.7,5.4159,5.0,287.0,19.6,393.68,5.08,24.6
326,0.30347,0.0,7.38,0.0,0.493,6.312,28.9,5.4159,5.0,287.0,19.6,396.9,6.15,23.0
327,0.24103,0.0,7.38,0.0,0.493,6.083,43.7,5.4159,5.0,287.0,19.6,396.9,12.79,22.2
328,0.06617,0.0,3.24,0.0,0.46,5.868,25.8,5.2146,4.0,430.0,16.9,382.44,9.97,19.3
329,0.06724,0.0,3.24,0.0,0.46,6.333,17.2,5.2146,4.0,430.0,16.9,375.21,7.34,22.6
330,0.04544,0.0,3.24,0.0,0.46,6.144,32.2,5.8736,4.0,430.0,16.9,368.57,9.09,19.8
331,0.05023,35.0,6.06,0.0,0.4379,5.706,28.4,6.6407,1.0,304.0,16.9,394.02,12.43,17.1
332,0.03466,35.0,6.06,0.0,0.4379,6.031,23.3,6.6407,1.0,304.0,16.9,362.25,7.83,19.4
333,0.05083,0.0,5.19,0.0,0.515,6.316,38.1,6.4584,5.0,224.0,20.2,389.71,5.68,22.2
334,0.03738,0.0,5.19,0.0,0.515,6.31,38.5,6.4584,5.0,224.0,20.2,389.4,6.75,20.7
335,0.03961,0.0,5.19,0.0,0.515,6.037,34.5,5.9853,5.0,224.0,20.2,396.9,8.01,21.1
336,0.03427,0.0,5.19,0.0,0.515,5.869,46.3,5.2311,5.0,224.0,20.2,396.9,9.8,19.5
337,0.03041,0.0,5.19,0.0,0.515,5.895,59.6,5.615,5.0,224.0,20.2,394.81,10.56,18.5
338,0.03306,0.0,5.19,0.0,0.515,6.059,37.3,4.8122,5.0,224.0,20.2,396.14,8.51,20.6
339,0.05497,0.0,5.19,0.0,0.515,5.985,45.4,4.8122,5.0,224.0,20.2,396.9,9.74,19.0
340,0.06151,0.0,5.19,0.0,0.515,5.968,58.5,4.8122,5.0,224.0,20.2,396.9,9.29,18.7
341,0.01301,35.0,1.52,0.0,0.442,7.241,49.3,7.0379,1.0,284.0,15.5,394.74,5.49,32.7
342,0.02498,0.0,1.89,0.0,0.518,6.54,59.7,6.2669,1.0,422.0,15.9,389.96,8.65,16.5
343,0.02543,55.0,3.78,0.0,0.484,6.696,56.4,5.7321,5.0,370.0,17.6,396.9,7.18,23.9
344,0.03049,55.0,3.78,0.0,0.484,6.874,28.1,6.4654,5.0,370.0,17.6,387.97,4.61,31.2
345,0.03113,0.0,4.39,0.0,0.442,6.014,48.5,8.0136,3.0,352.0,18.8,385.64,10.53,17.5
346,0.06162,0.0,4.39,0.0,0.442,5.898,52.3,8.0136,3.0,352.0,18.8,364.61,12.67,17.2
347,0.0187,85.0,4.15,0.0,0.429,6.516,27.7,8.5353,4.0,351.0,17.9,392.43,6.36,23.1
348,0.01501,80.0,2.01,0.0,0.435,6.635,29.7,8.344,4.0,280.0,17.0,390.94,5.99,24.5
349,0.02899,40.0,1.25,0.0,0.429,6.939,34.5,8.7921,1.0,335.0,19.7,389.85,5.89,26.6
350,0.06211,40.0,1.25,0.0,0.429,6.49,44.4,8.7921,1.0,335.0,19.7,396.9,5.98,22.9
351,0.0795,60.0,1.69,0.0,0.411,6.579,35.9,10.7103,4.0,411.0,18.3,370.78,5.49,24.1
352,0.07244,60.0,1.69,0.0,0.411,5.884,18.5,10.7103,4.0,411.0,18.3,392.33,7.79,18.6
353,0.01709,90.0,2.02,0.0,0.41,6.728,36.1,12.1265,5.0,187.0,17.0,384.46,4.5,30.1
354,0.04301,80.0,1.91,0.0,0.413,5.663,21.9,10.5857,4.0,334.0,22.0,382.8,8.05,18.2
355,0.10659,80.0,1.91,0.0,0.413,5.936,19.5,10.5857,4.0,334.0,22.0,376.04,5.57,20.6
356,8.98296,0.0,18.1,1.0,0.77,6.212,97.4,2.1222,24.0,666.0,20.2,377.73,17.6,17.8
357,3.8497,0.0,18.1,1.0,0.77,6.395,91.0,2.5052,24.0,666.0,20.2,391.34,13.27,21.7
358,5.20177,0.0,18.1,1.0,0.77,6.127,83.4,2.7227,24.0,666.0,20.2,395.43,11.48,22.7
359,4.26131,0.0,18.1,0.0,0.77,6.112,81.3,2.5091,24.0,666.0,20.2,390.74,12.67,22.6
360,4.54192,0.0,18.1,0.0,0.77,6.398,88.0,2.5182,24.0,666.0,20.2,374.56,7.79,25.0
361,3.83684,0.0,18.1,0.0,0.77,6.251,91.1,2.2955,24.0,666.0,20.2,350.65,14.19,19.9
362,3.67822,0.0,18.1,0.0,0.77,5.362,96.2,2.1036,24.0,666.0,20.2,380.79,10.19,20.8
363,4.22239,0.0,18.1,1.0,0.77,5.803,89.0,1.9047,24.0,666.0,20.2,353.04,14.64,16.8
364,3.47428,0.0,18.1,1.0,0.718,8.78,82.9,1.9047,24.0,666.0,20.2,354.55,5.29,21.9
365,4.55587,0.0,18.1,0.0,0.718,3.561,87.9,1.6132,24.0,666.0,20.2,354.7,7.12,27.5
366,3.69695,0.0,18.1,0.0,0.718,4.963,91.4,1.7523,24.0,666.0,20.2,316.03,14.0,21.9
367,13.5222,0.0,18.1,0.0,0.631,3.863,100.0,1.5106,24.0,666.0,20.2,131.42,13.33,23.1
368,4.89822,0.0,18.1,0.0,0.631,4.97,100.0,1.3325,24.0,666.0,20.2,375.52,3.26,50.0
369,5.66998,0.0,18.1,1.0,0.631,6.683,96.8,1.3567,24.0,666.0,20.2,375.33,3.73,50.0
370,6.53876,0.0,18.1,1.0,0.631,7.016,97.5,1.2024,24.0,666.0,20.2,392.05,2.96,50.0
371,9.2323,0.0,18.1,0.0,0.631,6.216,100.0,1.1691,24.0,666.0,20.2,366.15,9.53,50.0
372,8.26725,0.0,18.1,1.0,0.668,5.875,89.6,1.1296,24.0,666.0,20.2,347.88,8.88,50.0
373,11.1081,0.0,18.1,0.0,0.668,4.906,100.0,1.1742,24.0,666.0,20.2,396.9,34.77,13.8
374,18.4982,0.0,18.1,0.0,0.668,4.138,100.0,1.137,24.0,666.0,20.2,396.9,37.97,13.8
375,19.6091,0.0,18.1,0.0,0.671,7.313,97.9,1.3163,24.0,666.0,20.2,396.9,13.44,15.0
376,15.288,0.0,18.1,0.0,0.671,6.649,93.3,1.3449,24.0,666.0,20.2,363.02,23.24,13.9
377,9.82349,0.0,18.1,0.0,0.671,6.794,98.8,1.358,24.0,666.0,20.2,396.9,21.24,13.3
378,23.6482,0.0,18.1,0.0,0.671,6.38,96.2,1.3861,24.0,666.0,20.2,396.9,23.69,13.1
379,17.8667,0.0,18.1,0.0,0.671,6.223,100.0,1.3861,24.0,666.0,20.2,393.74,21.78,10.2
380,88.9762,0.0,18.1,0.0,0.671,6.968,91.9,1.4165,24.0,666.0,20.2,396.9,17.21,10.4
381,15.8744,0.0,18.1,0.0,0.671,6.545,99.1,1.5192,24.0,666.0,20.2,396.9,21.08,10.9
382,9.18702,0.0,18.1,0.0,0.7,5.536,100.0,1.5804,24.0,666.0,20.2,396.9,23.6,11.3
383,7.99248,0.0,18.1,0.0,0.7,5.52,100.0,1.5331,24.0,666.0,20.2,396.9,24.56,12.3
384,20.0849,0.0,18.1,0.0,0.7,4.368,91.2,1.4395,24.0,666.0,20.2,285.83,30.63,8.8
385,16.8118,0.0,18.1,0.0,0.7,5.277,98.1,1.4261,24.0,666.0,20.2,396.9,30.81,7.2
386,24.3938,0.0,18.1,0.0,0.7,4.652,100.0,1.4672,24.0,666.0,20.2,396.9,28.28,10.5
387,22.5971,0.0,18.1,0.0,0.7,5.0,89.5,1.5184,24.0,666.0,20.2,396.9,31.99,7.4
388,14.3337,0.0,18.1,0.0,0.7,4.88,100.0,1.5895,24.0,666.0,20.2,372.92,30.62,10.2
389,8.15174,0.0,18.1,0.0,0.7,5.39,98.9,1.7281,24.0,666.0,20.2,396.9,20.85,11.5
390,6.96215,0.0,18.1,0.0,0.7,5.713,97.0,1.9265,24.0,666.0,20.2,394.43,17.11,15.1
391,5.29305,0.0,18.1,0.0,0.7,6.051,82.5,2.1678,24.0,666.0,20.2,378.38,18.76,23.2
392,11.5779,0.0,18.1,0.0,0.7,5.036,97.0,1.77,24.0,666.0,20.2,396.9,25.68,9.7
393,8.64476,0.0,18.1,0.0,0.693,6.193,92.6,1.7912,24.0,666.0,20.2,396.9,15.17,13.8
394,13.3598,0.0,18.1,0.0,0.693,5.887,94.7,1.7821,24.0,666.0,20.2,396.9,16.35,12.7
395,8.71675,0.0,18.1,0.0,0.693,6.471,98.8,1.7257,24.0,666.0,20.2,391.98,17.12,13.1
396,5.87205,0.0,18.1,0.0,0.693,6.405,96.0,1.6768,24.0,666.0,20.2,396.9,19.37,12.5
397,7.67202,0.0,18.1,0.0,0.693,5.747,98.9,1.6334,24.0,666.0,20.2,393.1,19.92,8.5
398,38.3518,0.0,18.1,0.0,0.693,5.453,100.0,1.4896,24.0,666.0,20.2,396.9,30.59,5.0
399,9.91655,0.0,18.1,0.0,0.693,5.852,77.8,1.5004,24.0,666.0,20.2,338.16,29.97,6.3
400,25.0461,0.0,18.1,0.0,0.693,5.987,100.0,1.5888,24.0,666.0,20.2,396.9,26.77,5.6
401,14.2362,0.0,18.1,0.0,0.693,6.343,100.0,1.5741,24.0,666.0,20.2,396.9,20.32,7.2
402,9.59571,0.0,18.1,0.0,0.693,6.404,100.0,1.639,24.0,666.0,20.2,376.11,20.31,12.1
403,24.8017,0.0,18.1,0.0,0.693,5.349,96.0,1.7028,24.0,666.0,20.2,396.9,19.77,8.3
404,41.5292,0.0,18.1,0.0,0.693,5.531,85.4,1.6074,24.0,666.0,20.2,329.46,27.38,8.5
405,67.9208,0.0,18.1,0.0,0.693,5.683,100.0,1.4254,24.0,666.0,20.2,384.97,22.98,5.0
406,20.7162,0.0,18.1,0.0,0.659,4.138,100.0,1.1781,24.0,666.0,20.2,370.22,23.34,11.9
407,11.9511,0.0,18.1,0.0,0.659,5.608,100.0,1.2852,24.0,666.0,20.2,332.09,12.13,27.9
408,7.40389,0.0,18.1,0.0,0.597,5.617,97.9,1.4547,24.0,666.0,20.2,314.64,26.4,17.2
409,14.4383,0.0,18.1,0.0,0.597,6.852,100.0,1.4655,24.0,666.0,20.2,179.36,19.78,27.5
410,51.1358,0.0,18.1,0.0,0.597,5.757,100.0,1.413,24.0,666.0,20.2,2.6,10.11,15.0
411,14.0507,0.0,18.1,0.0,0.597,6.657,100.0,1.5275,24.0,666.0,20.2,35.05,21.22,17.2
412,18.811,0.0,18.1,0.0,0.597,4.628,100.0,1.5539,24.0,666.0,20.2,28.79,34.37,17.9
413,28.6558,0.0,18.1,0.0,0.597,5.155,100.0,1.5894,24.0,666.0,20.2,210.97,20.08,16.3
414,45.7461,0.0,18.1,0.0,0.693,4.519,100.0,1.6582,24.0,666.0,20.2,88.27,36.98,7.0
415,18.0846,0.0,18.1,0.0,0.679,6.434,100.0,1.8347,24.0,666.0,20.2,27.25,29.05,7.2
416,10.8342,0.0,18.1,0.0,0.679,6.782,90.8,1.8195,24.0,666.0,20.2,21.57,25.79,7.5
417,25.9406,0.0,18.1,0.0,0.679,5.304,89.1,1.6475,24.0,666.0,20.2,127.36,26.64,10.4
418,73.5341,0.0,18.1,0.0,0.679,5.957,100.0,1.8026,24.0,666.0,20.2,16.45,20.62,8.8
419,11.8123,0.0,18.1,0.0,0.718,6.824,76.5,1.794,24.0,666.0,20.2,48.45,22.74,8.4
420,11.0874,0.0,18.1,0.0,0.718,6.411,100.0,1.8589,24.0,666.0,20.2,318.75,15.02,16.7
421,7.02259,0.0,18.1,0.0,0.718,6.006,95.3,1.8746,24.0,666.0,20.2,319.98,15.7,14.2
422,12.0482,0.0,18.1,0.0,0.614,5.648,87.6,1.9512,24.0,666.0,20.2,291.55,14.1,20.8
423,7.05042,0.0,18.1,0.0,0.614,6.103,85.1,2.0218,24.0,666.0,20.2,2.52,23.29,13.4
424,8.79212,0.0,18.1,0.0,0.584,5.565,70.6,2.0635,24.0,666.0,20.2,3.65,17.16,11.7
425,15.8603,0.0,18.1,0.0,0.679,5.896,95.4,1.9096,24.0,666.0,20.2,7.68,24.39,8.3
426,12.2472,0.0,18.1,0.0,0.584,5.837,59.7,1.9976,24.0,666.0,20.2,24.65,15.69,10.2
427,37.6619,0.0,18.1,0.0,0.679,6.202,78.7,1.8629,24.0,666.0,20.2,18.82,14.52,10.9
428,7.36711,0.0,18.1,0.0,0.679,6.193,78.1,1.9356,24.0,666.0,20.2,96.73,21.52,11.0
429,9.33889,0.0,18.1,0.0,0.679,6.38,95.6,1.9682,24.0,666.0,20.2,60.72,24.08,9.5
430,8.49213,0.0,18.1,0.0,0.584,6.348,86.1,2.0527,24.0,666.0,20.2,83.45,17.64,14.5
431,10.0623,0.0,18.1,0.0,0.584,6.833,94.3,2.0882,24.0,666.0,20.2,81.33,19.69,14.1
432,6.44405,0.0,18.1,0.0,0.584,6.425,74.8,2.2004,24.0,666.0,20.2,97.95,12.03,16.1
433,5.58107,0.0,18.1,0.0,0.713,6.436,87.9,2.3158,24.0,666.0,20.2,100.19,16.22,14.3
434,13.9134,0.0,18.1,0.0,0.713,6.208,95.0,2.2222,24.0,666.0,20.2,100.63,15.17,11.7
435,11.1604,0.0,18.1,0.0,0.74,6.629,94.6,2.1247,24.0,666.0,20.2,109.85,23.27,13.4
436,14.4208,0.0,18.1,0.0,0.74,6.461,93.3,2.0026,24.0,666.0,20.2,27.49,18.05,9.6
437,15.1772,0.0,18.1,0.0,0.74,6.152,100.0,1.9142,24.0,666.0,20.2,9.32,26.45,8.7
438,13.6781,0.0,18.1,0.0,0.74,5.935,87.9,1.8206,24.0,666.0,20.2,68.95,34.02,8.4
439,9.39063,0.0,18.1,0.0,0.74,5.627,93.9,1.8172,24.0,666.0,20.2,396.9,22.88,12.8
440,22.0511,0.0,18.1,0.0,0.74,5.818,92.4,1.8662,24.0,666.0,20.2,391.45,22.11,10.5
441,9.72418,0.0,18.1,0.0,0.74,6.406,97.2,2.0651,24.0,666.0,20.2,385.96,19.52,17.1
442,5.66637,0.0,18.1,0.0,0.74,6.219,100.0,2.0048,24.0,666.0,20.2,395.69,16.59,18.4
443,9.96654,0.0,18.1,0.0,0.74,6.485,100.0,1.9784,24.0,666.0,20.2,386.73,18.85,15.4
444,12.8023,0.0,18.1,0.0,0.74,5.854,96.6,1.8956,24.0,666.0,20.2,240.52,23.79,10.8
445,0.6718,0.0,18.1,0.0,0.74,6.459,94.8,1.9879,24.0,666.0,20.2,43.06,23.98,11.8
446,6.28807,0.0,18.1,0.0,0.74,6.341,96.4,2.072,24.0,666.0,20.2,318.01,17.79,14.9
447,9.92485,0.0,18.1,0.0,0.74,6.251,96.6,2.198,24.0,666.0,20.2,388.52,16.44,12.6
448,9.32909,0.0,18.1,0.0,0.713,6.185,98.7,2.2616,24.0,666.0,20.2,396.9,18.13,14.1
449,7.52601,0.0,18.1,0.0,0.713,6.417,98.3,2.185,24.0,666.0,20.2,304.21,19.31,13.0
450,6.71772,0.0,18.1,0.0,0.713,6.749,92.6,2.3236,24.0,666.0,20.2,0.32,17.44,13.4
451,5.44114,0.0,18.1,0.0,0.713,6.655,98.2,2.3552,24.0,666.0,20.2,355.29,17.73,15.2
452,5.09017,0.0,18.1,0.0,0.713,6.297,91.8,2.3682,24.0,666.0,20.2,385.09,17.27,16.1
453,8.24809,0.0,18.1,0.0,0.713,7.393,99.3,2.4527,24.0,666.0,20.2,375.87,16.74,17.8
454,9.51363,0.0,18.1,0.0,0.713,6.728,94.1,2.4961,24.0,666.0,20.2,6.68,18.71,14.9
455,4.75237,0.0,18.1,0.0,0.713,6.525,86.5,2.4358,24.0,666.0,20.2,50.92,18.13,14.1
456,4.66883,0.0,18.1,0.0,0.713,5.976,87.9,2.5806,24.0,666.0,20.2,10.48,19.01,12.7
457,8.20058,0.0,18.1,0.0,0.713,5.936,80.3,2.7792,24.0,666.0,20.2,3.5,16.94,13.5
458,7.75223,0.0,18.1,0.0,0.713,6.301,83.7,2.7831,24.0,666.0,20.2,272.21,16.23,14.9
459,6.80117,0.0,18.1,0.0,0.713,6.081,84.4,2.7175,24.0,666.0,20.2,396.9,14.7,20.0
460,4.81213,0.0,18.1,0.0,0.713,6.701,90.0,2.5975,24.0,666.0,20.2,255.23,16.42,16.4
461,3.69311,0.0,18.1,0.0,0.713,6.376,88.4,2.5671,24.0,666.0,20.2,391.43,14.65,17.7
462,6.65492,0.0,18.1,0.0,0.713,6.317,83.0,2.7344,24.0,666.0,20.2,396.9,13.99,19.5
463,5.82115,0.0,18.1,0.0,0.713,6.513,89.9,2.8016,24.0,666.0,20.2,393.82,10.29,20.2
464,7.83932,0.0,18.1,0.0,0.655,6.209,65.4,2.9634,24.0,666.0,20.2,396.9,13.22,21.4
465,3.1636,0.0,18.1,0.0,0.655,5.759,48.2,3.0665,24.0,666.0,20.2,334.4,14.13,19.9
466,3.77498,0.0,18.1,0.0,0.655,5.952,84.7,2.8715,24.0,666.0,20.2,22.01,17.15,19.0
467,4.42228,0.0,18.1,0.0,0.584,6.003,94.5,2.5403,24.0,666.0,20.2,331.29,21.32,19.1
468,15.5757,0.0,18.1,0.0,0.58,5.926,71.0,2.9084,24.0,666.0,20.2,368.74,18.13,19.1
469,13.0751,0.0,18.1,0.0,0.58,5.713,56.7,2.8237,24.0,666.0,20.2,396.9,14.76,20.1
470,4.34879,0.0,18.1,0.0,0.58,6.167,84.0,3.0334,24.0,666.0,20.2,396.9,16.29,19.9
471,4.03841,0.0,18.1,0.0,0.532,6.229,90.7,3.0993,24.0,666.0,20.2,395.33,12.87,19.6
472,3.56868,0.0,18.1,0.0,0.58,6.437,75.0,2.8965,24.0,666.0,20.2,393.37,14.36,23.2
473,4.64689,0.0,18.1,0.0,0.614,6.98,67.6,2.5329,24.0,666.0,20.2,374.68,11.66,29.8
474,8.05579,0.0,18.1,0.0,0.584,5.427,95.4,2.4298,24.0,666.0,20.2,352.58,18.14,13.8
475,6.39312,0.0,18.1,0.0,0.584,6.162,97.4,2.206,24.0,666.0,20.2,302.76,24.1,13.3
476,4.87141,0.0,18.1,0.0,0.614,6.484,93.6,2.3053,24.0,666.0,20.2,396.21,18.68,16.7
477,15.0234,0.0,18.1,0.0,0.614,5.304,97.3,2.1007,24.0,666.0,20.2,349.48,24.91,12.0
478,10.233,0.0,18.1,0.0,0.614,6.185,96.7,2.1705,24.0,666.0,20.2,379.7,18.03,14.6
479,14.3337,0.0,18.1,0.0,0.614,6.229,88.0,1.9512,24.0,666.0,20.2,383.32,13.11,21.4
480,5.82401,0.0,18.1,0.0,0.532,6.242,64.7,3.4242,24.0,666.0,20.2,396.9,10.74,23.0
481,5.70818,0.0,18.1,0.0,0.532,6.75,74.9,3.3317,24.0,666.0,20.2,393.07,7.74,23.7
482,5.73116,0.0,18.1,0.0,0.532,7.061,77.0,3.4106,24.0,666.0,20.2,395.28,7.01,25.0
483,2.81838,0.0,18.1,0.0,0.532,5.762,40.3,4.0983,24.0,666.0,20.2,392.92,10.42,21.8
484,2.37857,0.0,18.1,0.0,0.583,5.871,41.9,3.724,24.0,666.0,20.2,370.73,13.34,20.6
485,3.67367,0.0,18.1,0.0,0.583,6.312,51.9,3.9917,24.0,666.0,20.2,388.62,10.58,21.2
486,5.69175,0.0,18.1,0.0,0.583,6.114,79.8,3.5459,24.0,666.0,20.2,392.68,14.98,19.1
487,4.83567,0.0,18.1,0.0,0.583,5.905,53.2,3.1523,24.0,666.0,20.2,388.22,11.45,20.6
488,0.15086,0.0,27.74,0.0,0.609,5.454,92.7,1.8209,4.0,711.0,20.1,395.09,18.06,15.2
489,0.18337,0.0,27.74,0.0,0.609,5.414,98.3,1.7554,4.0,711.0,20.1,344.05,23.97,7.0
490,0.20746,0.0,27.74,0.0,0.609,5.093,98.0,1.8226,4.0,711.0,20.1,318.43,29.68,8.1
491,0.10574,0.0,27.74,0.0,0.609,5.983,98.8,1.8681,4.0,711.0,20.1,390.11,18.07,13.6
492,0.11132,0.0,27.74,0.0,0.609,5.983,83.5,2.1099,4.0,711.0,20.1,396.9,13.35,20.1
493,0.17331,0.0,9.69,0.0,0.585,5.707,54.0,2.3817,6.0,391.0,19.2,396.9,12.01,21.8
494,0.27957,0.0,9.69,0.0,0.585,5.926,42.6,2.3817,6.0,391.0,19.2,396.9,13.59,24.5
495,0.17899,0.0,9.69,0.0,0.585,5.67,28.8,2.7986,6.0,391.0,19.2,393.29,17.6,23.1
496,0.2896,0.0,9.69,0.0,0.585,5.39,72.9,2.7986,6.0,391.0,19.2,396.9,21.14,19.7
497,0.26838,0.0,9.69,0.0,0.585,5.794,70.6,2.8927,6.0,391.0,19.2,396.9,14.1,18.3
498,0.23912,0.0,9.69,0.0,0.585,6.019,65.3,2.4091,6.0,391.0,19.2,396.9,12.92,21.2
499,0.17783,0.0,9.69,0.0,0.585,5.569,73.5,2.3999,6.0,391.0,19.2,395.77,15.1,17.5
500,0.22438,0.0,9.69,0.0,0.585,6.027,79.7,2.4982,6.0,391.0,19.2,396.9,14.33,16.8
501,0.06263,0.0,11.93,0.0,0.573,6.593,69.1,2.4786,1.0,273.0,21.0,391.99,9.67,22.4
502,0.04527,0.0,11.93,0.0,0.573,6.12,76.7,2.2875,1.0,273.0,21.0,396.9,9.08,20.6
503,0.06076,0.0,11.93,0.0,0.573,6.976,91.0,2.1675,1.0,273.0,21.0,396.9,5.64,23.9
504,0.10959,0.0,11.93,0.0,0.573,6.794,89.3,2.3889,1.0,273.0,21.0,393.45,6.48,22.0
505,0.04741,0.0,11.93,0.0,0.573,6.03,80.8,2.505,1.0,273.0,21.0,396.9,7.88,11.9
================================================
FILE: examples/data/census_1k.csv
================================================
"YEAR","DATANUM","SERIAL","CBSERIAL","HHWT","CPI99","GQ","QGQ","PERNUM","PERWT","SEX","AGE","EDUC","EDUCD","INCTOT","SEX_HEAD","SEX_MOM","SEX_POP","SEX_SP","SEX_MOM2","SEX_POP2","AGE_HEAD","AGE_MOM","AGE_POP","AGE_SP","AGE_MOM2","AGE_POP2","EDUC_HEAD","EDUC_MOM","EDUC_POP","EDUC_SP","EDUC_MOM2","EDUC_POP2","EDUCD_HEAD","EDUCD_MOM","EDUCD_POP","EDUCD_SP","EDUCD_MOM2","EDUCD_POP2","INCTOT_HEAD","INCTOT_MOM","INCTOT_POP","INCTOT_SP","INCTOT_MOM2","INCTOT_POP2"
1970,2,1,,100,4.54,1,0,1,100,1,39,6,60,12450,1,,,2,,,39,,,36,,,6,,,3,,,60,,,30,,,12450,,,3450,,
1970,2,1,,100,4.54,1,0,2,100,2,36,3,30,3450,1,,,1,,,39,,,39,,,6,,,6,,,60,,,60,,,12450,,,12450,,
1970,2,2,,100,4.54,1,0,1,100,1,56,7,70,9050,1,,,2,,,56,,,54,,,7,,,6,,,70,,,60,,,9050,,,0,,
1970,2,2,,100,4.54,1,0,2,100,2,54,6,60,0,1,,,1,,,56,,,56,,,7,,,7,,,70,,,70,,,9050,,,9050,,
1970,2,4,,100,4.54,1,0,1,100,1,82,1,17,7450,1,,,2,,,82,,,74,,,1,,,2,,,17,,,23,,,7450,,,650,,
1970,2,4,,100,4.54,1,0,2,100,2,74,2,23,650,1,,,1,,,82,,,82,,,1,,,1,,,17,,,17,,,7450,,,7450,,
1970,2,5,,100,4.54,1,0,1,100,1,66,10,100,6950,1,,,2,,,66,,,62,,,10,,,6,,,100,,,60,,,6950,,,250,,
1970,2,5,,100,4.54,1,0,2,100,2,62,6,60,250,1,,,1,,,66,,,66,,,10,,,10,,,100,,,100,,,6950,,,6950,,
1970,2,6,,100,4.54,1,0,1,100,2,70,4,40,1250,2,,,,,,70,,,,,,4,,,,,,40,,,,,,1250,,,,,
1970,2,7,,100,4.54,1,0,1,100,1,25,6,60,11150,1,,,2,,,25,,,22,,,6,,,6,,,60,,,60,,,11150,,,4050,,
1970,2,7,,100,4.54,1,0,2,100,2,22,6,60,4050,1,,,1,,,25,,,25,,,6,,,6,,,60,,,60,,,11150,,,11150,,
1970,2,7,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,25,22,25,,,,6,6,6,,,,60,60,60,,,,11150,4050,11150,,,
1970,2,8,,100,4.54,3,0,1,100,2,98,2,26,550,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,9,,100,4.54,1,0,1,100,1,25,10,100,6150,1,,,2,,,25,,,27,,,10,,,9,,,100,,,90,,,6150,,,1050,,
1970,2,9,,100,4.54,1,0,2,100,2,27,9,90,1050,1,,,1,,,25,,,25,,,10,,,10,,,100,,,100,,,6150,,,6150,,
1970,2,10,,100,4.54,1,0,1,100,1,41,11,111,8050,1,2,,,,,41,78,,,,,11,6,,,,,111,60,,,,,8050,0,,,,
1970,2,10,,100,4.54,1,0,2,100,2,78,6,60,0,1,,,,,,41,,,,,,11,,,,,,111,,,,,,8050,,,,,
1970,2,10,,100,4.54,1,0,3,100,2,38,6,60,7150,1,2,,,,,41,78,,,,,11,6,,,,,111,60,,,,,8050,0,,,,
1970,2,11,,100,4.54,1,0,1,100,1,20,6,60,2050,1,,,,,,20,,,,,,6,,,,,,60,,,,,,2050,,,,,
1970,2,13,,100,4.54,1,0,1,100,1,37,6,65,16850,1,,,2,,,37,,,30,,,6,,,8,,,65,,,80,,,16850,,,350,,
1970,2,13,,100,4.54,1,0,2,100,2,30,8,80,350,1,,,1,,,37,,,37,,,6,,,6,,,65,,,65,,,16850,,,16850,,
1970,2,13,,100,4.54,1,0,3,100,1,5,0,2,9999999,1,2,1,,,,37,30,37,,,,6,8,6,,,,65,80,65,,,,16850,350,16850,,,
1970,2,13,,100,4.54,1,0,4,100,2,1,0,1,9999999,1,2,1,,,,37,30,37,,,,6,8,6,,,,65,80,65,,,,16850,350,16850,,,
1970,2,14,,100,4.54,1,0,1,100,1,49,2,23,8850,1,,,2,,,49,,,35,,,2,,,3,,,23,,,30,,,8850,,,4850,,
1970,2,14,,100,4.54,1,0,2,100,2,35,3,30,4850,1,,,1,,,49,,,49,,,2,,,2,,,23,,,23,,,8850,,,8850,,
1970,2,14,,100,4.54,1,0,3,100,2,17,3,30,250,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,
1970,2,14,,100,4.54,1,0,4,100,2,14,2,25,0,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,
1970,2,14,,100,4.54,1,0,5,100,1,10,1,15,9999999,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,
1970,2,14,,100,4.54,1,0,6,100,2,8,1,14,9999999,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,
1970,2,14,,100,4.54,1,0,7,100,1,0,0,1,9999999,1,2,1,,,,49,35,49,,,,2,3,2,,,,23,30,23,,,,8850,4850,8850,,,
1970,2,15,,100,4.54,1,0,1,100,2,62,7,70,7750,2,,,,,,62,,,,,,7,,,,,,70,,,,,,7750,,,,,
1970,2,15,,100,4.54,1,0,2,100,1,35,11,111,5350,2,2,,,,,62,62,,,,,7,7,,,,,70,70,,,,,7750,7750,,,,
1970,2,16,,100,4.54,1,0,1,100,1,57,4,40,11250,1,,,2,,,57,,,54,,,4,,,2,,,40,,,26,,,11250,,,150,,
1970,2,16,,100,4.54,1,0,2,100,2,54,2,26,150,1,2,,1,,,57,86,,57,,,4,2,,4,,,40,26,,40,,,11250,1250,,11250,,
1970,2,16,,100,4.54,1,0,3,100,2,86,2,26,1250,1,,,,,,57,,,,,,4,,,,,,40,,,,,,11250,,,,,
1970,2,17,,100,4.54,1,0,1,100,1,54,6,60,6050,1,,,,,,54,,,,,,6,,,,,,60,,,,,,6050,,,,,
1970,2,17,,100,4.54,1,0,2,100,2,64,2,26,0,1,,,,,,54,,,,,,6,,,,,,60,,,,,,6050,,,,,
1970,2,18,,100,4.54,1,0,1,100,1,52,7,70,12050,1,,,2,,,52,,,44,,,7,,,6,,,70,,,60,,,12050,,,650,,
1970,2,18,,100,4.54,1,0,2,100,2,44,6,60,650,1,,,1,,,52,,,52,,,7,,,7,,,70,,,70,,,12050,,,12050,,
1970,2,18,,100,4.54,1,0,3,100,2,16,4,40,950,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,
1970,2,18,,100,4.54,1,0,4,100,2,15,3,30,350,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,
1970,2,18,,100,4.54,1,0,5,100,1,14,2,25,350,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,
1970,2,18,,100,4.54,1,0,6,100,1,12,2,22,9999999,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,
1970,2,18,,100,4.54,1,0,7,100,1,6,1,12,9999999,1,2,1,,,,52,44,52,,,,7,6,7,,,,70,60,70,,,,12050,650,12050,,,
1970,2,19,,100,4.54,1,0,1,100,1,77,2,26,250,1,,,2,,,77,,,79,,,2,,,2,,,26,,,26,,,250,,,0,,
1970,2,19,,100,4.54,1,0,2,100,2,79,2,26,0,1,,,1,,,77,,,77,,,2,,,2,,,26,,,26,,,250,,,250,,
1970,2,20,,100,4.54,1,0,1,100,1,36,6,60,11450,1,,,2,,,36,,,32,,,6,,,6,,,60,,,60,,,11450,,,5550,,
1970,2,20,,100,4.54,1,0,2,100,2,32,6,60,5550,1,,,1,,,36,,,36,,,6,,,6,,,60,,,60,,,11450,,,11450,,
1970,2,20,,100,4.54,1,0,3,100,2,9,1,16,9999999,1,2,1,,,,36,32,36,,,,6,6,6,,,,60,60,60,,,,11450,5550,11450,,,
1970,2,21,,100,4.54,1,0,1,100,1,21,4,40,2450,1,,,2,,,21,,,20,,,4,,,4,,,40,,,40,,,2450,,,4550,,
1970,2,21,,100,4.54,1,0,2,100,2,20,4,40,4550,1,,,1,,,21,,,21,,,4,,,4,,,40,,,40,,,2450,,,2450,,
1970,2,21,,100,4.54,1,0,3,100,1,5,1,12,9999999,1,2,1,,,,21,20,21,,,,4,4,4,,,,40,40,40,,,,2450,4550,2450,,,
1970,2,21,,100,4.54,1,0,4,100,1,4,1,11,9999999,1,2,1,,,,21,20,21,,,,4,4,4,,,,40,40,40,,,,2450,4550,2450,,,
1970,2,22,,100,4.54,1,0,1,100,1,23,2,26,5050,1,,,,,,23,,,,,,2,,,,,,26,,,,,,5050,,,,,
1970,2,22,,100,4.54,1,0,2,100,2,23,3,30,1850,1,,,,,,23,,,,,,2,,,,,,26,,,,,,5050,,,,,
1970,2,23,,100,4.54,1,0,1,100,1,63,6,60,5050,1,,,,,,63,,,,,,6,,,,,,60,,,,,,5050,,,,,
1970,2,24,,100,4.54,1,0,1,100,2,68,3,30,2150,2,,,,,,68,,,,,,3,,,,,,30,,,,,,2150,,,,,
1970,2,25,,100,4.54,1,0,1,100,1,65,2,22,4850,1,,,2,,,65,,,61,,,2,,,4,,,22,,,40,,,4850,,,4350,,
1970,2,25,,100,4.54,1,0,2,100,2,61,4,40,4350,1,,,1,,,65,,,65,,,2,,,2,,,22,,,22,,,4850,,,4850,,
1970,2,26,,100,4.54,1,0,1,100,1,61,8,80,2150,1,,,2,,,61,,,66,,,8,,,6,,,80,,,60,,,2150,,,5650,,
1970,2,26,,100,4.54,1,0,2,100,2,66,6,60,5650,1,,,1,,,61,,,61,,,8,,,8,,,80,,,80,,,2150,,,2150,,
1970,2,27,,100,4.54,1,0,1,100,2,77,1,14,4050,2,,,,,,77,,,,,,1,,,,,,14,,,,,,4050,,,,,
1970,2,27,,100,4.54,1,0,2,100,1,75,1,14,2050,2,,,,,,77,,,,,,1,,,,,,14,,,,,,4050,,,,,
1970,2,28,,100,4.54,1,0,1,100,1,32,8,80,5050,1,,,,,,32,,,,,,8,,,,,,80,,,,,,5050,,,,,
1970,2,29,,100,4.54,1,0,1,100,1,59,5,50,15050,1,,,2,,,59,,,55,,,5,,,6,,,50,,,60,,,15050,,,0,,
1970,2,29,,100,4.54,1,0,2,100,2,55,6,60,0,1,,,1,,,59,,,59,,,5,,,5,,,50,,,50,,,15050,,,15050,,
1970,2,30,,100,4.54,1,0,1,100,2,47,6,60,0,2,,,,,,47,,,,,,6,,,,,,60,,,,,,0,,,,,
1970,2,31,,100,4.54,1,0,1,100,1,43,8,80,7050,1,,,2,,,43,,,41,,,8,,,6,,,80,,,60,,,7050,,,2050,,
1970,2,31,,100,4.54,1,0,2,100,2,41,6,60,2050,1,,,1,,,43,,,43,,,8,,,8,,,80,,,80,,,7050,,,7050,,
1970,2,31,,100,4.54,1,0,3,100,2,18,6,65,4050,1,2,1,,,,43,41,43,,,,8,6,8,,,,80,60,80,,,,7050,2050,7050,,,
1970,2,31,,100,4.54,1,0,4,100,2,15,2,26,0,1,2,1,,,,43,41,43,,,,8,6,8,,,,80,60,80,,,,7050,2050,7050,,,
1970,2,31,,100,4.54,1,0,5,100,2,10,1,17,9999999,1,2,1,,,,43,41,43,,,,8,6,8,,,,80,60,80,,,,7050,2050,7050,,,
1970,2,32,,100,4.54,1,0,1,100,1,40,10,100,13350,1,,,2,,,40,,,36,,,10,,,10,,,100,,,100,,,13350,,,0,,
1970,2,32,,100,4.54,1,0,2,100,2,36,10,100,0,1,,,1,,,40,,,40,,,10,,,10,,,100,,,100,,,13350,,,13350,,
1970,2,32,,100,4.54,1,0,3,100,1,14,2,25,0,1,2,1,,,,40,36,40,,,,10,10,10,,,,100,100,100,,,,13350,0,13350,,,
1970,2,32,,100,4.54,1,0,4,100,1,10,1,16,9999999,1,2,1,,,,40,36,40,,,,10,10,10,,,,100,100,100,,,,13350,0,13350,,,
1970,2,32,,100,4.54,1,0,5,100,2,4,1,11,9999999,1,2,1,,,,40,36,40,,,,10,10,10,,,,100,100,100,,,,13350,0,13350,,,
1970,2,33,,100,4.54,1,0,1,100,1,40,11,111,25050,1,,,2,,,40,,,32,,,11,,,10,,,111,,,100,,,25050,,,0,,
1970,2,33,,100,4.54,1,0,2,100,2,32,10,100,0,1,,,1,,,40,,,40,,,11,,,11,,,111,,,111,,,25050,,,25050,,
1970,2,33,,100,4.54,1,0,3,100,1,5,0,2,9999999,1,2,1,,,,40,32,40,,,,11,10,11,,,,111,100,111,,,,25050,0,25050,,,
1970,2,33,,100,4.54,1,0,4,100,1,3,0,2,9999999,1,2,1,,,,40,32,40,,,,11,10,11,,,,111,100,111,,,,25050,0,25050,,,
1970,2,34,,100,4.54,1,0,1,100,1,31,11,111,19350,1,,,2,,,31,,,31,,,11,,,10,,,111,,,100,,,19350,,,0,,
1970,2,34,,100,4.54,1,0,2,100,2,31,10,100,0,1,,,1,,,31,,,31,,,11,,,11,,,111,,,111,,,19350,,,19350,,
1970,2,34,,100,4.54,1,0,3,100,2,3,0,2,9999999,1,2,1,,,,31,31,31,,,,11,10,11,,,,111,100,111,,,,19350,0,19350,,,
1970,2,35,,100,4.54,1,0,1,100,1,64,11,111,17150,1,,,,,,64,,,,,,11,,,,,,111,,,,,,17150,,,,,
1970,2,36,,100,4.54,1,0,1,100,1,55,3,30,9050,1,,,2,,,55,,,51,,,3,,,6,,,30,,,60,,,9050,,,2950,,
1970,2,36,,100,4.54,1,0,2,100,2,51,6,60,2950,1,,,1,,,55,,,55,,,3,,,3,,,30,,,30,,,9050,,,9050,,
1970,2,37,,100,4.54,1,0,1,100,1,43,11,111,50000,1,,,2,,,43,,,40,,,11,,,10,,,111,,,100,,,50000,,,1150,,
1970,2,37,,100,4.54,1,0,2,100,2,40,10,100,1150,1,,,1,,,43,,,43,,,11,,,11,,,111,,,111,,,50000,,,50000,,
1970,2,37,,100,4.54,1,0,3,100,1,16,4,40,250,1,2,1,,,,43,40,43,,,,11,10,11,,,,111,100,111,,,,50000,1150,50000,,,
1970,2,37,,100,4.54,1,0,4,100,2,15,2,26,50,1,2,1,,,,43,40,43,,,,11,10,11,,,,111,100,111,,,,50000,1150,50000,,,
1970,2,37,,100,4.54,1,0,5,100,1,12,2,22,9999999,1,2,1,,,,43,40,43,,,,11,10,11,,,,111,100,111,,,,50000,1150,50000,,,
1970,2,38,,100,4.54,1,0,1,100,1,32,10,100,22150,1,,,2,,,32,,,31,,,10,,,7,,,100,,,70,,,22150,,,0,,
1970,2,38,,100,4.54,1,0,2,100,2,31,7,70,0,1,,,1,,,32,,,32,,,10,,,10,,,100,,,100,,,22150,,,22150,,
1970,2,38,,100,4.54,1,0,3,100,2,5,0,2,9999999,1,2,1,,,,32,31,32,,,,10,7,10,,,,100,70,100,,,,22150,0,22150,,,
1970,2,38,,100,4.54,1,0,4,100,1,4,0,2,9999999,1,2,1,,,,32,31,32,,,,10,7,10,,,,100,70,100,,,,22150,0,22150,,,
1970,2,38,,100,4.54,1,0,5,100,2,2,0,1,9999999,1,2,1,,,,32,31,32,,,,10,7,10,,,,100,70,100,,,,22150,0,22150,,,
1970,2,39,,100,4.54,1,0,1,100,1,54,8,80,17850,1,,,2,,,54,,,47,,,8,,,6,,,80,,,60,,,17850,,,0,,
1970,2,39,,100,4.54,1,0,2,100,2,47,6,60,0,1,,,1,,,54,,,54,,,8,,,8,,,80,,,80,,,17850,,,17850,,
1970,2,39,,100,4.54,1,0,3,100,1,19,6,60,2750,1,2,1,,,,54,47,54,,,,8,6,8,,,,80,60,80,,,,17850,0,17850,,,
1970,2,39,,100,4.54,1,0,4,100,1,12,2,23,9999999,1,2,1,,,,54,47,54,,,,8,6,8,,,,80,60,80,,,,17850,0,17850,,,
1970,2,40,,100,4.54,1,0,1,100,1,46,6,60,17150,1,,,2,,,46,,,46,,,6,,,6,,,60,,,60,,,17150,,,5050,,
1970,2,40,,100,4.54,1,0,2,100,2,46,6,60,5050,1,,,1,,,46,,,46,,,6,,,6,,,60,,,60,,,17150,,,17150,,
1970,2,41,,100,4.54,1,0,1,100,1,70,6,60,11450,1,,,,,,70,,,,,,6,,,,,,60,,,,,,11450,,,,,
1970,2,41,,100,4.54,1,0,2,100,2,28,5,50,2150,1,,,,,,70,,,,,,6,,,,,,60,,,,,,11450,,,,,
1970,2,41,,100,4.54,1,0,3,100,1,33,8,80,9850,1,,1,,,,70,,70,,,,6,,6,,,,60,,60,,,,11450,,11450,,,
1970,2,41,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,,1,,,,70,,33,,,,6,,8,,,,60,,80,,,,11450,,9850,,,
1970,2,41,,100,4.54,1,0,5,100,2,5,0,2,9999999,1,,1,,,,70,,33,,,,6,,8,,,,60,,80,,,,11450,,9850,,,
1970,2,42,,100,4.54,1,0,1,100,1,53,3,30,10050,1,,,2,,,53,,,51,,,3,,,6,,,30,,,60,,,10050,,,0,,
1970,2,42,,100,4.54,1,0,2,100,2,51,6,60,0,1,,,1,,,53,,,53,,,3,,,3,,,30,,,30,,,10050,,,10050,,
1970,2,42,,100,4.54,1,0,3,100,2,14,2,26,0,1,2,1,,,,53,51,53,,,,3,6,3,,,,30,60,30,,,,10050,0,10050,,,
1970,2,43,,100,4.54,1,0,1,100,1,41,11,111,32050,1,,,2,,,41,,,40,,,11,,,11,,,111,,,111,,,32050,,,250,,
1970,2,43,,100,4.54,1,0,2,100,2,40,11,111,250,1,,,1,,,41,,,41,,,11,,,11,,,111,,,111,,,32050,,,32050,,
1970,2,43,,100,4.54,1,0,3,100,2,10,1,17,9999999,1,2,1,,,,41,40,41,,,,11,11,11,,,,111,111,111,,,,32050,250,32050,,,
1970,2,43,,100,4.54,1,0,4,100,2,6,1,12,9999999,1,2,1,,,,41,40,41,,,,11,11,11,,,,111,111,111,,,,32050,250,32050,,,
1970,2,44,,100,4.54,1,0,1,100,1,47,2,26,14050,1,,,2,,,47,,,44,,,2,,,2,,,26,,,26,,,14050,,,0,,
1970,2,44,,100,4.54,1,0,2,100,2,44,2,26,0,1,,,1,,,47,,,47,,,2,,,2,,,26,,,26,,,14050,,,14050,,
1970,2,44,,100,4.54,1,0,3,100,2,21,8,80,2050,1,2,1,,,,47,44,47,,,,2,2,2,,,,26,26,26,,,,14050,0,14050,,,
1970,2,44,,100,4.54,1,0,4,100,1,18,5,50,1750,1,2,1,,,,47,44,47,,,,2,2,2,,,,26,26,26,,,,14050,0,14050,,,
1970,2,44,,100,4.54,1,0,5,100,1,4,0,2,9999999,1,2,1,,,,47,44,47,,,,2,2,2,,,,26,26,26,,,,14050,0,14050,,,
1970,2,45,,100,4.54,1,0,1,100,1,60,8,80,16550,1,,,2,,,60,,,50,,,8,,,6,,,80,,,60,,,16550,,,2950,,
1970,2,45,,100,4.54,1,0,2,100,2,50,6,60,2950,1,,,1,,,60,,,60,,,8,,,8,,,80,,,80,,,16550,,,16550,,
1970,2,46,,100,4.54,1,0,1,100,1,47,10,100,16250,1,2,,,,,47,83,,,,,10,2,,,,,100,22,,,,,16250,1250,,,,
1970,2,46,,100,4.54,1,0,2,100,2,83,2,22,1250,1,,,,,,47,,,,,,10,,,,,,100,,,,,,16250,,,,,
1970,2,46,,100,4.54,1,0,3,100,2,83,2,22,1450,1,,,,,,47,,,,,,10,,,,,,100,,,,,,16250,,,,,
1970,2,48,,100,4.54,1,0,1,100,1,49,6,60,8850,1,2,,2,,,49,89,,48,,,6,2,,6,,,60,26,,60,,,8850,450,,6050,,
1970,2,48,,100,4.54,1,0,2,100,2,48,6,60,6050,1,,,1,,,49,,,49,,,6,,,6,,,60,,,60,,,8850,,,8850,,
1970,2,48,,100,4.54,1,0,3,100,2,89,2,26,450,1,,,,,,49,,,,,,6,,,,,,60,,,,,,8850,,,,,
1970,2,49,,100,4.54,1,0,1,100,1,30,10,100,12050,1,,,2,,,30,,,30,,,10,,,10,,,100,,,100,,,12050,,,5650,,
1970,2,49,,100,4.54,1,0,2,100,2,30,10,100,5650,1,,,1,,,30,,,30,,,10,,,10,,,100,,,100,,,12050,,,12050,,
1970,2,49,,100,4.54,1,0,3,100,1,3,0,2,9999999,1,2,1,,,,30,30,30,,,,10,10,10,,,,100,100,100,,,,12050,5650,12050,,,
1970,2,49,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,30,30,30,,,,10,10,10,,,,100,100,100,,,,12050,5650,12050,,,
1970,2,50,,100,4.54,1,0,1,100,1,54,8,80,10150,1,,,,,,54,,,,,,8,,,,,,80,,,,,,10150,,,,,
1970,2,51,,100,4.54,1,0,1,100,2,64,10,100,8650,2,,,,,,64,,,,,,10,,,,,,100,,,,,,8650,,,,,
1970,2,52,,100,4.54,1,0,1,100,2,37,7,70,11350,2,,,,,,37,,,,,,7,,,,,,70,,,,,,11350,,,,,
1970,2,52,,100,4.54,1,0,2,100,2,6,1,12,9999999,2,2,,,,,37,37,,,,,7,7,,,,,70,70,,,,,11350,11350,,,,
1970,2,53,,100,4.54,1,0,1,100,2,34,6,60,8050,2,,,,,,34,,,,,,6,,,,,,60,,,,,,8050,,,,,
1970,2,53,,100,4.54,1,0,2,100,1,11,2,22,9999999,2,2,,,,,34,34,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,
1970,2,53,,100,4.54,1,0,3,100,2,9,1,15,9999999,2,2,,,,,34,34,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,
1970,2,54,,100,4.54,1,0,1,100,1,48,0,2,0,1,,,2,,,48,,,51,,,0,,,2,,,2,,,26,,,0,,,0,,
1970,2,54,,100,4.54,1,0,2,100,2,51,2,26,0,1,,,1,,,48,,,48,,,0,,,0,,,2,,,2,,,0,,,0,,
1970,2,55,,100,4.54,1,0,1,100,2,27,2,25,150,2,,,,,,27,,,,,,2,,,,,,25,,,,,,150,,,,,
1970,2,55,,100,4.54,1,0,2,100,2,12,2,23,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,
1970,2,55,,100,4.54,1,0,3,100,2,7,1,14,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,
1970,2,55,,100,4.54,1,0,4,100,2,6,1,12,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,
1970,2,55,,100,4.54,1,0,5,100,2,5,1,11,9999999,2,2,,,,,27,27,,,,,2,2,,,,,25,25,,,,,150,150,,,,
1970,2,56,,100,4.54,1,0,1,100,1,58,2,22,8050,1,,,2,,,58,,,52,,,2,,,6,,,22,,,60,,,8050,,,0,,
1970,2,56,,100,4.54,1,0,2,100,2,52,6,60,0,1,,,1,,,58,,,58,,,2,,,2,,,22,,,22,,,8050,,,8050,,
1970,2,56,,100,4.54,1,0,3,100,2,23,9,90,0,1,2,1,,,,58,52,58,,,,2,6,2,,,,22,60,22,,,,8050,0,8050,,,
1970,2,57,,100,4.54,1,0,1,100,1,32,2,26,7050,1,,,2,,,32,,,32,,,2,,,5,,,26,,,50,,,7050,,,5050,,
1970,2,57,,100,4.54,1,0,2,100,2,32,5,50,5050,1,,,1,,,32,,,32,,,2,,,2,,,26,,,26,,,7050,,,7050,,
1970,2,57,,100,4.54,1,0,3,100,2,5,0,2,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,57,,100,4.54,1,0,4,100,2,3,0,2,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,57,,100,4.54,1,0,5,100,2,13,2,23,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,57,,100,4.54,1,0,6,100,1,10,1,17,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,57,,100,4.54,1,0,7,100,2,9,1,16,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,57,,100,4.54,1,0,8,100,2,8,1,15,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,57,,100,4.54,1,0,9,100,1,6,1,12,9999999,1,2,1,,,,32,32,32,,,,2,5,2,,,,26,50,26,,,,7050,5050,7050,,,
1970,2,58,,100,4.54,1,0,1,100,1,24,11,110,1350,1,,,2,,,24,,,25,,,11,,,11,,,110,,,110,,,1350,,,8150,,
1970,2,58,,100,4.54,1,0,2,100,2,25,11,110,8150,1,,,1,,,24,,,24,,,11,,,11,,,110,,,110,,,1350,,,1350,,
1970,2,58,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,24,25,24,,,,11,11,11,,,,110,110,110,,,,1350,8150,1350,,,
1970,2,59,,100,4.54,1,0,1,100,1,34,5,50,14150,1,,,2,,,34,,,31,,,5,,,6,,,50,,,60,,,14150,,,5050,,
1970,2,59,,100,4.54,1,0,2,100,2,31,6,60,5050,1,,,1,,,34,,,34,,,5,,,5,,,50,,,50,,,14150,,,14150,,
1970,2,59,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,
1970,2,59,,100,4.54,1,0,4,100,1,11,2,22,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,
1970,2,59,,100,4.54,1,0,5,100,2,10,1,17,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,
1970,2,59,,100,4.54,1,0,6,100,1,6,1,12,9999999,1,2,1,,,,34,31,34,,,,5,6,5,,,,50,60,50,,,,14150,5050,14150,,,
1970,2,60,,100,4.54,1,0,1,100,1,35,10,100,14050,1,,,,,,35,,,,,,10,,,,,,100,,,,,,14050,,,,,
1970,2,62,,100,4.54,1,0,1,100,1,53,6,60,10650,1,,,2,,,53,,,49,,,6,,,6,,,60,,,60,,,10650,,,6650,,
1970,2,62,,100,4.54,1,0,2,100,2,49,6,60,6650,1,,,1,,,53,,,53,,,6,,,6,,,60,,,60,,,10650,,,10650,,
1970,2,63,,100,4.54,1,0,1,100,1,78,2,26,1250,1,,,,,,78,,,,,,2,,,,,,26,,,,,,1250,,,,,
1970,2,63,,100,4.54,1,0,2,100,2,38,7,70,7050,1,,1,,,,78,,78,,,,2,,2,,,,26,,26,,,,1250,,1250,,,
1970,2,64,,100,4.54,1,0,1,100,1,37,6,60,7050,1,,,2,,,37,,,36,,,6,,,6,,,60,,,60,,,7050,,,0,,
1970,2,64,,100,4.54,1,0,2,100,2,36,6,60,0,1,,,1,,,37,,,37,,,6,,,6,,,60,,,60,,,7050,,,7050,,
1970,2,64,,100,4.54,1,0,3,100,1,13,2,25,9999999,1,2,1,,,,37,36,37,,,,6,6,6,,,,60,60,60,,,,7050,0,7050,,,
1970,2,64,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,37,36,37,,,,6,6,6,,,,60,60,60,,,,7050,0,7050,,,
1970,2,64,,100,4.54,1,0,5,100,1,5,0,2,9999999,1,2,1,,,,37,36,37,,,,6,6,6,,,,60,60,60,,,,7050,0,7050,,,
1970,2,65,,100,4.54,1,0,1,100,2,33,8,80,8050,2,,,,,,33,,,,,,8,,,,,,80,,,,,,8050,,,,,
1970,2,66,,100,4.54,1,0,1,100,1,25,4,40,8050,1,,,2,,,25,,,23,,,4,,,6,,,40,,,60,,,8050,,,6050,,
1970,2,66,,100,4.54,1,0,2,100,2,23,6,60,6050,1,,,1,,,25,,,25,,,4,,,4,,,40,,,40,,,8050,,,8050,,
1970,2,66,,100,4.54,1,0,3,100,2,4,1,11,9999999,1,2,1,,,,25,23,25,,,,4,6,4,,,,40,60,40,,,,8050,6050,8050,,,
1970,2,66,,100,4.54,1,0,4,100,1,2,0,1,9999999,1,2,1,,,,25,23,25,,,,4,6,4,,,,40,60,40,,,,8050,6050,8050,,,
1970,2,68,,100,4.54,1,0,1,100,1,40,5,50,3850,1,,,2,,,40,,,34,,,5,,,10,,,50,,,100,,,3850,,,850,,
1970,2,68,,100,4.54,1,0,2,100,2,34,10,100,850,1,,,1,,,40,,,40,,,5,,,5,,,50,,,50,,,3850,,,3850,,
1970,2,68,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,40,34,40,,,,5,10,5,,,,50,100,50,,,,3850,850,3850,,,
1970,2,68,,100,4.54,1,0,4,100,2,0,0,1,9999999,1,2,1,,,,40,34,40,,,,5,10,5,,,,50,100,50,,,,3850,850,3850,,,
1970,2,68,,100,4.54,1,0,5,100,2,5,1,11,9999999,1,2,1,,,,40,34,40,,,,5,10,5,,,,50,100,50,,,,3850,850,3850,,,
1970,2,69,,100,4.54,1,0,1,100,2,35,4,40,3850,2,,,,,,35,,,,,,4,,,,,,40,,,,,,3850,,,,,
1970,2,69,,100,4.54,1,0,2,100,2,8,1,15,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,
1970,2,69,,100,4.54,1,0,3,100,2,7,1,15,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,
1970,2,69,,100,4.54,1,0,4,100,2,6,1,14,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,
1970,2,69,,100,4.54,1,0,5,100,2,2,0,1,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,
1970,2,69,,100,4.54,1,0,6,100,1,13,2,23,9999999,2,2,,,,,35,35,,,,,4,4,,,,,40,40,,,,,3850,3850,,,,
1970,2,70,,100,4.54,1,0,1,100,1,29,2,26,1750,1,,,,,,29,,,,,,2,,,,,,26,,,,,,1750,,,,,
1970,2,71,,100,4.54,1,0,1,100,1,35,7,70,12050,1,,,,,,35,,,,,,7,,,,,,70,,,,,,12050,,,,,
1970,2,71,,100,4.54,1,0,2,100,2,31,10,100,5450,1,,,,,,35,,,,,,7,,,,,,70,,,,,,12050,,,,,
1970,2,72,,100,4.54,1,0,1,100,2,52,4,40,1550,2,,,,,,52,,,,,,4,,,,,,40,,,,,,1550,,,,,
1970,2,72,,100,4.54,1,0,2,100,2,26,5,50,2450,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,
1970,2,72,,100,4.54,1,0,3,100,2,6,1,14,9999999,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,
1970,2,72,,100,4.54,1,0,4,100,2,2,0,1,9999999,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,
1970,2,72,,100,4.54,1,0,5,100,1,1,0,1,9999999,2,2,,,,,52,52,,,,,4,4,,,,,40,40,,,,,1550,1550,,,,
1970,2,74,,100,4.54,1,0,1,100,2,33,6,60,10050,2,,,,,,33,,,,,,6,,,,,,60,,,,,,10050,,,,,
1970,2,74,,100,4.54,1,0,2,100,1,13,2,22,9999999,2,2,,,,,33,33,,,,,6,6,,,,,60,60,,,,,10050,10050,,,,
1970,2,74,,100,4.54,1,0,3,100,1,11,1,17,9999999,2,2,,,,,33,33,,,,,6,6,,,,,60,60,,,,,10050,10050,,,,
1970,2,75,,100,4.54,1,0,1,100,2,26,9,90,10550,2,,,,,,26,,,,,,9,,,,,,90,,,,,,10550,,,,,
1970,2,76,,100,4.54,1,0,1,100,1,35,11,111,17250,1,,,,,,35,,,,,,11,,,,,,111,,,,,,17250,,,,,
1970,2,77,,100,4.54,1,0,1,100,1,49,6,60,11050,1,,,2,,,49,,,49,,,6,,,6,,,60,,,60,,,11050,,,0,,
1970,2,77,,100,4.54,1,0,2,100,2,49,6,60,0,1,,,1,,,49,,,49,,,6,,,6,,,60,,,60,,,11050,,,11050,,
1970,2,77,,100,4.54,1,0,3,100,2,16,4,40,3550,1,2,1,,,,49,49,49,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,
1970,2,77,,100,4.54,1,0,4,100,1,12,2,23,9999999,1,2,1,,,,49,49,49,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,
1970,2,77,,100,4.54,1,0,5,100,2,20,6,60,6050,1,2,1,,,,49,49,49,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,
1970,2,78,,100,4.54,1,0,1,100,1,72,2,26,2050,1,,,2,,,72,,,65,,,2,,,2,,,26,,,26,,,2050,,,750,,
1970,2,78,,100,4.54,1,0,2,100,2,65,2,26,750,1,,,1,,,72,,,72,,,2,,,2,,,26,,,26,,,2050,,,2050,,
1970,2,79,,100,4.54,1,0,1,100,1,37,6,60,7550,1,,,2,,,37,,,35,,,6,,,9,,,60,,,90,,,7550,,,5550,,
1970,2,79,,100,4.54,1,0,2,100,2,35,9,90,5550,1,,,1,,,37,,,37,,,6,,,6,,,60,,,60,,,7550,,,7550,,
1970,2,80,,100,4.54,1,0,1,100,1,44,6,60,6050,1,,,2,,,44,,,45,,,6,,,6,,,60,,,60,,,6050,,,0,,
1970,2,80,,100,4.54,1,0,2,100,2,45,6,60,0,1,,,1,,,44,,,44,,,6,,,6,,,60,,,60,,,6050,,,6050,,
1970,2,80,,100,4.54,1,0,3,100,1,25,6,60,5050,1,2,1,,,,44,45,44,,,,6,6,6,,,,60,60,60,,,,6050,0,6050,,,
1970,2,81,,100,4.54,1,0,1,100,1,35,2,22,4050,1,,,2,,,35,,,31,,,2,,,2,,,22,,,23,,,4050,,,0,,
1970,2,81,,100,4.54,1,0,2,100,2,31,2,23,0,1,,,1,,,35,,,35,,,2,,,2,,,22,,,22,,,4050,,,4050,,
1970,2,81,,100,4.54,1,0,3,100,2,13,2,23,9999999,1,2,1,,,,35,31,35,,,,2,2,2,,,,22,23,22,,,,4050,0,4050,,,
1970,2,81,,100,4.54,1,0,4,100,2,4,0,2,9999999,1,2,1,,,,35,31,35,,,,2,2,2,,,,22,23,22,,,,4050,0,4050,,,
1970,2,81,,100,4.54,1,0,5,100,2,1,0,1,9999999,1,2,1,,,,35,31,35,,,,2,2,2,,,,22,23,22,,,,4050,0,4050,,,
1970,2,82,,100,4.54,1,0,1,100,1,45,6,60,10050,1,,,2,,,45,,,46,,,6,,,7,,,60,,,70,,,10050,,,4050,,
1970,2,82,,100,4.54,1,0,2,100,2,46,7,70,4050,1,,,1,,,45,,,45,,,6,,,6,,,60,,,60,,,10050,,,10050,,
1970,2,82,,100,4.54,1,0,3,100,1,19,6,60,2050,1,2,1,,,,45,46,45,,,,6,7,6,,,,60,70,60,,,,10050,4050,10050,,,
1970,2,82,,100,4.54,1,0,4,100,2,13,2,23,9999999,1,2,1,,,,45,46,45,,,,6,7,6,,,,60,70,60,,,,10050,4050,10050,,,
1970,2,83,,100,4.54,1,0,1,100,1,58,2,26,9650,1,,,2,,,58,,,59,,,2,,,4,,,26,,,40,,,9650,,,0,,
1970,2,83,,100,4.54,1,0,2,100,2,59,4,40,0,1,,,1,,,58,,,58,,,2,,,2,,,26,,,26,,,9650,,,9650,,
1970,2,83,,100,4.54,1,0,3,100,1,23,10,100,2050,1,2,1,,,,58,59,58,,,,2,4,2,,,,26,40,26,,,,9650,0,9650,,,
1970,2,84,,100,4.54,1,0,1,100,1,68,2,26,5050,1,,,,,,68,,,,,,2,,,,,,26,,,,,,5050,,,,,
1970,2,85,,100,4.54,1,0,1,100,1,40,2,22,8050,1,,,2,,,40,,,35,,,2,,,1,,,22,,,17,,,8050,,,0,,
1970,2,85,,100,4.54,1,0,2,100,2,35,1,17,0,1,,,1,,,40,,,40,,,2,,,2,,,22,,,22,,,8050,,,8050,,
1970,2,85,,100,4.54,1,0,3,100,2,1,0,1,9999999,1,2,1,,,,40,35,40,,,,2,1,2,,,,22,17,22,,,,8050,0,8050,,,
1970,2,85,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,40,35,40,,,,2,1,2,,,,22,17,22,,,,8050,0,8050,,,
1970,2,86,,100,4.54,1,0,1,100,1,56,2,23,7950,1,,,,,,56,,,,,,2,,,,,,23,,,,,,7950,,,,,
1970,2,86,,100,4.54,1,0,2,100,2,82,2,26,1150,1,,1,,,,56,,56,,,,2,,2,,,,23,,23,,,,7950,,7950,,,
1970,2,87,,100,4.54,1,0,1,100,1,28,6,60,12150,1,,,2,,,28,,,28,,,6,,,6,,,60,,,60,,,12150,,,0,,
1970,2,87,,100,4.54,1,0,2,100,2,28,6,60,0,1,,,1,,,28,,,28,,,6,,,6,,,60,,,60,,,12150,,,12150,,
1970,2,87,,100,4.54,1,0,3,100,1,1,0,1,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12150,0,12150,,,
1970,2,88,,100,4.54,1,0,1,100,2,70,2,26,3550,2,,,,,,70,,,,,,2,,,,,,26,,,,,,3550,,,,,
1970,2,89,,100,4.54,1,0,1,100,1,54,2,26,7150,1,,,2,,,54,,,55,,,2,,,4,,,26,,,40,,,7150,,,0,,
1970,2,89,,100,4.54,1,0,2,100,2,55,4,40,0,1,,,1,,,54,,,54,,,2,,,2,,,26,,,26,,,7150,,,7150,,
1970,2,90,,100,4.54,1,0,1,100,2,42,6,60,0,2,,,,,,42,,,,,,6,,,,,,60,,,,,,0,,,,,
1970,2,90,,100,4.54,1,0,2,100,1,14,2,26,0,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,
1970,2,90,,100,4.54,1,0,3,100,1,13,2,23,9999999,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,
1970,2,90,,100,4.54,1,0,4,100,1,11,2,22,9999999,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,
1970,2,90,,100,4.54,1,0,5,100,2,8,1,14,9999999,2,2,,,,,42,42,,,,,6,6,,,,,60,60,,,,,0,0,,,,
1970,2,91,,100,4.54,1,0,1,100,2,36,6,60,3250,2,,,,,,36,,,,,,6,,,,,,60,,,,,,3250,,,,,
1970,2,92,,100,4.54,1,0,1,100,1,42,4,40,6250,1,,,2,,,42,,,33,,,4,,,4,,,40,,,40,,,6250,,,750,,
1970,2,92,,100,4.54,1,0,2,100,2,33,4,40,750,1,,,1,,,42,,,42,,,4,,,4,,,40,,,40,,,6250,,,6250,,
1970,2,92,,100,4.54,1,0,3,100,2,13,2,25,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,
1970,2,92,,100,4.54,1,0,4,100,1,12,1,17,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,
1970,2,92,,100,4.54,1,0,5,100,1,10,1,15,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,
1970,2,92,,100,4.54,1,0,6,100,2,6,1,12,9999999,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,
1970,2,92,,100,4.54,1,0,7,100,2,95,2,23,1750,1,2,1,,,,42,33,42,,,,4,4,4,,,,40,40,40,,,,6250,750,6250,,,
1970,2,93,,100,4.54,1,0,1,100,1,35,10,100,12050,1,,,2,,,35,,,31,,,10,,,6,,,100,,,60,,,12050,,,3250,,
1970,2,93,,100,4.54,1,0,2,100,2,31,6,60,3250,1,,,1,,,35,,,35,,,10,,,10,,,100,,,100,,,12050,,,12050,,
1970,2,93,,100,4.54,1,0,3,100,1,14,2,26,0,1,2,1,,,,35,31,35,,,,10,6,10,,,,100,60,100,,,,12050,3250,12050,,,
1970,2,93,,100,4.54,1,0,4,100,2,12,2,23,9999999,1,2,1,,,,35,31,35,,,,10,6,10,,,,100,60,100,,,,12050,3250,12050,,,
1970,2,93,,100,4.54,1,0,5,100,2,10,1,16,9999999,1,2,1,,,,35,31,35,,,,10,6,10,,,,100,60,100,,,,12050,3250,12050,,,
1970,2,94,,100,4.54,1,0,1,100,2,72,9,90,6350,2,,,,,,72,,,,,,9,,,,,,90,,,,,,6350,,,,,
1970,2,94,,100,4.54,1,0,2,100,2,32,11,111,8650,2,2,,,,,72,72,,,,,9,9,,,,,90,90,,,,,6350,6350,,,,
1970,2,94,,100,4.54,1,0,3,100,2,42,6,60,7050,2,2,,,,,72,72,,,,,9,9,,,,,90,90,,,,,6350,6350,,,,
1970,2,95,,100,4.54,1,0,1,100,1,50,5,50,16150,1,,,2,,,50,,,48,,,5,,,6,,,50,,,60,,,16150,,,50,,
1970,2,95,,100,4.54,1,0,2,100,2,48,6,60,50,1,,,1,,,50,,,50,,,5,,,5,,,50,,,50,,,16150,,,16150,,
1970,2,95,,100,4.54,1,0,3,100,2,15,3,30,0,1,2,1,,,,50,48,50,,,,5,6,5,,,,50,60,50,,,,16150,50,16150,,,
1970,2,95,,100,4.54,1,0,4,100,1,13,2,23,9999999,1,2,1,,,,50,48,50,,,,5,6,5,,,,50,60,50,,,,16150,50,16150,,,
1970,2,96,,100,4.54,1,0,1,100,1,21,4,40,12050,1,,,2,,,21,,,19,,,4,,,6,,,40,,,60,,,12050,,,12050,,
1970,2,96,,100,4.54,1,0,2,100,2,19,6,60,12050,1,,,1,,,21,,,21,,,4,,,4,,,40,,,40,,,12050,,,12050,,
1970,2,97,,100,4.54,1,0,1,100,1,66,4,40,7150,1,,,2,,,66,,,64,,,4,,,2,,,40,,,23,,,7150,,,550,,
1970,2,97,,100,4.54,1,0,2,100,2,64,2,23,550,1,,,1,,,66,,,66,,,4,,,4,,,40,,,40,,,7150,,,7150,,
1970,2,98,,100,4.54,1,0,1,100,1,56,6,60,11050,1,,,2,,,56,,,53,,,6,,,6,,,60,,,60,,,11050,,,0,,
1970,2,98,,100,4.54,1,0,2,100,2,53,6,60,0,1,,,1,,,56,,,56,,,6,,,6,,,60,,,60,,,11050,,,11050,,
1970,2,98,,100,4.54,1,0,3,100,1,29,7,70,5050,1,2,1,,,,56,53,56,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,
1970,2,98,,100,4.54,1,0,4,100,1,18,5,50,0,1,2,1,,,,56,53,56,,,,6,6,6,,,,60,60,60,,,,11050,0,11050,,,
1970,2,99,,100,4.54,1,0,1,100,1,51,8,80,12050,1,,,2,,,51,,,55,,,8,,,4,,,80,,,40,,,12050,,,0,,
1970,2,99,,100,4.54,1,0,2,100,2,55,4,40,0,1,,,1,,,51,,,51,,,8,,,8,,,80,,,80,,,12050,,,12050,,
1970,2,99,,100,4.54,1,0,3,100,2,11,2,22,9999999,1,2,1,,,,51,55,51,,,,8,4,8,,,,80,40,80,,,,12050,0,12050,,,
1970,2,100,,100,4.54,1,0,1,100,2,56,4,40,6250,2,,,,,,56,,,,,,4,,,,,,40,,,,,,6250,,,,,
1970,2,101,,100,4.54,1,0,1,100,1,42,9,90,21850,1,,,2,,,42,,,39,,,9,,,6,,,90,,,60,,,21850,,,650,,
1970,2,101,,100,4.54,1,0,2,100,2,39,6,60,650,1,,,1,,,42,,,42,,,9,,,9,,,90,,,90,,,21850,,,21850,,
1970,2,101,,100,4.54,1,0,3,100,1,18,5,50,650,1,2,1,,,,42,39,42,,,,9,6,9,,,,90,60,90,,,,21850,650,21850,,,
1970,2,101,,100,4.54,1,0,4,100,2,11,2,22,9999999,1,2,1,,,,42,39,42,,,,9,6,9,,,,90,60,90,,,,21850,650,21850,,,
1970,2,102,,100,4.54,1,0,1,100,1,49,6,60,19150,1,,,2,,,49,,,46,,,6,,,6,,,60,,,60,,,19150,,,0,,
1970,2,102,,100,4.54,1,0,2,100,2,46,6,60,0,1,,,1,,,49,,,49,,,6,,,6,,,60,,,60,,,19150,,,19150,,
1970,2,102,,100,4.54,1,0,3,100,2,9,1,16,9999999,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,
1970,2,102,,100,4.54,1,0,4,100,1,6,1,12,9999999,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,
1970,2,102,,100,4.54,1,0,5,100,1,17,5,50,2050,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,
1970,2,102,,100,4.54,1,0,6,100,1,16,3,30,1450,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,
1970,2,102,,100,4.54,1,0,7,100,1,16,3,30,1450,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,
1970,2,102,,100,4.54,1,0,8,100,2,11,2,22,9999999,1,2,1,,,,49,46,49,,,,6,6,6,,,,60,60,60,,,,19150,0,19150,,,
1970,2,103,,100,4.54,1,0,1,100,1,59,7,70,8850,1,,,,,,59,,,,,,7,,,,,,70,,,,,,8850,,,,,
1970,2,103,,100,4.54,1,0,2,100,2,69,6,60,450,1,,,,,,59,,,,,,7,,,,,,70,,,,,,8850,,,,,
1970,2,104,,100,4.54,1,0,1,100,1,59,5,50,6750,1,,,2,,,59,,,59,,,5,,,2,,,50,,,26,,,6750,,,4850,,
1970,2,104,,100,4.54,1,0,2,100,2,59,2,26,4850,1,,,1,,,59,,,59,,,5,,,5,,,50,,,50,,,6750,,,6750,,
1970,2,104,,100,4.54,1,0,3,100,1,20,6,60,1650,1,2,1,,,,59,59,59,,,,5,2,5,,,,50,26,50,,,,6750,4850,6750,,,
1970,2,105,,100,4.54,1,0,1,100,2,55,7,70,9450,2,,,,,,55,,,,,,7,,,,,,70,,,,,,9450,,,,,
1970,2,106,,100,4.54,1,0,1,100,1,63,2,23,7950,1,,,2,,,63,,,57,,,2,,,4,,,23,,,40,,,7950,,,4650,,
1970,2,106,,100,4.54,1,0,2,100,2,57,4,40,4650,1,,,1,,,63,,,63,,,2,,,2,,,23,,,23,,,7950,,,7950,,
1970,2,106,,100,4.54,1,0,3,100,1,21,8,80,1450,1,2,1,,,,63,57,63,,,,2,4,2,,,,23,40,23,,,,7950,4650,7950,,,
1970,2,107,,100,4.54,1,0,1,100,1,40,3,30,18650,1,,,2,,,40,,,37,,,3,,,6,,,30,,,60,,,18650,,,0,,
1970,2,107,,100,4.54,1,0,2,100,2,37,6,60,0,1,,,1,,,40,,,40,,,3,,,3,,,30,,,30,,,18650,,,18650,,
1970,2,107,,100,4.54,1,0,3,100,1,14,2,26,0,1,2,1,,,,40,37,40,,,,3,6,3,,,,30,60,30,,,,18650,0,18650,,,
1970,2,107,,100,4.54,1,0,4,100,1,8,1,15,9999999,1,2,1,,,,40,37,40,,,,3,6,3,,,,30,60,30,,,,18650,0,18650,,,
1970,2,108,,100,4.54,1,0,1,100,1,36,6,65,4050,1,,,2,,,36,,,36,,,6,,,2,,,65,,,26,,,4050,,,1550,,
1970,2,108,,100,4.54,1,0,2,100,2,36,2,26,1550,1,,,1,,,36,,,36,,,6,,,6,,,65,,,65,,,4050,,,4050,,
1970,2,108,,100,4.54,1,0,3,100,2,17,5,50,950,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,
1970,2,108,,100,4.54,1,0,4,100,2,14,2,26,0,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,
1970,2,108,,100,4.54,1,0,5,100,1,10,1,17,9999999,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,
1970,2,108,,100,4.54,1,0,6,100,2,8,1,15,9999999,1,2,1,,,,36,36,36,,,,6,2,6,,,,65,26,65,,,,4050,1550,4050,,,
1970,2,109,,100,4.54,1,0,1,100,1,37,10,100,1550,1,,,,,,37,,,,,,10,,,,,,100,,,,,,1550,,,,,
1970,2,109,,100,4.54,1,0,2,100,1,48,10,100,11050,1,,,,,,37,,,,,,10,,,,,,100,,,,,,1550,,,,,
1970,2,110,,100,4.54,1,0,1,100,2,78,6,60,3950,2,,,,,,78,,,,,,6,,,,,,60,,,,,,3950,,,,,
1970,2,111,,100,4.54,1,0,1,100,1,32,4,40,6050,1,,,,,,32,,,,,,4,,,,,,40,,,,,,6050,,,,,
1970,2,112,,100,4.54,1,0,1,100,2,63,2,25,250,2,,,,,,63,,,,,,2,,,,,,25,,,,,,250,,,,,
1970,2,113,,100,4.54,1,0,1,100,1,42,5,50,10050,1,,,2,,,42,,,34,,,5,,,6,,,50,,,60,,,10050,,,550,,
1970,2,113,,100,4.54,1,0,2,100,2,34,6,60,550,1,,,1,,,42,,,42,,,5,,,5,,,50,,,50,,,10050,,,10050,,
1970,2,113,,100,4.54,1,0,3,100,1,13,2,23,9999999,1,2,1,,,,42,34,42,,,,5,6,5,,,,50,60,50,,,,10050,550,10050,,,
1970,2,113,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,42,34,42,,,,5,6,5,,,,50,60,50,,,,10050,550,10050,,,
1970,2,113,,100,4.54,1,0,5,100,2,11,1,17,9999999,1,2,1,,,,42,34,42,,,,5,6,5,,,,50,60,50,,,,10050,550,10050,,,
1970,2,114,,100,4.54,1,0,1,100,1,82,2,26,0,1,,,2,,,82,,,75,,,2,,,2,,,26,,,26,,,0,,,1450,,
1970,2,114,,100,4.54,1,0,2,100,2,75,2,26,1450,1,,,1,,,82,,,82,,,2,,,2,,,26,,,26,,,0,,,0,,
1970,2,114,,100,4.54,1,0,3,100,1,47,0,2,0,1,2,1,,,,82,75,82,,,,2,2,2,,,,26,26,26,,,,0,1450,0,,,
1970,2,115,,100,4.54,1,0,1,100,2,70,2,23,1250,2,,,,,,70,,,,,,2,,,,,,23,,,,,,1250,,,,,
1970,2,116,,100,4.54,1,0,1,100,1,65,3,30,3450,1,,,2,,,65,,,65,,,3,,,6,,,30,,,60,,,3450,,,350,,
1970,2,116,,100,4.54,1,0,2,100,2,65,6,60,350,1,,,1,,,65,,,65,,,3,,,3,,,30,,,30,,,3450,,,3450,,
1970,2,116,,100,4.54,1,0,3,100,1,42,7,70,6750,1,2,1,,,,65,65,65,,,,3,6,3,,,,30,60,30,,,,3450,350,3450,,,
1970,2,117,,100,4.54,1,0,1,100,1,34,6,60,14050,1,,,2,,,34,,,31,,,6,,,7,,,60,,,70,,,14050,,,0,,
1970,2,117,,100,4.54,1,0,2,100,2,31,7,70,0,1,,,1,,,34,,,34,,,6,,,6,,,60,,,60,,,14050,,,14050,,
1970,2,117,,100,4.54,1,0,3,100,1,8,1,15,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,
1970,2,117,,100,4.54,1,0,4,100,2,7,1,14,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,
1970,2,117,,100,4.54,1,0,5,100,2,5,1,11,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,
1970,2,117,,100,4.54,1,0,6,100,2,1,0,1,9999999,1,2,1,,,,34,31,34,,,,6,7,6,,,,60,70,60,,,,14050,0,14050,,,
1970,2,118,,100,4.54,1,0,1,100,1,23,6,60,6050,1,,,2,,,23,,,20,,,6,,,7,,,60,,,70,,,6050,,,6050,,
1970,2,118,,100,4.54,1,0,2,100,2,20,7,70,6050,1,,,1,,,23,,,23,,,6,,,6,,,60,,,60,,,6050,,,6050,,
1970,2,119,,100,4.54,1,0,1,100,1,24,11,110,8650,1,,,2,,,24,,,25,,,11,,,6,,,110,,,60,,,8650,,,4050,,
1970,2,119,,100,4.54,1,0,2,100,2,25,6,60,4050,1,,,1,,,24,,,24,,,11,,,11,,,110,,,110,,,8650,,,8650,,
1970,2,119,,100,4.54,1,0,3,100,2,0,0,1,9999999,1,2,1,,,,24,25,24,,,,11,6,11,,,,110,60,110,,,,8650,4050,8650,,,
1970,2,120,,100,4.54,1,0,1,100,1,35,2,26,10150,1,,,2,,,35,,,29,,,2,,,6,,,26,,,60,,,10150,,,2550,,
1970,2,120,,100,4.54,1,0,2,100,2,29,6,60,2550,1,,,1,,,35,,,35,,,2,,,2,,,26,,,26,,,10150,,,10150,,
1970,2,120,,100,4.54,1,0,3,100,2,2,0,1,9999999,1,2,1,,,,35,29,35,,,,2,6,2,,,,26,60,26,,,,10150,2550,10150,,,
1970,2,120,,100,4.54,1,0,4,100,1,4,0,2,9999999,1,2,1,,,,35,29,35,,,,2,6,2,,,,26,60,26,,,,10150,2550,10150,,,
1970,2,121,,100,4.54,1,0,1,100,1,45,4,40,12550,1,,,,,,45,,,,,,4,,,,,,40,,,,,,12550,,,,,
1970,2,122,,100,4.54,1,0,1,100,1,64,6,60,5550,1,,,2,,,64,,,62,,,6,,,2,,,60,,,26,,,5550,,,0,,
1970,2,122,,100,4.54,1,0,2,100,2,62,2,26,0,1,,,1,,,64,,,64,,,6,,,6,,,60,,,60,,,5550,,,5550,,
1970,2,123,,100,4.54,1,0,1,100,1,26,1,12,1950,1,,,2,,,26,,,27,,,1,,,8,,,12,,,80,,,1950,,,550,,
1970,2,123,,100,4.54,1,0,2,100,2,27,8,80,550,1,,,1,,,26,,,26,,,1,,,1,,,12,,,12,,,1950,,,1950,,
1970,2,123,,100,4.54,1,0,3,100,1,4,0,2,9999999,1,2,1,,,,26,27,26,,,,1,8,1,,,,12,80,12,,,,1950,550,1950,,,
1970,2,123,,100,4.54,1,0,4,100,1,4,0,2,9999999,1,2,1,,,,26,27,26,,,,1,8,1,,,,12,80,12,,,,1950,550,1950,,,
1970,2,123,,100,4.54,1,0,5,100,2,2,0,1,9999999,1,2,1,,,,26,27,26,,,,1,8,1,,,,12,80,12,,,,1950,550,1950,,,
1970,2,124,,100,4.54,1,0,1,100,1,44,2,23,10750,1,,,2,,,44,,,48,,,2,,,2,,,23,,,26,,,10750,,,0,,
1970,2,124,,100,4.54,1,0,2,100,2,48,2,26,0,1,,,1,,,44,,,44,,,2,,,2,,,23,,,23,,,10750,,,10750,,
1970,2,124,,100,4.54,1,0,3,100,2,10,1,17,9999999,1,2,1,,,,44,48,44,,,,2,2,2,,,,23,26,23,,,,10750,0,10750,,,
1970,2,125,,100,4.54,1,0,1,100,1,55,2,26,11450,1,,,2,,,55,,,50,,,2,,,7,,,26,,,70,,,11450,,,7050,,
1970,2,125,,100,4.54,1,0,2,100,2,50,7,70,7050,1,,,1,,,55,,,55,,,2,,,2,,,26,,,26,,,11450,,,11450,,
1970,2,126,,100,4.54,1,0,1,100,1,53,2,26,11150,1,,,2,,,53,,,49,,,2,,,2,,,26,,,26,,,11150,,,0,,
1970,2,126,,100,4.54,1,0,2,100,2,49,2,26,0,1,,,1,,,53,,,53,,,2,,,2,,,26,,,26,,,11150,,,11150,,
1970,2,126,,100,4.54,1,0,3,100,1,21,5,50,750,1,2,1,,,,53,49,53,,,,2,2,2,,,,26,26,26,,,,11150,0,11150,,,
1970,2,126,,100,4.54,1,0,4,100,2,15,3,30,0,1,2,1,,,,53,49,53,,,,2,2,2,,,,26,26,26,,,,11150,0,11150,,,
1970,2,127,,100,4.54,1,0,1,100,1,40,8,80,11550,1,,,2,,,40,,,39,,,8,,,6,,,80,,,65,,,11550,,,2950,,
1970,2,127,,100,4.54,1,0,2,100,2,39,6,65,2950,1,,,1,,,40,,,40,,,8,,,8,,,80,,,80,,,11550,,,11550,,
1970,2,127,,100,4.54,1,0,3,100,2,17,5,50,250,1,2,1,,,,40,39,40,,,,8,6,8,,,,80,65,80,,,,11550,2950,11550,,,
1970,2,127,,100,4.54,1,0,4,100,1,14,2,26,350,1,2,1,,,,40,39,40,,,,8,6,8,,,,80,65,80,,,,11550,2950,11550,,,
1970,2,127,,100,4.54,1,0,5,100,2,8,1,15,9999999,1,2,1,,,,40,39,40,,,,8,6,8,,,,80,65,80,,,,11550,2950,11550,,,
1970,2,128,,100,4.54,1,0,1,100,1,48,3,30,9050,1,,,2,,,48,,,44,,,3,,,3,,,30,,,30,,,9050,,,0,,
1970,2,128,,100,4.54,1,0,2,100,2,44,3,30,0,1,,,1,,,48,,,48,,,3,,,3,,,30,,,30,,,9050,,,9050,,
1970,2,128,,100,4.54,1,0,3,100,1,21,6,60,6850,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,
1970,2,128,,100,4.54,1,0,4,100,2,16,3,30,0,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,
1970,2,128,,100,4.54,1,0,5,100,2,15,3,30,0,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,
1970,2,128,,100,4.54,1,0,6,100,2,11,2,22,9999999,1,2,1,,,,48,44,48,,,,3,3,3,,,,30,30,30,,,,9050,0,9050,,,
1970,2,130,,100,4.54,1,0,1,100,1,61,2,22,10250,1,,,2,,,61,,,58,,,2,,,2,,,22,,,25,,,10250,,,3050,,
1970,2,130,,100,4.54,1,0,2,100,2,58,2,25,3050,1,,,1,,,61,,,61,,,2,,,2,,,22,,,22,,,10250,,,10250,,
1970,2,131,,100,4.54,1,0,1,100,1,59,4,40,6550,1,,,,,,59,,,,,,4,,,,,,40,,,,,,6550,,,,,
1970,2,132,,100,4.54,1,0,1,100,1,44,2,26,8850,1,,,2,,,44,,,39,,,2,,,1,,,26,,,17,,,8850,,,0,,
1970,2,132,,100,4.54,1,0,2,100,2,39,1,17,0,1,,,1,,,44,,,44,,,2,,,2,,,26,,,26,,,8850,,,8850,,
1970,2,132,,100,4.54,1,0,3,100,1,15,4,40,0,1,2,1,,,,44,39,44,,,,2,1,2,,,,26,17,26,,,,8850,0,8850,,,
1970,2,132,,100,4.54,1,0,4,100,2,10,1,17,9999999,1,2,1,,,,44,39,44,,,,2,1,2,,,,26,17,26,,,,8850,0,8850,,,
1970,2,134,,100,4.54,1,0,1,100,1,22,1,16,4850,1,,,,,,22,,,,,,1,,,,,,16,,,,,,4850,,,,,
1970,2,134,,100,4.54,1,0,2,100,1,20,2,23,4850,1,,,,,,22,,,,,,1,,,,,,16,,,,,,4850,,,,,
1970,2,135,,100,4.54,1,0,1,100,1,72,2,25,3350,1,,,2,,,72,,,75,,,2,,,2,,,25,,,26,,,3350,,,0,,
1970,2,135,,100,4.54,1,0,2,100,2,75,2,26,0,1,,,1,,,72,,,72,,,2,,,2,,,25,,,25,,,3350,,,3350,,
1970,2,136,,100,4.54,3,0,1,100,1,14,2,23,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,137,,100,4.54,1,0,1,100,1,26,6,60,650,1,,,2,,,26,,,23,,,6,,,6,,,60,,,60,,,650,,,0,,
1970,2,137,,100,4.54,1,0,2,100,2,23,6,60,0,1,,,1,,,26,,,26,,,6,,,6,,,60,,,60,,,650,,,650,,
1970,2,137,,100,4.54,1,0,3,100,1,3,0,2,9999999,1,2,1,,,,26,23,26,,,,6,6,6,,,,60,60,60,,,,650,0,650,,,
1970,2,137,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,26,23,26,,,,6,6,6,,,,60,60,60,,,,650,0,650,,,
1970,2,137,,100,4.54,1,0,5,100,1,0,0,1,9999999,1,2,1,,,,26,23,26,,,,6,6,6,,,,60,60,60,,,,650,0,650,,,
1970,2,138,,100,4.54,1,0,1,100,1,32,6,60,-850,1,,,,,,32,,,,,,6,,,,,,60,,,,,,-850,,,,,
1970,2,138,,100,4.54,1,0,2,100,2,38,6,60,2650,1,,,,,,32,,,,,,6,,,,,,60,,,,,,-850,,,,,
1970,2,140,,100,4.54,1,0,1,100,1,57,1,15,3050,1,,,2,,,57,,,52,,,1,,,2,,,15,,,25,,,3050,,,0,,
1970,2,140,,100,4.54,1,0,2,100,2,52,2,25,0,1,,,1,,,57,,,57,,,1,,,1,,,15,,,15,,,3050,,,3050,,
1970,2,140,,100,4.54,1,0,3,100,1,30,6,60,4050,1,2,1,,,,57,52,57,,,,1,2,1,,,,15,25,15,,,,3050,0,3050,,,
1970,2,141,,100,4.54,1,0,1,100,2,75,2,23,5550,2,,,,,,75,,,,,,2,,,,,,23,,,,,,5550,,,,,
1970,2,142,,100,4.54,1,0,1,100,1,76,2,22,750,1,,,,,,76,,,,,,2,,,,,,22,,,,,,750,,,,,
1970,2,142,,100,4.54,1,0,2,100,2,72,2,26,250,1,,,,,,76,,,,,,2,,,,,,22,,,,,,750,,,,,
1970,2,143,,100,4.54,1,0,1,100,1,58,6,60,7050,1,,,2,,,58,,,57,,,6,,,6,,,60,,,60,,,7050,,,6550,,
1970,2,143,,100,4.54,1,0,2,100,2,57,6,60,6550,1,,,1,,,58,,,58,,,6,,,6,,,60,,,60,,,7050,,,7050,,
1970,2,144,,100,4.54,1,0,1,100,1,30,11,110,15050,1,,,2,,,30,,,29,,,11,,,9,,,110,,,90,,,15050,,,0,,
1970,2,144,,100,4.54,1,0,2,100,2,29,9,90,0,1,,,1,,,30,,,30,,,11,,,11,,,110,,,110,,,15050,,,15050,,
1970,2,144,,100,4.54,1,0,3,100,2,3,0,2,9999999,1,2,1,,,,30,29,30,,,,11,9,11,,,,110,90,110,,,,15050,0,15050,,,
1970,2,145,,100,4.54,1,0,1,100,1,30,2,26,5150,1,,,2,,,30,,,29,,,2,,,4,,,26,,,40,,,5150,,,3850,,
1970,2,145,,100,4.54,1,0,2,100,2,29,4,40,3850,1,,,1,,,30,,,30,,,2,,,2,,,26,,,26,,,5150,,,5150,,
1970,2,145,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,
1970,2,145,,100,4.54,1,0,4,100,1,10,1,17,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,
1970,2,145,,100,4.54,1,0,5,100,2,8,1,14,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,
1970,2,145,,100,4.54,1,0,6,100,2,3,0,2,9999999,1,2,1,,,,30,29,30,,,,2,4,2,,,,26,40,26,,,,5150,3850,5150,,,
1970,2,146,,100,4.54,1,0,1,100,2,70,6,60,1950,2,,,,,,70,,,,,,6,,,,,,60,,,,,,1950,,,,,
1970,2,146,,100,4.54,1,0,2,100,1,37,2,26,7550,2,2,,,,,70,70,,,,,6,6,,,,,60,60,,,,,1950,1950,,,,
1970,2,147,,100,4.54,1,0,1,100,1,34,9,90,13050,1,,,2,,,34,,,34,,,9,,,11,,,90,,,110,,,13050,,,0,,
1970,2,147,,100,4.54,1,0,2,100,2,34,11,110,0,1,,,1,,,34,,,34,,,9,,,9,,,90,,,90,,,13050,,,13050,,
1970,2,147,,100,4.54,1,0,3,100,2,9,1,16,9999999,1,2,1,,,,34,34,34,,,,9,11,9,,,,90,110,90,,,,13050,0,13050,,,
1970,2,147,,100,4.54,1,0,4,100,1,7,1,14,9999999,1,2,1,,,,34,34,34,,,,9,11,9,,,,90,110,90,,,,13050,0,13050,,,
1970,2,148,,100,4.54,1,0,1,100,2,79,6,60,1150,2,,,,,,79,,,,,,6,,,,,,60,,,,,,1150,,,,,
1970,2,149,,100,4.54,1,0,1,100,1,70,2,26,11050,1,,,2,,,70,,,68,,,2,,,2,,,26,,,26,,,11050,,,0,,
1970,2,149,,100,4.54,1,0,2,100,2,68,2,26,0,1,,,1,,,70,,,70,,,2,,,2,,,26,,,26,,,11050,,,11050,,
1970,2,150,,100,4.54,1,0,1,100,1,35,6,60,15050,1,,,2,,,35,,,31,,,6,,,6,,,60,,,60,,,15050,,,1550,,
1970,2,150,,100,4.54,1,0,2,100,2,31,6,60,1550,1,,,1,,,35,,,35,,,6,,,6,,,60,,,60,,,15050,,,15050,,
1970,2,150,,100,4.54,1,0,3,100,1,8,1,14,9999999,1,2,1,,,,35,31,35,,,,6,6,6,,,,60,60,60,,,,15050,1550,15050,,,
1970,2,150,,100,4.54,1,0,4,100,2,5,1,11,9999999,1,2,1,,,,35,31,35,,,,6,6,6,,,,60,60,60,,,,15050,1550,15050,,,
1970,2,152,,100,4.54,1,0,1,100,1,58,6,60,12650,1,,,2,,,58,,,55,,,6,,,2,,,60,,,26,,,12650,,,0,,
1970,2,152,,100,4.54,1,0,2,100,2,55,2,26,0,1,,,1,,,58,,,58,,,6,,,6,,,60,,,60,,,12650,,,12650,,
1970,2,153,,100,4.54,1,0,1,100,2,49,4,40,4550,2,,,,,,49,,,,,,4,,,,,,40,,,,,,4550,,,,,
1970,2,153,,100,4.54,1,0,2,100,1,39,5,50,6050,2,2,,,,,49,49,,,,,4,4,,,,,40,40,,,,,4550,4550,,,,
1970,2,154,,100,4.54,3,0,1,100,1,73,2,26,4250,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,155,,100,4.54,1,0,1,100,1,88,4,40,1550,1,,,2,,,88,,,84,,,4,,,2,,,40,,,26,,,1550,,,850,,
1970,2,155,,100,4.54,1,0,2,100,2,84,2,26,850,1,,,1,,,88,,,88,,,4,,,4,,,40,,,40,,,1550,,,1550,,
1970,2,156,,100,4.54,1,0,1,100,1,25,5,50,10050,1,,,2,,,25,,,24,,,5,,,6,,,50,,,60,,,10050,,,750,,
1970,2,156,,100,4.54,1,0,2,100,2,24,6,60,750,1,,,1,,,25,,,25,,,5,,,5,,,50,,,50,,,10050,,,10050,,
1970,2,156,,100,4.54,1,0,3,100,2,6,1,12,9999999,1,2,1,,,,25,24,25,,,,5,6,5,,,,50,60,50,,,,10050,750,10050,,,
1970,2,156,,100,4.54,1,0,4,100,1,3,0,2,9999999,1,2,1,,,,25,24,25,,,,5,6,5,,,,50,60,50,,,,10050,750,10050,,,
1970,2,157,,100,4.54,1,0,1,100,2,56,0,2,3450,2,,,,,,56,,,,,,0,,,,,,2,,,,,,3450,,,,,
1970,2,158,,100,4.54,3,0,1,100,2,79,6,60,2250,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,159,,100,4.54,1,0,1,100,1,25,8,80,1850,1,,,2,,,25,,,36,,,8,,,6,,,80,,,60,,,1850,,,3250,,
1970,2,159,,100,4.54,1,0,2,100,2,36,6,60,3250,1,,,1,,,25,,,25,,,8,,,8,,,80,,,80,,,1850,,,1850,,
1970,2,160,,100,4.54,1,0,1,100,2,28,6,60,3550,2,,,,,,28,,,,,,6,,,,,,60,,,,,,3550,,,,,
1970,2,160,,100,4.54,1,0,2,100,1,7,1,12,9999999,2,2,,,,,28,28,,,,,6,6,,,,,60,60,,,,,3550,3550,,,,
1970,2,161,,100,4.54,1,0,1,100,2,21,8,80,1350,2,,,,,,21,,,,,,8,,,,,,80,,,,,,1350,,,,,
1970,2,161,,100,4.54,1,0,2,100,2,23,10,100,2750,2,,,,,,21,,,,,,8,,,,,,80,,,,,,1350,,,,,
1970,2,162,,100,4.54,1,0,1,100,1,22,10,100,2050,1,,,2,,,22,,,22,,,10,,,10,,,100,,,100,,,2050,,,1250,,
1970,2,162,,100,4.54,1,0,2,100,2,22,10,100,1250,1,,,1,,,22,,,22,,,10,,,10,,,100,,,100,,,2050,,,2050,,
1970,2,163,,100,4.54,1,0,1,100,2,60,6,60,4350,2,,,,,,60,,,,,,6,,,,,,60,,,,,,4350,,,,,
1970,2,164,,100,4.54,1,0,1,100,1,20,8,80,2250,1,,,,,,20,,,,,,8,,,,,,80,,,,,,2250,,,,,
1970,2,165,,100,4.54,1,0,1,100,1,78,2,23,3250,1,,,,,,78,,,,,,2,,,,,,23,,,,,,3250,,,,,
1970,2,166,,100,4.54,1,0,1,100,1,24,10,100,9050,1,,,2,,,24,,,24,,,10,,,8,,,100,,,80,,,9050,,,3250,,
1970,2,166,,100,4.54,1,0,2,100,2,24,8,80,3250,1,,,1,,,24,,,24,,,10,,,10,,,100,,,100,,,9050,,,9050,,
1970,2,166,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,24,24,24,,,,10,8,10,,,,100,80,100,,,,9050,3250,9050,,,
1970,2,167,,100,4.54,1,0,1,100,1,34,10,100,17050,1,,,2,,,34,,,33,,,10,,,10,,,100,,,100,,,17050,,,0,,
1970,2,167,,100,4.54,1,0,2,100,2,33,10,100,0,1,,,1,,,34,,,34,,,10,,,10,,,100,,,100,,,17050,,,17050,,
1970,2,167,,100,4.54,1,0,3,100,1,9,1,16,9999999,1,2,1,,,,34,33,34,,,,10,10,10,,,,100,100,100,,,,17050,0,17050,,,
1970,2,167,,100,4.54,1,0,4,100,2,7,1,12,9999999,1,2,1,,,,34,33,34,,,,10,10,10,,,,100,100,100,,,,17050,0,17050,,,
1970,2,168,,100,4.54,1,0,1,100,1,27,6,60,16650,1,,,2,,,27,,,26,,,6,,,6,,,60,,,60,,,16650,,,0,,
1970,2,168,,100,4.54,1,0,2,100,2,26,6,60,0,1,,,1,,,27,,,27,,,6,,,6,,,60,,,60,,,16650,,,16650,,
1970,2,168,,100,4.54,1,0,3,100,2,5,1,11,9999999,1,2,1,,,,27,26,27,,,,6,6,6,,,,60,60,60,,,,16650,0,16650,,,
1970,2,168,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,27,26,27,,,,6,6,6,,,,60,60,60,,,,16650,0,16650,,,
1970,2,168,,100,4.54,1,0,5,100,2,1,0,1,9999999,1,2,1,,,,27,26,27,,,,6,6,6,,,,60,60,60,,,,16650,0,16650,,,
1970,2,169,,100,4.54,1,0,1,100,1,28,6,65,11250,1,,,2,,,28,,,23,,,6,,,7,,,65,,,70,,,11250,,,6550,,
1970,2,169,,100,4.54,1,0,2,100,2,23,7,70,6550,1,,,1,,,28,,,28,,,6,,,6,,,65,,,65,,,11250,,,11250,,
1970,2,170,,100,4.54,1,0,1,100,1,45,6,60,12950,1,,,2,,,45,,,46,,,6,,,4,,,60,,,40,,,12950,,,0,,
1970,2,170,,100,4.54,1,0,2,100,2,46,4,40,0,1,,,1,,,45,,,45,,,6,,,6,,,60,,,60,,,12950,,,12950,,
1970,2,170,,100,4.54,1,0,3,100,1,18,6,65,1650,1,2,1,,,,45,46,45,,,,6,4,6,,,,60,40,60,,,,12950,0,12950,,,
1970,2,170,,100,4.54,1,0,4,100,2,23,6,60,5550,1,2,1,,,,45,46,45,,,,6,4,6,,,,60,40,60,,,,12950,0,12950,,,
1970,2,170,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,,,,,45,23,,,,,6,6,,,,,60,60,,,,,12950,5550,,,,
1970,2,171,,100,4.54,1,0,1,100,1,26,6,60,11450,1,,,2,,,26,,,27,,,6,,,6,,,60,,,60,,,11450,,,0,,
1970,2,171,,100,4.54,1,0,2,100,2,27,6,60,0,1,,,1,,,26,,,26,,,6,,,6,,,60,,,60,,,11450,,,11450,,
1970,2,171,,100,4.54,1,0,3,100,1,4,0,2,9999999,1,2,1,,,,26,27,26,,,,6,6,6,,,,60,60,60,,,,11450,0,11450,,,
1970,2,171,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,26,27,26,,,,6,6,6,,,,60,60,60,,,,11450,0,11450,,,
1970,2,171,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,1,,,,26,27,26,,,,6,6,6,,,,60,60,60,,,,11450,0,11450,,,
1970,2,172,,100,4.54,1,0,1,100,1,33,6,60,1450,1,,,2,,,33,,,37,,,6,,,2,,,60,,,26,,,1450,,,3450,,
1970,2,172,,100,4.54,1,0,2,100,2,37,2,26,3450,1,,,1,,,33,,,33,,,6,,,6,,,60,,,60,,,1450,,,1450,,
1970,2,172,,100,4.54,1,0,3,100,2,6,1,12,9999999,1,2,1,,,,33,37,33,,,,6,2,6,,,,60,26,60,,,,1450,3450,1450,,,
1970,2,172,,100,4.54,1,0,4,100,1,5,1,11,9999999,1,2,1,,,,33,37,33,,,,6,2,6,,,,60,26,60,,,,1450,3450,1450,,,
1970,2,174,,100,4.54,1,0,1,100,1,60,4,40,13850,1,,,2,,,60,,,48,,,4,,,6,,,40,,,60,,,13850,,,0,,
1970,2,174,,100,4.54,1,0,2,100,2,48,6,60,0,1,,,1,,,60,,,60,,,4,,,4,,,40,,,40,,,13850,,,13850,,
1970,2,175,,100,4.54,1,0,1,100,1,37,1,15,3550,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,
1970,2,175,,100,4.54,1,0,2,100,1,29,1,17,1550,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,
1970,2,175,,100,4.54,1,0,3,100,1,28,1,17,2050,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,
1970,2,175,,100,4.54,1,0,4,100,1,29,2,23,1350,1,,,,,,37,,,,,,1,,,,,,15,,,,,,3550,,,,,
1970,2,176,,100,4.54,4,0,1,100,1,34,11,111,11550,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,177,,100,4.54,1,0,1,100,2,75,8,80,50000,2,,,,,,75,,,,,,8,,,,,,80,,,,,,50000,,,,,
1970,2,177,,100,4.54,1,0,2,100,2,44,6,60,0,2,2,,,,,75,75,,,,,8,8,,,,,80,80,,,,,50000,50000,,,,
1970,2,178,,100,4.54,1,0,1,100,2,28,8,80,6050,2,,,,,,28,,,,,,8,,,,,,80,,,,,,6050,,,,,
1970,2,178,,100,4.54,1,0,2,100,1,5,1,11,9999999,2,2,,,,,28,28,,,,,8,8,,,,,80,80,,,,,6050,6050,,,,
1970,2,179,,100,4.54,1,0,1,100,1,35,11,111,19150,1,,,2,,,35,,,30,,,11,,,11,,,111,,,110,,,19150,,,4050,,
1970,2,179,,100,4.54,1,0,2,100,2,30,11,110,4050,1,,,1,,,35,,,35,,,11,,,11,,,111,,,111,,,19150,,,19150,,
1970,2,179,,100,4.54,1,0,3,100,1,4,1,11,9999999,1,2,1,,,,35,30,35,,,,11,11,11,,,,111,110,111,,,,19150,4050,19150,,,
1970,2,180,,100,4.54,1,0,1,100,1,28,10,100,2350,1,,,2,,,28,,,26,,,10,,,11,,,100,,,110,,,2350,,,7250,,
1970,2,180,,100,4.54,1,0,2,100,2,26,11,110,7250,1,,,1,,,28,,,28,,,10,,,10,,,100,,,100,,,2350,,,2350,,
1970,2,182,,100,4.54,1,0,1,100,1,73,2,25,1050,1,,,,,,73,,,,,,2,,,,,,25,,,,,,1050,,,,,
1970,2,183,,100,4.54,1,0,1,100,1,27,6,60,15050,1,,,2,,,27,,,27,,,6,,,6,,,60,,,60,,,15050,,,0,,
1970,2,183,,100,4.54,1,0,2,100,2,27,6,60,0,1,,,1,,,27,,,27,,,6,,,6,,,60,,,60,,,15050,,,15050,,
1970,2,183,,100,4.54,1,0,3,100,2,4,0,2,9999999,1,2,1,,,,27,27,27,,,,6,6,6,,,,60,60,60,,,,15050,0,15050,,,
1970,2,183,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,27,27,27,,,,6,6,6,,,,60,60,60,,,,15050,0,15050,,,
1970,2,184,,100,4.54,1,0,1,100,2,49,2,25,2550,2,,,,,,49,,,,,,2,,,,,,25,,,,,,2550,,,,,
1970,2,184,,100,4.54,1,0,2,100,2,48,2,25,0,2,,,,,,49,,,,,,2,,,,,,25,,,,,,2550,,,,,
1970,2,185,,100,4.54,1,0,1,100,1,29,6,60,8250,1,,,2,,,29,,,22,,,6,,,6,,,60,,,60,,,8250,,,5050,,
1970,2,185,,100,4.54,1,0,2,100,2,22,6,60,5050,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,8250,,,8250,,
1970,2,187,,100,4.54,1,0,1,100,1,51,8,80,10050,1,,,2,,,51,,,44,,,8,,,6,,,80,,,60,,,10050,,,1450,,
1970,2,187,,100,4.54,1,0,2,100,2,44,6,60,1450,1,,,1,,,51,,,51,,,8,,,8,,,80,,,80,,,10050,,,10050,,
1970,2,187,,100,4.54,1,0,3,100,2,16,4,40,0,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,
1970,2,187,,100,4.54,1,0,4,100,2,10,1,17,9999999,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,
1970,2,187,,100,4.54,1,0,5,100,2,9,1,15,9999999,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,
1970,2,187,,100,4.54,1,0,6,100,2,19,6,60,0,1,2,1,,,,51,44,51,,,,8,6,8,,,,80,60,80,,,,10050,1450,10050,,,
1970,2,187,,100,4.54,1,0,7,100,2,0,0,1,9999999,1,2,,,,,51,19,,,,,8,6,,,,,80,60,,,,,10050,0,,,,
1970,2,188,,100,4.54,1,0,1,100,1,55,6,60,11250,1,,,2,,,55,,,50,,,6,,,2,,,60,,,26,,,11250,,,650,,
1970,2,188,,100,4.54,1,0,2,100,2,50,2,26,650,1,,,1,,,55,,,55,,,6,,,6,,,60,,,60,,,11250,,,11250,,
1970,2,188,,100,4.54,1,0,3,100,2,10,2,22,9999999,1,2,1,,,,55,50,55,,,,6,2,6,,,,60,26,60,,,,11250,650,11250,,,
1970,2,189,,100,4.54,1,0,1,100,1,51,6,60,8050,1,,,2,,,51,,,51,,,6,,,4,,,60,,,40,,,8050,,,0,,
1970,2,189,,100,4.54,1,0,2,100,2,51,4,40,0,1,,,1,,,51,,,51,,,6,,,6,,,60,,,60,,,8050,,,8050,,
1970,2,189,,100,4.54,1,0,3,100,2,22,8,80,0,1,2,1,,,,51,51,51,,,,6,4,6,,,,60,40,60,,,,8050,0,8050,,,
1970,2,189,,100,4.54,1,0,4,100,1,17,5,50,1250,1,2,1,,,,51,51,51,,,,6,4,6,,,,60,40,60,,,,8050,0,8050,,,
1970,2,190,,100,4.54,1,0,1,100,1,43,11,111,21150,1,,,2,,,43,,,39,,,11,,,6,,,111,,,60,,,21150,,,0,,
1970,2,190,,100,4.54,1,0,2,100,2,39,6,60,0,1,,,1,,,43,,,43,,,11,,,11,,,111,,,111,,,21150,,,21150,,
1970,2,190,,100,4.54,1,0,3,100,1,15,2,26,0,1,2,1,,,,43,39,43,,,,11,6,11,,,,111,60,111,,,,21150,0,21150,,,
1970,2,190,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,43,39,43,,,,11,6,11,,,,111,60,111,,,,21150,0,21150,,,
1970,2,190,,100,4.54,1,0,5,100,1,9,1,16,9999999,1,2,1,,,,43,39,43,,,,11,6,11,,,,111,60,111,,,,21150,0,21150,,,
1970,2,191,,100,4.54,1,0,1,100,1,54,3,30,7550,1,,,2,,,54,,,50,,,3,,,2,,,30,,,26,,,7550,,,5050,,
1970,2,191,,100,4.54,1,0,2,100,2,50,2,26,5050,1,,,1,,,54,,,54,,,3,,,3,,,30,,,30,,,7550,,,7550,,
1970,2,192,,100,4.54,1,0,1,100,2,62,2,26,3550,2,,,,,,62,,,,,,2,,,,,,26,,,,,,3550,,,,,
1970,2,193,,100,4.54,1,0,1,100,1,26,4,40,11050,1,,,2,,,26,,,21,,,4,,,6,,,40,,,60,,,11050,,,0,,
1970,2,193,,100,4.54,1,0,2,100,2,21,6,60,0,1,,,1,,,26,,,26,,,4,,,4,,,40,,,40,,,11050,,,11050,,
1970,2,193,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,26,21,26,,,,4,6,4,,,,40,60,40,,,,11050,0,11050,,,
1970,2,194,,100,4.54,1,0,1,100,1,28,1,16,2550,1,,,2,,,28,,,23,,,1,,,0,,,16,,,2,,,2550,,,1550,,
1970,2,194,,100,4.54,1,0,2,100,2,23,0,2,1550,1,,,1,,,28,,,28,,,1,,,1,,,16,,,16,,,2550,,,2550,,
1970,2,194,,100,4.54,1,0,3,100,1,4,0,2,9999999,1,2,1,,,,28,23,28,,,,1,0,1,,,,16,2,16,,,,2550,1550,2550,,,
1970,2,194,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,28,23,28,,,,1,0,1,,,,16,2,16,,,,2550,1550,2550,,,
1970,2,195,,100,4.54,1,0,1,100,1,48,6,60,9450,1,,,2,,,48,,,45,,,6,,,4,,,60,,,40,,,9450,,,3650,,
1970,2,195,,100,4.54,1,0,2,100,2,45,4,40,3650,1,,,1,,,48,,,48,,,6,,,6,,,60,,,60,,,9450,,,9450,,
1970,2,196,,100,4.54,1,0,1,100,1,43,3,30,18050,1,,,2,,,43,,,37,,,3,,,6,,,30,,,60,,,18050,,,2550,,
1970,2,196,,100,4.54,1,0,2,100,2,37,6,60,2550,1,,,1,,,43,,,43,,,3,,,3,,,30,,,30,,,18050,,,18050,,
1970,2,196,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,43,37,43,,,,3,6,3,,,,30,60,30,,,,18050,2550,18050,,,
1970,2,196,,100,4.54,1,0,4,100,2,8,1,14,9999999,1,2,1,,,,43,37,43,,,,3,6,3,,,,30,60,30,,,,18050,2550,18050,,,
1970,2,196,,100,4.54,1,0,5,100,2,14,2,26,0,1,2,1,,,,43,37,43,,,,3,6,3,,,,30,60,30,,,,18050,2550,18050,,,
1970,2,197,,100,4.54,1,0,1,100,1,56,2,23,7650,1,,,,,,56,,,,,,2,,,,,,23,,,,,,7650,,,,,
1970,2,198,,100,4.54,1,0,1,100,1,62,2,22,12050,1,,,2,,,62,,,59,,,2,,,7,,,22,,,70,,,12050,,,2650,,
1970,2,198,,100,4.54,1,0,2,100,2,59,7,70,2650,1,,,1,,,62,,,62,,,2,,,2,,,22,,,22,,,12050,,,12050,,
1970,2,198,,100,4.54,1,0,3,100,2,22,7,70,4350,1,2,1,,,,62,59,62,,,,2,7,2,,,,22,70,22,,,,12050,2650,12050,,,
1970,2,198,,100,4.54,1,0,4,100,2,20,1,14,1050,1,2,1,,,,62,59,62,,,,2,7,2,,,,22,70,22,,,,12050,2650,12050,,,
1970,2,199,,100,4.54,1,0,1,100,1,50,2,26,12050,1,,,2,,,50,,,51,,,2,,,2,,,26,,,25,,,12050,,,0,,
1970,2,199,,100,4.54,1,0,2,100,2,51,2,25,0,1,,,1,,,50,,,50,,,2,,,2,,,26,,,26,,,12050,,,12050,,
1970,2,199,,100,4.54,1,0,3,100,2,18,5,50,750,1,2,1,,,,50,51,50,,,,2,2,2,,,,26,25,26,,,,12050,0,12050,,,
1970,2,200,,100,4.54,1,0,1,100,1,63,4,40,12350,1,,,2,,,63,,,59,,,4,,,6,,,40,,,60,,,12350,,,0,,
1970,2,200,,100,4.54,1,0,2,100,2,59,6,60,0,1,,,1,,,63,,,63,,,4,,,4,,,40,,,40,,,12350,,,12350,,
1970,2,201,,100,4.54,1,0,1,100,1,67,4,40,6350,1,,,2,,,67,,,61,,,4,,,2,,,40,,,26,,,6350,,,0,,
1970,2,201,,100,4.54,1,0,2,100,2,61,2,26,0,1,,,1,,,67,,,67,,,4,,,4,,,40,,,40,,,6350,,,6350,,
1970,2,202,,100,4.54,1,0,1,100,1,51,6,60,10050,1,,,2,,,51,,,45,,,6,,,6,,,60,,,60,,,10050,,,0,,
1970,2,202,,100,4.54,1,0,2,100,2,45,6,60,0,1,,,1,,,51,,,51,,,6,,,6,,,60,,,60,,,10050,,,10050,,
1970,2,202,,100,4.54,1,0,3,100,2,17,4,40,150,1,2,1,,,,51,45,51,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,
1970,2,203,,100,4.54,1,0,1,100,1,42,6,60,13750,1,,,2,,,42,,,43,,,6,,,3,,,60,,,30,,,13750,,,2750,,
1970,2,203,,100,4.54,1,0,2,100,2,43,3,30,2750,1,,,1,,,42,,,42,,,6,,,6,,,60,,,60,,,13750,,,13750,,
1970,2,203,,100,4.54,1,0,3,100,1,19,5,50,1550,1,2,1,,,,42,43,42,,,,6,3,6,,,,60,30,60,,,,13750,2750,13750,,,
1970,2,203,,100,4.54,1,0,4,100,1,15,3,30,50,1,2,1,,,,42,43,42,,,,6,3,6,,,,60,30,60,,,,13750,2750,13750,,,
1970,2,203,,100,4.54,1,0,5,100,2,11,1,17,9999999,1,2,1,,,,42,43,42,,,,6,3,6,,,,60,30,60,,,,13750,2750,13750,,,
1970,2,204,,100,4.54,1,0,1,100,1,46,2,23,5250,1,,,2,,,46,,,42,,,2,,,1,,,23,,,17,,,5250,,,0,,
1970,2,204,,100,4.54,1,0,2,100,2,42,1,17,0,1,,,1,,,46,,,46,,,2,,,2,,,23,,,23,,,5250,,,5250,,
1970,2,205,,100,4.54,1,0,1,100,2,47,2,26,3550,2,,,,,,47,,,,,,2,,,,,,26,,,,,,3550,,,,,
1970,2,205,,100,4.54,1,0,2,100,1,8,1,14,9999999,2,2,,,,,47,28,,,,,2,5,,,,,26,50,,,,,3550,6050,,,,
1970,2,205,,100,4.54,1,0,3,100,2,28,5,50,6050,2,2,,,,,47,47,,,,,2,2,,,,,26,26,,,,,3550,3550,,,,
1970,2,205,,100,4.54,1,0,4,100,1,1,0,1,9999999,2,2,,,,,47,28,,,,,2,5,,,,,26,50,,,,,3550,6050,,,,
1970,2,206,,100,4.54,1,0,1,100,1,29,2,23,7250,1,,,2,,,29,,,24,,,2,,,2,,,23,,,26,,,7250,,,0,,
1970,2,206,,100,4.54,1,0,2,100,2,24,2,26,0,1,,,1,,,29,,,29,,,2,,,2,,,23,,,23,,,7250,,,7250,,
1970,2,206,,100,4.54,1,0,3,100,1,2,0,1,9999999,1,2,1,,,,29,24,29,,,,2,2,2,,,,23,26,23,,,,7250,0,7250,,,
1970,2,206,,100,4.54,1,0,4,100,1,0,0,1,9999999,1,2,1,,,,29,24,29,,,,2,2,2,,,,23,26,23,,,,7250,0,7250,,,
1970,2,207,,100,4.54,1,0,1,100,2,49,6,60,7450,2,,,,,,49,,,,,,6,,,,,,60,,,,,,7450,,,,,
1970,2,208,,100,4.54,1,0,1,100,1,35,4,40,7250,1,,,2,,,35,,,29,,,4,,,2,,,40,,,22,,,7250,,,0,,
1970,2,208,,100,4.54,1,0,2,100,2,29,2,22,0,1,,,1,,,35,,,35,,,4,,,4,,,40,,,40,,,7250,,,7250,,
1970,2,208,,100,4.54,1,0,3,100,2,7,1,16,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,
1970,2,208,,100,4.54,1,0,4,100,1,8,1,14,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,
1970,2,208,,100,4.54,1,0,5,100,1,5,0,2,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,
1970,2,208,,100,4.54,1,0,6,100,1,3,0,2,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,
1970,2,208,,100,4.54,1,0,7,100,1,0,0,1,9999999,1,2,1,,,,35,29,35,,,,4,2,4,,,,40,22,40,,,,7250,0,7250,,,
1970,2,209,,100,4.54,1,0,1,100,1,29,4,40,7050,1,,,2,,,29,,,40,,,4,,,0,,,40,,,2,,,7050,,,0,,
1970,2,209,,100,4.54,1,0,2,100,2,40,0,2,0,1,,,1,,,29,,,29,,,4,,,4,,,40,,,40,,,7050,,,7050,,
1970,2,209,,100,4.54,1,0,3,100,1,7,1,14,9999999,1,2,1,,,,29,40,29,,,,4,0,4,,,,40,2,40,,,,7050,0,7050,,,
1970,2,210,,100,4.54,1,0,1,100,1,77,1,15,1750,1,,,2,,,77,,,64,,,1,,,2,,,15,,,26,,,1750,,,1050,,
1970,2,210,,100,4.54,1,0,2,100,2,64,2,26,1050,1,,,1,,,77,,,77,,,1,,,1,,,15,,,15,,,1750,,,1750,,
1970,2,210,,100,4.54,1,0,3,100,1,26,6,65,5350,1,,,,,,77,,,,,,1,,,,,,15,,,,,,1750,,,,,
1970,2,211,,100,4.54,1,0,1,100,1,92,0,2,2250,1,,,2,,,92,,,76,,,0,,,1,,,2,,,16,,,2250,,,1150,,
1970,2,211,,100,4.54,1,0,2,100,2,76,1,16,1150,1,,,1,,,92,,,92,,,0,,,0,,,2,,,2,,,2250,,,2250,,
1970,2,212,,100,4.54,1,0,1,100,1,41,6,60,7350,1,,,2,,,41,,,38,,,6,,,4,,,60,,,40,,,7350,,,0,,
1970,2,212,,100,4.54,1,0,2,100,2,38,4,40,0,1,,,1,,,41,,,41,,,6,,,6,,,60,,,60,,,7350,,,7350,,
1970,2,212,,100,4.54,1,0,3,100,1,14,2,26,650,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,
1970,2,212,,100,4.54,1,0,4,100,2,18,6,60,0,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,
1970,2,212,,100,4.54,1,0,5,100,1,17,4,40,0,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,
1970,2,212,,100,4.54,1,0,6,100,2,12,2,23,9999999,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,
1970,2,212,,100,4.54,1,0,7,100,1,10,1,16,9999999,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,
1970,2,212,,100,4.54,1,0,8,100,2,15,2,25,2150,1,2,1,,,,41,38,41,,,,6,4,6,,,,60,40,60,,,,7350,0,7350,,,
1970,2,213,,100,4.54,1,0,1,100,2,55,2,22,1650,2,,,,,,55,,,,,,2,,,,,,22,,,,,,1650,,,,,
1970,2,213,,100,4.54,1,0,2,100,1,23,6,60,0,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,
1970,2,213,,100,4.54,1,0,3,100,1,21,6,65,0,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,
1970,2,213,,100,4.54,1,0,4,100,1,16,4,40,0,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,
1970,2,213,,100,4.54,1,0,5,100,1,13,2,26,9999999,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,
1970,2,213,,100,4.54,1,0,6,100,1,12,2,23,9999999,2,2,,,,,55,55,,,,,2,2,,,,,22,22,,,,,1650,1650,,,,
1970,2,214,,100,4.54,1,0,1,100,1,37,2,25,9550,1,,,2,,,37,,,47,,,2,,,6,,,25,,,60,,,9550,,,10650,,
1970,2,214,,100,4.54,1,0,2,100,2,47,6,60,10650,1,,,1,,,37,,,37,,,2,,,2,,,25,,,25,,,9550,,,9550,,
1970,2,214,,100,4.54,1,0,3,100,1,19,6,65,1950,1,2,1,,,,37,47,37,,,,2,6,2,,,,25,60,25,,,,9550,10650,9550,,,
1970,2,215,,100,4.54,1,0,1,100,1,52,6,60,15350,1,,,2,,,52,,,51,,,6,,,6,,,60,,,60,,,15350,,,0,,
1970,2,215,,100,4.54,1,0,2,100,2,51,6,60,0,1,,,1,,,52,,,52,,,6,,,6,,,60,,,60,,,15350,,,15350,,
1970,2,215,,100,4.54,1,0,3,100,1,26,6,60,5750,1,2,1,,,,52,51,52,,,,6,6,6,,,,60,60,60,,,,15350,0,15350,,,
1970,2,216,,100,4.54,1,0,1,100,1,50,10,100,6450,1,,,2,,,50,,,38,,,10,,,6,,,100,,,60,,,6450,,,2450,,
1970,2,216,,100,4.54,1,0,2,100,2,38,6,60,2450,1,,,1,,,50,,,50,,,10,,,10,,,100,,,100,,,6450,,,6450,,
1970,2,216,,100,4.54,1,0,3,100,1,17,4,40,650,1,2,1,,,,50,38,50,,,,10,6,10,,,,100,60,100,,,,6450,2450,6450,,,
1970,2,216,,100,4.54,1,0,4,100,1,16,4,40,950,1,2,1,,,,50,38,50,,,,10,6,10,,,,100,60,100,,,,6450,2450,6450,,,
1970,2,216,,100,4.54,1,0,5,100,1,14,2,26,150,1,2,1,,,,50,38,50,,,,10,6,10,,,,100,60,100,,,,6450,2450,6450,,,
1970,2,217,,100,4.54,1,0,1,100,1,39,6,60,11050,1,,,2,,,39,,,41,,,6,,,6,,,60,,,60,,,11050,,,1250,,
1970,2,217,,100,4.54,1,0,2,100,2,41,6,60,1250,1,,,1,,,39,,,39,,,6,,,6,,,60,,,60,,,11050,,,11050,,
1970,2,217,,100,4.54,1,0,3,100,1,17,4,40,0,1,2,1,,,,39,41,39,,,,6,6,6,,,,60,60,60,,,,11050,1250,11050,,,
1970,2,218,,100,4.54,1,0,1,100,1,29,6,60,17750,1,,,2,,,29,,,29,,,6,,,6,,,60,,,60,,,17750,,,0,,
1970,2,218,,100,4.54,1,0,2,100,2,29,6,60,0,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,17750,,,17750,,
1970,2,218,,100,4.54,1,0,3,100,1,6,1,12,9999999,1,2,1,,,,29,29,29,,,,6,6,6,,,,60,60,60,,,,17750,0,17750,,,
1970,2,218,,100,4.54,1,0,4,100,2,3,0,2,9999999,1,2,1,,,,29,29,29,,,,6,6,6,,,,60,60,60,,,,17750,0,17750,,,
1970,2,219,,100,4.54,1,0,1,100,1,30,9,90,10050,1,,,2,,,30,,,22,,,9,,,6,,,90,,,60,,,10050,,,6050,,
1970,2,219,,100,4.54,1,0,2,100,2,22,6,60,6050,1,,,1,,,30,,,30,,,9,,,9,,,90,,,90,,,10050,,,10050,,
1970,2,220,,100,4.54,1,0,1,100,1,34,10,100,12050,1,,,2,,,34,,,30,,,10,,,10,,,100,,,100,,,12050,,,0,,
1970,2,220,,100,4.54,1,0,2,100,2,30,10,100,0,1,,,1,,,34,,,34,,,10,,,10,,,100,,,100,,,12050,,,12050,,
1970,2,221,,100,4.54,1,0,1,100,1,28,10,100,35050,1,,,,,,28,,,,,,10,,,,,,100,,,,,,35050,,,,,
1970,2,222,,100,4.54,1,0,1,100,1,31,6,60,9850,1,,,2,,,31,,,26,,,6,,,7,,,60,,,70,,,9850,,,0,,
1970,2,222,,100,4.54,1,0,2,100,2,26,7,70,0,1,,,1,,,31,,,31,,,6,,,6,,,60,,,60,,,9850,,,9850,,
1970,2,222,,100,4.54,1,0,3,100,1,2,0,1,9999999,1,2,1,,,,31,26,31,,,,6,7,6,,,,60,70,60,,,,9850,0,9850,,,
1970,2,222,,100,4.54,1,0,4,100,2,0,0,1,9999999,1,2,1,,,,31,26,31,,,,6,7,6,,,,60,70,60,,,,9850,0,9850,,,
1970,2,223,,100,4.54,1,0,1,100,1,34,6,60,13250,1,,,2,,,34,,,30,,,6,,,6,,,60,,,60,,,13250,,,0,,
1970,2,223,,100,4.54,1,0,2,100,2,30,6,60,0,1,,,1,,,34,,,34,,,6,,,6,,,60,,,60,,,13250,,,13250,,
1970,2,223,,100,4.54,1,0,3,100,1,7,1,14,9999999,1,2,1,,,,34,30,34,,,,6,6,6,,,,60,60,60,,,,13250,0,13250,,,
1970,2,223,,100,4.54,1,0,4,100,1,5,0,2,9999999,1,2,1,,,,34,30,34,,,,6,6,6,,,,60,60,60,,,,13250,0,13250,,,
1970,2,223,,100,4.54,1,0,5,100,1,2,0,1,9999999,1,2,1,,,,34,30,34,,,,6,6,6,,,,60,60,60,,,,13250,0,13250,,,
1970,2,224,,100,4.54,1,0,1,100,1,52,8,80,15250,1,,,2,,,52,,,56,,,8,,,7,,,80,,,70,,,15250,,,0,,
1970,2,224,,100,4.54,1,0,2,100,2,56,7,70,0,1,,,1,,,52,,,52,,,8,,,8,,,80,,,80,,,15250,,,15250,,
1970,2,224,,100,4.54,1,0,3,100,2,19,6,60,3950,1,2,1,,,,52,56,52,,,,8,7,8,,,,80,70,80,,,,15250,0,15250,,,
1970,2,225,,100,4.54,1,0,1,100,1,34,7,70,14050,1,,,2,,,34,,,32,,,7,,,8,,,70,,,80,,,14050,,,0,,
1970,2,225,,100,4.54,1,0,2,100,2,32,8,80,0,1,,,1,,,34,,,34,,,7,,,7,,,70,,,70,,,14050,,,14050,,
1970,2,225,,100,4.54,1,0,3,100,2,8,1,15,9999999,1,2,1,,,,34,32,34,,,,7,8,7,,,,70,80,70,,,,14050,0,14050,,,
1970,2,225,,100,4.54,1,0,4,100,1,2,0,1,9999999,1,2,1,,,,34,32,34,,,,7,8,7,,,,70,80,70,,,,14050,0,14050,,,
1970,2,226,,100,4.54,1,0,1,100,1,39,7,70,15650,1,,,2,,,39,,,38,,,7,,,3,,,70,,,30,,,15650,,,0,,
1970,2,226,,100,4.54,1,0,2,100,2,38,3,30,0,1,,,1,,,39,,,39,,,7,,,7,,,70,,,70,,,15650,,,15650,,
1970,2,226,,100,4.54,1,0,3,100,2,11,1,17,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,
1970,2,226,,100,4.54,1,0,4,100,2,9,1,15,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,
1970,2,226,,100,4.54,1,0,5,100,2,5,0,2,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,
1970,2,226,,100,4.54,1,0,6,100,1,2,0,1,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,
1970,2,226,,100,4.54,1,0,7,100,1,1,0,1,9999999,1,2,1,,,,39,38,39,,,,7,3,7,,,,70,30,70,,,,15650,0,15650,,,
1970,2,227,,100,4.54,1,0,1,100,2,45,6,60,8050,2,,,,,,45,,,,,,6,,,,,,60,,,,,,8050,,,,,
1970,2,227,,100,4.54,1,0,2,100,2,17,5,50,1350,2,2,,,,,45,45,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,
1970,2,227,,100,4.54,1,0,3,100,2,20,8,80,2050,2,2,,,,,45,45,,,,,6,6,,,,,60,60,,,,,8050,8050,,,,
1970,2,228,,100,4.54,1,0,1,100,1,60,8,80,18150,1,,,2,,,60,,,56,,,8,,,6,,,80,,,60,,,18150,,,250,,
1970,2,228,,100,4.54,1,0,2,100,2,56,6,60,250,1,,,1,,,60,,,60,,,8,,,8,,,80,,,80,,,18150,,,18150,,
1970,2,228,,100,4.54,1,0,3,100,2,62,2,26,0,1,,,,,,60,,,,,,8,,,,,,80,,,,,,18150,,,,,
1970,2,230,,100,4.54,1,0,1,100,2,63,7,70,4250,2,,,,,,63,,,,,,7,,,,,,70,,,,,,4250,,,,,
1970,2,230,,100,4.54,1,0,2,100,2,57,6,60,4350,2,,,,,,63,,,,,,7,,,,,,70,,,,,,4250,,,,,
1970,2,231,,100,4.54,1,0,1,100,2,65,5,50,2250,2,,,,,,65,,,,,,5,,,,,,50,,,,,,2250,,,,,
1970,2,232,,100,4.54,1,0,1,100,1,75,2,26,23850,1,,,2,,,75,,,58,,,2,,,6,,,26,,,60,,,23850,,,1850,,
1970,2,232,,100,4.54,1,0,2,100,2,58,6,60,1850,1,,,1,,,75,,,75,,,2,,,2,,,26,,,26,,,23850,,,23850,,
1970,2,233,,100,4.54,4,4,1,100,1,42,2,22,150,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,234,,100,4.54,1,0,1,100,1,50,1,17,3850,1,,,,,,50,,,,,,1,,,,,,17,,,,,,3850,,,,,
1970,2,235,,100,4.54,1,0,1,100,2,22,5,50,1050,2,,,,,,22,,,,,,5,,,,,,50,,,,,,1050,,,,,
1970,2,235,,100,4.54,1,0,2,100,1,3,0,2,9999999,2,2,,,,,22,22,,,,,5,5,,,,,50,50,,,,,1050,1050,,,,
1970,2,235,,100,4.54,1,0,3,100,2,20,6,60,3450,2,,,,,,22,,,,,,5,,,,,,50,,,,,,1050,,,,,
1970,2,235,,100,4.54,1,0,4,100,1,1,0,1,9999999,2,2,,,,,22,20,,,,,5,6,,,,,50,60,,,,,1050,3450,,,,
1970,2,236,,100,4.54,1,0,1,100,2,79,2,26,2150,2,,,,,,79,,,,,,2,,,,,,26,,,,,,2150,,,,,
1970,2,237,,100,4.54,1,0,1,100,2,45,3,30,2150,2,,,,,,45,,,,,,3,,,,,,30,,,,,,2150,,,,,
1970,2,237,,100,4.54,1,0,2,100,2,12,2,22,9999999,2,,,,,,45,,,,,,3,,,,,,30,,,,,,2150,,,,,
1970,2,238,,100,4.54,1,0,1,100,2,40,6,60,2750,2,,,,,,40,,,,,,6,,,,,,60,,,,,,2750,,,,,
1970,2,238,,100,4.54,1,0,2,100,1,18,6,60,650,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,
1970,2,238,,100,4.54,1,0,3,100,1,17,3,30,350,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,
1970,2,238,,100,4.54,1,0,4,100,1,13,2,25,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,
1970,2,238,,100,4.54,1,0,5,100,1,11,1,17,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,
1970,2,238,,100,4.54,1,0,6,100,2,10,1,17,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,
1970,2,238,,100,4.54,1,0,7,100,2,9,1,15,9999999,2,2,,,,,40,40,,,,,6,6,,,,,60,60,,,,,2750,2750,,,,
1970,2,240,,100,4.54,1,0,1,100,1,49,4,40,350,1,,,2,,,49,,,54,,,4,,,5,,,40,,,50,,,350,,,750,,
1970,2,240,,100,4.54,1,0,2,100,2,54,5,50,750,1,,,1,,,49,,,49,,,4,,,4,,,40,,,40,,,350,,,350,,
1970,2,240,,100,4.54,1,0,3,100,2,2,0,1,9999999,1,2,1,,,,49,54,49,,,,4,5,4,,,,40,50,40,,,,350,750,350,,,
1970,2,241,,100,4.54,1,0,1,100,1,65,2,26,5550,1,,,2,,,65,,,49,,,2,,,4,,,26,,,40,,,5550,,,5350,,
1970,2,241,,100,4.54,1,0,2,100,2,49,4,40,5350,1,,,1,,,65,,,65,,,2,,,2,,,26,,,26,,,5550,,,5550,,
1970,2,241,,100,4.54,1,0,3,100,1,21,6,65,7550,1,2,1,,,,65,49,65,,,,2,4,2,,,,26,40,26,,,,5550,5350,5550,,,
1970,2,241,,100,4.54,1,0,4,100,2,19,5,50,550,1,2,1,,,,65,49,65,,,,2,4,2,,,,26,40,26,,,,5550,5350,5550,,,
1970,2,241,,100,4.54,1,0,5,100,2,74,6,60,650,1,,,,,,65,,,,,,2,,,,,,26,,,,,,5550,,,,,
1970,2,242,,100,4.54,1,0,1,100,1,72,7,70,1250,1,,,2,,,72,,,62,,,7,,,2,,,70,,,23,,,1250,,,350,,
1970,2,242,,100,4.54,1,0,2,100,2,62,2,23,350,1,,,1,,,72,,,72,,,7,,,7,,,70,,,70,,,1250,,,1250,,
1970,2,243,,100,4.54,1,0,1,100,1,28,6,60,12050,1,,,2,,,28,,,28,,,6,,,6,,,60,,,60,,,12050,,,1550,,
1970,2,243,,100,4.54,1,0,2,100,2,28,6,60,1550,1,,,1,,,28,,,28,,,6,,,6,,,60,,,60,,,12050,,,12050,,
1970,2,243,,100,4.54,1,0,3,100,1,6,1,11,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12050,1550,12050,,,
1970,2,243,,100,4.54,1,0,4,100,2,4,1,11,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12050,1550,12050,,,
1970,2,243,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,1,,,,28,28,28,,,,6,6,6,,,,60,60,60,,,,12050,1550,12050,,,
1970,2,244,,100,4.54,1,0,1,100,1,28,6,60,8050,1,,,,,,28,,,,,,6,,,,,,60,,,,,,8050,,,,,
1970,2,244,,100,4.54,1,0,2,100,2,25,6,60,6550,1,,,,,,28,,,,,,6,,,,,,60,,,,,,8050,,,,,
1970,2,245,,100,4.54,1,0,1,100,1,44,4,40,10050,1,,,2,,,44,,,43,,,4,,,4,,,40,,,40,,,10050,,,1450,,
1970,2,245,,100,4.54,1,0,2,100,2,43,4,40,1450,1,,,1,,,44,,,44,,,4,,,4,,,40,,,40,,,10050,,,10050,,
1970,2,245,,100,4.54,1,0,3,100,2,18,6,65,2750,1,2,1,,,,44,43,44,,,,4,4,4,,,,40,40,40,,,,10050,1450,10050,,,
1970,2,245,,100,4.54,1,0,4,100,1,11,1,17,9999999,1,2,1,,,,44,43,44,,,,4,4,4,,,,40,40,40,,,,10050,1450,10050,,,
1970,2,246,,100,4.54,1,0,1,100,1,27,10,100,5050,1,,,,,,27,,,,,,10,,,,,,100,,,,,,5050,,,,,
1970,2,246,,100,4.54,1,0,2,100,1,23,9,90,7350,1,,,,,,27,,,,,,10,,,,,,100,,,,,,5050,,,,,
1970,2,247,,100,4.54,1,0,1,100,1,59,4,40,350,1,,,,,,59,,,,,,4,,,,,,40,,,,,,350,,,,,
1970,2,247,,100,4.54,1,0,2,100,2,52,4,40,7350,1,,,,,,59,,,,,,4,,,,,,40,,,,,,350,,,,,
1970,2,248,,100,4.54,1,0,1,100,1,57,4,40,10750,1,,,2,,,57,,,38,,,4,,,6,,,40,,,60,,,10750,,,0,,
1970,2,248,,100,4.54,1,0,2,100,2,38,6,60,0,1,,,1,,,57,,,57,,,4,,,4,,,40,,,40,,,10750,,,10750,,
1970,2,248,,100,4.54,1,0,3,100,2,16,2,23,0,1,2,1,,,,57,38,57,,,,4,6,4,,,,40,60,40,,,,10750,0,10750,,,
1970,2,248,,100,4.54,1,0,4,100,2,4,0,2,9999999,1,2,1,,,,57,38,57,,,,4,6,4,,,,40,60,40,,,,10750,0,10750,,,
1970,2,248,,100,4.54,1,0,5,100,2,8,1,14,9999999,1,2,1,,,,57,38,57,,,,4,6,4,,,,40,60,40,,,,10750,0,10750,,,
1970,2,250,,100,4.54,1,0,1,100,1,59,6,60,7050,1,,,,,,59,,,,,,6,,,,,,60,,,,,,7050,,,,,
1970,2,251,,100,4.54,1,0,1,100,1,41,6,60,10050,1,,,2,,,41,,,33,,,6,,,6,,,60,,,60,,,10050,,,0,,
1970,2,251,,100,4.54,1,0,2,100,2,33,6,60,0,1,,,1,,,41,,,41,,,6,,,6,,,60,,,60,,,10050,,,10050,,
1970,2,251,,100,4.54,1,0,3,100,1,10,2,22,9999999,1,2,1,,,,41,33,41,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,
1970,2,251,,100,4.54,1,0,4,100,1,6,1,11,9999999,1,2,1,,,,41,33,41,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,
1970,2,252,,100,4.54,1,0,1,100,1,51,2,26,10050,1,,,2,,,51,,,46,,,2,,,2,,,26,,,26,,,10050,,,0,,
1970,2,252,,100,4.54,1,0,2,100,2,46,2,26,0,1,,,1,,,51,,,51,,,2,,,2,,,26,,,26,,,10050,,,10050,,
1970,2,252,,100,4.54,1,0,3,100,1,15,2,26,0,1,2,1,,,,51,46,51,,,,2,2,2,,,,26,26,26,,,,10050,0,10050,,,
1970,2,253,,100,4.54,1,0,1,100,2,47,6,60,8050,2,,,,,,47,,,,,,6,,,,,,60,,,,,,8050,,,,,
1970,2,254,,100,4.54,1,0,1,100,1,33,8,80,6050,1,,,2,,,33,,,41,,,8,,,6,,,80,,,60,,,6050,,,5050,,
1970,2,254,,100,4.54,1,0,2,100,2,41,6,60,5050,1,,,1,,,33,,,33,,,8,,,8,,,80,,,80,,,6050,,,6050,,
1970,2,254,,100,4.54,1,0,3,100,1,15,3,30,150,1,2,1,,,,33,41,33,,,,8,6,8,,,,80,60,80,,,,6050,5050,6050,,,
1970,2,254,,100,4.54,1,0,4,100,1,14,2,26,0,1,2,1,,,,33,41,33,,,,8,6,8,,,,80,60,80,,,,6050,5050,6050,,,
1970,2,254,,100,4.54,1,0,5,100,2,12,2,23,9999999,1,2,1,,,,33,41,33,,,,8,6,8,,,,80,60,80,,,,6050,5050,6050,,,
1970,2,254,,100,4.54,1,0,6,100,2,47,8,80,6050,1,,,,,,33,,,,,,8,,,,,,80,,,,,,6050,,,,,
1970,2,255,,100,4.54,1,0,1,100,1,42,6,60,20050,1,,,2,,,42,,,36,,,6,,,10,,,60,,,100,,,20050,,,13150,,
1970,2,255,,100,4.54,1,0,2,100,2,36,10,100,13150,1,,,1,,,42,,,42,,,6,,,6,,,60,,,60,,,20050,,,20050,,
1970,2,256,,100,4.54,1,0,1,100,1,62,11,111,16950,1,,,2,,,62,,,57,,,11,,,10,,,111,,,100,,,16950,,,50,,
1970,2,256,,100,4.54,1,0,2,100,2,57,10,100,50,1,,,1,,,62,,,62,,,11,,,11,,,111,,,111,,,16950,,,16950,,
1970,2,256,,100,4.54,1,0,3,100,2,19,7,70,850,1,2,1,,,,62,57,62,,,,11,10,11,,,,111,100,111,,,,16950,50,16950,,,
1970,2,257,,100,4.54,1,0,1,100,1,38,11,111,23050,1,,,2,,,38,,,37,,,11,,,9,,,111,,,90,,,23050,,,0,,
1970,2,257,,100,4.54,1,0,2,100,2,37,9,90,0,1,,,1,,,38,,,38,,,11,,,11,,,111,,,111,,,23050,,,23050,,
1970,2,257,,100,4.54,1,0,3,100,2,14,2,26,0,1,2,1,,,,38,37,38,,,,11,9,11,,,,111,90,111,,,,23050,0,23050,,,
1970,2,257,,100,4.54,1,0,4,100,2,11,2,22,9999999,1,2,1,,,,38,37,38,,,,11,9,11,,,,111,90,111,,,,23050,0,23050,,,
1970,2,257,,100,4.54,1,0,5,100,1,9,1,16,9999999,1,2,1,,,,38,37,38,,,,11,9,11,,,,111,90,111,,,,23050,0,23050,,,
1970,2,258,,100,4.54,1,0,1,100,1,38,6,65,3050,1,,,,,,38,,,,,,6,,,,,,65,,,,,,3050,,,,,
1970,2,259,,100,4.54,1,0,1,100,2,69,3,30,4150,2,,,,,,69,,,,,,3,,,,,,30,,,,,,4150,,,,,
1970,2,259,,100,4.54,1,0,2,100,1,26,2,23,1650,2,,,,,,69,,,,,,3,,,,,,30,,,,,,4150,,,,,
1970,2,260,,100,4.54,3,0,1,100,2,89,6,60,850,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,261,,100,4.54,1,0,1,100,1,36,6,60,11450,1,,,2,,,36,,,35,,,6,,,6,,,60,,,60,,,11450,,,250,,
1970,2,261,,100,4.54,1,0,2,100,2,35,6,60,250,1,,,1,,,36,,,36,,,6,,,6,,,60,,,60,,,11450,,,11450,,
1970,2,261,,100,4.54,1,0,3,100,2,8,1,14,9999999,1,2,1,,,,36,35,36,,,,6,6,6,,,,60,60,60,,,,11450,250,11450,,,
1970,2,261,,100,4.54,1,0,4,100,2,5,0,2,9999999,1,2,1,,,,36,35,36,,,,6,6,6,,,,60,60,60,,,,11450,250,11450,,,
1970,2,262,,100,4.54,1,0,1,100,1,59,6,60,18250,1,,,2,,,59,,,51,,,6,,,6,,,60,,,60,,,18250,,,0,,
1970,2,262,,100,4.54,1,0,2,100,2,51,6,60,0,1,,,1,,,59,,,59,,,6,,,6,,,60,,,60,,,18250,,,18250,,
1970,2,263,,100,4.54,1,0,1,100,1,48,6,60,10050,1,,,2,,,48,,,47,,,6,,,3,,,60,,,30,,,10050,,,0,,
1970,2,263,,100,4.54,1,0,2,100,2,47,3,30,0,1,,,1,,,48,,,48,,,6,,,6,,,60,,,60,,,10050,,,10050,,
1970,2,263,,100,4.54,1,0,3,100,1,20,6,60,0,1,2,1,,,,48,47,48,,,,6,3,6,,,,60,30,60,,,,10050,0,10050,,,
1970,2,263,,100,4.54,1,0,4,100,1,14,2,23,0,1,2,1,,,,48,47,48,,,,6,3,6,,,,60,30,60,,,,10050,0,10050,,,
1970,2,263,,100,4.54,1,0,5,100,2,11,1,16,9999999,1,2,1,,,,48,47,48,,,,6,3,6,,,,60,30,60,,,,10050,0,10050,,,
1970,2,264,,100,4.54,1,0,1,100,1,34,11,110,10850,1,,,2,,,34,,,34,,,11,,,11,,,110,,,110,,,10850,,,150,,
1970,2,264,,100,4.54,1,0,2,100,2,34,11,110,150,1,,,1,,,34,,,34,,,11,,,11,,,110,,,110,,,10850,,,10850,,
1970,2,264,,100,4.54,1,0,3,100,2,5,1,11,9999999,1,2,1,,,,34,34,34,,,,11,11,11,,,,110,110,110,,,,10850,150,10850,,,
1970,2,264,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,34,34,34,,,,11,11,11,,,,110,110,110,,,,10850,150,10850,,,
1970,2,265,,100,4.54,1,0,1,100,1,67,8,80,5450,1,,,2,,,67,,,62,,,8,,,7,,,80,,,70,,,5450,,,11550,,
1970,2,265,,100,4.54,1,0,2,100,2,62,7,70,11550,1,,,1,,,67,,,67,,,8,,,8,,,80,,,80,,,5450,,,5450,,
1970,2,265,,100,4.54,1,0,3,100,1,21,8,80,0,1,2,1,,,,67,62,67,,,,8,7,8,,,,80,70,80,,,,5450,11550,5450,,,
1970,2,266,,100,4.54,1,0,1,100,1,77,4,40,15750,1,,,2,,,77,,,74,,,4,,,6,,,40,,,60,,,15750,,,2250,,
1970,2,266,,100,4.54,1,0,2,100,2,74,6,60,2250,1,,,1,,,77,,,77,,,4,,,4,,,40,,,40,,,15750,,,15750,,
1970,2,267,,100,4.54,1,0,1,100,1,53,6,60,12050,1,,,2,,,53,,,52,,,6,,,6,,,60,,,60,,,12050,,,3550,,
1970,2,267,,100,4.54,1,0,2,100,2,52,6,60,3550,1,,,1,,,53,,,53,,,6,,,6,,,60,,,60,,,12050,,,12050,,
1970,2,267,,100,4.54,1,0,3,100,2,20,8,80,2550,1,2,1,,,,53,52,53,,,,6,6,6,,,,60,60,60,,,,12050,3550,12050,,,
1970,2,268,,100,4.54,1,0,1,100,1,39,11,111,1650,1,,,2,,,39,,,37,,,11,,,10,,,111,,,100,,,1650,,,3350,,
1970,2,268,,100,4.54,1,0,2,100,2,37,10,100,3350,1,,,1,,,39,,,39,,,11,,,11,,,111,,,111,,,1650,,,1650,,
1970,2,268,,100,4.54,1,0,3,100,1,6,1,12,9999999,1,2,1,,,,39,37,39,,,,11,10,11,,,,111,100,111,,,,1650,3350,1650,,,
1970,2,269,,100,4.54,1,0,1,100,2,50,2,26,2850,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,
1970,2,269,,100,4.54,1,0,2,100,2,17,3,30,2750,2,2,,,,,50,50,,,,,2,2,,,,,26,26,,,,,2850,2850,,,,
1970,2,269,,100,4.54,1,0,3,100,1,26,2,25,0,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,
1970,2,269,,100,4.54,1,0,4,100,1,26,1,17,13250,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,
1970,2,269,,100,4.54,1,0,5,100,1,26,1,15,13250,2,,,,,,50,,,,,,2,,,,,,26,,,,,,2850,,,,,
1970,2,270,,100,4.54,1,0,1,100,1,44,7,70,7150,1,,,2,,,44,,,48,,,7,,,8,,,70,,,80,,,7150,,,2650,,
1970,2,270,,100,4.54,1,0,2,100,2,48,8,80,2650,1,,,1,,,44,,,44,,,7,,,7,,,70,,,70,,,7150,,,7150,,
1970,2,270,,100,4.54,1,0,3,100,2,12,2,22,9999999,1,2,1,,,,44,48,44,,,,7,8,7,,,,70,80,70,,,,7150,2650,7150,,,
1970,2,270,,100,4.54,1,0,4,100,2,10,1,17,9999999,1,2,1,,,,44,48,44,,,,7,8,7,,,,70,80,70,,,,7150,2650,7150,,,
1970,2,271,,100,4.54,1,0,1,100,1,34,11,111,5550,1,,,2,,,34,,,29,,,11,,,11,,,111,,,111,,,5550,,,8250,,
1970,2,271,,100,4.54,1,0,2,100,2,29,11,111,8250,1,,,1,,,34,,,34,,,11,,,11,,,111,,,111,,,5550,,,5550,,
1970,2,271,,100,4.54,1,0,3,100,2,2,0,1,9999999,1,2,1,,,,34,29,34,,,,11,11,11,,,,111,111,111,,,,5550,8250,5550,,,
1970,2,272,,100,4.54,1,0,1,100,1,43,11,110,10850,1,,,2,,,43,,,34,,,11,,,11,,,110,,,111,,,10850,,,11550,,
1970,2,272,,100,4.54,1,0,2,100,2,34,11,111,11550,1,,,1,,,43,,,43,,,11,,,11,,,110,,,110,,,10850,,,10850,,
1970,2,272,,100,4.54,1,0,3,100,2,9,1,17,9999999,1,2,1,,,,43,34,43,,,,11,11,11,,,,110,111,110,,,,10850,11550,10850,,,
1970,2,273,,100,4.54,1,0,1,100,2,28,2,26,4150,2,2,,,,,28,59,,,,,2,2,,,,,26,23,,,,,4150,0,,,,
1970,2,273,,100,4.54,1,0,2,100,1,7,1,14,9999999,2,2,,,,,28,28,,,,,2,2,,,,,26,26,,,,,4150,4150,,,,
1970,2,273,,100,4.54,1,0,3,100,2,59,2,23,0,2,,,,,,28,,,,,,2,,,,,,26,,,,,,4150,,,,,
1970,2,274,,100,4.54,1,0,1,100,1,38,2,26,5050,1,,,2,,,38,,,35,,,2,,,6,,,26,,,60,,,5050,,,4050,,
1970,2,274,,100,4.54,1,0,2,100,2,35,6,60,4050,1,,,1,,,38,,,38,,,2,,,2,,,26,,,26,,,5050,,,5050,,
1970,2,274,,100,4.54,1,0,3,100,2,15,3,30,350,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,
1970,2,274,,100,4.54,1,0,4,100,2,14,2,25,0,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,
1970,2,274,,100,4.54,1,0,5,100,1,9,1,15,9999999,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,
1970,2,274,,100,4.54,1,0,6,100,1,8,1,14,9999999,1,2,1,,,,38,35,38,,,,2,6,2,,,,26,60,26,,,,5050,4050,5050,,,
1970,2,275,,100,4.54,1,0,1,100,1,72,1,14,0,1,,,,,,72,,,,,,1,,,,,,14,,,,,,0,,,,,
1970,2,276,,100,4.54,1,0,1,100,2,27,3,30,3250,2,,,,,,27,,,,,,3,,,,,,30,,,,,,3250,,,,,
1970,2,276,,100,4.54,1,0,2,100,2,9,1,16,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,
1970,2,276,,100,4.54,1,0,3,100,1,8,1,15,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,
1970,2,276,,100,4.54,1,0,4,100,1,7,1,14,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,
1970,2,276,,100,4.54,1,0,5,100,1,6,1,11,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,
1970,2,276,,100,4.54,1,0,6,100,1,4,0,2,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,
1970,2,276,,100,4.54,1,0,7,100,2,3,0,2,9999999,2,2,,,,,27,27,,,,,3,3,,,,,30,30,,,,,3250,3250,,,,
1970,2,277,,100,4.54,1,0,1,100,1,74,2,26,4450,1,,,2,,,74,,,58,,,2,,,2,,,26,,,26,,,4450,,,4450,,
1970,2,277,,100,4.54,1,0,2,100,2,58,2,26,4450,1,,,1,,,74,,,74,,,2,,,2,,,26,,,26,,,4450,,,4450,,
1970,2,278,,100,4.54,1,0,1,100,1,34,3,30,9350,1,,,2,,,34,,,24,,,3,,,6,,,30,,,60,,,9350,,,0,,
1970,2,278,,100,4.54,1,0,2,100,2,24,6,60,0,1,,,1,,,34,,,34,,,3,,,3,,,30,,,30,,,9350,,,9350,,
1970,2,279,,100,4.54,1,0,1,100,2,64,6,60,7550,2,,,,,,64,,,,,,6,,,,,,60,,,,,,7550,,,,,
1970,2,280,,100,4.54,1,0,1,100,1,48,6,60,7550,1,,,,,,48,,,,,,6,,,,,,60,,,,,,7550,,,,,
1970,2,281,,100,4.54,1,0,1,100,1,47,11,110,9250,1,,,2,,,47,,,40,,,11,,,6,,,110,,,60,,,9250,,,4450,,
1970,2,281,,100,4.54,1,0,2,100,2,40,6,60,4450,1,,,1,,,47,,,47,,,11,,,11,,,110,,,110,,,9250,,,9250,,
1970,2,281,,100,4.54,1,0,3,100,1,16,4,40,0,1,2,1,,,,47,40,47,,,,11,6,11,,,,110,60,110,,,,9250,4450,9250,,,
1970,2,281,,100,4.54,1,0,4,100,1,12,2,22,9999999,1,2,1,,,,47,40,47,,,,11,6,11,,,,110,60,110,,,,9250,4450,9250,,,
1970,2,282,,100,4.54,1,0,1,100,1,36,2,26,5250,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,
1970,2,282,,100,4.54,1,0,2,100,1,39,2,26,5050,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,
1970,2,282,,100,4.54,1,0,3,100,2,77,2,23,650,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,
1970,2,282,,100,4.54,1,0,4,100,1,47,4,40,0,1,,,,,,36,,,,,,2,,,,,,26,,,,,,5250,,,,,
1970,2,282,,100,4.54,1,0,5,100,2,14,2,26,2450,1,,1,,,,36,,47,,,,2,,4,,,,26,,40,,,,5250,,0,,,
1970,2,282,,100,4.54,1,0,6,100,2,10,1,16,9999999,1,,1,,,,36,,47,,,,2,,4,,,,26,,40,,,,5250,,0,,,
1970,2,282,,100,4.54,1,0,7,100,1,8,1,14,9999999,1,,1,,,,36,,47,,,,2,,4,,,,26,,40,,,,5250,,0,,,
1970,2,283,,100,4.54,1,0,1,100,2,23,8,80,7850,2,,,,,,23,,,,,,8,,,,,,80,,,,,,7850,,,,,
1970,2,283,,100,4.54,1,0,2,100,2,18,4,40,0,2,,,,,,23,,,,,,8,,,,,,80,,,,,,7850,,,,,
1970,2,284,,100,4.54,1,0,1,100,2,31,1,16,6250,2,,,,,,31,,,,,,1,,,,,,16,,,,,,6250,,,,,
1970,2,284,,100,4.54,1,0,2,100,1,5,0,2,9999999,2,2,,,,,31,31,,,,,1,1,,,,,16,16,,,,,6250,6250,,,,
1970,2,284,,100,4.54,1,0,3,100,2,4,0,2,9999999,2,2,,,,,31,31,,,,,1,1,,,,,16,16,,,,,6250,6250,,,,
1970,2,285,,100,4.54,1,0,1,100,1,49,2,26,9050,1,,,2,,,49,,,50,,,2,,,2,,,26,,,26,,,9050,,,1550,,
1970,2,285,,100,4.54,1,0,2,100,2,50,2,26,1550,1,,,1,,,49,,,49,,,2,,,2,,,26,,,26,,,9050,,,9050,,
1970,2,286,,100,4.54,1,0,1,100,2,61,2,26,8450,2,,,,,,61,,,,,,2,,,,,,26,,,,,,8450,,,,,
1970,2,287,,100,4.54,1,0,1,100,2,14,2,26,0,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,
1970,2,287,,100,4.54,1,0,2,100,1,10,1,17,9999999,2,2,,,,,14,14,,,,,2,2,,,,,26,26,,,,,0,0,,,,
1970,2,287,,100,4.54,1,0,3,100,1,12,1,17,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,
1970,2,287,,100,4.54,1,0,4,100,1,9,1,16,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,
1970,2,287,,100,4.54,1,0,5,100,1,5,1,12,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,
1970,2,287,,100,4.54,1,0,6,100,1,48,2,25,6450,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,
1970,2,287,,100,4.54,1,0,7,100,2,2,0,1,9999999,2,,,,,,14,,,,,,2,,,,,,26,,,,,,0,,,,,
1970,2,288,,100,4.54,1,0,1,100,1,58,4,40,8050,1,,,2,,,58,,,53,,,4,,,6,,,40,,,60,,,8050,,,7050,,
1970,2,288,,100,4.54,1,0,2,100,2,53,6,60,7050,1,,,1,,,58,,,58,,,4,,,4,,,40,,,40,,,8050,,,8050,,
1970,2,289,,100,4.54,1,0,1,100,1,67,3,30,14750,1,,,,,,67,,,,,,3,,,,,,30,,,,,,14750,,,,,
1970,2,290,,100,4.54,1,0,1,100,1,73,2,25,1950,1,,,2,,,73,,,68,,,2,,,2,,,25,,,25,,,1950,,,650,,
1970,2,290,,100,4.54,1,0,2,100,2,68,2,25,650,1,2,,1,,,73,89,,73,,,2,2,,2,,,25,25,,25,,,1950,850,,1950,,
1970,2,290,,100,4.54,1,0,3,100,2,89,2,25,850,1,,,,,,73,,,,,,2,,,,,,25,,,,,,1950,,,,,
1970,2,291,,100,4.54,1,0,1,100,1,68,2,26,3450,1,,,2,,,68,,,65,,,2,,,4,,,26,,,40,,,3450,,,1750,,
1970,2,291,,100,4.54,1,0,2,100,2,65,4,40,1750,1,,,1,,,68,,,68,,,2,,,2,,,26,,,26,,,3450,,,3450,,
1970,2,292,,100,4.54,1,0,1,100,1,29,6,60,12050,1,,,2,,,29,,,28,,,6,,,6,,,60,,,60,,,12050,,,4050,,
1970,2,292,,100,4.54,1,0,2,100,2,28,6,60,4050,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,12050,,,12050,,
1970,2,292,,100,4.54,1,0,3,100,2,7,1,12,9999999,1,2,1,,,,29,28,29,,,,6,6,6,,,,60,60,60,,,,12050,4050,12050,,,
1970,2,292,,100,4.54,1,0,4,100,1,6,1,11,9999999,1,2,1,,,,29,28,29,,,,6,6,6,,,,60,60,60,,,,12050,4050,12050,,,
1970,2,292,,100,4.54,1,0,5,100,1,3,0,2,9999999,1,2,1,,,,29,28,29,,,,6,6,6,,,,60,60,60,,,,12050,4050,12050,,,
1970,2,293,,100,4.54,1,0,1,100,2,63,6,60,9950,2,,,,,,63,,,,,,6,,,,,,60,,,,,,9950,,,,,
1970,2,294,,100,4.54,1,0,1,100,1,42,6,60,30150,1,,,2,,,42,,,39,,,6,,,7,,,60,,,70,,,30150,,,1050,,
1970,2,294,,100,4.54,1,0,2,100,2,39,7,70,1050,1,,,1,,,42,,,42,,,6,,,6,,,60,,,60,,,30150,,,30150,,
1970,2,294,,100,4.54,1,0,3,100,2,16,4,40,450,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,
1970,2,294,,100,4.54,1,0,4,100,1,15,3,30,450,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,
1970,2,294,,100,4.54,1,0,5,100,1,14,2,26,0,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,
1970,2,294,,100,4.54,1,0,6,100,1,11,2,22,9999999,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,
1970,2,294,,100,4.54,1,0,7,100,2,9,1,16,9999999,1,2,1,,,,42,39,42,,,,6,7,6,,,,60,70,60,,,,30150,1050,30150,,,
1970,2,295,,100,4.54,1,0,1,100,1,70,5,50,450,1,,,2,,,70,,,70,,,5,,,5,,,50,,,50,,,450,,,150,,
1970,2,295,,100,4.54,1,0,2,100,2,70,5,50,150,1,,,1,,,70,,,70,,,5,,,5,,,50,,,50,,,450,,,450,,
1970,2,296,,100,4.54,1,0,1,100,1,47,5,50,7050,1,,,2,,,47,,,44,,,5,,,5,,,50,,,50,,,7050,,,4550,,
1970,2,296,,100,4.54,1,0,2,100,2,44,5,50,4550,1,2,,1,,,47,71,,47,,,5,4,,5,,,50,40,,50,,,7050,650,,7050,,
1970,2,296,,100,4.54,1,0,3,100,2,71,4,40,650,1,,,,,,47,,,,,,5,,,,,,50,,,,,,7050,,,,,
1970,2,296,,100,4.54,1,0,4,100,1,17,5,50,550,1,2,1,,,,47,44,47,,,,5,5,5,,,,50,50,50,,,,7050,4550,7050,,,
1970,2,297,,100,4.54,1,0,1,100,1,33,10,100,17650,1,,,2,,,33,,,33,,,10,,,10,,,100,,,100,,,17650,,,0,,
1970,2,297,,100,4.54,1,0,2,100,2,33,10,100,0,1,,,1,,,33,,,33,,,10,,,10,,,100,,,100,,,17650,,,17650,,
1970,2,297,,100,4.54,1,0,3,100,1,8,1,15,9999999,1,2,1,,,,33,33,33,,,,10,10,10,,,,100,100,100,,,,17650,0,17650,,,
1970,2,297,,100,4.54,1,0,4,100,2,1,0,1,9999999,1,2,1,,,,33,33,33,,,,10,10,10,,,,100,100,100,,,,17650,0,17650,,,
1970,2,297,,100,4.54,1,0,5,100,2,4,1,11,9999999,1,2,1,,,,33,33,33,,,,10,10,10,,,,100,100,100,,,,17650,0,17650,,,
1970,2,298,,100,4.54,1,0,1,100,1,49,2,26,14050,1,,,2,,,49,,,46,,,2,,,6,,,26,,,60,,,14050,,,7450,,
1970,2,298,,100,4.54,1,0,2,100,2,46,6,60,7450,1,,,1,,,49,,,49,,,2,,,2,,,26,,,26,,,14050,,,14050,,
1970,2,298,,100,4.54,1,0,3,100,2,13,2,23,9999999,1,2,1,,,,49,46,49,,,,2,6,2,,,,26,60,26,,,,14050,7450,14050,,,
1970,2,299,,100,4.54,1,0,1,100,1,47,6,60,15750,1,,,2,,,47,,,46,,,6,,,6,,,60,,,60,,,15750,,,4850,,
1970,2,299,,100,4.54,1,0,2,100,2,46,6,60,4850,1,,,1,,,47,,,47,,,6,,,6,,,60,,,60,,,15750,,,15750,,
1970,2,299,,100,4.54,1,0,3,100,2,18,6,65,2550,1,2,1,,,,47,46,47,,,,6,6,6,,,,60,60,60,,,,15750,4850,15750,,,
1970,2,300,,100,4.54,1,0,1,100,1,64,5,50,2450,1,,,2,,,64,,,63,,,5,,,2,,,50,,,22,,,2450,,,0,,
1970,2,300,,100,4.54,1,0,2,100,2,63,2,22,0,1,,,1,,,64,,,64,,,5,,,5,,,50,,,50,,,2450,,,2450,,
1970,2,301,,100,4.54,1,0,1,100,1,20,7,70,8050,1,,,2,,,20,,,21,,,7,,,7,,,70,,,70,,,8050,,,3050,,
1970,2,301,,100,4.54,1,0,2,100,2,21,7,70,3050,1,,,1,,,20,,,20,,,7,,,7,,,70,,,70,,,8050,,,8050,,
1970,2,301,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,20,21,20,,,,7,7,7,,,,70,70,70,,,,8050,3050,8050,,,
1970,2,302,,100,4.54,1,0,1,100,2,23,2,26,4450,2,,,,,,23,,,,,,2,,,,,,26,,,,,,4450,,,,,
1970,2,303,,100,4.54,1,0,1,100,1,45,6,60,11750,1,,,2,,,45,,,41,,,6,,,6,,,60,,,60,,,11750,,,0,,
1970,2,303,,100,4.54,1,0,2,100,2,41,6,60,0,1,2,1,1,,,45,67,71,45,,,6,2,2,6,,,60,26,23,60,,,11750,1450,3350,11750,,
1970,2,303,,100,4.54,1,0,3,100,2,21,9,90,550,1,2,1,,,,45,41,45,,,,6,6,6,,,,60,60,60,,,,11750,0,11750,,,
1970,2,303,,100,4.54,1,0,4,100,2,17,5,50,550,1,2,1,,,,45,41,45,,,,6,6,6,,,,60,60,60,,,,11750,0,11750,,,
1970,2,303,,100,4.54,1,0,5,100,1,15,3,30,0,1,2,1,,,,45,41,45,,,,6,6,6,,,,60,60,60,,,,11750,0,11750,,,
1970,2,303,,100,4.54,1,0,6,100,1,71,2,23,3350,1,,,2,,,45,,,67,,,6,,,2,,,60,,,26,,,11750,,,1450,,
1970,2,303,,100,4.54,1,0,7,100,2,67,2,26,1450,1,,,1,,,45,,,71,,,6,,,2,,,60,,,23,,,11750,,,3350,,
1970,2,304,,100,4.54,1,0,1,100,1,75,2,26,3650,1,,,2,,,75,,,75,,,2,,,2,,,26,,,26,,,3650,,,850,,
1970,2,304,,100,4.54,1,0,2,100,2,75,2,26,850,1,,,1,,,75,,,75,,,2,,,2,,,26,,,26,,,3650,,,3650,,
1970,2,305,,100,4.54,1,0,1,100,1,38,6,60,10050,1,,,2,,,38,,,38,,,6,,,6,,,60,,,60,,,10050,,,0,,
1970,2,305,,100,4.54,1,0,2,100,2,38,6,60,0,1,,,1,,,38,,,38,,,6,,,6,,,60,,,60,,,10050,,,10050,,
1970,2,305,,100,4.54,1,0,3,100,1,11,2,22,9999999,1,2,1,,,,38,38,38,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,
1970,2,305,,100,4.54,1,0,4,100,1,9,1,16,9999999,1,2,1,,,,38,38,38,,,,6,6,6,,,,60,60,60,,,,10050,0,10050,,,
1970,2,306,,100,4.54,1,0,1,100,1,47,3,30,11050,1,,,2,,,47,,,40,,,3,,,6,,,30,,,60,,,11050,,,0,,
1970,2,306,,100,4.54,1,0,2,100,2,40,6,60,0,1,,,1,,,47,,,47,,,3,,,3,,,30,,,30,,,11050,,,11050,,
1970,2,306,,100,4.54,1,0,3,100,1,16,3,30,0,1,2,1,,,,47,40,47,,,,3,6,3,,,,30,60,30,,,,11050,0,11050,,,
1970,2,306,,100,4.54,1,0,4,100,2,9,1,16,9999999,1,2,1,,,,47,40,47,,,,3,6,3,,,,30,60,30,,,,11050,0,11050,,,
1970,2,306,,100,4.54,1,0,5,100,2,7,1,14,9999999,1,2,1,,,,47,40,47,,,,3,6,3,,,,30,60,30,,,,11050,0,11050,,,
1970,2,307,,100,4.54,1,0,1,100,1,35,6,60,150,1,2,,,,,35,73,,,,,6,2,,,,,60,26,,,,,150,1250,,,,
1970,2,307,,100,4.54,1,0,2,100,2,73,2,26,1250,1,,,,,,35,,,,,,6,,,,,,60,,,,,,150,,,,,
1970,2,308,,100,4.54,1,0,1,100,1,77,2,22,4050,1,,,,,,77,,,,,,2,,,,,,22,,,,,,4050,,,,,
1970,2,308,,100,4.54,1,0,2,100,1,50,4,40,650,1,,1,,,,77,,77,,,,2,,2,,,,22,,22,,,,4050,,4050,,,
1970,2,308,,100,4.54,1,0,3,100,1,41,2,26,6550,1,,1,,,,77,,77,,,,2,,2,,,,22,,22,,,,4050,,4050,,,
1970,2,309,,100,4.54,1,0,1,100,1,61,5,50,3550,1,,,,,,61,,,,,,5,,,,,,50,,,,,,3550,,,,,
1970,2,309,,100,4.54,1,0,2,100,1,17,5,50,550,1,,1,,,,61,,61,,,,5,,5,,,,50,,50,,,,3550,,3550,,,
1970,2,310,,100,4.54,1,0,1,100,1,56,6,60,7050,1,,,2,,,56,,,61,,,6,,,6,,,60,,,60,,,7050,,,0,,
1970,2,310,,100,4.54,1,0,2,100,2,61,6,60,0,1,,,1,,,56,,,56,,,6,,,6,,,60,,,60,,,7050,,,7050,,
1970,2,311,,100,4.54,1,0,1,100,1,43,11,110,11850,1,,,2,,,43,,,42,,,11,,,10,,,110,,,100,,,11850,,,250,,
1970,2,311,,100,4.54,1,0,2,100,2,42,10,100,250,1,,,1,,,43,,,43,,,11,,,11,,,110,,,110,,,11850,,,11850,,
1970,2,312,,100,4.54,1,0,1,100,2,26,6,60,4050,2,,,,,,26,,,,,,6,,,,,,60,,,,,,4050,,,,,
1970,2,312,,100,4.54,1,0,2,100,1,9,1,15,9999999,2,2,,,,,26,26,,,,,6,6,,,,,60,60,,,,,4050,4050,,,,
1970,2,312,,100,4.54,1,0,3,100,1,7,1,12,9999999,2,2,,,,,26,26,,,,,6,6,,,,,60,60,,,,,4050,4050,,,,
1970,2,313,,100,4.54,1,0,1,100,1,22,10,100,0,1,,,,,,22,,,,,,10,,,,,,100,,,,,,0,,,,,
1970,2,314,,100,4.54,1,0,1,100,1,27,6,60,7050,1,,,,,,27,,,,,,6,,,,,,60,,,,,,7050,,,,,
1970,2,315,,100,4.54,3,0,1,100,2,61,8,80,150,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
1970,2,316,,100,4.54,1,0,1,100,1,62,4,40,12050,1,,,2,,,62,,,61,,,4,,,4,,,40,,,40,,,12050,,,1350,,
1970,2,316,,100,4.54,1,0,2,100,2,61,4,40,1350,1,,,1,,,62,,,62,,,4,,,4,,,40,,,40,,,12050,,,12050,,
1970,2,317,,100,4.54,1,0,1,100,2,45,8,80,5050,2,,,,,,45,,,,,,8,,,,,,80,,,,,,5050,,,,,
1970,2,318,,100,4.54,1,0,1,100,2,23,8,80,3050,2,,,,,,23,,,,,,8,,,,,,80,,,,,,3050,,,,,
1970,2,318,,100,4.54,1,0,2,100,1,2,0,1,9999999,2,2,,,,,23,23,,,,,8,8,,,,,80,80,,,,,3050,3050,,,,
1970,2,319,,100,4.54,1,0,1,100,1,43,10,100,22050,1,,,2,,,43,,,37,,,10,,,10,,,100,,,100,,,22050,,,150,,
1970,2,319,,100,4.54,1,0,2,100,2,37,10,100,150,1,,,1,,,43,,,43,,,10,,,10,,,100,,,100,,,22050,,,22050,,
1970,2,319,,100,4.54,1,0,3,100,2,14,2,26,0,1,2,1,,,,43,37,43,,,,10,10,10,,,,100,100,100,,,,22050,150,22050,,,
1970,2,319,,100,4.54,1,0,4,100,1,11,1,17,9999999,1,2,1,,,,43,37,43,,,,10,10,10,,,,100,100,100,,,,22050,150,22050,,,
1970,2,320,,100,4.54,1,0,1,100,2,79,2,26,750,2,,,,,,79,,,,,,2,,,,,,26,,,,,,750,,,,,
1970,2,321,,100,4.54,1,0,1,100,1,24,6,60,5050,1,,,2,,,24,,,21,,,6,,,6,,,60,,,60,,,5050,,,4050,,
1970,2,321,,100,4.54,1,0,2,100,2,21,6,60,4050,1,,,1,,,24,,,24,,,6,,,6,,,60,,,60,,,5050,,,5050,,
1970,2,321,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,24,21,24,,,,6,6,6,,,,60,60,60,,,,5050,4050,5050,,,
1970,2,322,,100,4.54,1,0,1,100,2,60,6,60,6550,2,,,,,,60,,,,,,6,,,,,,60,,,,,,6550,,,,,
1970,2,322,,100,4.54,1,0,2,100,1,24,6,60,1350,2,2,,,,,60,60,,,,,6,6,,,,,60,60,,,,,6550,6550,,,,
1970,2,323,,100,4.54,1,0,1,100,1,77,7,70,10050,1,,,2,,,77,,,71,,,7,,,3,,,70,,,30,,,10050,,,0,,
1970,2,323,,100,4.54,1,0,2,100,2,71,3,30,0,1,,,1,,,77,,,77,,,7,,,7,,,70,,,70,,,10050,,,10050,,
1970,2,324,,100,4.54,1,0,1,100,1,66,3,30,8050,1,,,2,,,66,,,60,,,3,,,5,,,30,,,50,,,8050,,,5050,,
1970,2,324,,100,4.54,1,0,2,100,2,60,5,50,5050,1,,,1,,,66,,,66,,,3,,,3,,,30,,,30,,,8050,,,8050,,
1970,2,325,,100,4.54,1,0,1,100,1,40,2,25,7050,1,,,2,,,40,,,34,,,2,,,4,,,25,,,40,,,7050,,,6050,,
1970,2,325,,100,4.54,1,0,2,100,2,34,4,40,6050,1,,,1,,,40,,,40,,,2,,,2,,,25,,,25,,,7050,,,7050,,
1970,2,325,,100,4.54,1,0,3,100,2,16,3,30,3850,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,
1970,2,325,,100,4.54,1,0,4,100,1,15,2,25,0,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,
1970,2,325,,100,4.54,1,0,5,100,2,13,2,25,9999999,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,
1970,2,325,,100,4.54,1,0,6,100,2,13,2,22,9999999,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,
1970,2,325,,100,4.54,1,0,7,100,2,0,0,1,9999999,1,2,1,,,,40,34,40,,,,2,4,2,,,,25,40,25,,,,7050,6050,7050,,,
1970,2,326,,100,4.54,1,0,1,100,1,37,6,60,4850,1,,,2,,,37,,,40,,,6,,,5,,,60,,,50,,,4850,,,0,,
1970,2,326,,100,4.54,1,0,2,100,2,40,5,50,0,1,,,1,,,37,,,37,,,6,,,6,,,60,,,60,,,4850,,,4850,,
1970,2,326,,100,4.54,1,0,3,100,2,19,6,60,2650,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,
1970,2,326,,100,4.54,1,0,4,100,2,17,5,50,0,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,
1970,2,326,,100,4.54,1,0,5,100,2,15,2,26,0,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,
1970,2,326,,100,4.54,1,0,6,100,1,14,2,25,0,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,
1970,2,326,,100,4.54,1,0,7,100,2,13,2,23,9999999,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,
1970,2,326,,100,4.54,1,0,8,100,2,12,2,22,9999999,1,2,1,,,,37,40,37,,,,6,5,6,,,,60,50,60,,,,4850,0,4850,,,
1970,2,326,,100,4.54,1,0,9,100,1,1,0,1,9999999,1,2,,,,,37,19,,,,,6,6,,,,,60,60,,,,,4850,2650,,,,
1970,2,327,,100,4.54,1,0,1,100,2,53,2,23,550,2,,,,,,53,,,,,,2,,,,,,23,,,,,,550,,,,,
1970,2,329,,100,4.54,1,0,1,100,1,33,1,17,7050,1,,,2,,,33,,,27,,,1,,,2,,,17,,,23,,,7050,,,2050,,
1970,2,329,,100,4.54,1,0,2,100,2,27,2,23,2050,1,,,1,,,33,,,33,,,1,,,1,,,17,,,17,,,7050,,,7050,,
1970,2,329,,100,4.54,1,0,3,100,2,0,0,1,9999999,1,2,1,,,,33,27,33,,,,1,2,1,,,,17,23,17,,,,7050,2050,7050,,,
1970,2,330,,100,4.54,1,0,1,100,1,29,3,30,9550,1,,,2,,,29,,,24,,,3,,,3,,,30,,,30,,,9550,,,0,,
1970,2,330,,100,4.54,1,0,2,100,2,24,3,30,0,1,,,1,,,29,,,29,,,3,,,3,,,30,,,30,,,9550,,,9550,,
1970,2,330,,100,4.54,1,0,3,100,2,3,0,2,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,
1970,2,330,,100,4.54,1,0,4,100,2,2,0,1,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,
1970,2,330,,100,4.54,1,0,5,100,1,0,0,1,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,
1970,2,330,,100,4.54,1,0,6,100,1,0,0,1,9999999,1,2,1,,,,29,24,29,,,,3,3,3,,,,30,30,30,,,,9550,0,9550,,,
1970,2,331,,100,4.54,1,0,1,100,2,45,1,17,0,2,,,,,,45,,,,,,1,,,,,,17,,,,,,0,,,,,
1970,2,331,,100,4.54,1,0,2,100,1,45,1,16,6150,2,,,,,,45,,,,,,1,,,,,,17,,,,,,0,,,,,
1970,2,331,,100,4.54,1,0,3,100,2,20,2,22,4150,2,2,,,,,45,45,,,,,1,1,,,,,17,17,,,,,0,0,,,,
1970,2,332,,100,4.54,1,0,1,100,1,39,6,60,6550,1,,,,,,39,,,,,,6,,,,,,60,,,,,,6550,,,,,
1970,2,333,,100,4.54,1,0,1,100,1,19,6,65,1750,1,,,,,,19,,,,,,6,,,,,,65,,,,,,1750,,,,,
1970,2,333,,100,4.54,1,0,2,100,1,18,7,70,2050,1,,,,,,19,,,,,,6,,,,,,65,,,,,,1750,,,,,
1970,2,333,,100,4.54,1,0,3,100,1,21,9,90,1950,1,,,,,,19,,,,,,6,,,,,,65,,,,,,1750,,,,,
1970,2,334,,100,4.54,1,0,1,100,1,40,3,30,9450,1,,,2,,,40,,,42,,,3,,,2,,,30,,,26,,,9450,,,0,,
1970,2,334,,100,4.54,1,0,2,100,2,42,2,26,0,1,,,1,,,40,,,40,,,3,,,3,,,30,,,30,,,9450,,,9450,,
1970,2,336,,100,4.54,1,0,1,100,2,29,5,50,3750,2,,,,,,29,,,,,,5,,,,,,50,,,,,,3750,,,,,
1970,2,336,,100,4.54,1,0,2,100,2,5,0,2,9999999,2,2,,,,,29,29,,,,,5,5,,,,,50,50,,,,,3750,3750,,,,
1970,2,336,,100,4.54,1,0,3,100,2,4,0,2,9999999,2,2,,,,,29,29,,,,,5,5,,,,,50,50,,,,,3750,3750,,,,
1970,2,337,,100,4.54,1,0,1,100,1,48,11,110,15050,1,,,2,,,48,,,49,,,11,,,2,,,110,,,23,,,15050,,,0,,
1970,2,337,,100,4.54,1,0,2,100,2,49,2,23,0,1,,,1,,,48,,,48,,,11,,,11,,,110,,,110,,,15050,,,15050,,
1970,2,337,,100,4.54,1,0,3,100,1,18,6,60,1750,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,
1970,2,337,,100,4.54,1,0,4,100,1,17,5,50,1050,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,
1970,2,337,,100,4.54,1,0,5,100,1,14,2,25,50,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,
1970,2,337,,100,4.54,1,0,6,100,2,11,2,22,9999999,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,
1970,2,337,,100,4.54,1,0,7,100,1,10,1,17,9999999,1,2,1,,,,48,49,48,,,,11,2,11,,,,110,23,110,,,,15050,0,15050,,,
1970,2,338,,100,4.54,1,0,1,100,1,57,9,90,10050,1,,,2,,,57,,,49,,,9,,,6,,,90,,,60,,,10050,,,250,,
1970,2,338,,100,4.54,1,0,2,100,2,49,6,60,250,1,,,1,,,57,,,57,,,9,,,9,,,90,,,90,,,10050,,,10050,,
1970,2,338,,100,4.54,1,0,3,100,1,16,4,40,1450,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,
1970,2,338,,100,4.54,1,0,4,100,2,14,2,26,0,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,
1970,2,338,,100,4.54,1,0,5,100,1,11,2,22,9999999,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,
1970,2,338,,100,4.54,1,0,6,100,1,6,1,12,9999999,1,2,1,,,,57,49,57,,,,9,6,9,,,,90,60,90,,,,10050,250,10050,,,
1970,2,339,,100,4.54,1,0,1,100,1,30,6,60,12050,1,,,2,,,30,,,28,,,6,,,6,,,60,,,60,,,12050,,,0,,
1970,2,339,,100,4.54,1,0,2,100,2,28,6,60,0,1,,,1,,,30,,,30,,,6,,,6,,,60,,,60,,,12050,,,12050,,
1970,2,339,,100,4.54,1,0,3,100,1,7,1,14,9999999,1,2,1,,,,30,28,30,,,,6,6,6,,,,60,60,60,,,,12050,0,12050,,,
1970,2,339,,100,4.54,1,0,4,100,1,5,1,11,9999999,1,2,1,,,,30,28,30,,,,6,6,6,,,,60,60,60,,,,12050,0,12050,,,
1970,2,339,,100,4.54,1,0,5,100,2,0,0,1,9999999,1,2,1,,,,30,28,30,,,,6,6,6,,,,60,60,60,,,,12050,0,12050,,,
1970,2,340,,100,4.54,1,0,1,100,1,57,3,30,5250,1,,,2,,,57,,,62,,,3,,,2,,,30,,,26,,,5250,,,0,,
1970,2,340,,100,4.54,1,0,2,100,2,62,2,26,0,1,,,1,,,57,,,57,,,3,,,3,,,30,,,30,,,5250,,,5250,,
1970,2,341,,100,4.54,1,0,1,100,1,73,1,16,1750,1,,,2,,,73,,,67,,,1,,,1,,,16,,,16,,,1750,,,750,,
1970,2,341,,100,4.54,1,0,2,100,2,67,1,16,750,1,,,1,,,73,,,73,,,1,,,1,,,16,,,16,,,1750,,,1750,,
1970,2,342,,100,4.54,1,0,1,100,1,29,6,60,7050,1,,,2,,,29,,,21,,,6,,,5,,,60,,,50,,,7050,,,3350,,
1970,2,342,,100,4.54,1,0,2,100,2,21,5,50,3350,1,,,1,,,29,,,29,,,6,,,6,,,60,,,60,,,7050,,,7050,,
1970,2,342,,100,4.54,1,0,3,100,1,0,0,1,9999999,1,2,1,,,,29,21,29,,,,6,5,6,,,,60,50,60,,,,7050,3350,7050,,,
1970,2,343,,100,4.54,1,0,1,100,1,36,11,111,24250,1,,,2,,,36,,,30,,,11,,,10,,,111,,,100,,,24250,,,7050,,
1970,2,343,,100,4.54,1,0,2,100,2,30,10,100,7050,1,,,1,,,36,,,36,,,11,,,11,,,111,,,111,,,24250,,,24250,,
1970,2,343,,100,4.54,1,0,3,100,2,7,1,14,9999999,1,2,1,,,,36,30,36,,,,11,10,11,,,,111,100,111,,,,24250,7050,24250,,,
1970,2,343,,100,4.54,1,0,4,100,2,3,0,2,9999999,1,2,1,,,,36,30,36,,,,11,10,11,,,,111,100,111,,,,24250,7050,24250,,,
1970,2,344,,100,4.54,1,0,1,100,1,24,11,111,250,1,,,,,,24,,,,,,11,,,,,,111,,,,,,250,,,,,
1970,2,344,,100,4.54,1,0,2,100,1,6,1,12,9999999,1,,1,,,,24,,24,,,,11,,11,,,,111,,111,,,,250,,250,,,
1970,2,344,,100,4.54,1,0,3,100,1,8,1,15,9999999,1,,1,,,,24,,24,,,,11,,11,,,,111,,111,,,,250,,250,,,
1970,2,345,,100,4.54,1,0,1,100,1,58,8,80,11850,1,,,2,,,58,,,57,,,8,,,6,,,80,,,60,,,11850,,,6450,,
1970,2,345,,100,4.54,1,0,2,100,2,57,6,60,6450,1,,,1,,,58,,,58,,,8,,,8,,,80,,,80,,,11850,,,11850,,
1970,2,345,,100,4.54,1,0,3,100,2,28,7,70,6150,1,2,1,,,,58,57,58,,,,8,6,8,,,,80,60,80,,,,11850,6450,11850,,,
1970,2,346,,100,4.54,1,0,1,100,1,39,6,60,12050,1,,,2,,,39,,,33,,,6,,,6,,,60,,,60,,,12050,,,3050,,
1970,2,346,,100,4.54,1,0,2,100,2,33,6,60,3050,1,,,1,,,39,,,39,,,6,,,6,,,60,,,60,,,12050,,,12050,,
1970,2,346,,100,4.54,1,0,3,100,1,16,3,30,0,1,2,1,,,,39,33,39,,,,6,6,6,,,,60,60,60,,,,12050,3050,12050,,,
1970,2,346,,100,4.54,1,0,4,100,1,14,2,25,0,1,2,1,,,,39,33,39,,,,6,6,6,,,,60,60,60,,,,12050,3050,12050,,,
1970,2,346,,100,4.54,1,0,5,100,1,4,1,11,9999999,1,2,1,,,,39,33,39,,,,6,6,6,,,,60,60,60,,,,12050,3050,12050,,,
1970,2,347,,100,4.54,1,0,1,100,1,38,11,111,46550,1,,,2,,,38,,,33,,,11,,,6,,,111,,,60,,,46550,,,0,,
1970,2,347,,100,4.54,1,0,2,100,2,33,6,60,0,1,,,1,,,38,,,38,,,11,,,11,,,111,,,111,,,46550,,,46550,,
1970,2,347,,100,4.54,1,0,3,100,1,10,1,16,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,
1970,2,347,,100,4.54,1,0,4,100,1,9,1,15,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,
1970,2,347,,100,4.54,1,0,5,100,2,7,1,14,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,
1970,2,347,,100,4.54,1,0,6,100,1,5,1,11,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,
1970,2,347,,100,4.54,1,0,7,100,2,4,0,2,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,
1970,2,347,,100,4.54,1,0,8,100,1,1,0,1,9999999,1,2,1,,,,38,33,38,,,,11,6,11,,,,111,60,111,,,,46550,0,46550,,,
1970,2,348,,100,4.54,1,0,1,100,2,57,8,80,9050,2,,,,,,57,,,,,,8,,,,,,80,,,,,,9050,,,,,
================================================
FILE: examples/data/nyc-taxi_1k.csv
================================================
1460000001,2,2017-12-15 00:00:28,2017-12-15 00:15:43,N,1,,,,,2,1.50,11,0.5,0.5,1.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000002,2,2017-12-15 00:33:12,2017-12-15 00:51:04,N,1,,,,,3,2.53,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000003,2,2017-12-15 00:56:59,2017-12-15 00:59:51,N,1,,,,,3,0.06,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000004,2,2017-12-15 00:09:19,2017-12-15 00:18:54,N,1,,,,,5,1.47,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000005,2,2017-12-15 00:21:02,2017-12-15 00:25:30,N,1,,,,,5,0.23,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000006,2,2017-12-15 00:35:38,2017-12-15 01:26:19,N,1,,,,,5,18.48,58,0.5,0.5,9.76,5.76,,0.3,76.77,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000007,1,2017-12-15 00:48:29,2017-12-15 01:01:42,N,1,,,,,1,2.60,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000008,2,2017-12-15 00:09:01,2017-12-15 00:11:57,N,1,,,,,1,0.85,4.5,0.5,0.5,0,0,,0.3,5.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000009,2,2017-12-15 00:32:00,2017-12-15 00:35:42,N,1,,,,,1,0.48,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000010,2,2017-12-15 00:42:13,2017-12-15 00:54:48,N,1,,,,,1,2.09,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000011,2,2017-12-15 00:05:46,2017-12-15 00:09:49,N,1,,,,,3,0.60,4.5,0.5,0.5,0.87,0,,0.3,6.67,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000012,2,2017-12-15 00:10:48,2017-12-15 00:20:24,N,1,,,,,3,0.97,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000013,2,2017-12-15 00:26:39,2017-12-15 00:39:18,N,1,,,,,3,1.82,9.5,0.5,0.5,3,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000014,2,2017-12-15 00:40:49,2017-12-15 01:16:48,N,1,,,,,3,8.99,31,0.5,0.5,6.46,0,,0.3,38.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000015,2,2017-12-15 00:24:04,2017-12-15 00:34:56,N,1,,,,,1,5.04,16,0.5,0.5,5.19,0,,0.3,22.49,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000016,1,2017-12-15 00:06:12,2017-12-15 00:35:33,N,1,,,,,1,5.30,21.5,0.5,0.5,1.5,0,,0.3,24.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000017,1,2017-12-15 00:38:38,2017-12-15 00:54:18,N,1,,,,,1,0.90,10.5,0.5,0.5,1,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000018,1,2017-12-15 00:33:57,2017-12-15 00:47:58,N,1,,,,,1,2.10,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000019,1,2017-12-15 00:00:23,2017-12-15 00:13:38,N,1,,,,,1,1.40,9.5,0.5,0.5,1.5,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000020,1,2017-12-15 00:21:30,2017-12-15 00:31:23,N,1,,,,,1,1.50,8,0.5,0.5,2.3,0,,0.3,11.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000021,1,2017-12-15 00:47:17,2017-12-15 01:02:26,N,1,,,,,1,2.80,12,0.5,0.5,1,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000022,1,2017-12-15 00:22:56,2017-12-15 00:29:31,N,1,,,,,1,1.50,7,0.5,0.5,3.5,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000023,1,2017-12-15 00:44:16,2017-12-15 00:50:13,N,1,,,,,1,0.90,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000024,2,2017-12-15 00:21:58,2017-12-15 00:44:00,N,1,,,,,1,13.41,37,0.5,0.5,0,0,,0.3,38.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000025,2,2017-12-15 00:18:22,2017-12-15 00:33:48,N,1,,,,,1,1.71,11,0.5,0.5,0,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000026,2,2017-12-15 00:39:06,2017-12-15 00:52:25,N,1,,,,,1,1.01,9,0.5,0.5,1.54,0,,0.3,11.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000027,2,2017-12-15 01:00:32,2017-12-15 01:20:29,N,1,,,,,1,3.69,16,0.5,0.5,2.6,0,,0.3,19.9,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000028,1,2017-12-15 00:09:13,2017-12-15 00:25:51,N,1,,,,,1,1.90,12,0.5,0.5,1,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000029,1,2017-12-15 00:27:17,2017-12-15 00:35:39,N,1,,,,,1,1.00,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000030,1,2017-12-15 00:37:01,2017-12-15 00:41:47,N,1,,,,,1,0.60,5,0.5,0.5,1.25,0,,0.3,7.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000031,1,2017-12-15 00:44:51,2017-12-15 00:52:31,N,1,,,,,1,1.80,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000032,1,2017-12-15 00:58:04,2017-12-15 01:42:27,N,1,,,,,1,12.10,41,0.5,0.5,2,0,,0.3,44.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000033,1,2017-12-15 00:04:01,2017-12-15 00:20:26,N,1,,,,,1,1.70,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000034,1,2017-12-15 00:21:36,2017-12-15 00:23:09,N,1,,,,,1,0.20,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000035,1,2017-12-15 00:24:05,2017-12-15 00:38:33,N,1,,,,,1,3.10,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000036,1,2017-12-15 00:49:14,2017-12-15 01:05:23,N,1,,,,,1,3.90,15,0.5,0.5,3.25,0,,0.3,19.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000037,1,2017-12-15 00:19:03,2017-12-15 00:44:02,N,1,,,,,1,3.50,17,0.5,0.5,2.75,0,,0.3,21.05,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000038,1,2017-12-15 00:47:45,2017-12-15 01:01:16,N,1,,,,,1,4.90,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000039,2,2017-12-15 00:23:06,2017-12-15 00:24:06,N,1,,,,,1,0.36,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000040,2,2017-12-15 00:27:28,2017-12-15 00:30:57,N,1,,,,,1,0.99,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000041,2,2017-12-15 00:18:20,2017-12-15 00:41:29,N,1,,,,,1,3.74,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000042,1,2017-12-15 00:08:31,2017-12-15 00:11:07,N,1,,,,,1,0.50,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000043,1,2017-12-15 00:13:03,2017-12-15 00:33:47,N,1,,,,,1,7.20,22.5,0.5,0.5,0,5.76,,0.3,29.56,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000044,2,2017-12-15 00:30:10,2017-12-15 00:45:02,N,1,,,,,1,3.02,12.5,0.5,0.5,0,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000045,2,2017-12-15 00:49:42,2017-12-15 01:13:51,N,1,,,,,1,5.27,20,0.5,0.5,2,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000046,2,2017-12-15 00:05:58,2017-12-15 00:25:43,N,1,,,,,1,4.15,17,0.5,0.5,4.58,0,,0.3,22.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000047,2,2017-12-15 00:37:28,2017-12-15 00:42:26,N,1,,,,,1,1.00,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000048,1,2017-12-15 00:15:17,2017-12-15 00:37:01,N,1,,,,,1,4.00,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000049,1,2017-12-15 00:42:53,2017-12-15 00:59:39,N,1,,,,,1,3.30,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000050,2,2017-12-15 00:33:52,2017-12-15 00:40:49,N,1,,,,,1,0.40,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000051,2,2017-12-15 00:46:35,2017-12-15 00:56:49,N,1,,,,,1,1.26,8.5,0.5,0.5,2,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000052,1,2017-12-15 00:06:04,2017-12-15 00:31:04,N,1,,,,,1,4.50,20,0.5,0.5,4.26,0,,0.3,25.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000053,2,2017-12-15 00:30:11,2017-12-15 00:34:09,N,1,,,,,3,1.14,5.5,0.5,0.5,2.04,0,,0.3,8.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000054,2,2017-12-15 00:45:10,2017-12-15 01:11:33,N,1,,,,,5,10.50,31.5,0.5,0.5,0,5.76,,0.3,38.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000055,2,2017-12-15 00:31:27,2017-12-15 00:53:23,N,1,,,,,2,4.40,18,0.5,0.5,4.82,0,,0.3,24.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000056,2,2017-12-15 00:23:15,2017-12-15 00:33:59,N,1,,,,,1,1.26,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000057,1,2017-12-15 00:18:49,2017-12-15 00:25:53,N,1,,,,,2,1.10,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000058,1,2017-12-15 00:28:53,2017-12-15 00:35:26,N,1,,,,,1,0.80,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000059,1,2017-12-15 00:36:04,2017-12-15 00:58:54,N,1,,,,,1,4.00,17,0.5,0.5,3.65,0,,0.3,21.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000060,1,2017-12-15 00:52:09,2017-12-15 00:59:47,N,1,,,,,1,1.00,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000061,2,2017-12-15 00:07:57,2017-12-15 00:25:09,N,1,,,,,1,2.79,13,0.5,0.5,0,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000062,2,2017-12-15 00:26:09,2017-12-15 00:29:52,N,1,,,,,1,0.68,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000063,2,2017-12-15 00:31:59,2017-12-15 00:40:18,N,1,,,,,1,1.31,7.5,0.5,0.5,1.25,0,,0.3,10.05,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000064,2,2017-12-15 00:49:06,2017-12-15 01:07:13,N,1,,,,,1,4.21,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000065,2,2017-12-15 00:56:03,2017-12-15 01:37:49,N,1,,,,,1,22.40,62.5,0.5,0.5,17.39,5.76,,0.3,86.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000066,1,2017-12-15 00:01:19,2017-12-15 00:12:58,N,1,,,,,1,1.80,10,0.5,0.5,1,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000067,1,2017-12-15 00:18:24,2017-12-15 01:04:16,N,5,,,,,1,5.00,0,0,0,0,10.5,,0.3,10.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000068,2,2017-12-15 00:08:27,2017-12-15 00:38:37,N,1,,,,,1,4.88,21.5,0.5,0.5,2.28,0,,0.3,25.08,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000069,2,2017-12-15 00:42:27,2017-12-15 00:53:50,N,1,,,,,1,5.28,17,0.5,0.5,4.81,5.76,,0.3,28.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000070,2,2017-12-15 00:05:56,2017-12-15 00:22:42,N,1,,,,,2,2.29,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000071,2,2017-12-15 00:27:09,2017-12-15 00:31:47,N,1,,,,,2,0.77,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000072,2,2017-12-15 00:42:03,2017-12-15 01:00:47,N,1,,,,,2,3.28,14.5,0.5,0.5,3.95,0,,0.3,19.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000073,1,2017-12-15 00:45:21,2017-12-15 00:50:56,N,1,,,,,1,1.50,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000074,2,2017-12-15 00:08:09,2017-12-15 00:21:11,N,1,,,,,5,1.97,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000075,2,2017-12-15 00:21:48,2017-12-15 00:57:13,N,1,,,,,5,5.40,24,0.5,0.5,0,0,,0.3,25.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000076,1,2017-12-15 00:09:11,2017-12-15 00:35:18,N,1,,,,,4,5.20,21.5,0.5,0.5,0,0,,0.3,22.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000077,1,2017-12-15 00:42:08,2017-12-15 00:42:09,N,1,,,,,1,5.30,2.5,0.5,0.5,0,0,,0.3,3.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000078,1,2017-12-15 00:45:28,2017-12-15 01:05:27,N,1,,,,,1,6.90,22.5,0.5,0.5,3,5.76,,0.3,32.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000079,2,2017-12-15 00:03:48,2017-12-15 00:38:38,N,1,,,,,6,7.86,29,0.5,0.5,9.09,0,,0.3,39.39,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000080,1,2017-12-15 00:25:04,2017-12-15 00:28:18,N,1,,,,,2,0.60,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000081,1,2017-12-15 00:34:23,2017-12-15 00:59:40,N,1,,,,,1,3.60,17.5,0.5,0.5,3,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000082,2,2017-12-15 00:53:37,2017-12-15 01:11:06,N,1,,,,,1,2.77,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000083,2,2017-12-15 00:09:00,2017-12-15 00:42:14,N,1,,,,,1,4.94,24.5,0.5,0.5,5.16,0,,0.3,30.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000084,1,2017-12-15 00:00:49,2017-12-15 00:07:51,N,1,,,,,1,0.50,6,0.5,0.5,1,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000085,1,2017-12-15 00:22:47,2017-12-15 00:30:21,N,1,,,,,4,0.60,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000086,1,2017-12-15 00:32:04,2017-12-15 00:39:38,N,1,,,,,1,0.90,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000087,1,2017-12-15 00:50:23,2017-12-15 01:05:29,N,1,,,,,1,3.00,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000088,2,2017-12-15 00:08:01,2017-12-15 00:25:35,N,1,,,,,1,2.80,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000089,2,2017-12-15 00:27:27,2017-12-15 00:55:33,N,1,,,,,1,4.05,19.5,0.5,0.5,5.2,0,,0.3,26,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000090,2,2017-12-15 00:11:22,2017-12-15 00:37:10,N,1,,,,,2,5.04,19.5,0.5,0.5,0,0,,0.3,20.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000091,2,2017-12-15 01:00:50,2017-12-15 01:29:18,N,1,,,,,2,7.11,24.5,0.5,0.5,0,0,,0.3,25.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000092,1,2017-12-15 00:26:01,2017-12-15 00:48:49,N,1,,,,,1,14.60,40,0.5,0.5,8.25,0,,0.3,49.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000093,1,2017-12-15 00:10:53,2017-12-15 00:48:57,N,1,,,,,1,14.30,43.5,0.5,0.5,8.95,0,,0.3,53.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000094,2,2017-12-15 00:31:37,2017-12-15 00:37:18,N,1,,,,,1,0.48,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000095,2,2017-12-15 00:41:42,2017-12-15 00:51:54,N,1,,,,,1,1.66,9,0.5,0.5,1,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000096,2,2017-12-15 00:53:10,2017-12-15 01:12:40,N,1,,,,,1,3.08,15.5,0.5,0.5,1,0,,0.3,17.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000097,2,2017-12-15 00:17:32,2017-12-15 00:30:33,N,1,,,,,5,1.14,9,0.5,0.5,1,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000098,2,2017-12-15 00:57:51,2017-12-15 01:11:23,N,1,,,,,5,2.01,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000099,1,2017-12-15 00:43:52,2017-12-15 00:52:43,N,1,,,,,4,1.60,8,0.5,0.5,0,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000100,1,2017-12-15 00:02:26,2017-12-15 00:13:59,N,1,,,,,1,1.10,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000101,1,2017-12-15 00:14:19,2017-12-15 00:56:54,N,5,,,,,1,18.50,100,0,0,15,0,,0.3,115.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000102,1,2017-12-15 00:40:45,2017-12-15 00:54:30,N,1,,,,,1,2.10,12,0.5,0.5,2.2,0,,0.3,15.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000103,2,2017-12-15 00:37:36,2017-12-15 00:53:05,N,1,,,,,1,3.60,14,0.5,0.5,2,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000104,2,2017-12-15 00:53:59,2017-12-15 00:55:30,N,1,,,,,1,0.22,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000105,1,2017-12-15 00:36:23,2017-12-15 00:51:06,N,1,,,,,1,2.30,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000106,2,2017-12-15 00:19:27,2017-12-15 00:31:53,N,1,,,,,6,1.49,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000107,2,2017-12-15 00:39:17,2017-12-15 00:44:01,N,1,,,,,6,0.79,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000108,1,2017-12-15 00:09:38,2017-12-15 00:47:23,N,1,,,,,2,17.90,51,0.5,0.5,1,17.28,,0.3,70.58,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000109,1,2017-12-15 00:49:21,2017-12-15 01:11:02,N,1,,,,,1,14.00,38,0.5,0.5,1,0,,0.3,40.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000110,2,2017-12-15 00:32:24,2017-12-15 00:38:56,N,1,,,,,1,0.80,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000111,2,2017-12-15 00:40:45,2017-12-15 00:55:44,N,1,,,,,1,1.54,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000112,1,2017-12-15 00:34:39,2017-12-15 00:41:03,N,1,,,,,1,1.00,6.5,0.5,0.5,1.95,0,,0.3,9.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000113,1,2017-12-15 00:43:04,2017-12-15 00:58:19,N,1,,,,,1,2.80,12.5,0.5,0.5,2.75,0,,0.3,16.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000114,2,2017-12-15 00:32:50,2017-12-15 00:45:30,N,1,,,,,1,1.56,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000115,2,2017-12-15 00:50:04,2017-12-15 00:59:07,N,1,,,,,1,1.70,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000116,1,2017-12-15 00:31:07,2017-12-15 00:35:35,N,1,,,,,1,0.60,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000117,1,2017-12-15 00:44:08,2017-12-15 01:12:08,N,1,,,,,1,7.30,26.5,0.5,0.5,0,0,,0.3,27.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000118,2,2017-12-15 00:40:24,2017-12-15 00:45:14,N,1,,,,,1,1.47,6.5,0.5,0.5,0,0,,0.3,7.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000119,2,2017-12-15 00:46:19,2017-12-15 01:02:19,N,1,,,,,2,1.87,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000120,2,2017-12-15 00:02:07,2017-12-15 00:04:51,N,1,,,,,1,0.36,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000121,2,2017-12-15 00:22:58,2017-12-15 00:34:06,N,1,,,,,2,0.73,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000122,2,2017-12-15 00:41:31,2017-12-15 00:48:10,N,1,,,,,1,0.82,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000123,2,2017-12-15 00:52:19,2017-12-15 01:01:01,N,1,,,,,1,1.54,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000124,1,2017-12-15 00:31:59,2017-12-15 00:44:42,N,1,,,,,0,2.20,10.5,0.5,0.5,2,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000125,2,2017-12-15 00:38:43,2017-12-15 00:44:54,N,1,,,,,1,1.44,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000126,2,2017-12-15 00:51:10,2017-12-15 01:05:47,N,1,,,,,5,2.75,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000127,2,2017-12-15 00:09:57,2017-12-15 00:25:53,N,1,,,,,1,2.61,12.5,0.5,0.5,2.76,0,,0.3,16.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000128,2,2017-12-15 00:27:34,2017-12-15 00:44:22,N,1,,,,,1,7.11,22.5,0.5,0.5,3.8,0,,0.3,27.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000129,2,2017-12-15 00:29:13,2017-12-15 00:53:56,N,1,,,,,2,9.73,30,0.5,0.5,0,0,,0.3,31.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000130,2,2017-12-15 00:28:32,2017-12-15 00:33:57,N,1,,,,,6,0.59,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000131,2,2017-12-15 00:37:51,2017-12-15 01:55:38,N,1,,,,,6,15.34,58,0.5,0.5,0,0,,0.3,59.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000132,2,2017-12-15 00:04:24,2017-12-15 00:13:56,N,1,,,,,3,1.46,8,0.5,0.5,2,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000133,2,2017-12-15 00:38:48,2017-12-15 01:01:48,N,1,,,,,2,2.72,15.5,0.5,0.5,4.2,0,,0.3,21,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000134,1,2017-12-15 00:33:15,2017-12-15 00:47:55,N,1,,,,,1,9.80,27,0.5,0.5,8,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000135,2,2017-12-15 00:41:39,2017-12-15 01:14:47,N,1,,,,,1,8.90,30.5,0.5,0.5,0,0,,0.3,31.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000136,1,2017-12-15 00:50:41,2017-12-15 00:59:29,N,1,,,,,1,1.30,8,0.5,0.5,1.5,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000137,2,2017-12-15 00:49:15,2017-12-15 00:58:07,N,1,,,,,4,1.87,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000138,2,2017-12-14 23:51:20,2017-12-15 00:50:09,N,1,,,,,1,10.08,43.5,0.5,0.5,8.96,0,,0.3,53.76,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000139,2,2017-12-15 00:05:31,2017-12-15 00:19:55,N,1,,,,,5,2.97,12,0.5,0.5,2.66,0,,0.3,15.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000140,2,2017-12-15 00:29:06,2017-12-15 01:09:26,N,1,,,,,3,13.26,41.5,0.5,0.5,0,0,,0.3,42.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000141,2,2017-12-15 00:26:34,2017-12-15 00:51:30,N,1,,,,,1,5.36,21,0.5,0.5,2,0,,0.3,24.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000142,2,2017-12-15 00:04:47,2017-12-15 00:23:09,N,1,,,,,1,2.68,13.5,0.5,0.5,4.44,0,,0.3,19.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000143,2,2017-12-15 00:31:16,2017-12-15 00:44:45,N,1,,,,,1,2.40,11,0.5,0.5,1,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000144,2,2017-12-15 00:49:37,2017-12-15 01:21:04,N,1,,,,,1,6.83,25.5,0.5,0.5,5.36,0,,0.3,32.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000145,2,2017-12-15 00:10:01,2017-12-15 00:16:49,N,1,,,,,1,0.91,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000146,2,2017-12-15 00:19:03,2017-12-15 00:30:11,N,1,,,,,1,1.81,9.5,0.5,0.5,2,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000147,2,2017-12-15 00:31:07,2017-12-15 00:34:53,N,1,,,,,1,0.92,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000148,2,2017-12-15 00:43:48,2017-12-15 01:04:53,N,1,,,,,1,4.63,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000149,1,2017-12-15 00:22:11,2017-12-15 00:32:11,N,1,,,,,1,2.20,10,0.5,0.5,2.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000150,1,2017-12-15 00:48:02,2017-12-15 01:01:45,N,1,,,,,1,1.30,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000151,2,2017-12-15 00:05:43,2017-12-15 00:12:57,N,1,,,,,1,2.05,9,0.5,0.5,2.06,0,,0.3,14.31,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000152,2,2017-12-15 00:32:48,2017-12-15 00:55:01,N,1,,,,,1,9.23,28,0.5,0.5,6,5.76,,0.3,41.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000153,2,2017-12-15 01:00:10,2017-12-15 01:16:04,N,1,,,,,1,4.37,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000154,2,2017-12-15 00:08:13,2017-12-15 00:28:37,N,1,,,,,1,3.22,15,0.5,0.5,0,0,,0.3,16.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000155,2,2017-12-15 00:31:00,2017-12-15 00:41:23,N,1,,,,,1,1.60,8.5,0.5,0.5,0,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000156,2,2017-12-15 00:42:50,2017-12-15 00:53:34,N,1,,,,,1,1.37,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000157,2,2017-12-15 00:55:45,2017-12-15 00:59:31,N,1,,,,,1,1.08,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000158,2,2017-12-15 00:09:20,2017-12-15 00:23:24,N,1,,,,,1,2.62,11.5,0.5,0.5,3.2,0,,0.3,16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000159,2,2017-12-15 00:24:46,2017-12-15 00:36:33,N,1,,,,,1,1.35,9.5,0.5,0.5,0,0,,0.3,10.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000160,2,2017-12-15 00:40:16,2017-12-15 00:57:48,N,1,,,,,1,3.03,13.5,0.5,0.5,1,0,,0.3,15.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000161,1,2017-12-15 00:00:35,2017-12-15 00:12:21,N,1,,,,,1,2.00,9.5,0.5,0.5,3.2,0,,0.3,14,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000162,1,2017-12-15 00:13:48,2017-12-15 00:20:03,N,1,,,,,1,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000163,1,2017-12-15 00:20:47,2017-12-15 00:31:56,N,1,,,,,1,1.70,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000164,1,2017-12-15 00:33:29,2017-12-15 01:02:24,N,1,,,,,1,9.00,29,0.5,0.5,7.55,0,,0.3,37.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000165,1,2017-12-15 00:09:42,2017-12-15 00:18:12,N,1,,,,,2,2.40,9,0.5,0.5,0,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000166,1,2017-12-15 00:57:43,2017-12-15 01:16:05,N,1,,,,,1,3.30,13.5,0.5,0.5,2.95,0,,0.3,17.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000167,2,2017-12-15 00:09:55,2017-12-15 00:19:30,N,1,,,,,1,1.45,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000168,2,2017-12-15 00:20:11,2017-12-15 00:33:01,N,1,,,,,1,2.18,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000169,2,2017-12-15 00:35:09,2017-12-15 01:09:59,N,1,,,,,1,17.48,51,0.5,0.5,0,5.76,,0.3,58.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000170,2,2017-12-15 00:26:34,2017-12-15 00:41:45,N,1,,,,,1,1.81,11,0.5,0.5,3.69,0,,0.3,15.99,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000171,2,2017-12-15 00:46:38,2017-12-15 01:08:50,N,1,,,,,1,3.31,16,0.5,0.5,3.46,0,,0.3,20.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000172,2,2017-12-15 00:19:37,2017-12-15 00:35:33,N,1,,,,,2,1.61,10.5,0.5,0.5,5,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000173,2,2017-12-15 00:38:17,2017-12-15 00:45:48,N,1,,,,,2,1.01,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000174,2,2017-12-15 00:58:49,2017-12-15 01:35:08,N,1,,,,,2,5.57,26.5,0.5,0.5,5.56,0,,0.3,33.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000175,1,2017-12-15 00:17:10,2017-12-15 00:52:01,N,1,,,,,1,4.00,23,0.5,0.5,6,5.76,,0.3,36.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000176,2,2017-12-15 00:51:45,2017-12-15 01:16:36,N,2,,,,,2,16.55,52,0,0.5,10.56,0,,0.3,63.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000177,2,2017-12-15 00:53:53,2017-12-15 00:59:08,N,1,,,,,3,1.09,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000178,2,2017-12-15 00:17:56,2017-12-15 00:41:33,N,1,,,,,1,5.12,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000179,2,2017-12-15 00:45:41,2017-12-15 00:58:18,N,1,,,,,1,2.83,11.5,0.5,0.5,3.2,0,,0.3,16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000180,2,2017-12-15 00:01:12,2017-12-15 00:11:38,N,1,,,,,1,1.65,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000181,2,2017-12-15 00:12:58,2017-12-15 00:19:54,N,1,,,,,1,0.89,6.5,0.5,0.5,1.95,0,,0.3,9.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000182,2,2017-12-15 00:24:50,2017-12-15 00:26:01,N,5,,,,,1,0.00,100,0,0.5,8.2,0,,0.3,109,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000183,1,2017-12-15 00:32:04,2017-12-15 01:01:36,N,2,,,,,3,20.80,52,0,0.5,0,5.76,,0.3,58.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000184,2,2017-12-15 00:18:06,2017-12-15 01:07:36,N,1,,,,,1,25.24,71,0.5,0.5,15.61,5.76,,0.3,93.67,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000185,2,2017-12-15 00:10:06,2017-12-15 00:20:55,N,1,,,,,1,1.84,9.5,0.5,0.5,2.5,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000186,2,2017-12-15 00:31:27,2017-12-15 00:45:44,N,1,,,,,1,1.66,10.5,0.5,0.5,1.2,0,,0.3,13,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000187,2,2017-12-15 00:53:13,2017-12-15 01:13:27,N,1,,,,,1,2.40,14,0.5,0.5,0,0,,0.3,15.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000188,2,2017-12-15 00:03:43,2017-12-15 00:11:39,N,1,,,,,2,1.58,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000189,2,2017-12-15 00:18:02,2017-12-15 00:43:48,N,1,,,,,2,4.61,19,0.5,0.5,0,0,,0.3,20.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000190,2,2017-12-15 00:48:20,2017-12-15 01:09:19,N,1,,,,,2,2.67,15,0.5,0.5,1,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000191,1,2017-12-15 00:05:45,2017-12-15 00:26:31,N,1,,,,,2,3.50,17.5,0.5,0.5,2,0,,0.3,20.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000192,1,2017-12-15 00:35:30,2017-12-15 00:47:19,N,1,,,,,1,2.50,10.5,0.5,0.5,2,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000193,1,2017-12-15 00:55:01,2017-12-15 01:06:13,N,1,,,,,1,0.90,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000194,2,2017-12-15 00:23:31,2017-12-15 00:30:12,N,1,,,,,1,0.98,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000195,2,2017-12-15 00:33:17,2017-12-15 00:55:09,N,1,,,,,1,3.20,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000196,2,2017-12-15 00:56:39,2017-12-15 01:14:39,N,1,,,,,1,4.65,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000197,2,2017-12-15 00:12:57,2017-12-15 00:18:57,N,1,,,,,1,1.03,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000198,2,2017-12-15 00:20:41,2017-12-15 00:46:44,N,1,,,,,1,5.33,21,0.5,0.5,1,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000199,2,2017-12-14 23:56:15,2017-12-15 00:20:06,N,1,,,,,1,10.45,30.5,0.5,0.5,7.51,5.76,,0.3,45.07,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000200,2,2017-12-15 00:30:30,2017-12-15 01:22:30,N,1,,,,,1,13.51,48,0.5,0.5,0,0,,0.3,49.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000201,2,2017-12-15 00:03:15,2017-12-15 00:21:04,N,1,,,,,1,2.38,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000202,2,2017-12-15 00:23:15,2017-12-15 00:27:40,N,1,,,,,1,0.47,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000203,2,2017-12-15 00:31:32,2017-12-15 00:49:33,N,1,,,,,1,2.91,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000204,2,2017-12-15 00:08:42,2017-12-15 00:12:21,N,1,,,,,1,0.39,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000205,2,2017-12-15 00:13:31,2017-12-15 00:24:20,N,1,,,,,1,2.49,10.5,0.5,0.5,1.2,0,,0.3,13,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000206,2,2017-12-15 00:29:19,2017-12-15 00:58:28,N,1,,,,,1,3.72,20,0.5,0.5,4.26,0,,0.3,25.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000207,2,2017-12-15 00:28:14,2017-12-15 00:34:25,N,1,,,,,1,1.55,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000208,2,2017-12-15 00:36:22,2017-12-15 00:37:55,N,1,,,,,1,0.45,3.5,0.5,0.5,0.96,0,,0.3,5.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000209,2,2017-12-15 00:39:37,2017-12-15 01:35:03,N,1,,,,,1,13.96,47,0.5,0.5,5.08,5.76,,0.3,59.14,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000210,2,2017-12-15 00:25:04,2017-12-15 00:33:25,N,1,,,,,1,1.64,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000211,2,2017-12-15 00:34:48,2017-12-15 00:43:08,N,1,,,,,1,1.60,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000212,2,2017-12-15 01:01:52,2017-12-15 01:36:51,N,1,,,,,1,9.76,35,0.5,0.5,0,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000213,1,2017-12-15 00:46:44,2017-12-15 00:49:28,N,1,,,,,1,0.50,4,0.5,0.5,0,0,,0.3,5.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000214,2,2017-12-15 00:06:12,2017-12-15 00:31:47,N,1,,,,,1,4.02,18,0.5,0.5,1.5,0,,0.3,20.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000215,2,2017-12-15 00:44:58,2017-12-15 01:05:54,N,1,,,,,1,4.67,18,0.5,0.5,0,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000216,2,2017-12-15 00:17:30,2017-12-15 00:24:38,N,1,,,,,1,1.71,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000217,2,2017-12-15 00:34:51,2017-12-15 00:47:29,N,1,,,,,1,1.75,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000218,2,2017-12-15 00:50:18,2017-12-15 01:10:23,N,1,,,,,1,3.51,16,0.5,0.5,2.5,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000219,1,2017-12-15 00:21:43,2017-12-15 00:30:20,N,1,,,,,1,1.20,7.5,0.5,0.5,1.75,0,,0.3,10.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000220,1,2017-12-15 00:45:42,2017-12-15 01:20:41,N,1,,,,,1,6.70,27,0.5,0.5,6.8,5.76,,0.3,40.86,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000221,2,2017-12-15 00:16:52,2017-12-15 00:27:51,N,1,,,,,2,1.33,8.5,0.5,0.5,1,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000222,2,2017-12-15 00:31:34,2017-12-15 00:45:50,N,1,,,,,2,6.73,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000223,2,2017-12-15 00:02:41,2017-12-15 00:30:26,N,1,,,,,1,6.01,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000224,2,2017-12-15 00:47:01,2017-12-15 00:55:49,N,1,,,,,1,1.98,8.5,0.5,0.5,2,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000225,2,2017-12-15 00:58:15,2017-12-15 01:14:41,N,1,,,,,1,3.40,13.5,0.5,0.5,4.44,0,,0.3,19.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000226,2,2017-12-15 00:13:35,2017-12-15 00:27:29,N,1,,,,,1,2.59,11.5,0.5,0.5,3.84,0,,0.3,16.64,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000227,2,2017-12-15 00:38:30,2017-12-15 00:43:51,N,1,,,,,1,1.15,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000228,2,2017-12-15 00:45:24,2017-12-15 00:51:18,N,1,,,,,1,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000229,2,2017-12-15 00:55:04,2017-12-15 01:08:57,N,1,,,,,1,3.47,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000230,2,2017-12-15 00:04:17,2017-12-15 00:10:33,N,1,,,,,1,1.23,6.5,0.5,0.5,1.95,0,,0.3,9.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000231,2,2017-12-15 00:13:03,2017-12-15 00:26:55,N,1,,,,,1,2.78,12,0.5,0.5,3.32,0,,0.3,16.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000232,2,2017-12-15 00:30:01,2017-12-15 00:37:15,N,1,,,,,1,0.89,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000233,2,2017-12-15 00:38:43,2017-12-15 00:47:26,N,1,,,,,1,1.40,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000234,2,2017-12-15 00:49:45,2017-12-15 00:53:07,N,1,,,,,1,0.59,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000235,2,2017-12-15 00:56:01,2017-12-15 01:07:10,N,1,,,,,1,2.46,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000236,2,2017-12-15 00:33:40,2017-12-15 00:39:42,N,1,,,,,2,1.31,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000237,2,2017-12-15 00:55:08,2017-12-15 01:29:46,N,1,,,,,2,11.64,37,0.5,0.5,8.81,5.76,,0.3,52.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000238,2,2017-12-15 00:38:07,2017-12-15 00:42:29,N,1,,,,,6,0.95,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000239,1,2017-12-15 00:02:32,2017-12-15 00:14:54,N,1,,,,,2,2.40,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000240,1,2017-12-15 00:47:13,2017-12-15 01:23:55,N,1,,,,,2,9.40,32.5,0.5,0.5,0,0,,0.3,33.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000241,2,2017-12-15 00:07:00,2017-12-15 00:22:39,N,1,,,,,2,2.66,12,0.5,0.5,3.99,0,,0.3,17.29,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000242,2,2017-12-15 00:24:20,2017-12-15 00:39:27,N,1,,,,,2,3.08,12.5,0.5,0.5,1,0,,0.3,14.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000243,2,2017-12-15 00:42:39,2017-12-15 00:57:57,N,1,,,,,2,2.33,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000244,1,2017-12-15 00:00:39,2017-12-15 00:03:05,N,1,,,,,1,0.40,4,0.5,0.5,0,0,,0.3,5.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000245,2,2017-12-15 00:10:32,2017-12-15 00:43:16,N,1,,,,,5,8.04,28,0.5,0.5,3.7,0,,0.3,33,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000246,2,2017-12-15 00:35:42,2017-12-15 00:51:25,N,1,,,,,1,8.76,25,0.5,0.5,5,5.76,,0.3,37.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000247,2,2017-12-15 00:52:46,2017-12-15 01:04:05,N,1,,,,,3,1.78,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000248,2,2017-12-15 00:05:05,2017-12-15 00:11:34,N,1,,,,,1,1.08,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000249,2,2017-12-15 00:12:28,2017-12-15 00:37:12,N,1,,,,,1,3.21,16.5,0.5,0.5,3.56,0,,0.3,21.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000250,2,2017-12-15 00:44:06,2017-12-15 01:04:05,N,1,,,,,1,5.06,17,0.5,0.5,0,0,,0.3,18.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000251,1,2017-12-15 00:05:32,2017-12-15 00:09:10,N,1,,,,,1,0.60,4.5,0.5,0.5,1.15,0,,0.3,6.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000252,1,2017-12-15 00:27:29,2017-12-15 00:39:05,N,1,,,,,1,2.20,10,0.5,0.5,2.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000253,1,2017-12-15 00:07:54,2017-12-15 00:10:50,N,1,,,,,2,0.60,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000254,1,2017-12-15 00:12:02,2017-12-15 00:17:01,N,1,,,,,2,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000255,1,2017-12-15 00:24:58,2017-12-15 00:37:38,N,1,,,,,1,1.60,9.5,0.5,0.5,1.7,0,,0.3,12.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000256,1,2017-12-15 00:41:49,2017-12-15 00:53:48,N,1,,,,,1,3.90,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000257,2,2017-12-14 23:58:09,2017-12-15 00:02:07,N,1,,,,,1,0.49,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000258,2,2017-12-15 00:05:04,2017-12-15 00:18:50,N,1,,,,,1,2.02,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000259,2,2017-12-15 00:21:45,2017-12-15 00:55:32,N,1,,,,,1,5.18,23.5,0.5,0.5,4.96,0,,0.3,29.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000260,2,2017-12-15 00:04:25,2017-12-15 00:11:39,N,1,,,,,1,1.13,7,0.5,0.5,2.08,0,,0.3,10.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000261,2,2017-12-15 00:20:57,2017-12-15 00:31:42,N,1,,,,,1,1.08,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000262,2,2017-12-15 00:39:26,2017-12-15 00:50:31,N,1,,,,,1,2.09,9.5,0.5,0.5,0,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000263,2,2017-12-15 00:53:53,2017-12-15 01:04:58,N,1,,,,,1,1.76,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000264,2,2017-12-15 00:08:30,2017-12-15 00:20:52,N,1,,,,,1,1.75,9.5,0.5,0.5,0,0,,0.3,10.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000265,2,2017-12-15 00:23:40,2017-12-15 00:41:49,N,1,,,,,1,2.18,12.5,0.5,0.5,2.76,0,,0.3,16.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000266,2,2017-12-15 00:44:35,2017-12-15 00:49:32,N,1,,,,,1,0.79,5.5,0.5,0.5,2.04,0,,0.3,8.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000267,2,2017-12-15 00:52:36,2017-12-15 01:02:00,N,1,,,,,1,0.73,7.5,0.5,0.5,1.76,0,,0.3,10.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000268,2,2017-12-15 00:31:52,2017-12-15 00:47:40,N,1,,,,,1,1.80,11.5,0.5,0.5,1.08,0,,0.3,13.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000269,2,2017-12-15 00:53:43,2017-12-15 01:05:05,N,1,,,,,1,2.54,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000270,2,2017-12-15 00:06:34,2017-12-15 00:13:18,N,1,,,,,2,1.21,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000271,2,2017-12-15 00:18:09,2017-12-15 00:25:11,N,1,,,,,2,1.29,7,0.5,0.5,2.08,0,,0.3,10.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000272,2,2017-12-15 00:30:15,2017-12-15 00:38:42,N,1,,,,,2,1.38,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000273,2,2017-12-15 00:40:42,2017-12-15 00:44:57,N,1,,,,,2,1.32,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000274,1,2017-12-15 00:47:31,2017-12-15 00:49:29,N,1,,,,,3,0.20,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000275,1,2017-12-15 00:52:16,2017-12-15 01:28:36,N,1,,,,,1,7.30,29.5,0.5,0.5,0,0,,0.3,30.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000276,1,2017-12-15 00:01:57,2017-12-15 00:18:51,N,1,,,,,1,11.40,31.5,0.5,0.5,8.2,0,,0.3,41,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000277,1,2017-12-15 00:39:16,2017-12-15 01:04:21,N,1,,,,,1,10.10,30.5,0.5,0.5,7.5,5.76,,0.3,45.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000278,1,2017-12-15 00:18:06,2017-12-15 00:26:33,N,1,,,,,1,1.70,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000279,1,2017-12-15 00:28:42,2017-12-15 01:10:48,N,1,,,,,1,8.70,34.5,0.5,0.5,7.15,0,,0.3,42.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000280,2,2017-12-15 00:20:44,2017-12-15 00:47:16,N,1,,,,,1,3.10,17.5,0.5,0.5,3.76,0,,0.3,22.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000281,2,2017-12-15 00:48:38,2017-12-15 00:58:46,N,1,,,,,1,1.87,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000282,2,2017-12-15 00:04:31,2017-12-15 00:25:09,N,1,,,,,1,2.28,14.5,0.5,0.5,3.16,0,,0.3,18.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000283,2,2017-12-15 00:26:46,2017-12-15 00:33:08,N,1,,,,,1,0.56,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000284,2,2017-12-15 00:37:12,2017-12-15 01:04:43,N,1,,,,,1,6.54,24,0.5,0.5,0,0,,0.3,25.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000285,2,2017-12-15 00:32:16,2017-12-15 01:27:04,N,1,,,,,1,14.64,50,0.5,0.5,10.26,0,,0.3,61.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000286,2,2017-12-15 00:59:27,2017-12-15 01:20:51,N,1,,,,,1,5.95,20,0.5,0.5,0,0,,0.3,21.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000287,2,2017-12-15 00:48:18,2017-12-15 01:14:34,N,1,,,,,1,10.73,32.5,0.5,0.5,0,5.76,,0.3,39.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000288,1,2017-12-15 00:07:31,2017-12-15 00:19:09,N,1,,,,,2,2.20,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000289,1,2017-12-15 00:24:03,2017-12-15 00:42:25,N,1,,,,,1,2.30,13,0.5,0.5,3.55,0,,0.3,17.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000290,1,2017-12-15 00:44:12,2017-12-15 00:55:18,N,1,,,,,1,1.80,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000291,1,2017-12-15 00:56:48,2017-12-15 01:13:09,N,1,,,,,2,2.60,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000292,1,2017-12-15 00:19:32,2017-12-15 00:25:08,N,1,,,,,1,1.20,6,0.5,0.5,1,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000293,1,2017-12-15 00:27:00,2017-12-15 00:51:04,N,1,,,,,1,4.60,19,0.5,0.5,4.05,0,,0.3,24.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000294,1,2017-12-15 00:20:38,2017-12-15 00:25:15,N,1,,,,,1,1.20,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000295,1,2017-12-15 00:41:06,2017-12-15 00:52:18,N,1,,,,,1,1.80,9.5,0.5,0.5,2.7,0,,0.3,13.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000296,1,2017-12-15 00:55:55,2017-12-15 01:12:31,N,1,,,,,1,3.30,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000297,2,2017-12-15 00:05:55,2017-12-15 00:14:30,N,1,,,,,1,1.63,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000298,2,2017-12-15 00:27:26,2017-12-15 01:03:16,N,1,,,,,1,6.38,26,0.5,0.5,5.46,0,,0.3,32.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000299,1,2017-12-15 00:08:21,2017-12-15 00:28:24,N,1,,,,,1,3.50,15,0.5,0.5,0,0,,0.3,16.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000300,1,2017-12-15 00:32:08,2017-12-15 00:43:23,N,1,,,,,2,2.80,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000301,1,2017-12-15 00:49:47,2017-12-15 00:53:20,N,1,,,,,2,0.60,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000302,1,2017-12-15 00:30:10,2017-12-15 01:07:08,N,1,,,,,1,9.40,33.5,0.5,0.5,5,5.76,,0.3,45.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000303,1,2017-12-15 00:04:37,2017-12-15 00:11:07,N,1,,,,,1,1.00,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000304,1,2017-12-15 00:11:47,2017-12-15 00:44:18,N,1,,,,,1,5.70,24.5,0.5,0.5,0,0,,0.3,25.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000305,1,2017-12-15 00:58:49,2017-12-15 01:17:57,N,1,,,,,1,5.20,18.5,0.5,0.5,3.95,0,,0.3,23.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000306,1,2017-12-15 00:02:04,2017-12-15 00:06:33,N,1,,,,,1,0.80,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000307,1,2017-12-15 00:19:59,2017-12-15 00:38:04,Y,1,,,,,1,2.80,13,0.5,0.5,3,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000308,2,2017-12-15 00:04:13,2017-12-15 00:44:18,N,1,,,,,6,7.07,30,0.5,0.5,6.26,0,,0.3,37.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000309,2,2017-12-15 00:54:44,2017-12-15 01:03:50,N,1,,,,,5,1.55,8,0.5,0.5,2.32,0,,0.3,11.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000310,1,2017-12-15 00:43:05,2017-12-15 00:52:01,N,1,,,,,1,2.10,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000311,1,2017-12-15 00:05:03,2017-12-15 00:18:37,N,1,,,,,1,2.60,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000312,1,2017-12-15 00:20:35,2017-12-15 00:38:49,N,1,,,,,1,2.50,13.5,0.5,0.5,4.4,0,,0.3,19.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000313,1,2017-12-15 00:39:52,2017-12-15 00:44:38,N,1,,,,,1,1.10,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000314,1,2017-12-15 00:49:40,2017-12-15 01:00:41,N,1,,,,,1,1.70,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000315,1,2017-12-15 00:47:28,2017-12-15 00:52:07,N,1,,,,,1,1.20,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000316,2,2017-12-15 00:06:56,2017-12-15 00:44:48,N,1,,,,,5,3.86,23.5,0.5,0.5,4.96,0,,0.3,29.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000317,2,2017-12-15 00:55:16,2017-12-15 01:06:36,N,1,,,,,5,1.94,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000318,1,2017-12-15 00:30:37,2017-12-15 00:30:47,N,2,,,,,1,0.00,52,0,0.5,8,5.76,,0.3,66.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000319,1,2017-12-15 00:34:47,2017-12-15 01:55:20,N,1,,,,,1,20.40,75.5,0.5,0.5,5,5.76,,0.3,87.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000320,2,2017-12-14 23:51:49,2017-12-15 00:09:43,N,1,,,,,1,1.19,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000321,2,2017-12-15 00:19:42,2017-12-15 00:27:12,N,1,,,,,1,0.78,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000322,2,2017-12-15 00:28:34,2017-12-15 00:40:23,N,1,,,,,1,2.75,11.5,0.5,0.5,1.5,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000323,2,2017-12-15 00:35:15,2017-12-15 00:55:59,N,1,,,,,1,7.24,23.5,0.5,0.5,0,5.76,,0.3,30.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000324,2,2017-12-15 00:26:02,2017-12-15 00:29:23,N,1,,,,,1,0.35,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000325,2,2017-12-15 00:31:42,2017-12-15 00:36:19,N,1,,,,,1,1.07,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000326,2,2017-12-15 00:36:55,2017-12-15 00:43:24,N,1,,,,,1,1.38,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000327,2,2017-12-15 00:45:40,2017-12-15 01:24:42,N,1,,,,,1,6.01,27,0.5,0.5,4.24,0,,0.3,32.54,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000328,2,2017-12-15 00:11:05,2017-12-15 00:19:49,N,1,,,,,2,1.16,7.5,0.5,0.5,1,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000329,2,2017-12-15 00:20:45,2017-12-15 00:54:08,N,1,,,,,2,5.22,23.5,0.5,0.5,2.4,0,,0.3,27.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000330,2,2017-12-14 23:59:30,2017-12-15 00:27:27,N,1,,,,,6,4.86,21,0.5,0.5,5.58,0,,0.3,27.88,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000331,2,2017-12-15 00:32:34,2017-12-15 00:43:36,N,1,,,,,6,2.73,10.5,0.5,0.5,0.59,0,,0.3,12.39,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000332,2,2017-12-15 00:55:30,2017-12-15 01:14:02,N,1,,,,,6,3.22,14.5,0.5,0.5,1.5,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000333,2,2017-12-15 00:48:56,2017-12-15 00:57:15,N,1,,,,,6,1.23,7.5,0.5,0.5,1.76,0,,0.3,10.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000334,1,2017-12-15 00:17:24,2017-12-15 00:28:44,N,1,,,,,1,2.10,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000335,1,2017-12-15 00:29:20,2017-12-15 00:40:38,N,1,,,,,1,1.70,9,0.5,0.5,2.5,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000336,1,2017-12-15 00:43:13,2017-12-15 01:04:32,N,1,,,,,1,1.90,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000337,2,2017-12-15 00:18:55,2017-12-15 00:35:52,N,1,,,,,5,2.25,12.5,0.5,0.5,2,0,,0.3,15.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000338,2,2017-12-15 00:37:39,2017-12-15 00:52:21,N,1,,,,,5,1.76,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000339,2,2017-12-15 00:02:22,2017-12-15 00:18:02,N,1,,,,,2,1.79,11.5,0.5,0.5,1.5,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000340,2,2017-12-15 00:19:03,2017-12-15 00:55:49,N,1,,,,,2,8.96,33.5,0.5,0.5,6.96,0,,0.3,41.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000341,2,2017-12-15 00:38:47,2017-12-15 01:02:25,N,1,,,,,1,5.67,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000342,1,2017-12-15 00:54:07,2017-12-15 00:56:25,N,1,,,,,1,0.30,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000343,1,2017-12-15 00:59:33,2017-12-15 01:15:36,N,1,,,,,1,4.10,16,0.5,0.5,3.45,0,,0.3,20.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000344,1,2017-12-15 00:16:35,2017-12-15 00:38:22,N,1,,,,,1,3.70,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000345,1,2017-12-15 00:41:35,2017-12-15 00:46:07,N,1,,,,,1,1.10,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000346,1,2017-12-15 00:28:16,2017-12-15 00:49:43,N,1,,,,,1,2.10,14.5,0.5,0.5,1,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000347,1,2017-12-15 00:50:32,2017-12-15 01:02:25,N,1,,,,,1,2.00,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000348,2,2017-12-15 00:06:30,2017-12-15 00:32:46,N,2,,,,,2,18.00,52,0,0.5,8,0,,0.3,60.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000349,2,2017-12-15 00:35:57,2017-12-15 00:49:57,N,1,,,,,2,3.00,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000350,2,2017-12-15 00:51:07,2017-12-15 01:03:37,N,1,,,,,2,1.78,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000351,2,2017-12-14 23:58:18,2017-12-15 00:17:19,N,1,,,,,1,2.89,13.5,0.5,0.5,0,0,,0.3,14.8,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000352,2,2017-12-15 00:25:33,2017-12-15 00:37:17,N,1,,,,,1,1.84,9.5,0.5,0.5,2.7,0,,0.3,13.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000353,2,2017-12-15 00:38:50,2017-12-15 01:12:55,N,5,,,,,1,20.52,95,0,0.5,21.26,10.5,,0.3,127.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000354,1,2017-12-15 00:10:39,2017-12-15 00:23:43,N,1,,,,,0,1.80,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000355,1,2017-12-15 00:26:21,2017-12-15 00:46:41,N,1,,,,,0,3.90,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000356,1,2017-12-15 00:51:23,2017-12-15 01:23:51,N,1,,,,,0,4.60,21,0.5,0.5,3.2,0,,0.3,25.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000357,1,2017-12-15 00:01:01,2017-12-15 00:09:09,N,1,,,,,1,1.50,7.5,0.5,0.5,2.2,0,,0.3,11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000358,1,2017-12-15 00:11:16,2017-12-15 00:28:12,N,1,,,,,1,4.20,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000359,1,2017-12-15 00:41:38,2017-12-15 00:49:41,N,1,,,,,1,1.00,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000360,1,2017-12-15 00:53:12,2017-12-15 01:03:55,N,1,,,,,1,2.30,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000361,2,2017-12-15 00:17:03,2017-12-15 00:23:50,N,1,,,,,6,0.78,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000362,2,2017-12-15 00:26:35,2017-12-15 00:30:26,N,1,,,,,6,0.70,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000363,2,2017-12-15 00:33:08,2017-12-15 00:49:10,N,1,,,,,6,3.51,14.5,0.5,0.5,0,0,,0.3,15.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000364,2,2017-12-15 00:54:31,2017-12-15 01:18:59,N,1,,,,,6,6.01,21.5,0.5,0.5,0,0,,0.3,22.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000365,1,2017-12-15 00:06:33,2017-12-15 00:44:01,N,1,,,,,1,9.40,34,0.5,0.5,7.05,0,,0.3,42.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000366,1,2017-12-15 00:24:39,2017-12-15 00:49:07,N,1,,,,,1,2.60,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000367,1,2017-12-15 00:53:08,2017-12-15 01:07:56,N,1,,,,,1,3.30,13,0.5,0.5,3.55,0,,0.3,17.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000368,1,2017-12-15 00:02:28,2017-12-15 00:13:55,N,1,,,,,1,1.80,9.5,0.5,0.5,2,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000369,1,2017-12-15 00:33:04,2017-12-15 00:40:02,N,1,,,,,1,2.10,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000370,1,2017-12-15 00:03:29,2017-12-15 00:08:56,N,1,,,,,1,1.00,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000371,1,2017-12-15 00:10:46,2017-12-15 00:20:15,N,1,,,,,1,1.20,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000372,1,2017-12-15 00:22:56,2017-12-15 00:37:19,N,1,,,,,1,4.00,15,0.5,0.5,0,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000373,1,2017-12-15 00:42:52,2017-12-15 00:51:19,N,1,,,,,1,5.00,15.5,0.5,0.5,3.35,0,,0.3,20.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000374,1,2017-12-15 00:53:47,2017-12-15 01:23:10,N,1,,,,,1,4.70,21,0.5,0.5,0,0,,0.3,22.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000375,2,2017-12-15 00:36:27,2017-12-15 01:12:34,N,1,,,,,1,10.62,35.5,0.5,0.5,4.2,0,,0.3,41,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000376,1,2017-12-15 00:28:33,2017-12-15 00:45:42,N,1,,,,,1,3.10,14,0.5,0.5,3.8,0,,0.3,19.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000377,2,2017-12-15 00:15:28,2017-12-15 00:50:41,N,1,,,,,2,6.56,28,0.5,0.5,0,0,,0.3,29.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000378,2,2017-12-15 00:56:38,2017-12-15 01:09:17,N,1,,,,,2,6.86,20.5,0.5,0.5,1.1,0,,0.3,22.9,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000379,1,2017-12-15 00:13:36,2017-12-15 01:14:17,N,1,,,,,1,13.00,47.5,0.5,0.5,9.75,0,,0.3,58.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000380,1,2017-12-15 00:03:32,2017-12-15 00:28:38,N,1,,,,,1,4.60,19.5,0.5,0.5,1,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000381,1,2017-12-15 00:46:55,2017-12-15 01:10:49,N,1,,,,,1,5.00,20.5,0.5,0.5,0,5.76,,0.3,27.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000382,1,2017-12-15 00:09:42,2017-12-15 00:20:45,N,1,,,,,1,2.50,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000383,1,2017-12-15 00:23:14,2017-12-15 00:27:02,N,1,,,,,1,0.60,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000384,1,2017-12-15 00:31:40,2017-12-15 00:36:49,N,1,,,,,1,1.00,6,0.5,0.5,2,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000385,1,2017-12-15 00:52:15,2017-12-15 01:13:40,N,1,,,,,1,4.60,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000386,2,2017-12-15 00:13:40,2017-12-15 00:33:20,N,1,,,,,1,1.93,13.5,0.5,0.5,1.48,0,,0.3,16.28,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000387,2,2017-12-15 00:52:47,2017-12-15 01:19:23,N,1,,,,,1,5.91,23.5,0.5,0.5,7.44,0,,0.3,32.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000388,1,2017-12-15 00:38:36,2017-12-15 00:56:37,N,1,,,,,1,5.90,20,0.5,0.5,4.25,0,,0.3,25.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000389,2,2017-12-15 00:15:53,2017-12-15 00:48:54,N,1,,,,,1,3.61,22,0.5,0.5,1,0,,0.3,24.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000390,2,2017-12-15 00:20:20,2017-12-15 00:40:11,N,1,,,,,1,2.02,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000391,2,2017-12-15 00:41:22,2017-12-15 00:57:21,N,1,,,,,1,3.29,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000392,2,2017-12-15 00:59:16,2017-12-15 01:09:53,N,1,,,,,1,1.08,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000393,1,2017-12-15 00:44:17,2017-12-15 01:13:28,N,2,,,,,1,17.20,52,0,0.5,10.56,0,,0.3,63.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000394,2,2017-12-15 00:06:55,2017-12-15 00:17:28,N,1,,,,,2,1.58,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000395,2,2017-12-15 00:22:35,2017-12-15 00:34:52,N,1,,,,,2,1.84,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000396,2,2017-12-15 00:44:25,2017-12-15 00:54:41,N,1,,,,,2,0.92,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000397,2,2017-12-15 00:10:17,2017-12-15 00:44:44,N,1,,,,,2,4.78,23,0.5,0.5,4.86,0,,0.3,29.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000398,1,2017-12-15 00:07:26,2017-12-15 00:11:49,N,1,,,,,1,0.80,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000399,1,2017-12-15 00:16:45,2017-12-15 00:46:00,N,1,,,,,1,5.10,21.5,0.5,0.5,4.55,0,,0.3,27.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000400,1,2017-12-15 00:56:23,2017-12-15 00:59:44,N,1,,,,,1,0.80,4.5,0.5,0.5,1.45,0,,0.3,7.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000401,1,2017-12-15 00:02:45,2017-12-15 00:17:46,N,1,,,,,1,2.60,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000402,1,2017-12-15 00:33:58,2017-12-15 01:10:08,N,1,,,,,1,13.80,42.5,0.5,0.5,12,0,,0.3,55.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000403,1,2017-12-15 00:08:20,2017-12-15 00:19:54,N,1,,,,,1,2.20,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000404,1,2017-12-15 00:22:02,2017-12-15 00:24:32,N,1,,,,,1,0.40,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000405,1,2017-12-15 00:49:45,2017-12-15 01:07:55,N,1,,,,,1,4.30,16.5,0.5,0.5,3.55,0,,0.3,21.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000406,1,2017-12-15 00:19:42,2017-12-15 00:35:10,N,1,,,,,1,7.20,21.5,0.5,0.5,4,0,,0.3,26.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000407,1,2017-12-15 00:36:17,2017-12-15 00:39:28,N,1,,,,,1,0.60,4.5,0.5,0.5,1.2,0,,0.3,7,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000408,1,2017-12-15 00:44:47,2017-12-15 00:47:53,N,1,,,,,1,0.50,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000409,1,2017-12-15 00:06:26,2017-12-15 01:03:38,N,1,,,,,1,12.30,48,0.5,0.5,13.75,5.76,,0.3,68.81,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000410,1,2017-12-15 00:03:14,2017-12-15 00:10:39,N,1,,,,,1,1.30,7,0.5,0.5,1.5,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000411,1,2017-12-15 00:12:39,2017-12-15 00:37:19,N,1,,,,,1,4.20,18,0.5,0.5,2.5,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000412,1,2017-12-15 00:42:49,2017-12-15 00:46:16,N,1,,,,,1,0.80,5,0.5,0.5,1.55,0,,0.3,7.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000413,2,2017-12-14 19:10:23,2017-12-14 19:18:28,N,1,,,,,1,1.02,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000414,2,2017-12-14 19:23:38,2017-12-15 19:06:32,N,5,,,,,3,4.24,60,0,0.5,0,10.5,,0.3,71.3,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000415,2,2017-12-15 00:55:49,2017-12-15 01:11:02,N,1,,,,,2,7.83,23.5,0.5,0.5,0,0,,0.3,24.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000416,2,2017-12-15 00:05:09,2017-12-15 00:20:48,N,1,,,,,2,1.36,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000417,2,2017-12-15 00:29:27,2017-12-15 00:52:13,N,1,,,,,2,3.98,17.5,0.5,0.5,3.76,0,,0.3,22.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000418,2,2017-12-15 00:02:35,2017-12-15 00:21:33,N,1,,,,,1,2.34,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000419,2,2017-12-15 00:26:40,2017-12-15 01:10:02,N,1,,,,,1,9.00,34.5,0.5,0.5,2.2,0,,0.3,38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000420,2,2017-12-15 00:41:13,2017-12-15 00:47:16,N,1,,,,,5,0.95,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000421,2,2017-12-15 00:48:07,2017-12-15 00:51:55,N,1,,,,,5,0.82,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000422,2,2017-12-15 00:54:41,2017-12-15 01:36:33,N,1,,,,,5,8.13,31.5,0.5,0.5,6.56,0,,0.3,39.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000423,1,2017-12-15 00:10:16,2017-12-15 00:30:12,N,1,,,,,3,4.20,17,0.5,0.5,0,0,,0.3,18.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000424,1,2017-12-15 00:09:09,2017-12-15 00:36:56,N,1,,,,,1,4.20,20,0.5,0.5,0.01,0,,0.3,21.31,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000425,1,2017-12-15 00:54:57,2017-12-15 01:17:59,N,1,,,,,1,5.40,19.5,0.5,0.5,4.15,0,,0.3,24.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000426,2,2017-12-15 00:13:43,2017-12-15 00:42:45,N,1,,,,,1,6.63,25.5,0.5,0.5,9.77,5.76,,0.3,42.33,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000427,2,2017-12-15 00:09:34,2017-12-15 00:21:47,N,1,,,,,1,1.91,10,0.5,0.5,0,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000428,2,2017-12-15 00:26:37,2017-12-15 00:54:33,N,1,,,,,1,6.54,25,0.5,0.5,0,5.76,,0.3,32.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000429,1,2017-12-15 00:09:48,2017-12-15 00:19:19,N,1,,,,,1,2.60,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000430,1,2017-12-15 00:32:26,2017-12-15 00:36:48,N,1,,,,,1,0.50,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000431,1,2017-12-15 00:42:32,2017-12-15 01:03:48,N,1,,,,,1,3.90,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000432,2,2017-12-14 23:59:37,2017-12-15 00:04:06,N,1,,,,,1,0.61,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000433,2,2017-12-15 00:06:30,2017-12-15 00:32:11,N,1,,,,,1,4.48,18.5,0.5,0.5,2,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000434,2,2017-12-15 00:40:44,2017-12-15 01:00:28,N,1,,,,,1,4.72,16.5,0.5,0.5,4.71,5.76,,0.3,28.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000435,2,2017-12-15 00:10:53,2017-12-15 00:17:52,N,1,,,,,1,1.12,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000436,2,2017-12-15 00:35:36,2017-12-15 00:59:12,N,1,,,,,1,10.99,32.5,0.5,0.5,6.76,0,,0.3,40.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000437,2,2017-12-15 00:43:04,2017-12-15 01:16:19,N,2,,,,,1,17.90,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000438,1,2017-12-15 00:02:42,2017-12-15 00:10:12,N,1,,,,,1,1.30,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000439,1,2017-12-15 00:11:22,2017-12-15 00:40:14,N,1,,,,,1,7.30,25.5,0.5,0.5,6.7,0,,0.3,33.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000440,1,2017-12-15 00:20:08,2017-12-15 00:40:28,N,1,,,,,1,2.40,14,0.5,0.5,3.05,0,,0.3,18.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000441,1,2017-12-15 00:43:20,2017-12-15 01:08:32,N,1,,,,,2,1.30,15.5,0.5,0.5,5,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000442,1,2017-12-15 00:11:32,2017-12-15 00:47:05,N,1,,,,,1,5.10,26.5,0.5,0.5,8.3,0,,0.3,36.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000443,2,2017-12-15 00:09:29,2017-12-15 00:29:03,N,1,,,,,1,5.70,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000444,2,2017-12-15 00:30:13,2017-12-15 00:37:34,N,1,,,,,1,1.67,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000445,2,2017-12-14 23:58:56,2017-12-15 00:30:50,N,2,,,,,1,20.04,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000446,2,2017-12-15 00:36:57,2017-12-15 00:45:41,N,1,,,,,1,1.46,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000447,1,2017-12-15 00:04:54,2017-12-15 00:37:27,N,1,,,,,1,4.60,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000448,2,2017-12-15 00:48:43,2017-12-15 00:53:39,N,1,,,,,1,1.30,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000449,2,2017-12-15 00:57:01,2017-12-15 01:01:12,N,1,,,,,1,0.68,5,0.5,0.5,0.63,0,,0.3,6.93,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000450,1,2017-12-15 00:05:18,2017-12-15 00:49:34,N,1,,,,,1,15.40,51,0.5,0.5,5,0,,0.3,57.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000451,2,2017-12-15 00:45:45,2017-12-15 01:07:45,N,1,,,,,1,9.52,28,0.5,0.5,0,5.76,,0.3,35.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000452,2,2017-12-15 00:08:35,2017-12-15 00:56:01,N,1,,,,,5,7.01,31,0.5,0.5,9.69,0,,0.3,41.99,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000453,1,2017-12-15 00:10:24,2017-12-15 00:31:26,N,1,,,,,1,3.20,16,0.5,0.5,0,0,,0.3,17.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000454,2,2017-12-15 00:04:37,2017-12-15 00:53:48,N,1,,,,,1,11.07,40.5,0.5,0.5,5,5.76,,0.3,52.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000455,1,2017-12-15 00:05:10,2017-12-15 00:20:23,N,1,,,,,1,3.10,13,0.5,0.5,1.43,0,,0.3,15.73,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000456,1,2017-12-15 00:21:02,2017-12-15 00:36:30,N,1,,,,,2,2.30,11.5,0.5,0.5,1.92,0,,0.3,14.72,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000457,1,2017-12-15 00:38:38,2017-12-15 01:10:20,N,1,,,,,1,4.90,22.5,0.5,0.5,5.95,0,,0.3,29.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000458,1,2017-12-15 00:03:57,2017-12-15 00:07:19,N,1,,,,,2,1.10,5.5,0.5,0.5,1,0,,0.3,7.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000459,1,2017-12-15 00:26:00,2017-12-15 00:34:10,N,1,,,,,1,1.30,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000460,1,2017-12-15 00:36:16,2017-12-15 00:49:08,N,1,,,,,1,2.60,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000461,1,2017-12-15 00:53:25,2017-12-15 01:00:12,N,1,,,,,1,1.90,8,0.5,0.5,2.3,0,,0.3,11.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000462,2,2017-12-15 00:15:12,2017-12-15 00:26:54,N,1,,,,,1,1.17,8.5,0.5,0.5,2.94,0,,0.3,12.74,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000463,2,2017-12-15 00:47:47,2017-12-15 00:59:33,N,1,,,,,1,1.40,9,0.5,0.5,2.06,0,,0.3,12.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000464,2,2017-12-15 00:08:58,2017-12-15 00:43:22,N,1,,,,,5,6.10,25.5,0.5,0.5,8.04,0,,0.3,34.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000465,2,2017-12-15 00:48:26,2017-12-15 01:01:21,N,1,,,,,5,2.41,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000466,1,2017-12-15 00:42:54,2017-12-15 00:46:52,N,1,,,,,1,0.70,4.5,0.5,0.5,1.15,0,,0.3,6.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000467,1,2017-12-15 00:01:07,2017-12-15 00:05:50,N,1,,,,,1,0.90,5,0.5,0.5,0,0,,0.3,6.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000468,1,2017-12-15 00:09:05,2017-12-15 00:26:40,N,1,,,,,1,3.60,16,0.5,0.5,1,0,,0.3,18.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000469,1,2017-12-15 00:59:32,2017-12-15 01:16:09,N,1,,,,,1,7.70,23,0.5,0.5,7.5,5.76,,0.3,37.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000470,1,2017-12-15 00:15:12,2017-12-15 00:36:25,N,1,,,,,1,1.40,13.5,0.5,0.5,2.95,0,,0.3,17.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000471,1,2017-12-15 00:39:37,2017-12-15 01:19:53,N,1,,,,,1,8.00,31,0.5,0.5,8.05,0,,0.3,40.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000472,1,2017-12-15 00:04:23,2017-12-15 00:37:18,N,1,,,,,1,6.20,25,0.5,0.5,5.25,0,,0.3,31.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000473,1,2017-12-15 00:53:13,2017-12-15 00:59:08,N,1,,,,,1,1.20,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000474,1,2017-12-15 00:14:58,2017-12-15 00:43:09,N,1,,,,,1,4.20,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000475,1,2017-12-15 00:54:26,2017-12-15 01:01:22,N,1,,,,,1,1.10,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000476,2,2017-12-15 00:03:41,2017-12-15 00:33:29,N,1,,,,,1,6.35,25,0.5,0.5,5.26,0,,0.3,31.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000477,2,2017-12-15 00:55:24,2017-12-15 01:29:13,N,1,,,,,1,8.48,30,0.5,0.5,5,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000478,2,2017-12-15 00:13:25,2017-12-15 00:43:14,N,2,,,,,2,17.12,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000479,1,2017-12-15 00:10:35,2017-12-15 00:21:11,N,1,,,,,2,1.10,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000480,1,2017-12-15 00:22:05,2017-12-15 00:25:54,N,1,,,,,1,0.60,4.5,0.5,0.5,1.15,0,,0.3,6.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000481,1,2017-12-15 00:27:36,2017-12-15 00:40:36,N,1,,,,,1,1.00,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000482,1,2017-12-15 00:41:44,2017-12-15 00:47:23,N,1,,,,,1,1.20,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000483,1,2017-12-15 00:49:04,2017-12-15 01:03:47,N,1,,,,,1,2.10,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000484,1,2017-12-15 00:05:27,2017-12-15 00:29:12,N,1,,,,,0,10.50,31.5,0.5,0.5,9,5.76,,0.3,47.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000485,1,2017-12-15 00:35:54,2017-12-15 00:40:31,N,1,,,,,2,0.70,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000486,1,2017-12-15 00:43:11,2017-12-15 00:53:22,N,1,,,,,2,1.00,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000487,2,2017-12-15 00:31:19,2017-12-15 00:49:31,N,1,,,,,1,6.31,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000488,2,2017-12-15 00:19:14,2017-12-15 00:29:43,N,1,,,,,1,1.05,8,0.5,0.5,1.5,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000489,2,2017-12-15 00:32:35,2017-12-15 00:44:57,N,1,,,,,1,2.42,10.5,0.5,0.5,1,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000490,2,2017-12-15 00:48:40,2017-12-15 00:57:57,N,1,,,,,1,1.22,8,0.5,0.5,2.32,0,,0.3,11.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000491,2,2017-12-15 00:38:41,2017-12-15 01:21:39,N,1,,,,,2,17.74,52,0.5,0.5,11.81,5.76,,0.3,70.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000492,2,2017-12-15 00:08:26,2017-12-15 00:19:44,N,1,,,,,1,1.66,8.5,0.5,0.5,2.5,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000493,2,2017-12-15 00:22:53,2017-12-15 00:36:57,N,1,,,,,1,1.74,10,0.5,0.5,1,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000494,2,2017-12-15 01:00:01,2017-12-15 01:05:22,N,1,,,,,1,0.96,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000495,2,2017-12-15 00:02:53,2017-12-15 00:09:38,N,1,,,,,1,0.65,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000496,2,2017-12-15 00:11:31,2017-12-15 00:12:25,N,5,,,,,1,0.00,15,0,0.5,40,0,,0.3,55.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000497,2,2017-12-15 00:43:55,2017-12-15 00:56:49,N,1,,,,,1,1.94,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000498,1,2017-12-15 00:25:49,2017-12-15 00:35:05,N,1,,,,,1,1.10,7.5,0.5,0.5,1.75,0,,0.3,10.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000499,1,2017-12-15 00:52:06,2017-12-15 00:57:28,N,1,,,,,2,0.60,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000500,1,2017-12-15 00:34:18,2017-12-15 01:21:09,N,1,,,,,1,6.60,30,0.5,0.5,9.35,0,,0.3,40.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000501,2,2017-12-15 00:07:37,2017-12-15 00:59:00,N,1,,,,,1,10.17,42,0.5,0.5,0,0,,0.3,43.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000502,1,2017-12-15 00:07:42,2017-12-15 00:30:07,N,1,,,,,2,2.90,16,0.5,0.5,0,0,,0.3,17.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000503,1,2017-12-15 00:30:45,2017-12-15 00:38:40,N,1,,,,,1,1.30,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000504,1,2017-12-15 00:42:23,2017-12-15 00:55:29,N,1,,,,,2,1.80,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000505,1,2017-12-15 00:57:04,2017-12-15 00:57:04,N,1,,,,,1,0.00,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000506,1,2017-12-15 00:59:48,2017-12-15 01:21:26,N,1,,,,,1,4.50,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000507,2,2017-12-15 00:20:44,2017-12-15 00:30:11,N,1,,,,,2,0.78,7.5,0.5,0.5,0,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000508,2,2017-12-15 00:37:52,2017-12-15 00:52:16,N,1,,,,,2,1.16,10.5,0.5,0.5,1.5,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000509,2,2017-12-15 00:58:26,2017-12-15 01:04:02,N,1,,,,,2,1.00,5.5,0.5,0.5,1.36,0,,0.3,8.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000510,1,2017-12-15 00:05:09,2017-12-15 00:25:20,N,1,,,,,1,2.60,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000511,1,2017-12-15 00:29:09,2017-12-15 00:36:48,N,1,,,,,1,1.40,7.5,0.5,0.5,2.2,0,,0.3,11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000512,1,2017-12-15 00:38:25,2017-12-15 00:51:42,N,1,,,,,1,1.50,10,0.5,0.5,3.35,0,,0.3,14.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000513,1,2017-12-15 00:23:44,2017-12-15 00:49:22,N,1,,,,,1,4.40,20.5,0.5,0.5,4.35,0,,0.3,26.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000514,1,2017-12-15 00:09:29,2017-12-15 00:14:30,N,1,,,,,1,0.80,5.5,0.5,0.5,1.7,0,,0.3,8.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000515,1,2017-12-15 00:16:07,2017-12-15 00:35:34,N,1,,,,,1,4.60,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000516,1,2017-12-15 00:58:07,2017-12-15 01:05:15,N,1,,,,,2,1.40,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000517,2,2017-12-15 00:25:15,2017-12-15 00:34:38,N,1,,,,,1,1.21,8,0.5,0.5,2.79,0,,0.3,12.09,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000518,2,2017-12-15 00:40:54,2017-12-15 00:52:56,N,1,,,,,1,0.86,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000519,2,2017-12-15 00:54:22,2017-12-15 01:00:26,N,1,,,,,1,0.89,5.5,0.5,0.5,1,0,,0.3,7.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000520,1,2017-12-15 00:07:03,2017-12-15 00:17:51,N,1,,,,,1,1.00,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000521,1,2017-12-15 00:27:43,2017-12-15 01:02:24,N,1,,,,,1,6.80,27,0.5,0.5,0,0,,0.3,28.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000522,2,2017-12-15 00:14:12,2017-12-15 00:57:30,N,4,,,,,1,33.89,158,0.5,0.5,10,0,,0.3,169.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000523,2,2017-12-15 00:01:06,2017-12-15 00:16:46,N,1,,,,,1,3.00,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000524,2,2017-12-15 00:30:25,2017-12-15 00:41:04,N,1,,,,,1,3.33,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000525,2,2017-12-15 00:42:14,2017-12-15 00:52:16,N,1,,,,,1,4.32,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000526,1,2017-12-15 00:44:17,2017-12-15 00:53:15,N,1,,,,,1,2.10,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000527,1,2017-12-15 00:58:36,2017-12-15 01:17:56,N,1,,,,,2,5.00,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000528,2,2017-12-15 00:14:35,2017-12-15 00:18:59,N,1,,,,,1,0.79,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000529,2,2017-12-15 00:26:09,2017-12-15 01:04:29,N,1,,,,,1,11.54,37.5,0.5,0.5,7.76,0,,0.3,48.51,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000530,2,2017-12-15 00:14:54,2017-12-15 00:31:45,N,1,,,,,1,2.37,12.5,0.5,0.5,2.07,0,,0.3,15.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000531,2,2017-12-15 00:33:52,2017-12-15 00:40:05,N,1,,,,,2,1.43,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000532,2,2017-12-15 00:49:02,2017-12-15 00:52:27,N,1,,,,,2,1.01,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000533,2,2017-12-15 00:21:05,2017-12-15 00:32:58,N,1,,,,,2,1.94,10,0.5,0.5,2.82,0,,0.3,14.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000534,2,2017-12-15 00:36:08,2017-12-15 00:44:27,N,1,,,,,2,1.33,7.5,0.5,0.5,1.2,0,,0.3,10,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000535,2,2017-12-15 00:49:05,2017-12-15 01:22:21,N,1,,,,,2,6.61,25.5,0.5,0.5,5.36,0,,0.3,32.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000536,2,2017-12-15 00:43:23,2017-12-15 01:16:38,N,1,,,,,1,7.28,27,0.5,0.5,5,0,,0.3,33.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000537,1,2017-12-15 00:08:26,2017-12-15 00:11:22,N,1,,,,,2,0.40,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000538,1,2017-12-15 00:13:29,2017-12-15 00:46:41,N,1,,,,,1,6.30,25,0.5,0.5,5.25,0,,0.3,31.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000539,2,2017-12-15 00:11:39,2017-12-15 00:11:42,N,5,,,,,1,0.00,125,0,0,40,0,,0.3,165.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000540,2,2017-12-15 00:56:03,2017-12-15 01:00:44,N,1,,,,,1,0.97,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000541,1,2017-12-15 00:42:52,2017-12-15 01:02:34,N,1,,,,,1,13.00,36,0.5,0.5,6,0,,0.3,43.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000542,1,2017-12-15 00:03:04,2017-12-15 00:52:32,N,1,,,,,1,6.60,32,0.5,0.5,6.65,0,,0.3,39.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000543,1,2017-12-15 00:13:41,2017-12-15 00:28:38,N,1,,,,,1,4.20,14.5,0.5,0.5,3.5,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000544,1,2017-12-15 00:33:27,2017-12-15 00:40:04,N,1,,,,,1,1.70,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000545,1,2017-12-15 00:47:29,2017-12-15 00:58:09,N,1,,,,,1,2.20,10,0.5,0.5,2.8,0,,0.3,14.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000546,1,2017-12-15 00:05:19,2017-12-15 00:38:17,N,1,,,,,1,8.20,30,0.5,0.5,6.25,0,,0.3,37.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000547,2,2017-12-15 00:53:06,2017-12-15 01:14:50,N,1,,,,,4,12.29,35,0.5,0.5,0,0,,0.3,36.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000548,1,2017-12-15 00:02:50,2017-12-15 00:07:18,N,1,,,,,1,0.60,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000549,1,2017-12-15 00:12:02,2017-12-15 00:14:27,N,1,,,,,1,0.40,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000550,1,2017-12-15 00:15:16,2017-12-15 00:21:59,N,1,,,,,1,0.90,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000551,1,2017-12-15 00:24:38,2017-12-15 00:46:00,N,1,,,,,1,4.00,17,0.5,0.5,1,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000552,1,2017-12-15 00:53:46,2017-12-15 00:59:26,N,1,,,,,1,1.20,6.5,0.5,0.5,2,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000553,1,2017-12-15 00:00:57,2017-12-15 00:15:13,N,1,,,,,2,2.60,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000554,1,2017-12-15 00:19:11,2017-12-15 01:02:12,N,1,,,,,2,6.70,30,0.5,0.5,7.8,0,,0.3,39.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000555,2,2017-12-15 00:35:27,2017-12-15 00:54:46,N,1,,,,,1,2.59,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000556,2,2017-12-15 00:06:54,2017-12-15 00:37:31,N,1,,,,,1,9.21,31,0.5,0.5,8.08,0,,0.3,40.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000557,2,2017-12-15 00:38:32,2017-12-15 00:53:06,N,1,,,,,1,4.34,14.5,0.5,0.5,3.95,0,,0.3,19.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000558,2,2017-12-15 00:56:49,2017-12-15 01:03:38,N,1,,,,,1,1.81,7.5,0.5,0.5,0,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000559,2,2017-12-15 00:03:50,2017-12-15 00:13:29,N,1,,,,,1,2.34,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000560,2,2017-12-15 00:28:29,2017-12-15 00:35:00,N,1,,,,,1,1.32,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000561,2,2017-12-15 00:50:11,2017-12-15 01:05:29,N,1,,,,,1,3.22,13.5,0.5,0.5,4.44,0,,0.3,19.24,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000562,1,2017-12-15 00:02:47,2017-12-15 00:41:54,N,1,,,,,1,7.80,31.5,0.5,0.5,6.55,0,,0.3,39.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000563,2,2017-12-15 00:05:55,2017-12-15 00:55:30,N,1,,,,,1,14.85,47,0.5,0.5,0,0,,0.3,48.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000564,2,2017-12-15 00:57:55,2017-12-15 00:58:01,N,5,,,,,1,0.00,10,0,0.5,0,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000565,2,2017-12-15 00:13:16,2017-12-15 00:22:13,N,1,,,,,2,1.57,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000566,2,2017-12-15 00:22:47,2017-12-15 00:24:31,N,1,,,,,1,0.43,3.5,0.5,0.5,0.72,0,,0.3,5.52,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000567,2,2017-12-15 00:32:29,2017-12-15 00:35:46,N,1,,,,,2,0.72,4.5,0.5,0.5,1.45,0,,0.3,7.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000568,2,2017-12-15 00:36:36,2017-12-15 00:52:02,N,1,,,,,2,2.10,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000569,2,2017-12-15 00:54:25,2017-12-15 00:58:58,N,1,,,,,2,0.00,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000570,1,2017-12-15 00:10:19,2017-12-15 00:42:43,N,1,,,,,1,5.40,22.5,0.5,0.5,0,0,,0.3,23.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000571,2,2017-12-15 00:22:17,2017-12-15 00:54:37,N,1,,,,,1,5.00,24,0.5,0.5,4,0,,0.3,29.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000572,2,2017-12-14 23:58:52,2017-12-15 00:49:35,N,1,,,,,1,11.20,41,0.5,0.5,7.21,5.76,,0.3,55.27,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000573,1,2017-12-15 00:36:07,2017-12-15 01:03:18,Y,1,,,,,1,5.00,20,0.5,0.5,4.25,0,,0.3,25.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000574,1,2017-12-15 00:15:45,2017-12-15 00:50:45,N,1,,,,,1,9.20,32,0.5,0.5,0,5.76,,0.3,39.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000575,1,2017-12-15 00:22:12,2017-12-15 00:38:30,N,1,,,,,1,2.20,12.5,0.5,0.5,0,0,,0.3,13.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000576,1,2017-12-15 00:41:46,2017-12-15 01:01:28,Y,1,,,,,1,3.70,15.5,0.5,0.5,3.35,0,,0.3,20.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000577,1,2017-12-15 00:25:24,2017-12-15 01:02:48,N,1,,,,,1,6.00,27.5,0.5,0.5,5.75,0,,0.3,34.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000578,2,2017-12-15 00:07:55,2017-12-15 00:12:34,N,1,,,,,1,0.96,5.5,0.5,0.5,0,0,,0.3,6.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000579,2,2017-12-15 00:13:36,2017-12-15 00:45:06,N,1,,,,,1,4.72,22,0.5,0.5,0,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000580,2,2017-12-15 00:56:28,2017-12-15 01:00:43,N,1,,,,,1,0.79,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000581,2,2017-12-15 00:08:58,2017-12-15 00:16:04,N,1,,,,,1,0.73,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000582,2,2017-12-15 00:17:18,2017-12-15 00:50:54,N,1,,,,,1,3.66,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000583,2,2017-12-15 00:49:08,2017-12-15 01:09:20,N,1,,,,,1,4.16,17,0.5,0.5,3.66,0,,0.3,21.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000584,2,2017-12-15 00:50:52,2017-12-15 01:03:54,N,1,,,,,1,2.09,10,0.5,0.5,2.82,0,,0.3,14.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000585,2,2017-12-15 00:31:15,2017-12-15 00:35:58,N,1,,,,,1,1.11,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000586,2,2017-12-15 00:51:58,2017-12-15 01:15:57,N,1,,,,,1,2.79,16,0.5,0.5,0,5.76,,0.3,23.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000587,2,2017-12-15 00:03:58,2017-12-15 00:14:13,N,1,,,,,1,1.68,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000588,2,2017-12-15 00:16:35,2017-12-15 01:25:44,N,1,,,,,1,15.30,59,0.5,0.5,12.06,0,,0.3,72.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000589,2,2017-12-15 00:06:16,2017-12-15 00:20:06,N,1,,,,,1,2.16,11,0.5,0.5,0,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000590,2,2017-12-15 00:23:57,2017-12-15 00:34:07,N,1,,,,,1,1.73,8.5,0.5,0.5,2.45,0,,0.3,12.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000591,2,2017-12-15 00:35:00,2017-12-15 00:41:32,N,1,,,,,1,1.44,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000592,2,2017-12-15 00:42:57,2017-12-15 01:17:17,N,1,,,,,1,8.22,28.5,0.5,0.5,7.45,0,,0.3,37.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000593,2,2017-12-15 00:11:18,2017-12-15 00:41:56,N,1,,,,,1,4.03,20.5,0.5,0.5,3.5,0,,0.3,25.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000594,2,2017-12-15 00:50:15,2017-12-15 01:05:16,N,1,,,,,1,2.87,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000595,2,2017-12-15 00:44:35,2017-12-15 01:36:39,N,1,,,,,1,13.58,47,0.5,0.5,9,0,,0.3,57.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000596,2,2017-12-15 00:10:26,2017-12-15 01:14:11,N,2,,,,,5,22.01,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000597,1,2017-12-15 00:20:22,2017-12-15 00:59:40,N,2,,,,,2,18.00,52,0,0.5,0,0,,0.3,52.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000598,1,2017-12-15 00:11:10,2017-12-15 00:22:38,N,1,,,,,2,2.00,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000599,1,2017-12-15 00:25:37,2017-12-15 00:41:15,N,1,,,,,3,2.60,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000600,2,2017-12-15 00:09:00,2017-12-15 00:36:33,N,1,,,,,2,2.91,18.5,0.5,0.5,0,0,,0.3,19.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000601,2,2017-12-15 00:53:19,2017-12-15 01:19:39,N,1,,,,,2,7.12,24,0.5,0.5,6.32,0,,0.3,31.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000602,2,2017-12-15 00:29:14,2017-12-15 00:38:05,N,1,,,,,1,1.50,8,0.5,0.5,2.32,0,,0.3,11.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000603,2,2017-12-15 00:39:24,2017-12-15 00:57:43,N,1,,,,,1,3.18,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000604,2,2017-12-15 00:22:31,2017-12-15 00:29:31,N,1,,,,,3,1.44,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000605,2,2017-12-15 00:48:10,2017-12-15 01:20:50,N,1,,,,,3,6.85,25.5,0.5,0.5,0,0,,0.3,26.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000606,1,2017-12-15 00:17:11,2017-12-15 00:24:41,N,1,,,,,1,1.30,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000607,1,2017-12-15 00:26:31,2017-12-15 00:36:40,N,1,,,,,1,1.40,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000608,1,2017-12-15 00:43:07,2017-12-15 01:24:10,N,1,,,,,1,10.60,37.5,0.5,0.5,2,0,,0.3,40.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000609,2,2017-12-15 00:22:39,2017-12-15 00:43:55,N,2,,,,,1,8.03,52,0,0.5,2,5.76,,0.3,60.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000610,1,2017-12-15 00:02:34,2017-12-15 00:11:48,N,1,,,,,1,4.90,15.5,0.5,0.5,3,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000611,1,2017-12-15 00:29:52,2017-12-15 00:32:26,N,1,,,,,1,0.30,3.5,0.5,0.5,1.4,0,,0.3,6.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000612,1,2017-12-15 00:37:00,2017-12-15 00:40:47,N,1,,,,,1,0.70,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000613,1,2017-12-15 00:23:17,2017-12-15 00:56:40,N,1,,,,,1,3.90,21.5,0.5,0.5,4.55,0,,0.3,27.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000614,2,2017-12-15 00:45:58,2017-12-15 01:16:25,N,1,,,,,1,15.59,44.5,0.5,0.5,0,5.76,,0.3,51.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000615,2,2017-12-15 00:10:34,2017-12-15 00:11:56,N,1,,,,,1,0.08,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000616,2,2017-12-15 00:17:20,2017-12-15 00:36:29,N,1,,,,,1,2.99,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000617,2,2017-12-15 00:10:12,2017-12-15 00:30:28,N,1,,,,,1,2.17,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000618,2,2017-12-15 00:35:21,2017-12-15 00:55:02,N,1,,,,,1,6.77,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000619,1,2017-12-15 00:46:25,2017-12-15 01:09:39,N,1,,,,,1,5.70,20.5,0.5,0.5,0,0,,0.3,21.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000620,1,2017-12-15 00:04:11,2017-12-15 00:18:22,N,1,,,,,1,3.10,12,0.5,0.5,2,0,,0.3,15.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000621,1,2017-12-15 00:31:24,2017-12-15 00:42:28,N,1,,,,,1,0.60,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000622,1,2017-12-15 00:44:08,2017-12-15 00:54:43,N,1,,,,,1,1.60,9,0.5,0.5,3.05,0,,0.3,13.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000623,2,2017-12-15 00:19:42,2017-12-15 00:47:45,N,1,,,,,1,13.76,39,0.5,0.5,0,5.76,,0.3,46.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000624,1,2017-12-15 00:52:27,2017-12-15 01:08:45,N,1,,,,,1,7.70,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000625,1,2017-12-15 00:19:22,2017-12-15 00:34:50,N,1,,,,,1,3.10,14,0.5,0.5,2,0,,0.3,17.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000626,1,2017-12-15 00:44:21,2017-12-15 01:23:25,N,1,,,,,1,10.70,35,0.5,0.5,7,0,,0.3,43.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000627,1,2017-12-15 00:50:13,2017-12-15 01:15:53,N,1,,,,,1,16.10,44,0.5,0.5,11.3,0,,0.3,56.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000628,2,2017-12-14 23:58:56,2017-12-15 00:03:51,N,1,,,,,3,1.18,6,0.5,0.5,1.02,0,,0.3,8.32,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000629,2,2017-12-15 00:05:10,2017-12-15 00:41:52,N,1,,,,,4,9.78,33.5,0.5,0.5,2,5.76,,0.3,42.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000630,1,2017-12-15 00:02:37,2017-12-15 00:09:45,N,1,,,,,1,1.30,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000631,1,2017-12-15 00:16:03,2017-12-15 00:22:28,N,1,,,,,1,0.50,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000632,1,2017-12-15 00:24:13,2017-12-15 00:37:26,N,1,,,,,2,1.90,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000633,1,2017-12-15 00:48:50,2017-12-15 00:52:32,N,1,,,,,1,0.50,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000634,2,2017-12-15 00:10:46,2017-12-15 00:39:54,N,1,,,,,2,4.28,21,0.5,0.5,4.46,0,,0.3,26.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000635,1,2017-12-15 00:03:20,2017-12-15 00:09:54,N,1,,,,,5,1.00,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000636,1,2017-12-15 00:11:44,2017-12-15 00:13:57,N,1,,,,,2,0.50,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000637,1,2017-12-15 00:14:52,2017-12-15 00:31:37,N,1,,,,,1,3.60,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000638,1,2017-12-15 00:28:49,2017-12-15 00:35:36,N,1,,,,,1,1.30,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000639,1,2017-12-15 00:46:13,2017-12-15 01:11:12,N,1,,,,,1,4.90,19,0.5,0.5,0,0,,0.3,20.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000640,1,2017-12-15 00:52:19,2017-12-15 01:28:24,N,1,,,,,1,6.60,29,0.5,0.5,6.05,0,,0.3,36.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000641,1,2017-12-15 00:09:02,2017-12-15 00:15:32,N,1,,,,,2,1.30,7,0.5,0.5,0,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000642,1,2017-12-15 00:32:12,2017-12-15 00:38:02,N,1,,,,,1,1.30,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000643,1,2017-12-15 00:57:40,2017-12-15 01:05:08,N,1,,,,,1,1.40,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000644,2,2017-12-15 00:07:59,2017-12-15 00:37:06,N,1,,,,,1,3.73,19,0.5,0.5,1,0,,0.3,21.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000645,2,2017-12-15 00:49:50,2017-12-15 00:56:39,N,1,,,,,1,1.95,8.5,0.5,0.5,0,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000646,2,2017-12-15 00:01:20,2017-12-15 00:25:50,N,1,,,,,5,5.33,20.5,0.5,0.5,4.36,0,,0.3,26.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000647,2,2017-12-15 00:28:14,2017-12-15 00:29:15,N,1,,,,,5,0.03,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000648,2,2017-12-15 00:39:06,2017-12-15 00:53:21,N,1,,,,,5,1.21,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000649,2,2017-12-15 00:54:22,2017-12-15 01:20:16,N,1,,,,,5,4.22,18.5,0.5,0.5,4.95,0,,0.3,24.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000650,2,2017-12-15 00:21:40,2017-12-15 00:42:49,N,1,,,,,2,3.51,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000651,2,2017-12-15 01:00:21,2017-12-15 01:20:15,N,1,,,,,2,8.83,26.5,0.5,0.5,0,5.76,,0.3,33.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000652,1,2017-12-15 00:51:00,2017-12-15 01:03:07,N,1,,,,,1,2.10,10,0.5,0.5,2.3,0,,0.3,13.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000653,1,2017-12-15 00:00:05,2017-12-15 00:14:23,N,1,,,,,1,2.60,12,0.5,0.5,3.95,0,,0.3,17.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000654,1,2017-12-15 00:26:10,2017-12-15 00:30:41,N,1,,,,,1,0.70,5,0.5,0.5,0.95,0,,0.3,7.25,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000655,1,2017-12-15 00:32:06,2017-12-15 00:42:31,N,1,,,,,1,1.70,9,0.5,0.5,0,0,,0.3,10.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000656,1,2017-12-15 00:43:47,2017-12-15 01:04:09,N,1,,,,,1,3.90,15.5,0.5,0.5,0,0,,0.3,16.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000657,2,2017-12-15 00:00:11,2017-12-15 00:29:57,N,1,,,,,3,5.36,24,0.5,0.5,7.59,0,,0.3,32.89,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000658,2,2017-12-15 00:55:15,2017-12-15 01:12:55,N,1,,,,,4,2.32,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000659,2,2017-12-15 00:05:39,2017-12-15 00:32:38,N,1,,,,,1,3.69,19,0.5,0.5,6.09,0,,0.3,26.39,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000660,2,2017-12-15 00:36:55,2017-12-15 01:15:17,N,1,,,,,1,10.12,36,0.5,0.5,7.46,0,,0.3,44.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000661,2,2017-12-15 00:11:13,2017-12-15 00:26:46,N,1,,,,,3,4.16,15.5,0.5,0.5,3.36,0,,0.3,20.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000662,2,2017-12-15 00:29:43,2017-12-15 01:09:14,N,1,,,,,3,4.33,25.5,0.5,0.5,3,0,,0.3,29.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000663,2,2017-12-15 00:04:03,2017-12-15 00:18:22,N,1,,,,,2,1.95,11,0.5,0.5,0,0,,0.3,12.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000664,2,2017-12-15 00:23:19,2017-12-15 00:25:03,N,1,,,,,2,0.55,3.5,0.5,0.5,1.2,0,,0.3,6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000665,2,2017-12-15 00:28:30,2017-12-15 00:48:26,N,1,,,,,2,3.86,16.5,0.5,0.5,2,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000666,2,2017-12-15 00:54:59,2017-12-16 00:48:26,N,1,,,,,2,6.51,27,0.5,0.5,0,0,,0.3,28.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000667,1,2017-12-15 00:00:19,2017-12-15 00:07:29,N,1,,,,,1,1.30,6.5,0.5,0.5,0.5,0,,0.3,8.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000668,1,2017-12-15 00:14:16,2017-12-15 00:24:38,N,1,,,,,1,1.70,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000669,1,2017-12-15 00:28:03,2017-12-15 00:48:26,N,1,,,,,1,3.70,15.5,0.5,0.5,1.5,0,,0.3,18.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000670,2,2017-12-15 00:04:25,2017-12-15 00:10:06,N,1,,,,,1,0.92,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000671,2,2017-12-15 00:11:22,2017-12-15 00:36:07,N,1,,,,,1,4.65,20,0.5,0.5,5.32,0,,0.3,26.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000672,2,2017-12-15 00:49:56,2017-12-15 00:56:19,N,1,,,,,1,1.85,7.5,0.5,0.5,1.76,0,,0.3,10.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000673,1,2017-12-15 00:54:32,2017-12-15 01:06:37,N,1,,,,,2,4.40,15,0.5,0.5,0,0,,0.3,16.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000674,2,2017-12-15 00:10:56,2017-12-15 00:49:42,N,2,,,,,1,19.22,52,0,0.5,0,5.76,,0.3,58.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000675,2,2017-12-15 00:53:14,2017-12-15 01:10:10,N,1,,,,,1,2.44,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000676,1,2017-12-15 00:24:30,2017-12-15 00:27:49,N,1,,,,,1,0.80,5,0.5,0.5,1,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000677,1,2017-12-15 00:35:28,2017-12-15 00:43:21,N,1,,,,,1,1.20,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000678,1,2017-12-15 00:45:35,2017-12-15 00:55:48,N,1,,,,,1,1.60,8.5,0.5,0.5,2.9,0,,0.3,12.7,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000679,2,2017-12-15 00:07:29,2017-12-15 00:38:34,N,1,,,,,1,13.41,39.5,0.5,0.5,9.31,5.76,,0.3,55.87,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000680,2,2017-12-15 00:48:16,2017-12-15 01:03:17,N,1,,,,,1,3.08,13.5,0.5,0.5,0,0,,0.3,14.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000681,1,2017-12-15 00:22:27,2017-12-15 00:38:33,N,1,,,,,1,1.20,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000682,1,2017-12-15 00:41:29,2017-12-15 00:46:11,N,1,,,,,1,0.80,5.5,0.5,0.5,12.3,0,,0.3,19.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000683,1,2017-12-15 00:57:43,2017-12-15 01:22:11,N,1,,,,,1,6.00,21.5,0.5,0.5,0,0,,0.3,22.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000684,2,2017-12-15 00:36:07,2017-12-15 01:08:34,N,1,,,,,1,5.41,22.5,0.5,0.5,0,0,,0.3,23.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000685,2,2017-12-15 00:32:43,2017-12-15 00:38:26,N,1,,,,,1,1.00,6.5,0.5,0.5,1,0,,0.3,8.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000686,2,2017-12-15 00:57:05,2017-12-15 01:05:25,N,1,,,,,1,1.67,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000687,2,2017-12-15 00:19:35,2017-12-15 00:31:13,N,1,,,,,1,2.63,11,0.5,0.5,0,0,,0.3,12.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000688,1,2017-12-15 00:04:18,2017-12-15 00:12:26,N,1,,,,,1,0.90,7,0.5,0.5,1.65,0,,0.3,9.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000689,1,2017-12-15 00:14:22,2017-12-15 00:59:18,N,1,,,,,1,6.60,29.5,0.5,0.5,6.15,0,,0.3,36.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000690,1,2017-12-15 00:00:52,2017-12-15 00:49:27,N,1,,,,,1,8.70,40,0.5,0.5,0,5.76,,0.3,47.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000691,1,2017-12-15 00:17:13,2017-12-15 00:37:07,N,1,,,,,1,3.10,14.5,0.5,0.5,3.15,0,,0.3,18.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000692,1,2017-12-15 00:38:56,2017-12-15 00:49:41,N,1,,,,,1,1.10,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000693,1,2017-12-15 00:52:01,2017-12-15 01:17:31,N,1,,,,,1,4.70,19.5,0.5,0.5,4,0,,0.3,24.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000694,2,2017-12-15 00:03:29,2017-12-15 00:03:31,N,5,,,,,1,0.06,75,0,0,15.06,0,,0.3,90.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000873,1,2017-12-15 00:48:42,2017-12-15 00:51:50,Y,1,,,,,1,0.40,4,0.5,0.5,0.79,0,,0.3,6.09,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000695,2,2017-12-15 00:22:23,2017-12-15 00:29:43,N,1,,,,,2,1.23,6.5,0.5,0.5,2.34,0,,0.3,10.14,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000696,2,2017-12-15 00:44:53,2017-12-15 00:48:01,N,1,,,,,2,0.38,4,0.5,0.5,1,0,,0.3,6.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000697,1,2017-12-15 00:19:26,2017-12-15 00:28:01,N,1,,,,,1,1.40,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000698,1,2017-12-15 00:39:46,2017-12-15 01:14:14,N,1,,,,,1,7.80,28,0.5,0.5,3,0,,0.3,32.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000699,1,2017-12-15 00:16:44,2017-12-15 00:45:52,N,2,,,,,3,17.30,52,0,0.5,0,5.76,,0.3,58.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000700,2,2017-12-15 00:02:11,2017-12-15 00:29:09,N,1,,,,,1,6.58,24,0.5,0.5,5.06,0,,0.3,30.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000701,2,2017-12-15 00:30:14,2017-12-15 00:46:04,N,1,,,,,1,2.80,12.5,0.5,0.5,0,0,,0.3,13.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000702,1,2017-12-15 00:10:46,2017-12-15 00:40:01,N,1,,,,,1,9.90,31,0.5,0.5,7.6,5.76,,0.3,45.66,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000703,1,2017-12-15 00:48:41,2017-12-15 01:06:29,N,1,,,,,1,3.50,15,0.5,0.5,4.05,0,,0.3,20.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000704,2,2017-12-15 00:22:48,2017-12-15 00:42:03,N,1,,,,,1,3.05,15,0.5,0.5,2.44,0,,0.3,18.74,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000705,2,2017-12-15 00:46:30,2017-12-15 01:13:28,N,1,,,,,1,3.78,19,0.5,0.5,5.08,0,,0.3,25.38,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000706,2,2017-12-15 00:01:57,2017-12-15 00:25:18,N,1,,,,,1,4.73,19.5,0.5,0.5,6.24,0,,0.3,27.04,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000707,2,2017-12-15 00:01:06,2017-12-15 00:05:42,N,1,,,,,2,1.25,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000708,2,2017-12-15 00:28:01,2017-12-15 00:35:18,N,1,,,,,2,1.25,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000709,2,2017-12-15 00:36:39,2017-12-15 00:50:41,N,1,,,,,2,3.16,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000710,2,2017-12-15 00:02:19,2017-12-15 00:22:36,N,1,,,,,1,3.68,15.5,0.5,0.5,3.36,0,,0.3,22.11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000711,2,2017-12-15 00:23:44,2017-12-15 00:33:57,N,1,,,,,1,1.77,9,0.5,0.5,2.58,0,,0.3,12.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000712,2,2017-12-15 00:37:09,2017-12-15 00:56:53,N,1,,,,,1,2.92,15,0.5,0.5,2,0,,0.3,18.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000713,2,2017-12-15 00:12:13,2017-12-15 00:29:34,N,1,,,,,1,12.02,33,0.5,0.5,8.58,0,,0.3,42.88,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000714,2,2017-12-15 00:11:36,2017-12-15 00:40:50,N,1,,,,,3,5.63,23,0.5,0.5,4.86,0,,0.3,29.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000715,2,2017-12-15 00:43:02,2017-12-15 00:49:08,N,1,,,,,3,1.68,7,0.5,0.5,0,0,,0.3,8.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000716,1,2017-12-15 00:05:41,2017-12-15 00:32:29,N,1,,,,,2,4.20,20,0.5,0.5,5.3,0,,0.3,26.6,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000717,1,2017-12-15 00:44:27,2017-12-15 01:04:20,N,1,,,,,1,4.90,17,0.5,0.5,5.45,0,,0.3,23.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000718,1,2017-12-15 00:30:44,2017-12-15 00:40:04,N,1,,,,,1,1.50,8.5,0.5,0.5,1.5,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000719,1,2017-12-15 00:41:48,2017-12-15 00:50:54,N,1,,,,,1,1.50,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000720,1,2017-12-15 00:59:42,2017-12-15 01:12:12,N,1,,,,,1,4.20,13.5,0.5,0.5,2,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000721,2,2017-12-15 00:08:17,2017-12-15 00:23:49,N,1,,,,,2,6.81,21.5,0.5,0.5,3.5,0,,0.3,26.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000722,2,2017-12-15 00:54:07,2017-12-15 01:12:23,N,1,,,,,2,8.71,25,0.5,0.5,0,5.76,,0.3,32.06,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000723,1,2017-12-15 00:10:46,2017-12-15 00:23:46,N,1,,,,,1,1.80,10.5,0.5,0.5,1,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000724,1,2017-12-15 00:25:04,2017-12-15 00:40:45,N,1,,,,,1,3.10,13.5,0.5,0.5,3.7,0,,0.3,18.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000725,1,2017-12-15 00:46:29,2017-12-15 01:22:33,N,1,,,,,1,5.80,26.5,0.5,0.5,5.55,0,,0.3,33.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000726,2,2017-12-15 00:07:51,2017-12-15 00:14:32,N,1,,,,,1,0.81,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000727,2,2017-12-15 00:17:00,2017-12-15 00:37:28,N,1,,,,,1,1.28,13,0.5,0.5,2.86,0,,0.3,17.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000728,2,2017-12-15 00:19:24,2017-12-15 00:34:41,N,1,,,,,1,4.52,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000729,2,2017-12-15 00:01:16,2017-12-15 00:17:43,N,1,,,,,1,1.05,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000730,2,2017-12-15 00:18:40,2017-12-15 00:31:35,N,1,,,,,1,1.22,9.5,0.5,0.5,0,0,,0.3,10.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000731,1,2017-12-15 00:31:52,2017-12-15 00:38:45,N,1,,,,,1,1.30,6.5,0.5,0.5,1.55,0,,0.3,9.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000732,1,2017-12-15 00:18:16,2017-12-15 00:43:11,N,1,,,,,1,7.00,25,0.5,0.5,7,0,,0.3,33.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000733,2,2017-12-15 00:05:53,2017-12-15 00:35:02,N,1,,,,,1,6.85,26,0.5,0.5,0,0,,0.3,27.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000734,2,2017-12-15 00:53:47,2017-12-15 01:24:10,N,1,,,,,1,8.55,29.5,0.5,0.5,3,5.76,,0.3,39.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000735,1,2017-12-15 00:18:57,2017-12-15 00:36:41,N,1,,,,,1,2.40,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000736,1,2017-12-15 00:39:32,2017-12-15 00:51:27,N,1,,,,,1,2.20,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000737,1,2017-12-15 00:09:35,2017-12-15 00:10:30,N,4,,,,,0,0.00,3,0.5,0.5,0,0,,0.3,4.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000738,1,2017-12-15 00:10:38,2017-12-15 00:50:50,N,2,,,,,2,18.30,52,0,0.5,10,0,,0.3,62.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000739,1,2017-12-15 00:59:40,2017-12-15 01:12:42,N,1,,,,,1,1.90,10,0.5,0.5,2.25,0,,0.3,13.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000740,2,2017-12-15 00:10:27,2017-12-15 00:32:38,N,1,,,,,1,15.36,41.5,0.5,0.5,8.56,0,,0.3,51.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000741,2,2017-12-15 00:43:57,2017-12-15 01:14:33,N,1,,,,,1,7.44,28,0.5,0.5,8.76,5.76,,0.3,43.82,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000742,1,2017-12-15 00:23:14,2017-12-15 00:40:55,N,1,,,,,1,3.10,14,0.5,0.5,1.5,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000743,1,2017-12-15 00:45:14,2017-12-15 01:06:11,N,1,,,,,1,4.60,19.5,0.5,0.5,4.15,0,,0.3,24.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000744,1,2017-12-15 00:05:13,2017-12-15 00:31:01,N,1,,,,,1,4.90,20.5,0.5,0.5,4.35,0,,0.3,26.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000745,1,2017-12-15 00:55:03,2017-12-15 01:00:44,N,1,,,,,1,0.90,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000746,1,2017-12-15 00:08:52,2017-12-15 00:44:05,N,1,,,,,1,3.80,23,0.5,0.5,6.05,0,,0.3,30.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000747,2,2017-12-15 00:33:05,2017-12-15 00:48:27,N,1,,,,,1,2.99,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000748,2,2017-12-15 00:15:29,2017-12-15 00:54:51,N,1,,,,,1,4.36,26,0.5,0.5,5.46,0,,0.3,32.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000749,1,2017-12-15 00:01:36,2017-12-15 00:04:58,N,1,,,,,1,0.80,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000750,1,2017-12-15 00:15:24,2017-12-15 00:21:09,N,1,,,,,1,1.80,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000751,1,2017-12-15 00:40:23,2017-12-15 00:46:29,N,1,,,,,1,1.00,6,0.5,0.5,0,0,,0.3,7.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000752,1,2017-12-15 00:49:17,2017-12-15 01:00:42,N,1,,,,,1,1.30,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000753,2,2017-12-15 00:11:14,2017-12-15 00:23:43,N,1,,,,,3,2.00,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000754,2,2017-12-15 00:31:31,2017-12-15 00:45:43,N,1,,,,,2,2.13,11.5,0.5,0.5,1,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000755,2,2017-12-15 00:40:55,2017-12-15 01:00:47,N,1,,,,,1,2.88,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000756,2,2017-12-15 00:08:22,2017-12-15 00:10:21,N,1,,,,,1,0.68,4,0.5,0.5,0,0,,0.3,5.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000757,2,2017-12-15 00:11:45,2017-12-15 00:32:57,N,1,,,,,1,4.32,18,0.5,0.5,3.86,0,,0.3,23.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000758,2,2017-12-15 00:45:14,2017-12-15 01:04:07,N,1,,,,,1,4.99,18,0.5,0.5,5.79,0,,0.3,25.09,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000759,1,2017-12-15 00:31:29,2017-12-15 00:31:58,N,5,,,,,1,0.00,18,0,0,3.65,0,,0.3,21.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000760,1,2017-12-15 00:34:33,2017-12-15 00:39:54,N,1,,,,,1,1.60,7,0.5,0.5,2.05,0,,0.3,10.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000761,1,2017-12-15 00:42:43,2017-12-15 00:51:05,N,1,,,,,1,1.50,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000762,1,2017-12-15 00:17:16,2017-12-15 00:26:19,N,1,,,,,1,2.30,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000763,1,2017-12-15 00:35:40,2017-12-15 00:49:25,N,1,,,,,1,3.20,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000764,2,2017-12-15 00:23:04,2017-12-15 00:42:31,N,1,,,,,1,8.98,26.5,0.5,0.5,2,0,,0.3,29.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000765,1,2017-12-15 00:46:24,2017-12-15 01:01:21,N,1,,,,,1,2.80,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000766,2,2017-12-14 23:57:15,2017-12-15 00:15:42,N,1,,,,,1,5.55,20,0.5,0.5,4.26,0,,0.3,25.56,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000767,2,2017-12-15 00:23:19,2017-12-15 00:34:54,N,1,,,,,1,1.26,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000768,2,2017-12-15 00:41:52,2017-12-15 00:48:13,N,1,,,,,1,0.82,6,0.5,0.5,1.82,0,,0.3,9.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000769,2,2017-12-15 00:49:53,2017-12-15 01:02:43,N,1,,,,,1,2.31,11,0.5,0.5,3.69,0,,0.3,15.99,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000770,2,2017-12-15 00:29:46,2017-12-15 01:15:02,N,1,,,,,5,9.27,37.5,0.5,0.5,8.91,5.76,,0.3,53.47,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000771,2,2017-12-15 00:03:21,2017-12-15 00:16:01,N,1,,,,,5,2.26,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000772,2,2017-12-15 00:16:49,2017-12-15 00:24:40,N,1,,,,,5,1.39,7.5,0.5,0.5,1,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000773,2,2017-12-15 00:29:56,2017-12-15 00:42:19,N,1,,,,,5,1.61,9.5,0.5,0.5,2.16,0,,0.3,12.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000774,2,2017-12-15 00:43:51,2017-12-15 00:54:08,N,1,,,,,5,1.04,7,0.5,0.5,0.7,0,,0.3,9,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000775,2,2017-12-15 00:55:15,2017-12-15 00:59:45,N,1,,,,,5,0.38,4.5,0.5,0.5,0,0,,0.3,5.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000776,2,2017-12-15 00:15:43,2017-12-15 00:43:10,N,1,,,,,3,3.34,19,0.5,0.5,4.06,0,,0.3,24.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000777,2,2017-12-15 00:54:12,2017-12-15 01:05:13,N,1,,,,,3,2.16,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000778,2,2017-12-15 00:22:21,2017-12-15 00:48:52,N,1,,,,,1,9.82,30.5,0.5,0.5,0,0,,0.3,31.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000779,2,2017-12-15 00:53:35,2017-12-15 00:55:32,N,1,,,,,1,0.36,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000780,2,2017-12-15 00:33:11,2017-12-15 00:39:57,N,1,,,,,1,1.35,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000781,2,2017-12-15 00:40:32,2017-12-15 00:59:34,N,1,,,,,1,4.29,16,0.5,0.5,0,0,,0.3,17.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000782,2,2017-12-15 00:21:31,2017-12-15 00:25:39,N,1,,,,,1,0.27,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000783,2,2017-12-15 00:22:28,2017-12-15 00:57:26,N,1,,,,,1,5.58,26,0.5,0.5,1,0,,0.3,28.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000784,1,2017-12-15 00:44:31,2017-12-15 00:48:39,N,1,,,,,3,0.60,4.5,0.5,0.5,1.7,0,,0.3,7.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000785,1,2017-12-15 00:54:03,2017-12-15 01:08:58,N,1,,,,,1,2.60,12,0.5,0.5,2.65,0,,0.3,15.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000786,1,2017-12-15 00:17:08,2017-12-15 00:38:33,N,1,,,,,1,9.00,27.5,0.5,0.5,6,5.76,,0.3,40.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000787,1,2017-12-15 00:40:42,2017-12-15 00:58:49,N,1,,,,,1,3.70,15,0.5,0.5,0.02,0,,0.3,16.32,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000788,1,2017-12-15 00:07:49,2017-12-15 00:32:05,N,1,,,,,1,3.90,17.5,0.5,0.5,1.88,0,,0.3,20.68,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000789,1,2017-12-15 00:36:09,2017-12-15 00:37:14,N,1,,,,,1,0.30,3,0.5,0.5,0.85,0,,0.3,5.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000790,1,2017-12-15 00:41:57,2017-12-15 01:13:05,N,1,,,,,1,8.30,29.5,0.5,0.5,7.7,0,,0.3,38.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000791,2,2017-12-15 00:33:18,2017-12-15 00:39:28,N,1,,,,,2,1.87,8,0.5,0.5,1,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000792,2,2017-12-15 00:41:31,2017-12-15 01:14:47,N,1,,,,,2,9.43,32.5,0.5,0.5,6.76,0,,0.3,40.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000793,2,2017-12-15 00:51:28,2017-12-15 01:18:40,N,1,,,,,3,15.96,45,0.5,0.5,0,0,,0.3,46.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000794,2,2017-12-15 00:24:56,2017-12-15 00:57:40,N,1,,,,,1,5.40,23.5,0.5,0.5,0,5.76,,0.3,30.56,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000795,2,2017-12-15 00:50:36,2017-12-15 01:11:29,N,1,,,,,1,14.03,38,0.5,0.5,7.86,0,,0.3,47.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000796,2,2017-12-15 00:16:48,2017-12-15 00:25:42,N,1,,,,,1,3.31,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000797,2,2017-12-15 00:28:58,2017-12-15 00:45:19,N,1,,,,,1,2.56,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000798,2,2017-12-15 00:47:35,2017-12-15 01:25:59,N,1,,,,,1,5.91,26,0.5,0.5,2,5.76,,0.3,35.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000799,1,2017-12-15 00:04:08,2017-12-15 00:15:31,N,1,,,,,1,1.90,9,0.5,0.5,2.55,0,,0.3,12.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000800,1,2017-12-15 00:17:18,2017-12-15 00:31:04,N,1,,,,,1,2.50,11.5,0.5,0.5,1.5,0,,0.3,14.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000801,2,2017-12-15 00:17:03,2017-12-15 00:54:29,N,1,,,,,2,6.26,25.5,0.5,0.5,0,0,,0.3,26.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000802,1,2017-12-15 00:18:30,2017-12-15 00:32:04,N,1,,,,,2,1.40,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000803,1,2017-12-15 00:40:55,2017-12-15 00:52:32,N,1,,,,,2,1.70,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000804,1,2017-12-15 00:58:53,2017-12-15 01:14:54,N,1,,,,,1,3.00,13,0.5,0.5,3.55,0,,0.3,17.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000805,1,2017-12-15 00:21:00,2017-12-15 00:39:48,N,1,,,,,1,4.40,16.5,0.5,0.5,0,0,,0.3,17.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000806,1,2017-12-15 00:41:50,2017-12-15 00:54:39,N,1,,,,,1,5.00,16.5,0.5,0.5,0,0,,0.3,17.8,4,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000807,1,2017-12-15 00:16:11,2017-12-15 00:23:24,N,1,,,,,1,1.10,6.5,0.5,0.5,1.4,0,,0.3,9.2,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000808,1,2017-12-15 00:36:58,2017-12-15 01:23:38,N,1,,,,,3,4.50,29.5,0.5,0.5,6.15,0,,0.3,36.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000809,1,2017-12-15 00:02:18,2017-12-15 00:27:13,N,1,,,,,1,5.20,20,0.5,0.5,4.25,0,,0.3,25.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000810,1,2017-12-15 00:45:21,2017-12-15 00:53:02,N,1,,,,,1,1.40,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000811,1,2017-12-15 00:54:15,2017-12-15 01:08:34,N,1,,,,,1,2.90,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000812,1,2017-12-15 00:05:06,2017-12-15 00:10:53,N,1,,,,,1,1.60,7,0.5,0.5,3,0,,0.3,11.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000813,1,2017-12-15 00:13:19,2017-12-15 00:26:11,N,1,,,,,1,2.40,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000814,1,2017-12-15 00:56:51,2017-12-15 01:03:21,N,1,,,,,1,0.50,6,0.5,0.5,0,0,,0.3,7.3,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000815,1,2017-12-15 00:05:46,2017-12-15 00:43:24,N,1,,,,,1,3.20,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000816,1,2017-12-15 00:49:45,2017-12-15 01:01:15,N,1,,,,,1,1.80,9.5,0.5,0.5,2.15,0,,0.3,12.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000817,1,2017-12-15 00:06:47,2017-12-15 00:19:20,N,1,,,,,1,2.60,11,0.5,0.5,2.45,0,,0.3,14.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000818,1,2017-12-15 00:21:08,2017-12-15 00:39:13,N,1,,,,,1,4.20,16,0.5,0.5,2,0,,0.3,19.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000819,1,2017-12-15 00:41:09,2017-12-15 00:46:56,N,1,,,,,1,0.70,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000820,1,2017-12-15 00:49:48,2017-12-15 00:56:43,N,1,,,,,4,0.80,6,0.5,0.5,1.8,0,,0.3,9.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000821,1,2017-12-15 00:57:42,2017-12-15 01:16:36,N,1,,,,,1,3.00,14,0.5,0.5,2.5,0,,0.3,17.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000822,1,2017-12-15 00:04:26,2017-12-15 00:19:40,N,1,,,,,2,1.40,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000823,1,2017-12-15 00:26:21,2017-12-15 00:34:11,N,1,,,,,1,1.10,7,0.5,0.5,2,0,,0.3,10.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000824,1,2017-12-15 00:39:20,2017-12-15 00:56:50,N,1,,,,,1,5.00,17,0.5,0.5,3.65,0,,0.3,21.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000825,2,2017-12-15 00:12:02,2017-12-15 00:26:03,N,1,,,,,1,2.27,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000826,2,2017-12-15 00:32:43,2017-12-15 00:43:57,N,1,,,,,1,3.13,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000827,2,2017-12-15 00:45:55,2017-12-15 00:54:31,N,1,,,,,1,1.89,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000828,2,2017-12-15 00:04:28,2017-12-15 00:48:05,N,1,,,,,1,7.08,31,0.5,0.5,6.46,0,,0.3,38.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000829,1,2017-12-15 00:01:24,2017-12-15 00:38:04,N,1,,,,,2,6.60,27.5,0.5,0.5,5.75,0,,0.3,34.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000830,1,2017-12-15 00:17:14,2017-12-15 00:26:11,N,1,,,,,1,2.00,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000831,1,2017-12-15 00:30:59,2017-12-15 00:47:45,N,1,,,,,1,3.00,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000832,1,2017-12-15 00:51:48,2017-12-15 01:05:41,N,1,,,,,1,6.70,20.5,0.5,0.5,4.35,0,,0.3,26.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000833,2,2017-12-15 00:04:55,2017-12-15 00:17:37,N,1,,,,,1,2.77,12,0.5,0.5,0,0,,0.3,13.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000834,2,2017-12-15 00:27:10,2017-12-15 00:40:45,N,1,,,,,1,4.32,15,0.5,0.5,3.26,0,,0.3,19.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000835,2,2017-12-15 00:47:01,2017-12-15 01:00:18,N,1,,,,,1,1.12,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000836,2,2017-12-15 00:34:54,2017-12-15 00:51:54,N,1,,,,,1,1.76,11.5,0.5,0.5,2.56,0,,0.3,15.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000837,1,2017-12-15 00:41:18,2017-12-15 00:54:16,N,1,,,,,1,2.70,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000838,1,2017-12-15 00:16:37,2017-12-15 00:25:14,N,1,,,,,1,1.00,7.5,0.5,0.5,2.2,0,,0.3,11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000839,1,2017-12-15 00:29:29,2017-12-15 00:35:48,N,1,,,,,1,0.80,6,0.5,0.5,1.45,0,,0.3,8.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000840,1,2017-12-15 00:44:11,2017-12-15 00:54:00,N,1,,,,,1,1.30,8,0.5,0.5,1.85,0,,0.3,11.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000841,2,2017-12-15 00:03:15,2017-12-15 00:12:39,N,1,,,,,1,1.44,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000842,2,2017-12-15 00:13:34,2017-12-15 00:25:05,N,1,,,,,1,0.70,8,0.5,0.5,0,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000843,2,2017-12-15 00:29:16,2017-12-15 00:32:52,N,1,,,,,1,0.63,4.5,0.5,0.5,1.74,0,,0.3,7.54,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000844,2,2017-12-15 00:44:44,2017-12-15 00:55:18,N,1,,,,,1,2.22,10,0.5,0.5,2.26,0,,0.3,13.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000845,2,2017-12-15 00:56:56,2017-12-15 01:08:12,N,1,,,,,1,2.25,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000846,1,2017-12-15 00:05:27,2017-12-15 00:40:39,N,1,,,,,1,4.10,23,0.5,0.5,3,0,,0.3,27.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000847,1,2017-12-15 00:50:51,2017-12-15 00:55:54,N,1,,,,,1,1.00,5.5,0.5,0.5,1.35,0,,0.3,8.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000848,1,2017-12-15 00:59:52,2017-12-15 01:11:28,N,1,,,,,2,1.30,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000849,2,2017-12-15 00:43:01,2017-12-15 01:04:17,N,1,,,,,1,4.68,18,0.5,0.5,2.5,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000850,1,2017-12-15 00:52:23,2017-12-15 01:16:27,N,1,,,,,1,4.00,17.5,0.5,0.5,1.2,0,,0.3,20,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000851,1,2017-12-15 00:50:17,2017-12-15 01:18:00,N,1,,,,,1,7.70,26,0.5,0.5,5.45,0,,0.3,32.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000852,2,2017-12-15 00:29:48,2017-12-15 00:47:32,N,1,,,,,5,4.13,15.5,0.5,0.5,4.2,0,,0.3,21,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000853,1,2017-12-15 00:20:05,2017-12-15 00:28:06,N,1,,,,,1,0.60,6.5,0.5,0.5,2,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000854,1,2017-12-15 00:30:22,2017-12-15 01:10:08,N,1,,,,,1,5.80,27,0.5,0.5,7.05,0,,0.3,35.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000855,2,2017-12-15 00:30:08,2017-12-15 00:36:48,N,1,,,,,1,0.72,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000856,2,2017-12-15 00:38:28,2017-12-15 00:57:44,N,1,,,,,1,1.65,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000857,1,2017-12-15 00:11:26,2017-12-15 00:33:14,N,1,,,,,1,4.30,18.5,0.5,0.5,2,0,,0.3,21.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000858,1,2017-12-15 00:53:08,2017-12-15 01:24:53,N,1,,,,,2,8.90,31,0.5,0.5,5,5.76,,0.3,43.06,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000859,1,2017-12-15 00:53:46,2017-12-15 01:07:14,N,1,,,,,2,1.90,10.5,0.5,0.5,1.2,0,,0.3,13,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000860,2,2017-12-15 00:31:50,2017-12-15 00:37:17,N,1,,,,,1,1.77,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000861,1,2017-12-15 00:08:40,2017-12-15 00:58:36,N,1,,,,,1,19.60,59.5,0.5,0.5,13,0,,0.3,73.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000862,2,2017-12-15 00:02:27,2017-12-15 00:06:04,N,1,,,,,5,0.60,4.5,0.5,0.5,1.16,0,,0.3,6.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000863,2,2017-12-15 00:17:42,2017-12-15 01:05:29,N,1,,,,,5,6.86,30,0.5,0.5,0,0,,0.3,31.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000864,2,2017-12-15 00:22:02,2017-12-15 00:40:51,N,1,,,,,1,3.48,15.5,0.5,0.5,0,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000865,2,2017-12-15 00:42:18,2017-12-15 00:49:35,N,1,,,,,1,1.42,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000866,2,2017-12-15 00:02:19,2017-12-15 00:24:09,N,1,,,,,1,10.38,31,0.5,0.5,6.46,0,,0.3,38.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000867,2,2017-12-15 00:21:42,2017-12-15 01:25:06,N,1,,,,,1,11.80,45.5,0.5,0.5,0,0,,0.3,46.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000868,2,2017-12-15 00:03:13,2017-12-15 00:11:15,N,1,,,,,1,1.05,7,0.5,0.5,1.66,0,,0.3,9.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000869,2,2017-12-15 00:13:04,2017-12-15 00:43:07,N,1,,,,,1,5.54,22,0.5,0.5,0,0,,0.3,23.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000870,2,2017-12-15 00:03:01,2017-12-15 00:17:43,N,1,,,,,1,1.14,10,0.5,0.5,0,0,,0.3,11.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000871,2,2017-12-15 00:19:26,2017-12-15 00:36:55,N,1,,,,,1,3.51,15,0.5,0.5,20,0,,0.3,36.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000872,1,2017-12-15 00:42:05,2017-12-15 00:46:23,Y,1,,,,,2,0.70,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000874,1,2017-12-15 00:32:08,2017-12-15 00:38:53,N,1,,,,,1,0.60,6,0.5,0.5,0,0,,0.3,7.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000875,1,2017-12-15 00:52:04,2017-12-15 01:04:53,N,1,,,,,1,2.00,10.5,0.5,0.5,2.35,0,,0.3,14.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000876,2,2017-12-15 00:39:40,2017-12-15 01:22:10,N,2,,,,,6,21.28,52,0,0.5,6,5.76,,0.3,64.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000877,1,2017-12-15 00:04:14,2017-12-15 00:13:52,N,1,,,,,2,2.10,9.5,0.5,0.5,1.2,0,,0.3,12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000878,1,2017-12-15 00:44:57,2017-12-15 00:57:59,N,1,,,,,2,1.40,9.5,0.5,0.5,2,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000879,2,2017-12-15 00:12:01,2017-12-15 00:18:24,N,1,,,,,1,1.32,7,0.5,0.5,0.1,0,,0.3,8.4,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000880,2,2017-12-15 00:22:58,2017-12-15 00:40:06,N,1,,,,,1,4.69,16.5,0.5,0.5,3.56,0,,0.3,21.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000881,2,2017-12-15 00:47:43,2017-12-15 00:54:11,N,1,,,,,1,1.30,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000882,2,2017-12-15 00:16:46,2017-12-15 00:24:04,N,1,,,,,1,1.86,8,0.5,0.5,1.86,0,,0.3,11.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000883,2,2017-12-15 00:29:13,2017-12-15 00:43:55,N,1,,,,,1,2.95,12.5,0.5,0.5,2.2,0,,0.3,16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000884,2,2017-12-15 00:48:04,2017-12-15 00:49:49,N,1,,,,,1,0.34,3.5,0.5,0.5,0,0,,0.3,4.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000885,2,2017-12-15 00:51:53,2017-12-15 00:55:13,N,1,,,,,1,0.86,5,0.5,0.5,1.89,0,,0.3,8.19,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000886,2,2017-12-15 00:21:27,2017-12-15 01:14:34,N,1,,,,,1,10.62,42.5,0.5,0.5,10.95,0,,0.3,54.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000887,1,2017-12-15 00:04:08,2017-12-15 00:36:58,N,1,,,,,2,5.10,23,0.5,0.5,0,0,,0.3,24.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000888,1,2017-12-15 00:40:51,2017-12-15 00:57:22,N,1,,,,,2,3.10,14,0.5,0.5,3.8,0,,0.3,19.1,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000889,1,2017-12-15 00:08:02,2017-12-15 00:29:01,N,1,,,,,1,2.50,14.5,0.5,0.5,1,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000890,1,2017-12-15 00:48:54,2017-12-15 01:33:00,N,1,,,,,1,15.60,48.5,0.5,0.5,5.2,0,,0.3,55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000891,2,2017-12-14 23:59:19,2017-12-15 00:23:03,N,1,,,,,1,3.84,17.5,0.5,0.5,1,0,,0.3,19.8,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000892,2,2017-12-15 00:42:48,2017-12-15 01:03:15,N,1,,,,,1,2.45,14.5,0.5,0.5,1,0,,0.3,16.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000893,2,2017-12-15 00:16:33,2017-12-15 00:26:08,N,1,,,,,1,1.67,8.5,0.5,0.5,1.96,0,,0.3,11.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000894,2,2017-12-15 00:38:41,2017-12-15 00:56:22,N,1,,,,,1,2.95,13.5,0.5,0.5,0,0,,0.3,14.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000895,2,2017-12-15 00:57:11,2017-12-15 01:14:48,N,1,,,,,1,0.82,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000896,2,2017-12-15 00:02:45,2017-12-15 00:36:35,N,2,,,,,1,18.85,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000897,2,2017-12-15 00:43:23,2017-12-15 01:00:33,N,1,,,,,1,2.82,13.5,0.5,0.5,2.96,0,,0.3,17.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000898,2,2017-12-15 00:08:07,2017-12-15 00:41:12,N,1,,,,,1,5.36,24,0.5,0.5,6.32,0,,0.3,31.62,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000899,2,2017-12-15 00:48:01,2017-12-15 00:54:37,N,1,,,,,1,0.87,6.5,0.5,0.5,1.56,0,,0.3,9.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000900,2,2017-12-15 01:00:05,2017-12-15 01:04:23,N,1,,,,,1,1.04,5.5,0.5,0.5,2.04,0,,0.3,8.84,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000901,1,2017-12-15 00:43:08,2017-12-15 00:50:53,N,1,,,,,1,2.10,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000902,1,2017-12-15 00:51:34,2017-12-15 00:54:56,N,1,,,,,1,1.10,5.5,0.5,0.5,0,0,,0.3,6.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000903,2,2017-12-15 00:53:48,2017-12-15 01:04:17,N,1,,,,,1,1.66,9,0.5,0.5,0,0,,0.3,10.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000904,1,2017-12-15 00:22:51,2017-12-15 00:50:12,N,2,,,,,1,18.40,52,0,0.5,10.55,0,,0.3,63.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000905,1,2017-12-15 00:02:29,2017-12-15 00:27:44,N,1,,,,,1,4.20,18,0.5,0.5,0,0,,0.3,19.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000906,1,2017-12-15 00:31:33,2017-12-15 00:50:28,N,1,,,,,1,3.40,15,0.5,0.5,4.05,0,,0.3,20.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000907,1,2017-12-15 00:53:36,2017-12-15 00:57:11,N,1,,,,,1,0.90,5,0.5,0.5,1.25,0,,0.3,7.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000908,2,2017-12-15 00:01:47,2017-12-15 00:35:22,N,1,,,,,1,6.78,26.5,0.5,0.5,2,0,,0.3,29.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000909,2,2017-12-15 00:55:50,2017-12-15 01:21:26,N,1,,,,,1,4.70,19,0.5,0.5,0,0,,0.3,20.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000910,1,2017-12-15 00:15:27,2017-12-15 00:20:38,N,1,,,,,1,1.40,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000911,1,2017-12-15 00:34:52,2017-12-15 01:01:21,N,1,,,,,1,5.10,21,0.5,0.5,3.35,0,,0.3,25.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000912,1,2017-12-15 00:15:57,2017-12-15 00:22:46,N,1,,,,,1,1.80,7.5,0.5,0.5,1.75,0,,0.3,10.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000913,1,2017-12-15 00:31:27,2017-12-15 01:11:20,N,1,,,,,1,5.30,26.5,0.5,0.5,1,0,,0.3,28.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000914,1,2017-12-15 00:14:15,2017-12-15 00:29:52,N,1,,,,,1,3.00,13,0.5,0.5,1,0,,0.3,15.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000915,1,2017-12-15 00:48:36,2017-12-15 01:24:10,N,1,,,,,1,4.90,26,0.5,0.5,8.15,0,,0.3,35.45,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000916,1,2017-12-15 00:22:04,2017-12-15 00:32:41,Y,1,,,,,2,1.90,9,0.5,0.5,2.05,0,,0.3,12.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000917,1,2017-12-15 00:33:58,2017-12-15 00:47:41,N,1,,,,,1,3.30,12.5,0.5,0.5,0,0,,0.3,13.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000918,1,2017-12-15 00:54:47,2017-12-15 00:55:21,N,5,,,,,1,0.00,14,0,0,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000919,1,2017-12-15 00:35:43,2017-12-15 01:08:42,N,1,,,,,1,11.70,37.5,0.5,0.5,9.7,0,,0.3,48.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000920,2,2017-12-15 00:13:56,2017-12-15 00:41:08,N,1,,,,,1,3.73,19,0.5,0.5,3,0,,0.3,23.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000921,2,2017-12-15 00:50:32,2017-12-15 01:06:42,N,1,,,,,1,2.81,12,0.5,0.5,2.66,0,,0.3,15.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000922,1,2017-12-15 00:39:58,2017-12-15 01:01:50,N,1,,,,,1,4.50,18,0.5,0.5,3.85,0,,0.3,23.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000923,1,2017-12-15 00:08:23,2017-12-15 01:07:29,N,1,,,,,1,7.70,38,0.5,0.5,0,0,,0.3,39.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000924,1,2017-12-15 00:00:04,2017-12-15 00:14:22,N,1,,,,,2,2.30,11.5,0.5,0.5,2.55,0,,0.3,15.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000925,1,2017-12-15 00:15:32,2017-12-15 00:39:24,N,1,,,,,2,3.10,16.5,0.5,0.5,2,0,,0.3,19.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000926,1,2017-12-15 00:58:47,2017-12-15 01:01:23,N,1,,,,,1,0.60,4,0.5,0.5,1.05,0,,0.3,6.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000927,2,2017-12-15 00:19:03,2017-12-15 00:26:33,N,1,,,,,1,1.07,7,0.5,0.5,1,0,,0.3,9.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000928,2,2017-12-15 00:27:47,2017-12-15 01:05:07,N,1,,,,,1,5.63,26,0.5,0.5,5.46,0,,0.3,32.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000929,1,2017-12-15 00:02:07,2017-12-15 00:28:16,N,1,,,,,0,6.70,23.5,0.5,0.5,0,0,,0.3,24.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000930,1,2017-12-15 00:33:18,2017-12-15 01:03:18,N,1,,,,,1,4.80,21.5,0.5,0.5,5.7,5.76,,0.3,34.26,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000931,1,2017-12-15 00:38:52,2017-12-15 00:38:57,N,2,,,,,1,4.60,52,0,0.5,0,0,,0.3,52.8,3,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000932,1,2017-12-15 00:43:53,2017-12-15 01:18:44,N,1,,,,,1,6.40,26.5,0,0.5,0,0,,0.3,27.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000933,1,2017-12-15 00:04:13,2017-12-15 00:51:06,N,1,,,,,2,9.20,36,0.5,0.5,7.45,0,,0.3,44.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000934,2,2017-12-15 00:09:49,2017-12-15 00:23:32,N,1,,,,,1,2.51,11.5,0.5,0.5,0,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000935,2,2017-12-15 00:28:33,2017-12-15 01:04:59,N,1,,,,,1,6.32,25.5,0.5,0.5,5.36,0,,0.3,32.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000936,2,2017-12-15 00:06:03,2017-12-15 00:22:31,N,1,,,,,5,3.26,14,0.5,0.5,3.06,0,,0.3,18.36,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000937,2,2017-12-15 00:37:08,2017-12-15 01:03:40,N,1,,,,,5,2.83,17.5,0.5,0.5,0,0,,0.3,18.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000938,2,2017-12-15 00:57:16,2017-12-15 01:16:42,N,1,,,,,1,2.93,14,0.5,0.5,1,0,,0.3,16.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000939,2,2017-12-15 00:10:53,2017-12-15 00:22:45,N,1,,,,,2,1.44,9,0.5,0.5,2.5,0,,0.3,12.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000940,2,2017-12-15 00:50:00,2017-12-15 01:26:28,N,1,,,,,2,10.03,35.5,0.5,0.5,0,0,,0.3,36.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000941,2,2017-12-15 00:07:21,2017-12-15 00:20:07,N,1,,,,,1,1.85,10.5,0.5,0.5,2.36,0,,0.3,14.16,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000942,2,2017-12-15 00:29:10,2017-12-15 00:36:06,N,1,,,,,1,1.68,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000943,2,2017-12-15 00:47:04,2017-12-15 01:20:23,N,1,,,,,1,15.51,45,0.5,0.5,9.26,0,,0.3,55.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000944,1,2017-12-15 00:19:02,2017-12-15 00:29:05,N,1,,,,,1,1.90,9.5,0.5,0.5,1,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000945,1,2017-12-15 00:31:52,2017-12-15 00:44:24,N,1,,,,,1,2.20,10.5,0.5,0.5,0,0,,0.3,11.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000946,1,2017-12-15 00:46:42,2017-12-15 01:05:12,N,1,,,,,1,6.20,21,0.5,0.5,3.35,0,,0.3,25.65,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000947,2,2017-12-15 00:11:42,2017-12-15 00:43:20,N,1,,,,,1,6.04,25.5,0.5,0.5,5.36,0,,0.3,34.11,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000948,1,2017-12-15 00:21:34,2017-12-15 00:41:16,N,1,,,,,1,2.30,14,0.5,0.5,0,0,,0.3,15.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000949,1,2017-12-15 00:42:11,2017-12-15 00:55:13,N,1,,,,,2,3.50,13,0.5,0.5,2.85,0,,0.3,17.15,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000950,2,2017-12-15 00:00:16,2017-12-15 00:17:07,N,1,,,,,2,4.05,15.5,0.5,0.5,0,0,,0.3,16.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000951,2,2017-12-15 00:47:25,2017-12-15 01:08:08,N,1,,,,,2,4.73,18,0.5,0.5,0,0,,0.3,19.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000952,2,2017-12-14 23:58:52,2017-12-15 23:48:39,N,1,,,,,2,0.27,3.5,0.5,0.5,0,0,,0.3,4.8,1,,,,yellow,0.09,1,1.2,32,22,5.82,,,,,,,,,,,,,,,,,,,,
1460000953,2,2017-12-15 00:02:43,2017-12-15 00:31:33,N,1,,,,,1,3.81,19.5,0.5,0.5,3,0,,0.3,23.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000954,2,2017-12-15 00:08:07,2017-12-15 00:22:27,N,1,,,,,1,1.58,10.5,0.5,0.5,1.5,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000955,2,2017-12-15 00:23:59,2017-12-15 00:43:51,N,1,,,,,1,3.09,14.5,0.5,0.5,4.74,0,,0.3,20.54,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000956,2,2017-12-15 00:53:01,2017-12-15 00:53:05,N,5,,,,,1,0.00,5,0,0.5,0,0,,0.3,5.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000957,2,2017-12-15 00:55:09,2017-12-15 01:07:32,N,1,,,,,1,2.52,11.5,0.5,0.5,0,0,,0.3,12.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000958,2,2017-12-15 00:22:57,2017-12-15 00:36:58,N,1,,,,,1,2.75,12,0.5,0.5,0.66,0,,0.3,13.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000959,2,2017-12-15 00:03:25,2017-12-15 00:13:02,N,1,,,,,1,0.99,7.5,0.5,0.5,0,0,,0.3,8.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000960,2,2017-12-15 00:17:06,2017-12-15 00:31:31,N,1,,,,,1,1.79,10.5,0.5,0.5,0,0,,0.3,11.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000961,2,2017-12-15 00:37:42,2017-12-15 01:37:47,N,5,,,,,1,16.84,80,0,0.5,0,0,,0.3,80.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000962,1,2017-12-15 00:24:46,2017-12-15 00:34:31,N,1,,,,,1,1.60,8.5,0.5,0.5,1.95,0,,0.3,11.75,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000963,1,2017-12-15 00:47:28,2017-12-15 01:07:04,N,1,,,,,1,4.20,16.5,0.5,0.5,5,0,,0.3,22.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000964,2,2017-12-15 00:01:11,2017-12-15 00:09:14,N,1,,,,,1,1.57,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000965,2,2017-12-15 00:30:12,2017-12-15 01:01:05,N,1,,,,,1,4.68,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000966,1,2017-12-15 00:24:00,2017-12-15 00:38:13,N,1,,,,,1,3.20,12.5,0.5,0.5,2.75,0,,0.3,16.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000967,2,2017-12-15 00:49:53,2017-12-15 01:17:25,N,1,,,,,3,6.37,22.5,0.5,0.5,4.76,0,,0.3,28.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000968,2,2017-12-15 00:05:30,2017-12-15 00:37:30,N,2,,,,,4,18.09,52,0,0.5,11.71,5.76,,0.3,70.27,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000969,2,2017-12-15 00:47:11,2017-12-15 01:02:37,N,1,,,,,4,3.35,13,0.5,0.5,0,0,,0.3,14.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000970,2,2017-12-15 00:32:31,2017-12-15 01:15:13,N,1,,,,,6,10.56,38.5,0.5,0.5,4,5.76,,0.3,49.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000971,1,2017-12-15 00:55:25,2017-12-15 01:00:03,N,1,,,,,1,0.50,5,0.5,0.5,0,0,,0.3,6.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000972,1,2017-12-15 00:00:54,2017-12-15 00:04:07,N,1,,,,,1,0.90,5,0.5,0.5,1.55,0,,0.3,7.85,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000973,1,2017-12-15 00:11:51,2017-12-15 00:24:10,N,1,,,,,1,1.70,10,0.5,0.5,2,0,,0.3,13.3,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000974,1,2017-12-15 00:33:26,2017-12-15 01:19:20,N,1,,,,,1,8.50,34.5,0.5,0.5,7.15,0,,0.3,42.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000975,1,2017-12-15 00:11:27,2017-12-15 00:54:09,N,1,,,,,2,7.90,31.5,0.5,0.5,6.55,0,,0.3,39.35,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000976,2,2017-12-15 00:10:38,2017-12-15 00:50:52,N,1,,,,,1,6.88,29.5,0.5,0.5,7.7,0,,0.3,38.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000977,2,2017-12-15 00:04:00,2017-12-15 00:21:02,N,1,,,,,6,1.51,12,0.5,0.5,3.99,0,,0.3,17.29,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000978,2,2017-12-15 00:21:43,2017-12-15 00:33:56,N,1,,,,,6,0.71,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000979,2,2017-12-15 00:36:45,2017-12-15 00:48:13,N,1,,,,,6,1.00,8.5,0.5,0.5,2.94,0,,0.3,12.74,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000980,2,2017-12-15 00:50:35,2017-12-15 01:14:51,N,1,,,,,6,4.34,18,0.5,0.5,4.82,0,,0.3,24.12,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000981,2,2017-12-15 00:35:10,2017-12-15 00:40:56,N,1,,,,,1,1.21,6,0.5,0.5,1.46,0,,0.3,8.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000982,2,2017-12-15 00:53:17,2017-12-15 01:09:14,N,1,,,,,1,3.38,13.5,0.5,0.5,3.7,0,,0.3,18.5,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000983,2,2017-12-15 00:48:43,2017-12-15 00:58:44,N,1,,,,,1,1.34,8.5,0.5,0.5,1,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000984,2,2017-12-15 00:37:51,2017-12-15 00:38:05,N,5,,,,,1,0.00,60,0,0,0,0,,0.3,60.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000985,2,2017-12-15 00:45:54,2017-12-15 00:52:31,N,1,,,,,1,1.92,7.5,0.5,0.5,2,0,,0.3,10.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000986,2,2017-12-15 00:07:43,2017-12-15 00:28:15,N,1,,,,,2,10.02,29.5,0.5,0.5,0,0,,0.3,30.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000987,2,2017-12-15 00:32:40,2017-12-15 00:41:01,N,1,,,,,2,1.29,7.5,0.5,0.5,1,0,,0.3,9.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000988,2,2017-12-15 00:41:52,2017-12-15 00:48:28,N,1,,,,,2,1.00,6.5,0.5,0.5,0,0,,0.3,7.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000989,2,2017-12-15 00:52:43,2017-12-15 01:37:15,N,5,,,,,2,4.32,70,0,0.5,0,0,,0.3,70.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000990,1,2017-12-15 00:03:36,2017-12-15 00:25:14,N,1,,,,,2,3.90,17.5,0.5,0.5,3.75,0,,0.3,22.55,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000991,1,2017-12-15 00:39:01,2017-12-15 01:05:04,N,1,,,,,2,4.70,19.5,0.5,0.5,3,0,,0.3,23.8,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000992,2,2017-12-15 00:04:21,2017-12-15 00:09:29,N,1,,,,,1,0.69,5,0.5,0.5,1.89,0,,0.3,8.19,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000993,2,2017-12-15 00:19:51,2017-12-15 00:24:23,N,1,,,,,1,0.79,5,0.5,0.5,1.26,0,,0.3,7.56,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000994,2,2017-12-15 00:25:57,2017-12-15 00:55:44,N,1,,,,,1,3.62,19.5,0.5,0.5,4.16,0,,0.3,24.96,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000995,1,2017-12-15 00:00:00,2017-12-15 00:15:00,N,1,,,,,1,2.20,12,0.5,0.5,2.65,0,,0.3,15.95,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000996,1,2017-12-15 00:16:51,2017-12-15 00:49:40,N,1,,,,,2,8.50,29,0.5,0.5,7.2,5.76,,0.3,43.26,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000997,2,2017-12-15 00:42:37,2017-12-15 01:24:17,N,1,,,,,1,8.46,31.5,0.5,0.5,8.2,0,,0.3,41,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000998,2,2017-12-15 00:06:02,2017-12-15 00:18:45,N,1,,,,,1,2.28,11,0.5,0.5,2.46,0,,0.3,14.76,1,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460000999,2,2017-12-15 00:21:04,2017-12-15 00:31:14,N,1,,,,,1,1.01,8,0.5,0.5,0,0,,0.3,9.3,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
1460001000,2,2017-12-15 00:33:21,2017-12-15 00:42:19,N,1,,,,,1,1.87,8.5,0.5,0.5,0,0,,0.3,9.8,2,,,,yellow,0.11,1,1.2,28,20,3.80,,,,,,,,,,,,,,,,,,,,
================================================
FILE: examples/data/plasticc_test_set_1k.csv
================================================
object_id,mjd,passband,flux,flux_err,detected
13,59798.3205,2,-1.299735,1.357315,0
13,59798.3281,1,-2.095392,1.148654,0
13,59798.3357,3,-0.923794,1.763655,0
13,59798.3466,4,-4.009815,2.602911,0
13,59798.3576,5,-3.403503,5.367328,0
13,59801.3553,2,-1.778855,2.448943,0
13,59801.3629,1,2.491993,3.540421,0
13,59801.3705,3,1.644129,2.284999,0
13,59801.3815,4,-0.158192,2.515900,0
13,59801.3924,5,-6.457387,5.381231,0
13,59818.2740,0,1.962846,1.795587,0
13,59819.2541,0,-1.697929,2.433431,0
13,59820.2522,0,-1.698675,1.898612,0
13,59821.2478,0,-0.776626,2.435191,0
13,59822.2433,0,-3.826187,2.853957,0
13,59823.2659,0,-0.837001,2.690573,0
13,59826.3105,2,0.529480,0.920972,0
13,59826.3181,1,-0.702092,0.923219,0
13,59826.3258,3,-0.797231,1.508073,0
13,59826.3367,4,-3.898993,2.156529,0
13,59826.3477,5,-2.883462,5.535779,0
13,59842.2456,2,-0.838558,0.849664,0
13,59842.2532,1,-0.038403,0.813772,0
13,59842.2608,3,2.863780,1.322607,0
13,59842.2718,4,0.088068,2.166394,0
13,59842.2827,5,-1.672899,6.068114,0
13,59851.1792,0,-0.737309,2.574759,0
13,59854.1485,2,0.040905,1.224270,0
13,59854.1563,1,-0.265884,1.336087,0
13,59854.1640,3,-0.369843,1.901700,0
13,59854.1750,4,2.415236,2.771065,0
13,59854.1860,5,-4.745292,5.788470,0
13,59857.1408,2,4.814427,2.132096,0
13,59857.1485,1,-2.618018,2.915296,0
13,59857.1563,3,-2.852325,2.092946,0
13,59857.1673,4,-1.584260,2.676820,0
13,59857.1782,5,-12.249563,6.628258,0
13,59867.1112,2,-1.164210,1.083173,0
13,59867.1189,1,-1.176300,0.974351,0
13,59867.1267,3,0.451480,1.538879,0
13,59867.1377,4,0.359918,2.357359,0
13,59867.1487,5,-2.699421,5.830664,0
13,59870.1049,2,1.381781,0.971909,0
13,59870.1126,1,0.387448,0.822428,0
13,59870.1204,3,0.817313,1.373219,0
13,59870.1314,4,0.151035,2.283455,0
13,59870.1424,5,2.986818,5.280643,0
13,59873.0971,2,-1.240265,1.040900,0
13,59873.1049,1,0.931574,1.042814,0
13,59873.1126,3,2.134935,1.442823,0
13,59873.1236,4,-0.530458,2.068875,0
13,59873.1346,5,-0.314489,4.587787,0
13,59874.1461,0,-2.490195,1.611522,0
13,59875.0995,0,-1.276708,2.019888,0
13,59876.0980,0,3.425512,2.443817,0
13,59877.0976,0,2.518677,1.776579,0
13,59878.0964,0,1.872855,2.020787,0
13,59879.0895,0,0.522355,1.864210,0
13,59880.1017,0,-0.761489,1.371363,0
13,59884.1760,2,0.408623,0.768234,0
13,59884.1836,1,0.610256,0.887502,0
13,59884.1913,3,-0.134252,1.138571,0
13,59884.2022,4,-0.009705,1.609182,0
13,59884.2132,5,6.633543,4.214399,0
13,59887.2856,2,1.784338,1.826830,0
13,59887.2933,1,1.591402,2.206727,0
13,59887.3009,3,0.109085,2.350564,0
13,59887.3118,4,-2.874674,3.145338,0
13,59887.3228,5,-2.586321,7.947824,0
13,59896.1307,2,1.084506,0.822793,0
13,59896.1384,1,-0.561922,0.703150,0
13,59896.1460,3,0.284255,1.393400,0
13,59896.1569,4,0.466466,2.192684,0
13,59896.1679,5,0.433727,5.948182,0
13,59899.1519,2,0.102152,1.365073,0
13,59899.1595,1,1.624047,1.388812,0
13,59899.1672,3,1.820581,2.378815,0
13,59899.1781,4,-3.816161,3.709328,0
13,59899.1891,5,-8.805643,7.905914,0
13,59902.1384,2,0.196845,1.186146,0
13,59902.1460,1,0.993784,1.079431,0
13,59902.1537,3,1.891725,1.943785,0
13,59902.1646,4,-1.455820,2.762865,0
13,59902.1755,5,-0.622898,7.024046,0
13,59904.1053,0,-4.067552,2.281505,0
13,59905.0555,0,0.633017,2.504281,0
13,59906.0562,0,1.127558,2.030226,0
13,59907.0567,0,0.456890,1.749502,0
13,59908.0681,0,-1.925056,2.683702,0
13,59909.0582,0,-0.678187,1.834134,0
13,59910.0503,0,1.194135,1.869489,0
13,59914.0526,2,-1.644144,1.438881,0
13,59914.0602,1,-1.866351,1.956770,0
13,59914.0678,3,-0.710913,1.686324,0
13,59914.0788,4,5.825469,2.269697,0
13,59914.0897,5,3.513655,5.356808,0
13,59924.1060,2,-0.028742,1.131780,0
13,59924.1136,1,-1.108478,1.574127,0
13,59924.1212,3,-1.323133,1.728779,0
13,59924.1322,4,-5.513097,2.527147,0
13,59924.1431,5,4.266651,5.633382,0
13,59927.1074,2,-0.400232,1.190879,0
13,59927.1151,1,1.286833,1.135341,0
13,59927.1227,3,2.155155,1.593354,0
13,59927.1336,4,2.415218,2.299777,0
13,59927.1446,5,3.778454,5.891465,0
13,59930.1236,2,1.373614,0.942084,0
13,59930.1312,1,-1.140287,0.828822,0
13,59930.1388,3,0.832613,1.283629,0
13,59930.1498,4,2.345280,2.033006,0
13,59930.1607,5,-4.211092,5.330056,0
13,59933.1249,2,0.395000,0.691634,0
13,59933.1325,1,-0.423847,0.693980,0
13,59933.1401,3,-1.108444,1.268825,0
13,59933.1511,4,-0.328867,1.913908,0
13,59933.1620,5,7.551563,5.429599,0
13,59934.0638,0,-1.113758,1.141001,0
13,59935.0646,0,-0.818122,1.122888,0
13,59936.0642,0,2.141036,1.531045,0
13,59937.0650,0,-1.277371,1.639949,0
13,59938.0647,0,4.045178,2.015162,0
13,59939.0650,0,-0.050647,2.194245,0
13,60165.3032,2,-0.244917,1.195023,0
13,60165.3109,1,-0.051452,1.347307,0
13,60165.3186,3,-1.446614,1.518360,0
13,60165.3295,4,-1.239533,2.022870,0
13,60165.3405,5,-2.233357,4.782032,0
13,60168.2892,2,-0.272922,0.834000,0
13,60168.2970,1,-0.551885,0.787909,0
13,60168.3047,3,0.215221,1.300493,0
13,60168.3157,4,-0.354714,2.065893,0
13,60168.3267,5,9.800998,4.821743,0
13,60176.2820,0,-2.711282,2.363240,0
13,60177.2726,0,2.907485,2.777568,0
13,60181.4088,2,-0.778592,0.924176,0
13,60181.4164,1,0.776635,1.262891,0
13,60181.4232,3,5.557160,3.239328,0
13,60183.2660,2,0.008992,1.915637,0
13,60183.2736,1,3.495092,2.936934,0
13,60183.2812,3,-4.313775,2.548015,0
13,60183.2922,4,4.681122,2.996080,0
13,60183.3031,5,-0.644280,6.178194,0
13,60195.2812,2,-0.992008,0.933265,0
13,60195.2888,1,-0.199896,1.020470,0
13,60195.2964,3,1.175473,1.626211,0
13,60195.3073,4,-0.928975,2.339218,0
13,60195.3183,5,3.537798,6.276374,0
13,60198.2690,0,0.879443,2.172960,0
13,60199.2186,0,1.393119,2.161035,0
13,60200.2139,0,0.226138,2.589170,0
13,60201.2072,0,0.024500,2.490959,0
13,60202.2089,0,-1.152337,2.483400,0
13,60209.1811,2,-0.534034,1.565458,0
13,60209.1888,1,0.822128,1.410793,0
13,60209.1965,3,-0.684217,1.831758,0
13,60209.2075,4,-3.066642,2.909104,0
13,60209.2184,5,0.602669,6.696602,0
13,60212.1675,2,1.117115,1.936030,0
13,60212.1753,1,1.782217,2.752773,0
13,60212.1830,3,0.321750,2.295181,0
13,60212.1941,4,-1.936185,2.590936,0
13,60212.2050,5,11.339810,5.666715,0
13,60223.2416,2,0.123572,1.182341,0
13,60223.2493,1,0.568967,1.241492,0
13,60223.2569,3,0.040453,1.648758,0
13,60223.2678,4,0.459117,2.472937,0
13,60223.2788,5,-7.159738,5.794432,0
13,60226.3337,2,0.034159,0.944593,0
13,60226.3413,1,1.465278,0.946326,0
13,60226.3489,3,0.247305,1.793391,0
13,60226.3599,4,4.326116,3.071592,0
13,60226.3708,5,1.558216,8.109918,0
13,60238.3197,2,-0.738155,1.053325,0
13,60238.3273,1,-0.990783,1.152318,0
13,60238.3349,3,1.786266,2.171606,0
13,60238.3459,4,3.042687,3.829919,0
13,60238.3568,5,8.879921,11.257108,0
13,60241.0870,2,-0.147198,1.999162,0
13,60241.0948,1,-1.383542,3.066069,0
13,60241.1025,3,-0.347256,2.444988,0
13,60241.1136,4,2.242954,2.917921,0
13,60241.1245,5,4.122897,5.941785,0
13,60250.1708,2,2.348668,2.103043,0
13,60250.1957,1,1.762817,2.811031,0
13,60250.2034,3,-4.162498,2.927869,0
13,60250.2143,4,3.563345,3.983845,0
13,60250.2253,5,5.870506,7.937796,0
13,60261.1296,0,2.993462,2.083503,0
13,60262.0550,0,1.952810,2.311998,0
13,60263.0556,0,1.620923,1.977922,0
13,60264.0559,0,3.218953,2.187246,0
13,60265.0780,0,-4.046216,2.407065,0
13,60268.0449,2,1.923153,1.404630,0
13,60268.0525,1,-0.380431,1.849085,0
13,60268.0601,3,1.653331,2.130050,0
13,60268.0711,4,-2.881836,2.734059,0
13,60268.0820,5,7.530805,6.747447,0
13,60278.0993,2,-1.574151,1.643888,0
13,60278.1069,1,-4.148108,2.273913,0
13,60278.1145,3,-1.136458,2.169661,0
13,60278.1255,4,-4.712543,3.059077,0
13,60278.1364,5,-4.843711,7.638536,0
13,60281.1023,2,-0.579702,0.798074,0
13,60281.1099,1,0.559643,0.795803,0
13,60281.1175,3,0.985490,1.369862,0
13,60281.1285,4,-0.125175,1.990803,0
13,60281.1394,5,-6.211294,5.176151,0
13,60284.1027,2,0.920098,0.837512,0
13,60284.1104,1,-0.531105,0.761165,0
13,60284.1180,3,-1.038009,1.160178,0
13,60284.1289,4,-0.543047,1.702779,0
13,60284.1399,5,-5.946274,4.435276,0
13,60287.1047,2,-0.388991,1.250426,0
13,60287.1123,1,0.622866,1.083712,0
13,60287.1200,3,-1.365885,1.796073,0
13,60287.1309,4,-1.566869,2.963949,0
13,60287.1418,5,-12.680235,7.013420,0
13,60290.0761,0,-0.996401,1.912932,0
13,60291.0689,0,-1.639395,1.661116,0
13,60292.0699,0,-0.087568,1.855572,0
13,60293.0699,0,3.384405,1.578842,0
13,60294.0708,0,0.056788,1.397227,0
13,60532.3019,2,24.529644,1.046373,1
13,60532.3097,1,23.404964,0.992504,1
13,60532.3173,3,36.069386,1.568371,1
13,60532.3282,4,42.765503,2.305008,1
13,60532.3392,5,36.567162,5.439748,1
13,60535.2802,2,23.662449,1.422315,1
13,60535.2879,1,20.202259,1.361849,1
13,60535.2957,3,39.966290,2.106815,1
13,60535.3068,4,39.323189,3.034410,1
13,60535.3177,5,25.412567,7.170496,1
13,60538.2826,2,19.899044,1.856537,1
13,60538.2903,1,14.983138,2.683123,1
13,60538.2980,3,32.445835,1.936947,1
13,60538.3089,4,37.566696,2.327296,1
13,60538.3199,5,36.802090,5.025538,1
13,60554.2651,0,2.890699,2.258298,0
13,60555.2411,0,1.421973,2.038275,0
13,60556.2370,0,5.816270,2.553418,0
13,60557.2322,0,2.976124,1.849058,0
13,60558.2332,0,3.465247,2.503530,0
13,60559.2274,0,2.022449,2.213130,0
13,60560.2268,0,3.503388,3.087183,0
13,60567.3291,2,14.211590,0.945795,1
13,60567.3368,1,8.278180,1.052801,1
13,60567.3444,3,24.237070,1.263879,1
13,60567.3553,4,31.986134,1.675546,1
13,60567.3663,5,26.146296,4.267163,1
13,60580.1736,2,10.443220,1.455966,1
13,60580.1813,1,4.004329,1.395011,0
13,60580.1889,3,21.601105,2.202044,1
13,60580.1999,4,33.927616,3.210988,1
13,60580.2108,5,31.062853,7.647927,0
13,60582.1681,0,1.094513,2.405975,0
13,60583.1640,0,-1.259893,2.979695,0
13,60584.1591,0,1.617939,2.344479,0
13,60585.1601,0,1.637716,1.874226,0
13,60586.1564,0,0.117864,1.815223,0
13,60587.1540,0,0.302183,2.121386,0
13,60588.1461,0,3.094076,1.792624,0
13,60593.1209,2,10.800429,1.153115,1
13,60593.1287,1,4.406773,1.308495,0
13,60593.1365,3,16.092447,1.545535,1
13,60593.1476,4,24.343342,2.181189,1
13,60593.1585,5,19.054943,5.111032,1
13,60596.1351,2,12.383826,2.473228,0
13,60596.1427,1,0.372005,3.515397,0
13,60596.1504,3,16.204580,2.651616,1
13,60596.1613,4,26.244993,3.235833,1
13,60596.1723,5,20.389833,7.186456,0
13,60605.0908,2,7.542615,0.977457,1
13,60605.0986,1,3.585346,0.917699,0
13,60605.1063,3,15.296724,1.463083,1
13,60605.1174,4,21.654158,2.154012,1
13,60605.1283,5,12.261124,5.138537,1
13,60608.0836,2,7.212741,0.886919,1
13,60608.0913,1,3.103603,0.823795,0
13,60608.0991,3,15.016788,1.333418,1
13,60608.1101,4,22.194675,1.964862,1
13,60608.1211,5,16.844475,4.664538,0
13,60611.0756,2,5.219585,0.772730,1
13,60611.0833,1,2.315308,0.703447,0
13,60611.0911,3,13.987851,1.177967,1
13,60611.1021,4,23.238731,1.752712,1
13,60611.1130,5,13.764248,4.170574,0
13,60612.0813,0,-0.500034,1.440229,0
13,60613.0818,0,1.002482,1.524789,0
13,60614.0803,0,-1.957333,1.736466,0
13,60615.0761,0,-1.864921,1.981273,0
13,60616.0769,0,-0.823935,2.007182,0
13,60617.0737,0,4.029193,1.584846,0
13,60620.1444,0,1.135807,1.713108,0
13,60621.2673,2,5.227184,0.942700,1
13,60621.2749,1,0.979372,0.945305,0
13,60621.2825,3,10.753699,1.650229,1
13,60621.2934,4,15.025331,2.743479,1
13,60621.3044,5,20.601212,7.516092,0
13,60624.1290,2,4.608896,1.191301,0
13,60624.1366,1,3.302126,1.565634,0
13,60624.1442,3,13.692765,1.502649,1
13,60624.1551,4,22.563152,2.025444,1
13,60624.1661,5,17.466129,4.765408,0
13,60633.0541,2,5.740388,1.090800,1
13,60633.0617,1,-0.248447,1.055601,0
13,60633.0693,3,11.721886,1.678442,1
13,60633.0803,4,21.894281,2.489034,1
13,60633.0912,5,8.198497,5.988672,0
13,60636.0482,2,4.445689,1.029320,0
13,60636.0558,1,0.510712,0.992636,0
13,60636.0635,3,8.838101,1.589973,0
13,60636.0744,4,18.146425,2.370814,1
13,60636.0854,5,20.537880,5.701680,0
13,60640.0972,2,2.708286,0.871447,0
13,60640.1049,1,0.387127,0.847005,0
13,60640.1125,3,11.844381,1.395034,1
13,60640.1234,4,16.336960,2.129130,1
13,60640.1344,5,22.784824,5.287734,0
13,60642.0643,0,-0.045857,2.075920,0
13,60643.0521,0,-0.339986,1.668015,0
13,60644.0621,0,-0.061904,1.665856,0
13,60645.0625,0,-1.683454,1.768579,0
13,60646.0636,0,-1.090810,1.782456,0
13,60647.0635,0,-0.917836,1.968023,0
13,60648.0642,0,-0.471162,1.443392,0
13,60652.1289,2,2.063019,0.939241,0
13,60652.1365,1,0.914091,1.117558,0
13,60652.1441,3,8.505517,1.381162,1
13,60652.1550,4,20.247869,2.050198,1
13,60652.1660,5,4.584575,5.200393,0
14,59798.3205,2,14.465278,1.364599,1
14,59798.3281,1,13.748290,1.165200,1
14,59798.3357,3,8.555202,1.766522,1
14,59798.3466,4,7.253281,2.603756,0
14,59798.3576,5,1.134257,5.365180,0
14,59801.3553,2,9.855556,2.449258,0
14,59801.3629,1,10.256726,3.538008,0
14,59801.3705,3,10.469539,2.286622,0
14,59801.3815,4,4.839510,2.516017,0
14,59801.3924,5,6.016302,5.381228,0
14,59818.2740,0,-1.523922,1.781903,0
14,59819.2541,0,6.043163,2.455132,0
14,59820.2522,0,-3.440181,1.895149,0
14,59821.2478,0,-1.079028,2.430641,0
14,59822.2433,0,-0.827454,2.848530,0
14,59823.2659,0,5.701578,2.706588,0
14,59826.3105,2,2.735408,0.921596,0
14,59826.3181,1,2.361817,0.925133,0
14,59826.3258,3,4.882393,1.509568,1
14,59826.3367,4,5.898125,2.157517,0
14,59826.3477,5,7.172944,5.536250,0
14,59842.2456,2,1.210632,0.849698,0
14,59842.2532,1,0.480856,0.813325,0
14,59842.2608,3,3.207553,1.321817,0
14,59842.2718,4,4.451898,2.166812,0
14,59842.2827,5,4.518466,6.066945,0
14,59851.1792,0,4.186630,2.586060,0
14,59854.1485,2,0.507158,1.223279,0
14,59854.1563,1,1.509088,1.335596,0
14,59854.1640,3,3.079645,1.901563,0
14,59854.1750,4,2.476080,2.769468,0
14,59854.1860,5,5.571058,5.787987,0
14,59857.1408,2,2.902585,2.129320,0
14,59857.1485,1,0.076045,2.911045,0
14,59857.1563,3,5.179635,2.093322,0
14,59857.1673,4,4.421298,2.676636,0
14,59857.1782,5,4.276851,6.626348,0
14,59867.1112,2,-0.034593,1.082066,0
14,59867.1189,1,-1.146521,0.972931,0
14,59867.1267,3,4.685015,1.540005,0
14,59867.1377,4,3.080778,2.356951,0
14,59867.1487,5,-3.318216,5.827800,0
14,59870.1049,2,0.554243,0.970361,0
14,59870.1126,1,0.375991,0.821212,0
14,59870.1204,3,4.517774,1.374431,0
14,59870.1314,4,1.906460,2.282754,0
14,59870.1424,5,-0.145795,5.276641,0
14,59873.0971,2,2.108685,1.041195,0
14,59873.1049,1,0.095400,1.040191,0
14,59873.1126,3,3.909108,1.442765,0
14,59873.1236,4,9.230726,2.071694,0
14,59873.1346,5,5.649127,4.588614,0
14,59874.1461,0,-3.214277,1.608558,0
14,59875.0995,0,2.425352,2.028404,0
14,59876.0980,0,3.646141,2.440229,0
14,59877.0976,0,-1.473760,1.759196,0
14,59878.0964,0,0.576515,2.010597,0
14,59879.0895,0,0.572945,1.861002,0
14,59880.1017,0,-0.224757,1.368790,0
14,59884.1760,2,-0.132164,0.767148,0
14,59884.1836,1,-0.991678,0.885456,0
14,59884.1913,3,2.358108,1.139322,0
14,59884.2022,4,3.557226,1.610064,0
14,59884.2132,5,1.323540,4.209188,0
14,59887.2856,2,-2.655194,1.824434,0
14,59887.2933,1,-3.118680,2.202677,0
14,59887.3009,3,4.665102,2.350384,0
14,59887.3118,4,1.135563,3.143775,0
14,59887.3228,5,4.491710,7.945457,0
14,59896.1307,2,-0.963295,0.821164,0
14,59896.1384,1,-0.071138,0.702119,0
14,59896.1460,3,2.277665,1.393450,0
14,59896.1569,4,7.248254,2.194077,0
14,59896.1679,5,4.658720,5.946997,0
14,59899.1519,2,-1.774250,1.363683,0
14,59899.1595,1,-1.174699,1.385324,0
14,59899.1672,3,-0.953666,2.376407,0
14,59899.1781,4,8.315199,3.709089,0
14,59899.1891,5,-5.444651,7.901953,0
14,59902.1384,2,0.053353,1.184878,0
14,59902.1460,1,-0.247240,1.076702,0
14,59902.1537,3,3.547660,1.943007,0
14,59902.1646,4,3.456085,2.762274,0
14,59902.1755,5,-2.950504,7.020600,0
14,59904.1053,0,4.177117,2.292928,0
14,59905.0555,0,-1.216755,2.497270,0
14,59906.0562,0,2.410480,2.032158,0
14,59907.0567,0,-1.391141,1.743943,0
14,59908.0681,0,3.657772,2.691214,0
14,59909.0582,0,0.521167,1.833233,0
14,59910.0503,0,-0.100852,1.860389,0
14,59914.0526,2,1.198847,1.437903,0
14,59914.0602,1,-4.574095,1.954059,0
14,59914.0678,3,0.367522,1.685196,0
14,59914.0788,4,2.276051,2.267016,0
14,59914.0897,5,10.662569,5.357585,0
14,59924.1060,2,0.898902,1.131097,0
14,59924.1136,1,-0.533751,1.571826,0
14,59924.1212,3,0.148105,1.727512,0
14,59924.1322,4,4.691455,2.527090,0
14,59924.1431,5,-2.120271,5.628747,0
14,59927.1074,2,-0.466719,1.189681,0
14,59927.1151,1,-0.370304,1.132207,0
14,59927.1227,3,2.069784,1.592095,0
14,59927.1336,4,2.988137,2.298649,0
14,59927.1446,5,2.182713,5.887877,0
14,59930.1236,2,0.240297,0.940370,0
14,59930.1312,1,-0.920183,0.827606,0
14,59930.1388,3,1.243185,1.282906,0
14,59930.1498,4,3.796952,2.032441,0
14,59930.1607,5,4.695318,5.329674,0
14,59933.1249,2,0.006509,0.690589,0
14,59933.1325,1,1.157228,0.695202,0
14,59933.1401,3,2.223172,1.269250,0
14,59933.1511,4,2.050447,1.913679,0
14,59933.1620,5,6.551545,5.426427,0
14,59934.0638,0,-1.038826,1.138865,0
14,59935.0646,0,-0.855022,1.120789,0
14,59936.0642,0,4.664287,1.541755,0
14,59937.0650,0,0.068496,1.637206,0
14,59938.0647,0,-0.037584,1.994424,0
14,59939.0650,0,0.482500,2.191997,0
14,60165.3032,2,-0.497132,1.193823,0
14,60165.3109,1,2.598282,1.347751,0
14,60165.3186,3,-2.635130,1.517240,0
14,60165.3295,4,5.500406,2.024001,0
14,60165.3405,5,-1.459476,4.779670,0
14,60168.2892,2,-0.132310,0.833159,0
14,60168.2970,1,1.305878,0.789044,0
14,60168.3047,3,1.487449,1.300279,0
14,60168.3157,4,2.190609,2.065560,0
14,60168.3267,5,-2.313916,4.814189,0
14,60176.2820,0,-2.258721,2.358804,0
14,60177.2726,0,-0.240173,2.761198,0
14,60181.4088,2,-0.118050,0.923237,0
14,60181.4164,1,-1.757872,1.260326,0
14,60181.4232,3,-0.566288,3.235162,0
14,60183.2660,2,-0.732747,1.913709,0
14,60183.2736,1,0.012732,2.931251,0
14,60183.2812,3,1.402672,2.546405,0
14,60183.2922,4,-1.725346,2.993038,0
14,60183.3031,5,0.494266,6.175346,0
14,60195.2812,2,3.029382,0.934420,0
14,60195.2888,1,-0.670324,1.018990,0
14,60195.2964,3,-0.662677,1.624419,0
14,60195.3073,4,-3.872562,2.337922,0
14,60195.3183,5,2.712977,6.272959,0
14,60198.2690,0,2.956521,2.177715,0
14,60199.2186,0,1.331341,2.156719,0
14,60200.2139,0,1.351745,2.588871,0
14,60201.2072,0,2.853797,2.498285,0
14,60202.2089,0,1.906982,2.486693,0
14,60209.1811,2,-1.602512,1.563901,0
14,60209.1888,1,-0.671917,1.407978,0
14,60209.1965,3,0.374204,1.830516,0
14,60209.2075,4,2.568392,2.908100,0
14,60209.2184,5,6.678216,6.695596,0
14,60212.1675,2,3.776216,1.935000,0
14,60212.1753,1,0.021062,2.747988,0
14,60212.1830,3,-1.807372,2.293340,0
14,60212.1941,4,2.568171,2.590221,0
14,60212.2050,5,10.840719,5.663691,0
14,60223.2416,2,-0.194212,1.181089,0
14,60223.2493,1,-0.036469,1.239146,0
14,60223.2569,3,1.530641,1.648199,0
14,60223.2678,4,1.801575,2.471934,0
14,60223.2788,5,4.744702,5.793488,0
14,60226.3337,2,-0.744695,0.943627,0
14,60226.3413,1,-0.106145,0.942899,0
14,60226.3489,3,2.442077,1.793019,0
14,60226.3599,4,1.509347,3.068994,0
14,60226.3708,5,-6.627614,8.105484,0
14,60238.3197,2,-0.863180,1.052267,0
14,60238.3273,1,1.557186,1.152551,0
14,60238.3349,3,-4.336288,2.169387,0
14,60238.3459,4,4.612921,3.828061,0
14,60238.3568,5,-2.021824,11.249375,0
14,60241.0870,2,0.738195,1.997376,0
14,60241.0948,1,-1.971425,3.061613,0
14,60241.1025,3,-2.025038,2.443142,0
14,60241.1136,4,-0.312927,2.915593,0
14,60241.1245,5,3.056638,5.938406,0
14,60250.1708,2,-2.482518,2.100286,0
14,60250.1957,1,-0.029834,2.806254,0
14,60250.2034,3,4.678575,2.926827,0
14,60250.2143,4,-3.449543,3.980825,0
14,60250.2253,5,-3.656875,7.932066,0
14,60261.1296,0,-3.352771,2.067494,0
14,60262.0550,0,-3.894481,2.299585,0
14,60263.0556,0,0.061202,1.966700,0
14,60264.0559,0,-0.441779,2.168793,0
14,60265.0780,0,-0.070926,2.402439,0
14,60268.0449,2,-2.115148,1.402396,0
14,60268.0525,1,-1.156284,1.846402,0
14,60268.0601,3,-0.248761,2.127820,0
14,60268.0711,4,1.514031,2.732877,0
14,60268.0820,5,-11.142164,6.741534,0
14,60278.0993,2,0.058940,1.642234,0
14,60278.1069,1,1.004120,2.270966,0
14,60278.1145,3,-0.427660,2.168000,0
14,60278.1255,4,0.148053,3.057248,0
14,60278.1364,5,14.839427,7.639682,0
14,60281.1023,2,0.190632,0.797410,0
14,60281.1099,1,-0.334674,0.793767,0
14,60281.1175,3,-1.839893,1.368291,0
14,60281.1285,4,-1.943368,1.989664,0
14,60281.1394,5,3.423336,5.175146,0
14,60284.1027,2,0.880585,0.836640,0
14,60284.1104,1,-1.071148,0.760074,0
14,60284.1180,3,-0.493422,1.159284,0
14,60284.1289,4,-1.769790,1.701805,0
14,60284.1399,5,10.491048,4.439059,0
14,60287.1047,2,-2.440768,1.249228,0
14,60287.1123,1,-0.587990,1.081420,0
14,60287.1200,3,-0.658860,1.794693,0
14,60287.1309,4,6.071679,2.963972,0
14,60287.1418,5,-0.912052,7.009711,0
14,60290.0761,0,0.791311,1.912837,0
14,60291.0689,0,-0.916465,1.657989,0
14,60292.0699,0,-0.940145,1.852110,0
14,60293.0699,0,-0.065888,1.558097,0
14,60294.0708,0,-0.362213,1.394291,0
14,60532.3019,2,-0.128229,1.022716,0
14,60532.3097,1,-0.763027,0.950918,0
14,60532.3173,3,-0.986068,1.539248,0
14,60532.3282,4,-0.845234,2.279716,0
14,60532.3392,5,8.809470,5.421995,0
14,60535.2802,2,-0.482045,1.404915,0
14,60535.2879,1,1.358840,1.336113,0
14,60535.2957,3,-0.531306,2.081220,0
14,60535.3068,4,-3.251427,3.016502,0
14,60535.3177,5,-1.253254,7.157213,0
14,60538.2826,2,1.802223,1.846149,0
14,60538.2903,1,-0.026881,2.671491,0
14,60538.2980,3,1.533429,1.916902,0
14,60538.3089,4,-0.856327,2.306201,0
14,60538.3199,5,-1.663799,5.002286,0
14,60554.2651,0,-1.160009,2.241721,0
14,60555.2411,0,-2.432893,2.027424,0
14,60556.2370,0,2.398724,2.534231,0
14,60557.2322,0,-1.568709,1.829169,0
14,60558.2332,0,-4.116055,2.484439,0
14,60559.2274,0,0.190265,2.200325,0
14,60560.2268,0,1.347325,3.073875,0
14,60567.3291,2,0.910389,0.934006,0
14,60567.3368,1,-1.444133,1.041686,0
14,60567.3444,3,-1.500671,1.243064,0
14,60567.3553,4,2.684830,1.653721,0
14,60567.3663,5,0.345307,4.248466,0
14,60580.1736,2,0.807077,1.449346,0
14,60580.1813,1,2.870951,1.391824,0
14,60580.1889,3,-0.262873,2.190222,0
14,60580.1999,4,-6.388680,3.196808,0
14,60580.2108,5,9.181289,7.635880,0
14,60582.1681,0,3.604234,2.412230,0
14,60583.1640,0,-2.491946,2.974151,0
14,60584.1591,0,0.669646,2.335869,0
14,60585.1601,0,0.574023,1.865054,0
14,60586.1564,0,-0.297618,1.811183,0
14,60587.1540,0,-1.218754,2.115987,0
14,60588.1461,0,-1.992268,1.771815,0
14,60593.1209,2,-0.511945,1.144706,0
14,60593.1287,1,-0.914986,1.302187,0
14,60593.1365,3,-1.629705,1.534286,0
14,60593.1476,4,-3.500163,2.167956,0
14,60593.1585,5,-7.225812,5.098632,0
14,60596.1351,2,1.942903,2.467501,0
14,60596.1427,1,-3.828583,3.510221,0
14,60596.1504,3,-0.379595,2.643848,0
14,60596.1613,4,-4.746652,3.225241,0
14,60596.1723,5,-0.740908,7.175334,0
14,60605.0908,2,-0.240784,0.970734,0
14,60605.0986,1,1.228715,0.912676,0
14,60605.1063,3,-0.173487,1.451932,0
14,60605.1174,4,-3.157050,2.142289,0
14,60605.1283,5,-4.353776,5.129833,0
14,60608.0836,2,-1.290518,0.880033,0
14,60608.0913,1,0.897031,0.818780,0
14,60608.0991,3,-1.789510,1.321652,0
14,60608.1101,4,-0.283683,1.951819,0
14,60608.1211,5,-1.071998,4.652730,0
14,60611.0756,2,0.428420,0.767492,0
14,60611.0833,1,-0.175538,0.697855,0
14,60611.0911,3,-1.351069,1.165860,0
14,60611.1021,4,3.187355,1.739082,0
14,60611.1130,5,-2.026892,4.159950,0
14,60612.0813,0,1.714202,1.449054,0
14,60613.0818,0,-1.208979,1.515573,0
14,60614.0803,0,2.366692,1.746634,0
14,60615.0761,0,0.562962,1.980418,0
14,60616.0769,0,-0.108613,2.003422,0
14,60617.0737,0,1.100774,1.564082,0
14,60620.1444,0,-3.990349,1.704574,0
14,60621.2673,2,-1.006580,0.937842,0
14,60621.2749,1,-0.619488,0.942508,0
14,60621.2825,3,1.860522,1.643903,0
14,60621.2934,4,1.823893,2.737007,0
14,60621.3044,5,-2.549210,7.504488,0
14,60624.1290,2,-1.234065,1.187694,0
14,60624.1366,1,0.769079,1.561521,0
14,60624.1442,3,-1.651142,1.493277,0
14,60624.1551,4,-1.015730,2.012733,0
14,60624.1661,5,-1.555369,4.753550,0
14,60633.0541,2,0.136783,1.086145,0
14,60633.0617,1,-0.152341,1.054063,0
14,60633.0693,3,-0.510173,1.670818,0
14,60633.0803,4,0.315384,2.478524,0
14,60633.0912,5,-6.652580,5.982395,0
14,60636.0482,2,-0.451914,1.025376,0
14,60636.0558,1,0.286247,0.990897,0
14,60636.0635,3,0.255769,1.584027,0
14,60636.0744,4,-0.340420,2.361790,0
14,60636.0854,5,9.500138,5.693579,0
14,60640.0972,2,0.332550,0.868813,0
14,60640.1049,1,-0.473409,0.845206,0
14,60640.1125,3,0.729877,1.386701,0
14,60640.1234,4,-1.934210,2.120461,0
14,60640.1344,5,-4.023204,5.273717,0
14,60642.0643,0,6.371965,2.098372,0
14,60643.0521,0,0.069668,1.665262,0
14,60644.0621,0,-0.031334,1.662740,0
14,60645.0625,0,0.388892,1.767081,0
14,60646.0636,0,-0.740859,1.779115,0
14,60647.0635,0,3.793745,1.980666,0
14,60648.0642,0,-2.506170,1.440771,0
14,60652.1289,2,2.536718,0.938627,0
14,60652.1365,1,-0.345612,1.114977,0
14,60652.1441,3,-0.218325,1.374793,0
14,60652.1550,4,3.384219,2.040273,0
14,60652.1660,5,5.932649,5.198509,0
17,59750.4229,2,0.384775,1.502702,0
17,59750.4306,1,2.970657,2.100801,0
17,59750.4383,3,-3.890317,2.298941,0
17,59750.4450,4,-7.424517,10.310197,0
17,59752.4070,2,1.180267,1.063318,0
17,59752.4147,1,-0.427451,1.103617,0
17,59752.4224,3,0.804619,1.459435,0
17,59752.4334,4,0.082026,2.345975,0
17,59752.4435,5,-12.127212,9.469489,0
17,59767.2968,2,0.300545,0.879810,0
17,59767.3045,1,-0.789039,0.757829,0
17,59767.3122,3,-2.516992,1.335309,0
17,59767.3233,4,-0.305087,2.137903,0
17,59767.3343,5,-2.361575,4.981940,0
17,59770.2179,2,2.652607,2.031536,0
17,59770.2256,1,0.104747,2.751110,0
17,59770.2334,3,-1.251170,2.138434,0
17,59770.2445,4,1.627103,2.665143,0
17,59770.2557,5,3.234555,6.275570,0
17,59779.3188,2,-1.560282,2.314641,0
17,59779.3265,1,1.138921,2.800010,0
17,59779.3342,3,1.399184,2.246128,0
17,59779.3452,4,-4.086601,2.966733,0
17,59779.3562,5,3.600588,7.075419,0
17,59782.1897,2,-0.205182,1.441666,0
17,59782.1974,1,-0.008989,1.378359,0
17,59782.2051,3,-3.370768,2.336834,0
17,59782.2162,4,-1.506593,3.089562,0
17,59782.2274,5,-1.340868,7.552328,0
17,59797.2861,2,0.574704,1.059202,0
17,59797.2938,1,-1.925344,1.095736,0
17,59797.3015,3,-1.303792,1.916743,0
17,59797.3126,4,-0.918545,2.786953,0
17,59797.3237,5,-7.784748,6.963860,0
17,59800.3168,2,0.594011,2.277387,0
17,59800.3244,1,6.002035,3.310603,0
17,59800.3320,3,0.828272,2.636445,0
17,59800.3429,4,-2.143990,3.045394,0
17,59800.3539,5,2.182031,6.526192,0
17,59807.1738,2,0.345367,1.600768,0
17,59807.1815,1,1.497795,2.061084,0
17,59807.1892,3,-1.738689,1.721755,0
17,59807.2003,4,-2.008608,2.187723,0
17,59807.2114,5,-4.591490,5.073585,0
17,59810.1045,2,-0.190427,1.024519,0
17,59810.1122,1,1.409750,0.987204,0
17,59810.1200,3,2.097727,1.766024,0
17,59810.1311,4,4.661858,2.723413,0
17,59810.1422,5,1.852001,6.541757,0
17,59813.1044,2,-0.609008,0.979608,0
17,59813.1122,1,-1.283370,0.947661,0
17,59813.1199,3,0.443231,1.563090,0
17,59813.1310,4,0.463331,2.578759,0
17,59813.1422,5,-4.206830,5.891186,0
17,59819.1532,0,-0.111748,1.900487,0
17,59820.1047,0,-1.127300,1.845100,0
17,59821.1026,0,-2.983045,2.653126,0
17,59822.1105,0,-3.786463,3.135153,0
17,59823.1505,0,-6.251946,2.924953,0
17,59835.0600,2,-0.105554,1.651957,0
17,59835.0678,1,-3.813735,2.461054,0
17,59835.0755,3,-5.183056,2.248735,0
17,59835.0866,4,-3.032220,2.768835,0
17,59835.0978,5,-1.849971,5.832443,0
17,59839.0306,2,1.615393,1.178528,0
17,59839.0384,1,-0.088986,1.136387,0
17,59839.0461,3,-3.522223,1.587615,0
17,59839.0573,4,-2.323542,2.134739,0
17,59839.0684,5,2.161493,5.221899,0
17,59842.0207,2,1.318505,1.033401,0
17,59842.0285,1,0.922532,1.039005,0
17,59842.0362,3,1.569138,1.599077,0
17,59842.0473,4,3.807324,2.315629,0
17,59842.0585,5,-1.946610,5.253097,0
17,59851.1114,0,0.466809,1.833261,0
17,59854.0796,2,-0.432977,1.106505,0
17,59854.0873,1,-0.349620,1.186553,0
17,59854.0950,3,-1.028248,1.475277,0
17,59854.1061,4,0.624902,2.189718,0
17,59854.1172,5,-6.577788,5.223208,0
17,59857.0453,2,1.048279,1.770880,0
17,59857.0531,1,4.293002,2.442868,0
17,59857.0608,3,0.670081,1.885174,0
17,59857.0719,4,-1.393756,2.439591,0
17,59857.0830,5,7.867962,5.441516,0
17,59864.0162,2,1.446581,0.850881,0
17,59864.0239,1,1.881013,0.881634,0
17,59864.0316,3,1.176724,1.403741,0
17,59864.0428,4,1.061918,2.333516,0
17,59864.0539,5,7.476207,5.917978,0
17,59867.0178,2,0.380713,1.309081,0
17,59867.0255,1,-3.510653,1.302897,0
17,59867.0332,3,-2.578208,2.183999,0
17,59867.0443,4,1.046922,3.227464,0
17,59867.0554,5,-14.202744,9.457583,0
17,59870.0194,2,-1.554325,0.927003,0
17,59870.0272,1,-0.831158,0.920545,0
17,59870.0349,3,-1.187623,1.500118,0
17,59870.0459,4,2.125045,2.528746,0
17,59870.0571,5,2.077787,5.911781,0
17,59873.0212,2,-0.457801,0.738598,0
17,59873.0289,1,2.796051,0.760300,0
17,59873.0366,3,0.138800,1.232345,0
17,59873.0477,4,0.273784,1.886916,0
17,59873.0588,5,-5.356641,4.801971,0
17,59874.0599,0,-2.398834,1.584590,0
17,59875.0311,0,-0.540180,1.315395,0
17,59876.0231,0,0.411913,2.120848,0
17,59877.0238,0,-0.891288,1.592397,0
17,59878.0246,0,0.252658,1.625712,0
17,59879.0248,0,0.494818,1.668736,0
17,59880.0258,0,1.609941,1.357399,0
17,59884.0823,2,1.431002,1.091537,0
17,59884.0900,1,-0.669697,1.493729,0
17,59884.0976,3,0.632236,1.522321,0
17,59884.1085,4,2.388964,2.008811,0
17,59884.1195,5,-3.279281,4.883777,0
17,59887.0298,2,-1.951323,1.872626,0
17,59887.0375,1,-0.140049,2.947258,0
17,59887.0451,3,-0.987463,2.277290,0
17,59887.0562,4,-1.669648,2.256846,0
17,59887.0673,5,4.776433,5.354571,0
17,60118.4163,0,1.237189,1.560750,0
17,60124.2541,2,-0.027002,2.279922,0
17,60124.2618,1,-1.582832,2.932222,0
17,60124.2695,3,-2.243939,2.501337,0
17,60124.2807,4,-2.010008,2.976800,0
17,60124.2918,5,-3.812106,6.686803,0
17,60140.2290,0,2.889142,1.885088,0
17,60141.2225,0,-2.675623,3.532677,0
17,60142.2202,0,3.701951,3.092139,0
17,60143.2212,0,0.107055,2.421978,0
17,60144.2186,0,2.121859,1.934957,0
17,60145.2123,0,3.519005,2.431906,0
17,60153.2274,2,1.331808,1.381732,0
17,60153.2351,1,1.678943,1.884794,0
17,60153.2428,3,-0.669065,1.646313,0
17,60153.2539,4,-2.336847,2.069594,0
17,60153.2650,5,-3.040281,5.269622,0
17,60162.1477,2,0.274822,1.883733,0
17,60162.1554,1,-1.090832,2.591505,0
17,60162.1631,3,-4.857473,2.765956,0
17,60162.1742,4,-10.275881,3.141578,0
17,60162.1853,5,-9.252888,7.983027,0
17,60165.1369,2,-0.468279,0.846974,0
17,60165.1446,1,-0.430437,0.708165,0
17,60165.1524,3,-0.581242,1.364931,0
17,60165.1635,4,3.187243,2.182051,0
17,60165.1746,5,6.814413,5.636336,0
17,60168.1260,2,1.795461,0.881347,0
17,60168.1337,1,-0.350579,0.854263,0
17,60168.1414,3,-2.100255,1.659393,0
17,60168.1525,4,-1.526452,2.377539,0
17,60168.1637,5,2.514493,5.409767,0
17,60176.1332,0,-1.022222,2.365174,0
17,60177.1370,0,-3.289955,2.790762,0
17,60181.3147,2,-1.503265,1.258997,0
17,60181.3223,1,1.620242,1.237350,0
17,60181.3299,3,2.305993,2.212692,0
17,60181.3409,4,-0.697285,4.016526,0
17,60181.3518,5,3.620666,8.753998,0
17,60184.3625,2,-0.132555,2.135256,0
17,60184.3701,1,-0.614091,2.560603,0
17,60184.3777,3,0.653611,2.486900,0
17,60184.3887,4,1.677764,2.891815,0
17,60184.3996,5,7.855544,6.898280,0
17,60194.1575,2,1.309505,1.067185,0
17,60194.1652,1,-0.309205,1.053532,0
17,60194.1729,3,-0.334617,1.786434,0
17,60194.1839,4,0.655740,3.139240,0
17,60194.1926,5,5.044827,11.278636,0
17,60197.1181,2,0.552177,1.097758,0
17,60197.1258,1,0.541816,0.909236,0
17,60197.1335,3,0.639729,1.336890,0
17,60197.1446,4,-0.978055,2.037800,0
17,60197.1557,5,4.619338,4.987423,0
17,60198.1077,0,2.149141,2.452106,0
17,60199.0914,0,5.486719,3.143868,0
17,60200.0650,0,1.228997,2.524841,0
17,60201.0680,0,6.056491,2.897650,0
17,60202.0552,0,3.771713,2.094126,0
17,60206.1107,0,1.964452,1.559410,0
17,60207.1469,0,1.315419,2.819893,0
17,60208.0229,2,-0.600505,1.423023,0
17,60208.0307,1,0.017527,1.343573,0
17,60208.0384,3,-1.550505,1.831250,0
17,60208.0495,4,3.882280,2.726225,0
17,60208.0606,5,9.559813,7.414577,0
17,60211.0124,2,1.860704,2.250480,0
17,60211.0202,1,-2.090354,3.034887,0
17,60211.0279,3,0.410192,2.599820,0
17,60211.0390,4,-1.062310,3.468099,0
17,60211.0502,5,8.321412,9.169344,0
17,60221.0153,2,0.448097,0.997295,0
17,60221.0230,1,0.082163,0.883182,0
17,60221.0308,3,1.515744,1.365631,0
17,60221.0419,4,-0.955922,2.171834,0
17,60221.0530,5,6.189006,5.659595,0
17,60224.0140,2,-0.977404,1.146213,0
17,60224.0217,1,-1.527085,1.044619,0
17,60224.0294,3,-2.026714,1.896492,0
17,60224.0405,4,-2.270058,2.563563,0
17,60224.0516,5,1.532559,5.857242,0
17,60227.0151,2,0.269555,1.054280,0
17,60227.0228,1,1.115057,0.999069,0
17,60227.0305,3,1.552577,1.598891,0
17,60227.0416,4,0.899977,2.226238,0
17,60227.0527,5,-13.314797,5.586868,0
17,60228.0187,0,1.561344,2.716171,0
17,60229.0162,0,2.115411,2.732387,0
17,60234.0265,0,-0.632177,2.254489,0
17,60237.2206,2,0.991648,0.921157,0
17,60237.2283,1,1.011999,0.870146,0
17,60237.2359,3,1.932343,1.516595,0
17,60237.2468,4,0.683186,2.506894,0
17,60237.2578,5,-2.942844,7.042232,0
17,60240.0223,2,0.455927,1.997859,0
17,60240.0300,1,2.247968,2.647892,0
17,60240.0377,3,6.028008,2.647428,0
17,60240.0488,4,2.535219,3.012484,0
17,60240.0598,5,8.448805,5.811503,0
17,60249.0338,2,0.460854,0.866548,0
17,60249.0415,1,0.452554,0.772836,0
17,60249.0492,3,-1.191506,1.286252,0
17,60249.0602,4,2.830095,2.116775,0
17,60249.0712,5,-4.378712,5.177276,0
17,60260.0423,0,-7.464523,2.644202,0
17,60261.0361,0,0.930443,1.914282,0
17,60262.0367,0,4.964463,1.854471,0
17,60263.0373,0,0.783741,2.196650,0
17,60264.0465,0,-3.080768,1.916776,0
17,60490.2647,2,-0.977129,1.482041,0
17,60490.2725,1,-2.611819,1.794772,0
17,60490.2802,3,1.078630,2.071858,0
17,60490.2913,4,-0.492853,2.955702,0
17,60490.3024,5,3.792550,7.104066,0
17,60493.2372,2,0.021979,1.054925,0
17,60493.2450,1,-1.376170,0.980580,0
17,60493.2527,3,1.015628,1.668951,0
17,60493.2639,4,-0.002133,2.546342,0
17,60493.2750,5,-11.551120,6.173368,0
17,60499.2467,0,0.438197,2.600874,0
17,60500.2437,0,-1.686091,2.074696,0
17,60501.2385,0,0.020948,2.299457,0
17,60502.2355,0,1.068030,2.629417,0
17,60508.2638,2,-2.790141,2.436066,0
17,60508.2715,1,5.700086,3.490426,0
17,60508.2792,3,0.113332,2.677060,0
17,60508.2903,4,-1.245703,3.280095,0
17,60508.3014,5,16.609587,7.427779,0
17,60524.2390,0,-0.490797,2.080026,0
17,60525.1736,0,2.666769,2.941305,0
17,60532.3489,2,-0.207433,0.864004,0
17,60532.3565,1,-0.134437,0.845488,0
17,60532.3641,3,-0.515472,1.462743,0
17,60532.3751,4,0.897895,2.323136,0
17,60532.3860,5,1.356515,5.973135,0
17,60535.1253,2,3.357187,1.895133,0
17,60535.1330,1,1.788090,2.447693,0
17,60535.1408,3,-0.746670,2.393641,0
17,60535.1519,4,1.761451,3.298109,0
17,60535.1630,5,-3.134825,7.769337,0
17,60538.2351,2,7.960939,1.803240,0
17,60538.2428,1,6.926830,2.602727,0
17,60538.2505,3,3.353295,1.972350,0
17,60538.2615,4,5.710919,2.318584,0
17,60538.2725,5,0.716567,5.113697,0
17,60546.3406,2,12.973317,1.903937,1
17,60546.3482,1,11.756248,2.837219,1
17,60546.3558,3,10.891992,2.308349,0
17,60546.3668,4,10.706098,3.126808,0
17,60546.3777,5,1.998101,7.646282,0
17,60549.0879,2,8.457420,1.114208,1
17,60549.0956,1,10.614448,1.048073,1
17,60549.1034,3,9.131392,1.747332,1
17,60549.1145,4,12.109291,2.651627,0
17,60549.1256,5,-0.454903,6.443316,0
17,60554.0964,0,3.385612,2.492159,0
17,60555.0951,0,-0.653454,2.008215,0
17,60556.0879,0,4.359657,2.500103,0
17,60557.0831,0,0.208935,1.804377,0
17,60558.1093,0,2.002906,2.298758,0
17,60559.1097,0,-1.713623,1.974990,0
17,60560.1065,0,-4.030066,2.814673,0
17,60567.2821,2,5.086202,1.698282,0
17,60567.2897,1,4.862064,2.316888,0
17,60567.2973,3,2.821583,1.933260,0
17,60567.3083,4,8.017381,2.382588,0
17,60567.3192,5,5.673804,5.440264,0
17,60574.1118,2,4.409955,1.550580,0
17,60574.1195,1,2.795052,2.196265,0
17,60574.1272,3,4.197219,1.987329,0
17,60574.1383,4,10.014592,2.528254,0
17,60574.1493,5,12.793771,5.745250,0
17,60577.0186,2,4.357023,1.470498,0
17,60577.0263,1,1.377891,1.400257,0
17,60577.0340,3,3.097299,2.332945,0
17,60577.0451,4,5.029422,3.520210,0
17,60577.0563,5,9.703295,8.530192,0
17,60580.0095,2,0.080061,1.449560,0
17,60580.0173,1,0.636153,1.387539,0
17,60580.0250,3,1.814262,2.309473,0
17,60580.0361,4,2.268791,3.480996,0
17,60580.0472,5,3.766253,8.433430,0
17,60582.0840,0,2.160855,2.057876,0
17,60583.0169,0,-2.377121,2.935992,0
17,60584.0117,0,-7.030778,2.272231,0
17,60585.0117,0,-2.432227,1.843772,0
17,60586.0123,0,0.942960,1.782520,0
17,60587.0127,0,-0.601708,2.076788,0
17,60588.0131,0,-2.069814,1.647215,0
17,60593.0636,2,0.346279,0.950896,0
17,60593.0713,1,1.045972,1.126506,0
17,60593.0790,3,1.954885,1.349871,0
17,60593.0901,4,4.170722,1.979371,0
17,60593.1012,5,5.737381,4.821476,0
17,60596.0304,2,1.884329,2.578470,0
17,60596.0381,1,-2.693286,3.627853,0
17,60596.0458,3,0.705422,2.812787,0
17,60596.0569,4,1.202975,3.396812,0
17,60596.0680,5,16.761280,7.451248,0
17,60603.0208,2,0.386061,0.711484,0
17,60603.0286,1,-0.007518,0.663680,0
17,60603.0363,3,0.688574,1.169908,0
17,60603.0473,4,4.215033,1.826377,0
17,60603.0584,5,4.616831,4.548802,0
17,60606.0225,2,-0.884437,1.024675,0
17,60606.0303,1,-0.808401,0.984848,0
17,60606.0379,3,1.421415,1.656912,0
17,60606.0490,4,-0.165550,2.551519,0
17,60606.0601,5,-0.249700,6.335219,0
17,60609.0247,2,0.342698,0.744517,0
17,60609.0323,1,0.365452,0.700571,0
17,60609.0400,3,3.474475,1.220512,0
17,60609.0510,4,5.966081,1.906627,0
17,60609.0621,5,-4.503197,4.756433,0
17,60612.0266,0,-1.850411,1.158724,0
17,60613.0269,0,-1.095945,1.252873,0
17,60614.0276,0,1.658578,1.438634,0
17,60615.0375,0,-1.868202,1.603315,0
================================================
FILE: examples/data/plasticc_test_set_metadata_1k.csv
================================================
object_id,ra,decl,gal_l,gal_b,ddf,hostgal_specz,hostgal_photoz,hostgal_photoz_err,distmod,mwebv
13,34.453125,-5.229529,169.987075,-59.956185,1,0.3048,0.3193,0.0542,41.1123,0.019
14,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.6323,0.0179,42.8774,0.018
17,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.8297,0.0605,43.6000,0.016
23,34.804688,-5.829153,171.307861,-60.174401,1,nan,0.6533,0.1479,42.9640,0.023
34,351.321442,-64.198746,317.458993,-50.429931,1,0.4557,0.4617,0.0122,42.0540,0.023
35,35.332031,-5.979157,172.286722,-59.931743,1,nan,0.8388,0.0375,43.6290,0.022
43,0.574468,-45.981140,327.041068,-68.778764,1,nan,0.6669,0.0546,43.0186,0.006
50,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.4663,0.0529,45.1281,0.006
60,346.562500,-63.448284,320.824720,-49.866957,1,nan,0.9462,0.0116,43.9519,0.021
69,349.160583,-64.760857,318.219706,-49.458924,1,nan,1.0432,0.1092,44.2138,0.020
88,349.160583,-64.760857,318.219706,-49.458924,1,0.1608,0.1650,0.0053,39.4929,0.020
96,151.171875,2.537361,237.288526,43.169764,1,0.3277,0.3680,0.0340,41.4711,0.024
106,1.666667,-44.399834,327.519190,-70.529554,1,nan,0.8532,0.0602,43.6747,0.009
114,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.7996,0.2747,43.5011,0.020
115,151.347656,4.181528,235.568369,44.259942,1,nan,0.8979,0.0515,43.8114,0.016
116,150.468750,1.641510,237.714575,42.075234,1,nan,1.1244,0.0363,44.4151,0.017
130,34.277344,-5.679190,170.314930,-60.410322,1,0.3395,0.3368,0.0728,41.2464,0.020
142,1.694561,-45.191612,326.278557,-69.858253,1,nan,1.2710,0.0796,44.7444,0.011
147,150.820312,1.641510,237.994507,42.358984,1,nan,0.2904,0.1155,40.8738,0.020
151,151.171875,1.342993,238.602520,42.464379,1,nan,0.5090,0.0122,42.3075,0.026
168,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.0000,0.0000,nan,0.020
171,52.910156,-27.953188,223.774083,-54.639214,1,nan,0.8623,0.0583,43.7031,0.007
173,150.996094,4.181528,235.291975,43.970869,1,nan,0.4490,0.0219,41.9820,0.015
176,52.910156,-27.953188,223.774083,-54.639214,1,0.3775,0.3642,0.0064,41.4450,0.007
184,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.9112,0.0513,43.8508,0.024
186,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.9302,0.0118,43.9062,0.011
195,152.050781,2.985506,237.495952,44.143927,1,nan,0.4658,0.0250,42.0768,0.019
198,1.694561,-45.191612,326.278557,-69.858253,1,0.4060,0.3959,0.0146,41.6579,0.011
204,349.046051,-61.943836,320.796530,-51.753706,1,0.5584,0.4997,0.0312,42.2594,0.017
211,53.613281,-27.953188,223.929533,-54.024772,1,0.5469,0.5644,0.0113,42.5781,0.007
216,150.820312,1.641510,237.994507,42.358984,1,nan,0.4056,0.0489,41.7202,0.020
236,2.457983,-45.389202,324.632685,-69.945696,1,0.3436,0.2885,0.0162,40.8574,0.011
240,151.171875,2.537361,237.288526,43.169764,1,nan,1.0936,0.0318,44.3405,0.024
260,150.820312,3.732834,235.666318,43.572109,1,nan,0.7554,0.0425,43.3496,0.016
268,149.589844,3.583322,234.885369,42.474696,1,nan,0.6234,0.0184,42.8401,0.024
272,149.414062,3.433834,234.919132,42.245550,1,nan,0.7059,0.0220,43.1693,0.027
277,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.8751,0.0187,43.7426,0.021
289,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.7442,0.0682,43.3099,0.006
306,148.886719,2.686724,235.347248,41.389003,1,0.7180,0.7265,0.0182,43.2458,0.028
316,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.5615,0.1166,42.5647,0.017
337,150.117188,2.836105,236.124718,42.483719,1,nan,1.4098,0.0499,45.0228,0.016
349,34.453125,-5.229529,169.987075,-59.956185,1,nan,0.7679,0.0318,43.3934,0.019
357,349.966217,-62.696659,319.542989,-51.376556,1,nan,0.8937,0.0213,43.7988,0.021
366,53.613281,-26.944359,222.237403,-53.863858,1,nan,1.3577,0.2274,44.9217,0.009
384,359.816315,-44.003082,331.451340,-70.123054,1,nan,0.8134,0.0374,43.5469,0.013
402,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.5684,0.0804,42.5965,0.023
406,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.8989,0.0967,43.8145,0.018
409,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.9110,0.0551,43.8503,0.017
413,349.429535,-62.508568,320.039643,-51.393745,1,0.6430,0.6411,0.0083,42.9139,0.020
443,150.996094,2.388015,237.313912,42.939977,1,0.3682,0.3649,0.0113,41.4497,0.021
451,349.615387,-63.636005,318.927246,-50.506542,1,nan,0.8853,0.0298,43.7734,0.018
455,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.4671,0.0117,42.0842,0.011
466,34.277344,-5.079716,169.526841,-59.956640,1,0.4986,0.5527,0.0171,42.5229,0.019
467,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.6573,0.0463,42.9800,0.009
478,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7283,0.0517,43.2524,0.008
483,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.7835,0.0575,43.4469,0.016
489,0.574468,-45.981140,327.041068,-68.778764,1,nan,0.5798,0.0174,42.6490,0.006
524,34.277344,-5.079716,169.526841,-59.956640,1,0.4067,0.3530,0.0854,41.3660,0.019
561,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.9267,0.0135,43.8959,0.023
565,152.050781,3.284369,237.157374,44.318466,1,nan,0.8253,0.0201,43.5857,0.019
568,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.0649,0.2186,44.2692,0.009
583,149.414062,2.238686,236.239766,41.565558,1,0.7070,0.7129,0.0137,43.1952,0.017
607,349.160583,-64.760857,318.219706,-49.458924,1,0.2424,0.4948,0.3294,42.2339,0.020
611,149.589844,3.583322,234.885369,42.474696,1,0.5068,0.5222,0.0459,42.3744,0.024
613,150.820312,1.641510,237.994507,42.358984,1,0.3014,0.6138,0.3121,42.7991,0.020
622,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.9127,0.3417,43.8552,0.009
639,52.207031,-28.630989,224.800211,-55.343637,1,nan,0.6900,0.0255,43.1086,0.009
662,33.750000,-4.630479,168.146242,-59.949072,1,0.4181,0.4889,0.0244,42.2029,0.019
670,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.3168,0.1915,44.8395,0.009
672,350.230255,-61.943836,320.053946,-52.070537,1,nan,0.9496,0.0789,43.9616,0.017
674,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.6032,0.0466,42.7531,0.018
680,32.871094,-4.780192,166.959493,-60.615132,1,nan,1.4883,0.2383,45.1681,0.017
683,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.8370,0.0432,43.6235,0.011
686,358.665253,-45.783966,330.353593,-68.203652,1,nan,1.3769,0.2502,44.9594,0.009
687,150.468750,3.732834,235.392208,43.283244,1,nan,0.3617,0.2373,41.4273,0.020
694,34.453125,-5.229529,169.987075,-59.956185,1,0.4544,0.4391,0.0361,41.9241,0.019
699,0.589520,-47.161343,325.385896,-67.769893,1,0.5659,0.5592,0.0064,42.5537,0.009
721,358.665253,-45.783966,330.353593,-68.203652,1,0.4664,0.4355,0.3217,41.9033,0.009
725,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.7186,0.0156,43.2165,0.010
729,52.207031,-26.610098,221.298836,-55.042928,1,nan,0.7994,0.0124,43.5005,0.014
731,346.276581,-64.011238,320.448031,-49.344136,1,0.5315,0.5418,0.0087,42.4710,0.019
734,34.804688,-5.829153,171.307861,-60.174401,1,nan,0.9281,0.0119,43.9001,0.023
747,54.667969,-27.615883,223.610785,-53.050840,1,nan,1.3670,0.0656,44.9399,0.009
759,151.347656,3.583322,236.252362,43.918627,1,nan,1.5633,0.2334,45.2997,0.015
779,347.861847,-61.943836,321.519104,-51.424048,1,0.7469,0.7381,0.0118,43.2877,0.017
793,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.6826,0.0160,43.0802,0.011
810,35.332031,-5.979157,172.286722,-59.931743,1,nan,0.9306,0.0189,43.9073,0.022
830,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.6893,0.0081,43.1060,0.011
833,150.292969,2.686724,236.427488,42.541447,1,nan,0.4998,0.0202,42.2604,0.016
834,359.058563,-45.191612,330.695783,-68.844915,1,0.5767,0.5866,0.0128,42.6794,0.011
843,53.789062,-27.784405,223.685697,-53.845803,1,nan,0.8523,0.0367,43.6717,0.009
868,35.332031,-5.979157,172.286722,-59.931743,1,0.3881,0.3855,1.3203,41.5900,0.022
883,53.261719,-27.615883,223.280041,-54.281374,1,nan,0.8622,0.0603,43.7027,0.006
886,33.574219,-4.780192,168.064587,-60.175886,1,0.4615,0.4476,0.7778,41.9738,0.019
887,358.648071,-46.375080,329.462659,-67.716008,1,nan,1.2642,0.2378,44.7300,0.009
888,359.814819,-44.399834,330.775011,-69.801007,1,nan,1.2159,0.1488,44.6254,0.009
905,0.189873,-45.586655,328.254458,-68.969298,1,nan,0.6630,0.0275,43.0028,0.007
916,150.292969,2.686724,236.427488,42.541447,1,0.4052,0.4393,0.0217,41.9253,0.016
917,1.666667,-44.399834,327.519190,-70.529554,1,nan,1.3559,0.2676,44.9180,0.009
943,2.457983,-45.389202,324.632685,-69.945696,1,0.8582,0.8663,0.0291,43.7153,0.011
946,359.446716,-44.201530,331.730015,-69.805709,1,nan,1.5205,0.0959,45.2254,0.010
960,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.6480,0.0129,42.9424,0.023
962,150.820312,1.641510,237.994507,42.358984,1,nan,0.6779,0.0117,43.0616,0.020
965,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.6636,0.0081,43.0054,0.011
968,52.910156,-27.953188,223.774083,-54.639214,1,nan,0.2472,0.0116,40.4743,0.007
978,34.101562,-5.829153,170.247753,-60.638325,1,0.4938,0.4954,0.0349,42.2373,0.019
979,148.710938,2.836105,235.050801,41.328739,1,nan,1.1407,0.1072,44.4539,0.031
983,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.3106,0.0065,41.0429,0.014
1017,351.299988,-62.320400,319.038597,-52.026867,1,nan,1.4323,0.0481,45.0652,0.018
1018,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.9262,0.0111,43.8945,0.018
1020,149.238281,3.882372,234.283829,42.351155,1,nan,1.0264,0.2278,44.1701,0.033
1030,352.711273,-63.823658,316.922299,-51.059403,1,nan,1.2671,0.0235,44.7362,0.024
1039,150.644531,3.583322,235.698235,43.342784,1,nan,0.7124,0.0358,43.1936,0.018
1049,1.723404,-45.981140,325.117958,-69.180825,1,nan,1.2628,0.1822,44.7271,0.010
1059,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.0223,0.0247,44.1595,0.007
1063,53.789062,-27.784405,223.685697,-53.845803,1,nan,0.0000,0.0000,nan,0.009
1065,53.613281,-26.944359,222.237403,-53.863858,1,0.4795,0.4443,0.0390,41.9546,0.009
1067,2.071130,-45.191612,325.606223,-69.989264,1,0.7417,0.8350,0.0544,43.6171,0.011
1084,0.965665,-46.375080,325.845907,-68.579427,1,0.2126,0.5647,0.4351,42.5797,0.007
1087,352.132874,-63.636005,317.424173,-51.095855,1,nan,1.1039,0.1661,44.3657,0.021
1088,34.277344,-5.679190,170.314930,-60.410322,1,0.7550,0.7142,0.0477,43.2001,0.020
1100,34.101562,-5.829153,170.247753,-60.638325,1,0.9332,0.8409,0.0631,43.6357,0.019
1106,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.9258,0.0219,43.8935,0.022
1108,51.855469,-26.276812,220.627031,-55.293792,1,0.6459,0.7378,0.0318,43.2867,0.014
1111,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.3981,0.7822,41.6724,0.007
1114,33.398438,-4.331149,167.226341,-59.936551,1,0.5401,0.5514,0.0091,42.5169,0.018
1115,150.820312,3.732834,235.666318,43.572109,1,nan,1.1354,0.0545,44.4412,0.016
1123,151.171875,2.238686,237.619933,42.994783,1,0.5819,0.5516,0.3972,42.5177,0.024
1127,0.965665,-46.375080,325.845907,-68.579427,1,nan,1.0761,0.1787,44.2973,0.007
1128,149.414062,1.940072,236.565366,41.393323,1,nan,0.3618,0.0387,41.4282,0.018
1138,359.816315,-44.003082,331.451340,-70.123054,1,nan,1.4331,0.2745,45.0668,0.013
1151,33.574219,-4.780192,168.064587,-60.175886,1,nan,0.9335,0.2471,43.9155,0.019
1168,347.861847,-61.943836,321.519104,-51.424048,1,nan,1.5435,0.1436,45.2656,0.017
1174,0.949367,-45.586655,326.991548,-69.251686,1,nan,1.0215,0.1042,44.1572,0.013
1193,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.7156,0.0493,43.2055,0.017
1216,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.9028,0.0148,43.8260,0.009
1245,53.085938,-28.122234,224.100909,-54.509752,1,0.4937,0.5333,0.0166,42.4294,0.007
1254,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.5079,0.3209,45.2031,0.023
1265,54.667969,-27.615883,223.610785,-53.050840,1,0.5310,0.6159,0.0352,42.8082,0.009
1266,149.414062,1.940072,236.565366,41.393323,1,nan,0.6497,0.0183,42.9492,0.018
1271,33.750000,-4.630479,168.146242,-59.949072,1,nan,1.0088,0.0104,44.1239,0.019
1274,0.189873,-45.586655,328.254458,-68.969298,1,nan,1.4221,0.2779,45.0460,0.007
1288,33.222656,-4.780192,167.515653,-60.396584,1,nan,0.9169,0.1056,43.8676,0.018
1289,346.276581,-64.011238,320.448031,-49.344136,1,nan,1.1128,0.0552,44.3872,0.019
1304,347.846710,-64.760857,318.929827,-49.143596,1,0.3102,0.3081,0.0119,41.0226,0.019
1321,151.523438,3.134927,236.900695,43.803170,1,nan,1.2033,0.1759,44.5975,0.019
1347,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.7305,0.0458,43.2603,0.020
1354,53.085938,-28.122234,224.100909,-54.509752,1,nan,0.8265,0.0527,43.5896,0.007
1365,1.708861,-45.586655,325.688716,-69.520253,1,0.6978,0.7233,0.0170,43.2338,0.011
1380,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.9291,0.0148,43.9029,0.020
1388,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.8363,0.0411,43.6210,0.008
1393,152.050781,2.985506,237.495952,44.143927,1,0.4202,2.8977,0.8121,46.9354,0.019
1415,51.328125,-27.784405,223.130589,-55.999499,1,0.7371,0.7551,0.0216,43.3483,0.013
1421,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.6717,0.0415,43.0374,0.009
1439,34.101562,-5.829153,170.247753,-60.638325,1,nan,1.0442,0.1545,44.2165,0.019
1450,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.8016,0.0528,43.5077,0.009
1463,1.363636,-46.768478,324.669342,-68.371416,1,0.2333,0.5070,0.1130,42.2975,0.008
1467,151.523438,3.134927,236.900695,43.803170,1,nan,0.8282,0.1183,43.5952,0.019
1468,346.562500,-63.448284,320.824720,-49.866957,1,nan,0.8677,0.0290,43.7199,0.021
1469,149.414062,2.238686,236.239766,41.565558,1,nan,1.0850,0.0150,44.3193,0.017
1487,150.468750,3.732834,235.392208,43.283244,1,0.6295,0.5829,0.0311,42.6627,0.020
1492,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.7396,0.0157,43.2934,0.021
1498,152.050781,3.284369,237.157374,44.318466,1,nan,0.8187,0.0238,43.5644,0.019
1500,149.414062,2.238686,236.239766,41.565558,1,nan,1.0392,0.0139,44.2035,0.017
1522,352.132874,-63.636005,317.424173,-51.095855,1,nan,1.2524,0.0277,44.7049,0.021
1523,150.468750,1.641510,237.714575,42.075234,1,0.2669,0.2407,0.0254,40.4086,0.017
1536,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.5789,0.1445,42.6449,0.016
1545,346.562500,-63.448284,320.824720,-49.866957,1,0.6731,0.7236,0.0459,43.2351,0.021
1567,152.050781,2.985506,237.495952,44.143927,1,0.5581,0.5870,0.0381,42.6815,0.019
1570,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.7967,0.0516,43.4914,0.017
1578,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.5831,0.1709,42.6637,0.009
1589,51.679688,-27.447618,222.618229,-55.642263,1,0.3366,0.3400,0.0077,41.2708,0.010
1593,150.996094,4.181528,235.291975,43.970869,1,nan,1.2366,0.1482,44.6708,0.015
1597,351.299988,-62.320400,319.038597,-52.026867,1,0.3161,0.3182,0.0159,41.1033,0.018
1599,52.207031,-26.610098,221.298836,-55.042928,1,nan,1.0088,0.2519,44.1237,0.014
1600,34.101562,-5.829153,170.247753,-60.638325,1,0.5596,0.5490,0.0474,42.5054,0.019
1601,53.085938,-28.122234,224.100909,-54.509752,1,0.7155,0.7376,0.0244,43.2861,0.007
1619,2.097458,-45.783966,324.737840,-69.478613,1,nan,1.1271,0.0568,44.4216,0.011
1630,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.7373,0.0283,43.2851,0.014
1636,0.965665,-46.375080,325.845907,-68.579427,1,nan,0.9029,0.0862,43.8264,0.007
1660,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.8022,0.0194,43.5100,0.010
1667,33.574219,-5.079716,168.448505,-60.407218,1,nan,1.0792,0.0660,44.3050,0.016
1676,0.965665,-46.375080,325.845907,-68.579427,1,nan,0.9666,0.2691,44.0090,0.007
1678,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.7378,0.0146,43.2866,0.011
1687,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.9107,0.0140,43.8494,0.006
1691,152.050781,2.985506,237.495952,44.143927,1,0.7346,0.7037,0.0529,43.1609,0.019
1695,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.9741,0.2405,44.0299,0.020
1702,150.996094,2.388015,237.313912,42.939977,1,nan,0.7806,0.0916,43.4371,0.021
1720,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.6746,0.0276,43.0488,0.011
1729,51.328125,-27.447618,222.535046,-55.950727,1,0.6110,0.5932,0.0122,42.7092,0.013
1730,150.117188,3.732834,235.120533,42.993809,1,nan,0.6999,0.0471,43.1464,0.020
1754,148.710938,2.836105,235.050801,41.328739,1,nan,0.9109,0.0431,43.8500,0.031
1763,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.8860,0.0855,43.7757,0.017
1770,0.190678,-45.783966,327.956322,-68.803772,1,0.2562,0.2577,0.0118,40.5774,0.005
1780,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.3954,0.0285,41.6551,0.011
1781,351.382965,-64.011238,317.574052,-50.604657,1,0.4976,0.4790,0.0223,42.1493,0.023
1830,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.7349,0.0181,43.2763,0.013
1833,32.871094,-4.780192,166.959493,-60.615132,1,0.3202,0.3262,0.0567,41.1659,0.017
1841,0.589520,-47.161343,325.385896,-67.769893,1,nan,0.9916,0.0165,44.0776,0.009
1871,33.574219,-4.780192,168.064587,-60.175886,1,nan,0.5952,0.0260,42.7180,0.019
1894,150.468750,1.641510,237.714575,42.075234,1,nan,1.2779,0.0607,44.7590,0.017
1914,359.816315,-44.003082,331.451340,-70.123054,1,0.6648,0.6345,0.0370,42.8867,0.013
1933,150.117188,2.836105,236.124718,42.483719,1,nan,0.4788,0.2350,42.1486,0.016
1934,351.734680,-62.884678,318.284128,-51.651217,1,0.6909,0.7567,0.0533,43.3540,0.019
1941,52.207031,-26.610098,221.298836,-55.042928,1,nan,0.9255,0.0135,43.8926,0.014
1948,359.814819,-44.399834,330.775011,-69.801007,1,nan,1.2610,0.0630,44.7233,0.009
1949,33.398438,-3.732834,166.492280,-59.466614,1,0.5070,0.5068,0.0084,42.2963,0.022
1971,149.414062,1.940072,236.565366,41.393323,1,nan,1.1763,0.1964,44.5365,0.018
1978,54.667969,-27.615883,223.610785,-53.050840,1,nan,0.8494,0.0510,43.6626,0.009
1999,349.160583,-64.760857,318.219706,-49.458924,1,0.6580,0.5147,0.5022,42.3368,0.020
2000,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.6559,0.0240,42.9743,0.017
2017,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.3325,0.1725,41.2145,0.010
2021,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.6515,0.0269,42.9566,0.014
2046,150.644531,3.583322,235.698235,43.342784,1,0.7425,0.7898,0.0207,43.4684,0.018
2054,152.050781,2.985506,237.495952,44.143927,1,nan,0.7912,0.0208,43.4730,0.019
2074,359.446716,-44.201530,331.730015,-69.805709,1,0.5175,0.4921,0.0570,42.2198,0.010
2093,150.996094,4.181528,235.291975,43.970869,1,nan,0.3252,0.0452,41.1578,0.015
2097,151.347656,4.181528,235.568369,44.259942,1,nan,0.6327,0.0348,42.8789,0.016
2106,1.694561,-45.191612,326.278557,-69.858253,1,nan,1.1768,0.0524,44.5376,0.011
2109,358.665253,-45.783966,330.353593,-68.203652,1,0.4692,0.4797,0.0133,42.1532,0.009
2133,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.6612,0.0157,42.9957,0.023
2145,51.328125,-27.447618,222.535046,-55.950727,1,nan,1.2076,0.0605,44.6070,0.013
2147,0.589520,-47.161343,325.385896,-67.769893,1,nan,1.5213,0.2618,45.2269,0.009
2160,34.277344,-5.079716,169.526841,-59.956640,1,nan,0.5716,0.0242,42.6114,0.019
2168,359.446716,-44.201530,331.730015,-69.805709,1,0.2214,0.2308,0.2438,40.3055,0.010
2172,0.189873,-45.586655,328.254458,-68.969298,1,0.6266,0.6681,0.0264,43.0232,0.007
2182,151.523438,3.134927,236.900695,43.803170,1,nan,0.8239,0.0320,43.5812,0.019
2183,148.710938,2.836105,235.050801,41.328739,1,0.2383,0.2618,0.0166,40.6165,0.031
2187,34.101562,-5.829153,170.247753,-60.638325,1,0.3599,0.3656,0.5424,41.4546,0.019
2193,347.846710,-64.760857,318.929827,-49.143596,1,nan,1.4268,0.0800,45.0550,0.019
2195,150.468750,3.732834,235.392208,43.283244,1,nan,0.4598,0.0125,42.0435,0.020
2198,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.9277,0.2414,43.8988,0.023
2206,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.5014,0.0616,45.1915,0.010
2208,352.398651,-62.696659,318.017427,-51.967966,1,0.7188,0.7126,0.0121,43.1942,0.020
2223,358.648071,-46.375080,329.462659,-67.716008,1,nan,1.1672,0.0321,44.5157,0.009
2228,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.6837,0.0267,43.0844,0.014
2243,150.820312,3.134927,236.341348,43.230123,1,nan,1.3813,0.1932,44.9680,0.016
2246,348.529419,-61.755440,321.293980,-51.763351,1,0.6467,0.7143,0.0413,43.2008,0.016
2252,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.5851,0.0120,42.6728,0.008
2265,359.058563,-45.191612,330.695783,-68.844915,1,0.2786,0.2719,0.0249,40.7105,0.011
2270,151.171875,2.238686,237.619933,42.994783,1,nan,0.0000,0.0000,nan,0.024
2276,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.6827,0.0726,43.0806,0.019
2281,346.655182,-63.260487,320.952196,-50.040935,1,0.3442,0.5476,0.1871,42.4986,0.019
2287,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.5387,0.0152,42.4558,0.019
2292,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.9430,0.0086,43.9427,0.014
2293,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.9148,0.0376,43.8614,0.010
2296,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.8033,0.0310,43.5136,0.011
2299,352.132874,-63.636005,317.424173,-51.095855,1,nan,0.8109,0.0190,43.5386,0.021
2304,1.708861,-45.586655,325.688716,-69.520253,1,nan,1.3102,0.0895,44.8261,0.011
2333,349.966217,-62.696659,319.542989,-51.376556,1,0.4284,0.4611,0.0224,42.0509,0.021
2348,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.5904,0.0092,42.6966,0.018
2351,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.7537,0.1688,43.3435,0.018
2354,51.679688,-27.447618,222.618229,-55.642263,1,0.3829,0.3935,0.0180,41.6425,0.010
2370,35.683594,-5.379379,171.992947,-59.253501,1,nan,0.6099,0.0096,42.7820,0.020
2372,34.453125,-5.229529,169.987075,-59.956185,1,0.3353,0.4757,0.6757,42.1315,0.019
2388,54.667969,-27.615883,223.610785,-53.050840,1,nan,1.0298,0.0768,44.1791,0.009
2395,359.814819,-44.399834,330.775011,-69.801007,1,0.6315,0.6359,0.0276,42.8924,0.009
2407,0.190678,-45.783966,327.956322,-68.803772,1,0.5859,0.5249,0.0829,42.3879,0.005
2410,52.207031,-26.610098,221.298836,-55.042928,1,0.3848,0.4258,0.0241,41.8449,0.014
2451,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.7416,0.0343,43.3003,0.009
2476,149.414062,1.940072,236.565366,41.393323,1,nan,0.4730,0.1009,42.1166,0.018
2538,346.130127,-63.072620,321.423103,-50.042305,1,0.6846,0.6643,0.0319,43.0081,0.020
2550,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.3640,0.1891,41.4439,0.020
2603,0.965665,-46.375080,325.845907,-68.579427,1,nan,1.2175,0.4670,44.6290,0.007
2615,52.910156,-26.276812,220.926149,-54.363918,1,nan,1.2323,0.0534,44.6613,0.008
2660,151.523438,3.134927,236.900695,43.803170,1,1.0412,1.3254,0.1765,44.8570,0.019
2661,346.130127,-63.072620,321.423103,-50.042305,1,0.5160,0.5185,0.0319,42.3560,0.020
2683,0.965665,-46.375080,325.845907,-68.579427,1,0.6609,0.6757,0.0091,43.0530,0.007
2687,348.529419,-61.755440,321.293980,-51.763351,1,0.6669,0.6531,0.0406,42.9628,0.016
2693,358.665253,-45.783966,330.353593,-68.203652,1,nan,1.5860,0.2426,45.3384,0.009
2701,349.891296,-64.573555,317.972107,-49.786192,1,0.4057,0.3811,0.0322,41.5609,0.023
2702,35.683594,-5.379379,171.992947,-59.253501,1,0.5081,0.5489,0.0165,42.5049,0.020
2707,33.398438,-4.331149,167.226341,-59.936551,1,0.1061,0.1359,0.0193,39.0322,0.018
2744,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.5735,0.0245,42.6203,0.021
2753,51.328125,-27.784405,223.130589,-55.999499,1,0.4302,0.3267,0.0389,41.1697,0.013
2759,148.886719,2.686724,235.347248,41.389003,1,nan,0.5974,0.0069,42.7274,0.028
2760,149.414062,3.433834,234.919132,42.245550,1,nan,0.3415,0.2635,41.2821,0.027
2766,150.468750,3.732834,235.392208,43.283244,1,nan,1.0362,0.0118,44.1956,0.020
2786,349.615387,-63.636005,318.927246,-50.506542,1,nan,0.3010,0.0504,40.9634,0.018
2790,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.0000,0.0000,nan,0.018
2794,359.415588,-46.768478,327.729895,-67.686097,1,0.6983,0.7077,0.0266,43.1760,0.009
2814,150.117188,2.836105,236.124718,42.483719,1,nan,0.7215,0.0782,43.2272,0.016
2818,52.207031,-28.291550,224.208534,-55.300157,1,nan,0.8251,0.0216,43.5851,0.007
2856,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.6348,0.0515,42.8877,0.009
2858,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.1145,0.0322,44.3914,0.023
2860,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.4398,0.2606,41.9285,0.016
2871,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.4647,0.0229,42.0709,0.019
2882,151.171875,1.342993,238.602520,42.464379,1,nan,2.7824,0.2944,46.8290,0.026
2886,148.710938,2.836105,235.050801,41.328739,1,nan,0.3475,0.0073,41.3259,0.031
2909,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.3229,0.3149,41.1406,0.010
2932,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.1911,0.0104,39.8466,0.024
2933,348.586945,-64.573555,318.693903,-49.477869,1,0.6336,0.6157,0.0140,42.8070,0.018
2943,0.589520,-47.161343,325.385896,-67.769893,1,0.7816,0.7511,0.0242,43.3344,0.009
2945,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.1553,0.0720,44.4880,0.010
2949,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.2125,0.0257,44.6178,0.010
2958,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.5145,0.0383,42.3356,0.009
2961,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.3771,0.0426,41.5336,0.019
2962,149.414062,3.433834,234.919132,42.245550,1,nan,0.7453,0.0637,43.3138,0.027
2970,53.085938,-27.111860,222.384291,-54.355086,1,0.4506,0.5123,0.0647,42.3247,0.007
2975,53.085938,-28.122234,224.100909,-54.509752,1,nan,1.4981,0.2615,45.1857,0.007
3004,349.966217,-62.696659,319.542989,-51.376556,1,0.4053,0.4978,0.0481,42.2498,0.021
3008,358.312500,-44.993881,332.185785,-68.685906,1,nan,0.1344,0.7139,39.0065,0.009
3012,150.117188,2.238686,236.784618,42.139082,1,nan,1.4853,0.0818,45.1627,0.016
3022,52.207031,-28.630989,224.800211,-55.343637,1,0.4266,0.3630,0.0443,41.4365,0.009
3025,359.415588,-46.768478,327.729895,-67.686097,1,0.6636,0.6827,0.0129,43.0806,0.009
3028,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.6075,0.2450,42.7719,0.017
3029,347.617462,-62.508568,321.121462,-50.904708,1,nan,0.7168,0.0344,43.2100,0.019
3037,150.820312,3.732834,235.666318,43.572109,1,0.3807,0.3856,0.0770,41.5907,0.016
3052,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.9395,0.0475,43.9327,0.019
3073,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.5457,0.7707,42.4896,0.014
3077,0.189873,-45.586655,328.254458,-68.969298,1,0.2853,0.3011,0.0497,40.9649,0.007
3133,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.6167,0.0739,45.3897,0.006
3163,34.277344,-5.079716,169.526841,-59.956640,1,0.6388,0.6322,0.0085,42.8770,0.019
3170,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.4754,0.0150,42.1298,0.018
3171,33.925781,-5.979157,170.179895,-60.866303,1,0.5550,0.5525,0.0096,42.5220,0.022
3175,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.0217,0.0243,44.1579,0.019
3176,150.820312,3.134927,236.341348,43.230123,1,nan,0.7794,0.0377,43.4329,0.016
3190,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.7705,0.0220,43.4023,0.022
3193,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.8700,0.0495,43.7269,0.021
3196,358.648071,-46.375080,329.462659,-67.716008,1,0.3112,0.2901,0.0115,40.8716,0.009
3206,53.085938,-27.111860,222.384291,-54.355086,1,0.6472,0.6860,0.0225,43.0931,0.007
3209,1.363636,-46.768478,324.669342,-68.371416,1,0.4369,0.4994,1.2314,42.2578,0.008
3210,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.7133,0.0739,43.1968,0.008
3214,348.908447,-63.823658,319.169886,-50.176186,1,0.2600,0.2876,0.3195,40.8500,0.018
3217,151.347656,4.181528,235.568369,44.259942,1,0.1826,0.4846,0.3727,42.1795,0.016
3247,1.708861,-45.586655,325.688716,-69.520253,1,nan,1.3152,0.1148,44.8363,0.011
3252,349.615387,-63.636005,318.927246,-50.506542,1,nan,0.8013,0.0111,43.5070,0.018
3278,53.613281,-27.953188,223.929533,-54.024772,1,nan,1.5343,0.1417,45.2497,0.007
3284,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.4121,1.2419,41.7612,0.011
3286,33.574219,-4.780192,168.064587,-60.175886,1,nan,0.7985,0.0224,43.4974,0.019
3288,51.855469,-26.276812,220.627031,-55.293792,1,nan,1.2928,0.0725,44.7902,0.014
3325,53.613281,-28.630989,225.073365,-54.119461,1,0.4825,0.5158,0.0204,42.3423,0.006
3336,34.453125,-5.229529,169.987075,-59.956185,1,nan,0.7542,0.0326,43.3453,0.019
3347,52.910156,-27.953188,223.774083,-54.639214,1,nan,0.5940,0.0536,42.7127,0.007
3351,359.811707,-45.191612,329.485675,-69.150905,1,0.5478,0.5828,0.0191,42.6626,0.010
3362,347.617462,-62.508568,321.121462,-50.904708,1,nan,0.6700,0.1306,43.0306,0.019
3373,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.4474,0.2471,41.9728,0.020
3395,53.613281,-27.953188,223.929533,-54.024772,1,0.3963,0.3657,0.8633,41.4551,0.007
3398,34.804688,-5.829153,171.307861,-60.174401,1,nan,1.0070,0.0407,44.1188,0.023
3410,349.615387,-63.636005,318.927246,-50.506542,1,nan,1.1667,0.0446,44.5144,0.018
3418,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.0109,0.0153,44.1294,0.005
3441,0.189873,-45.586655,328.254458,-68.969298,1,nan,0.4974,0.0111,42.2477,0.007
3457,151.347656,4.181528,235.568369,44.259942,1,nan,0.7204,0.0318,43.2232,0.016
3475,149.414062,3.433834,234.919132,42.245550,1,nan,0.7214,0.0251,43.2270,0.027
3498,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.9131,0.0795,43.8565,0.019
3499,150.996094,2.985506,236.647967,43.287350,1,0.9971,1.0190,0.0171,44.1508,0.020
3500,52.207031,-28.630989,224.800211,-55.343637,1,0.2306,0.4849,0.6744,42.1816,0.009
3504,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.5829,0.0306,42.6630,0.019
3509,53.261719,-27.615883,223.280041,-54.281374,1,0.5035,0.5169,0.2812,42.3478,0.006
3522,52.910156,-27.279613,222.625192,-54.536648,1,0.8636,0.9140,0.0250,43.8590,0.007
3529,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.9039,0.0247,43.8293,0.018
3535,359.814819,-44.399834,330.775011,-69.801007,1,nan,1.4756,0.1618,45.1450,0.009
3538,150.996094,2.985506,236.647967,43.287350,1,0.4615,0.4664,0.0103,42.0806,0.020
3547,2.097458,-45.783966,324.737840,-69.478613,1,nan,0.9467,0.0338,43.9533,0.011
3551,358.636353,-46.768478,328.890146,-67.388837,1,0.2573,0.2393,0.0145,40.3948,0.008
3566,151.171875,2.238686,237.619933,42.994783,1,0.6331,0.5448,0.6502,42.4852,0.024
3581,51.855469,-26.276812,220.627031,-55.293792,1,0.4844,0.4938,0.0183,42.2287,0.014
3585,150.820312,3.732834,235.666318,43.572109,1,nan,0.5231,0.1034,42.3791,0.016
3597,2.457983,-45.389202,324.632685,-69.945696,1,0.8409,0.8742,0.0306,43.7396,0.011
3620,35.859375,-4.630479,171.270769,-58.580806,1,nan,0.7190,0.0186,43.2181,0.022
3629,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.3951,0.1453,44.9947,0.009
3641,149.414062,3.433834,234.919132,42.245550,1,nan,0.5965,0.0131,42.7237,0.027
3645,346.562500,-63.448284,320.824720,-49.866957,1,0.5351,0.5274,0.0205,42.4003,0.021
3652,149.238281,3.882372,234.283829,42.351155,1,nan,0.8499,0.0477,43.6644,0.033
3657,34.277344,-5.679190,170.314930,-60.410322,1,0.3050,2.7750,0.9745,46.8221,0.020
3661,347.812500,-63.448284,320.128971,-50.202348,1,nan,1.3297,0.0487,44.8658,0.021
3666,150.117188,2.836105,236.124718,42.483719,1,nan,1.1820,0.1059,44.5495,0.016
3672,348.529419,-61.755440,321.293980,-51.763351,1,0.6020,0.5785,0.0108,42.6431,0.016
3681,150.996094,2.985506,236.647967,43.287350,1,0.6459,0.6429,0.0113,42.9215,0.020
3702,151.171875,2.238686,237.619933,42.994783,1,nan,0.8082,0.1270,43.5299,0.024
3706,51.328125,-27.784405,223.130589,-55.999499,1,0.3284,0.3150,0.1745,41.0780,0.013
3709,151.171875,2.238686,237.619933,42.994783,1,nan,0.8753,0.0139,43.7432,0.024
3710,1.666667,-44.399834,327.519190,-70.529554,1,nan,0.8356,0.0335,43.6190,0.009
3735,0.574468,-45.981140,327.041068,-68.778764,1,nan,0.7982,0.1565,43.4967,0.006
3737,150.292969,2.686724,236.427488,42.541447,1,nan,0.7567,0.0404,43.3543,0.016
3748,52.910156,-25.944481,220.366350,-54.301439,1,nan,1.0812,0.0621,44.3099,0.010
3749,359.415588,-46.768478,327.729895,-67.686097,1,0.7033,0.7840,0.0371,43.4487,0.009
3751,352.132874,-63.636005,317.424173,-51.095855,1,0.6527,0.6351,0.0190,42.8890,0.021
3753,346.562500,-63.448284,320.824720,-49.866957,1,0.1191,0.0761,0.0195,37.6870,0.021
3763,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.3349,0.0091,41.2320,0.017
3764,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.8530,0.0218,43.6740,0.021
3777,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.5510,0.0652,42.5152,0.016
3779,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.9939,0.2200,44.0838,0.021
3785,359.415588,-46.768478,327.729895,-67.686097,1,0.5052,0.5168,0.0229,42.3475,0.009
3794,52.910156,-26.276812,220.926149,-54.363918,1,0.6236,0.6316,0.0331,42.8743,0.008
3803,151.171875,1.342993,238.602520,42.464379,1,nan,0.7038,0.0688,43.1613,0.026
3821,52.910156,-27.953188,223.774083,-54.639214,1,nan,1.4631,0.2823,45.1222,0.007
3822,358.648071,-46.375080,329.462659,-67.716008,1,0.3455,2.9896,1.2696,47.0170,0.009
3824,1.753247,-46.768478,324.030235,-68.498041,1,0.8063,0.8352,0.0339,43.6178,0.014
3841,2.071130,-45.191612,325.606223,-69.989264,1,nan,0.5971,0.0990,42.7263,0.011
3844,2.457983,-45.389202,324.632685,-69.945696,1,0.4789,0.3748,0.4652,41.5181,0.011
3855,349.891296,-64.573555,317.972107,-49.786192,1,0.4507,0.3704,0.5053,41.4880,0.023
3857,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.7254,0.0296,43.2418,0.011
3875,52.207031,-28.630989,224.800211,-55.343637,1,0.6825,0.6468,0.0279,42.9374,0.009
3878,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.9110,0.2538,43.8503,0.008
3880,351.382965,-64.011238,317.574052,-50.604657,1,nan,1.5002,0.2529,45.1894,0.023
3889,53.964844,-28.630989,225.142950,-53.813613,1,0.1582,0.2015,1.1876,39.9743,0.009
3925,150.996094,4.181528,235.291975,43.970869,1,nan,0.2421,1.1561,40.4235,0.015
3928,150.996094,2.985506,236.647967,43.287350,1,nan,0.7806,0.0538,43.4371,0.020
3929,149.589844,3.583322,234.885369,42.474696,1,nan,1.8897,0.1427,45.8062,0.024
3936,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.0853,0.0192,44.3200,0.023
3950,352.132874,-63.636005,317.424173,-51.095855,1,0.7004,0.6924,0.0235,43.1180,0.021
3970,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.4637,0.1090,45.1234,0.014
3972,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.7985,0.0333,43.4976,0.016
4001,150.996094,4.181528,235.291975,43.970869,1,nan,1.5533,0.1631,45.2826,0.015
4007,150.820312,3.732834,235.666318,43.572109,1,nan,0.7855,0.0241,43.4536,0.016
4016,346.130127,-63.072620,321.423103,-50.042305,1,nan,1.2676,0.0738,44.7372,0.020
4023,0.589520,-47.161343,325.385896,-67.769893,1,nan,0.9574,0.1398,43.9835,0.009
4025,33.222656,-4.780192,167.515653,-60.396584,1,nan,0.0000,0.0000,nan,0.018
4038,35.859375,-4.630479,171.270769,-58.580806,1,nan,1.1349,0.0866,44.4401,0.022
4044,151.171875,2.238686,237.619933,42.994783,1,nan,0.6693,0.0222,43.0278,0.024
4054,34.453125,-5.229529,169.987075,-59.956185,1,0.5653,0.5478,0.0237,42.4998,0.019
4062,358.312500,-44.993881,332.185785,-68.685906,1,nan,1.1896,0.0988,44.5666,0.009
4063,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.6406,0.1744,42.9119,0.018
4065,35.332031,-5.979157,172.286722,-59.931743,1,nan,1.3023,0.0984,44.8098,0.022
4077,150.468750,1.641510,237.714575,42.075234,1,nan,1.1662,0.1634,44.5133,0.017
4103,349.160583,-64.760857,318.219706,-49.458924,1,0.3113,0.3331,0.0201,41.2185,0.020
4109,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.0000,0.0000,nan,0.008
4183,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.8906,0.0209,43.7897,0.009
4197,54.667969,-27.615883,223.610785,-53.050840,1,nan,0.7358,0.0471,43.2794,0.009
4201,34.453125,-5.229529,169.987075,-59.956185,1,0.2193,0.2338,0.1899,40.3374,0.019
4202,33.398438,-3.732834,166.492280,-59.466614,1,0.6916,0.7529,0.0348,43.3408,0.022
4216,53.964844,-28.630989,225.142950,-53.813613,1,0.4807,0.4515,0.0246,41.9964,0.009
4224,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.6240,0.0370,42.8426,0.022
4225,346.562500,-63.448284,320.824720,-49.866957,1,0.2704,0.3546,0.2924,41.3770,0.021
4249,51.679688,-27.447618,222.618229,-55.642263,1,nan,1.3003,0.1720,44.8058,0.010
4262,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.7108,0.0343,43.1877,0.020
4267,33.222656,-4.780192,167.515653,-60.396584,1,0.4186,0.3797,0.0298,41.5512,0.018
4272,53.613281,-26.944359,222.237403,-53.863858,1,0.2892,0.2718,0.0268,40.7089,0.009
4274,351.321442,-64.198746,317.458993,-50.429931,1,0.6479,0.6553,0.0091,42.9721,0.023
4278,149.414062,2.238686,236.239766,41.565558,1,nan,0.8546,0.0815,43.6790,0.017
4282,53.085938,-27.784405,223.525509,-54.460748,1,nan,0.4929,0.0100,42.2237,0.007
4283,52.910156,-27.279613,222.625192,-54.536648,1,0.3708,0.3947,0.0245,41.6505,0.007
4293,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.6305,0.1495,45.4125,0.014
4294,150.117188,2.836105,236.124718,42.483719,1,nan,0.8510,0.1009,43.6678,0.016
4300,51.855469,-26.276812,220.627031,-55.293792,1,0.6127,0.6028,0.0088,42.7511,0.014
4303,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.9797,0.0098,44.0451,0.018
4304,34.277344,-5.679190,170.314930,-60.410322,1,nan,0.2127,0.0101,40.1063,0.020
4306,151.171875,2.537361,237.288526,43.169764,1,nan,1.4930,0.1628,45.1765,0.024
4313,149.414062,3.433834,234.919132,42.245550,1,nan,1.1395,0.1626,44.4510,0.027
4322,151.347656,4.181528,235.568369,44.259942,1,0.7816,0.7648,0.0184,43.3824,0.016
4325,34.980469,-6.279288,172.180075,-60.389399,1,nan,0.6802,0.0118,43.0708,0.023
4330,53.613281,-28.630989,225.073365,-54.119461,1,nan,1.1409,0.1366,44.4543,0.006
4337,34.804688,-5.829153,171.307861,-60.174401,1,nan,0.6082,0.0100,42.7747,0.023
4347,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.5051,0.0376,45.1982,0.020
4369,53.964844,-28.630989,225.142950,-53.813613,1,nan,1.1974,0.0630,44.5843,0.009
4372,351.259003,-64.386185,317.344860,-50.255113,1,0.8588,0.8990,0.0160,43.8146,0.020
4376,33.222656,-4.780192,167.515653,-60.396584,1,nan,0.9810,0.0139,44.0488,0.018
4390,53.437500,-29.142223,225.908120,-54.336118,1,nan,1.0152,0.3139,44.1407,0.008
4424,151.699219,3.583322,236.533224,44.205648,1,nan,1.0317,0.0256,44.1840,0.016
4467,53.261719,-27.615883,223.280041,-54.281374,1,0.3489,0.3023,0.1233,40.9750,0.006
4469,150.820312,1.641510,237.994507,42.358984,1,nan,1.0352,0.2064,44.1930,0.020
4475,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.5952,0.0666,42.7177,0.014
4489,35.332031,-5.979157,172.286722,-59.931743,1,nan,0.3478,0.0152,41.3280,0.022
4490,1.723404,-45.981140,325.117958,-69.180825,1,nan,1.1304,0.0573,44.4295,0.010
4492,358.636353,-46.768478,328.890146,-67.388837,1,0.2686,0.6256,0.1162,42.8492,0.008
4494,2.097458,-45.783966,324.737840,-69.478613,1,0.4329,0.4492,0.0147,41.9831,0.011
4507,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.1883,0.0370,44.5636,0.020
4508,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.7777,0.0513,43.4272,0.021
4525,149.589844,3.583322,234.885369,42.474696,1,nan,0.7945,0.0729,43.4840,0.024
4528,359.811707,-45.191612,329.485675,-69.150905,1,0.6181,0.5632,0.0527,42.5723,0.010
4533,150.996094,2.985506,236.647967,43.287350,1,nan,0.9019,0.2483,43.8233,0.020
4551,358.312500,-44.993881,332.185785,-68.685906,1,0.2666,0.6085,0.0540,42.7761,0.009
4559,0.929752,-44.597992,328.531426,-70.083244,1,0.3229,0.3027,0.6578,40.9778,0.011
4561,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.7697,0.0339,43.3995,0.019
4576,150.292969,2.686724,236.427488,42.541447,1,0.5613,0.5636,0.0277,42.5742,0.016
4586,2.071130,-45.191612,325.606223,-69.989264,1,0.3810,0.3822,0.0308,41.5678,0.011
4589,150.820312,1.641510,237.994507,42.358984,1,nan,1.2748,0.1135,44.7525,0.020
4592,52.207031,-26.610098,221.298836,-55.042928,1,0.7073,0.7753,0.0429,43.4190,0.014
4597,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.6120,0.0070,42.7913,0.018
4611,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.8291,0.0669,43.5981,0.007
4625,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.7723,0.0357,43.4086,0.021
4644,35.859375,-4.630479,171.270769,-58.580806,1,0.4138,0.4027,0.0331,41.7017,0.022
4645,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.6546,0.1405,42.9690,0.009
4653,150.292969,2.686724,236.427488,42.541447,1,nan,0.9818,0.0169,44.0509,0.016
4673,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.5303,0.0066,42.4145,0.024
4677,53.085938,-27.784405,223.525509,-54.460748,1,0.3093,0.1395,0.2071,39.0948,0.007
4695,349.966217,-62.696659,319.542989,-51.376556,1,0.6195,0.6168,0.0156,42.8120,0.021
4707,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.8457,0.0478,43.6511,0.009
4712,0.190678,-45.783966,327.956322,-68.803772,1,0.1206,0.5167,0.0880,42.3468,0.005
4713,358.665253,-45.783966,330.353593,-68.203652,1,0.8042,0.7677,0.0483,43.3928,0.009
4719,0.965665,-46.375080,325.845907,-68.579427,1,0.3619,0.3727,0.0303,41.5035,0.007
4720,359.805206,-46.768478,327.135979,-67.829903,1,0.5530,0.5247,0.1218,42.3871,0.011
4724,352.398651,-62.696659,318.017427,-51.967966,1,0.5390,0.4826,0.1275,42.1692,0.020
4738,35.332031,-5.979157,172.286722,-59.931743,1,nan,1.0768,0.0572,44.2988,0.022
4739,51.855469,-26.276812,220.627031,-55.293792,1,nan,0.7661,0.0148,43.3871,0.014
4750,0.965665,-46.375080,325.845907,-68.579427,1,0.4822,0.2376,0.3459,40.3769,0.007
4759,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.7922,0.2472,43.4764,0.010
4760,350.230255,-61.943836,320.053946,-52.070537,1,nan,1.1776,0.0799,44.5394,0.017
4762,51.679688,-27.447618,222.618229,-55.642263,1,nan,1.0371,0.2322,44.1980,0.010
4768,349.285706,-62.884678,319.786163,-51.046461,1,nan,1.0012,0.0457,44.1033,0.018
4776,150.117188,2.238686,236.784618,42.139082,1,0.4602,0.4465,0.0429,41.9677,0.016
4812,151.171875,1.342993,238.602520,42.464379,1,nan,0.8278,0.0161,43.5940,0.026
4822,149.589844,3.583322,234.885369,42.474696,1,nan,0.8752,0.0381,43.7427,0.024
4824,346.655182,-63.260487,320.952196,-50.040935,1,0.4116,0.3961,1.0457,41.6592,0.019
4825,150.468750,3.732834,235.392208,43.283244,1,nan,0.8466,0.0165,43.6540,0.020
4830,1.666667,-44.399834,327.519190,-70.529554,1,nan,0.8267,0.0561,43.5904,0.009
4833,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.6975,0.0469,43.1373,0.011
4834,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.6570,0.0487,42.9789,0.008
4840,151.171875,2.537361,237.288526,43.169764,1,nan,0.9905,0.0337,44.0747,0.024
4844,34.277344,-5.679190,170.314930,-60.410322,1,nan,1.0144,0.1816,44.1385,0.020
4853,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.8895,0.1178,43.7862,0.017
4854,0.949367,-45.586655,326.991548,-69.251686,1,nan,0.7034,0.0178,43.1597,0.013
4864,151.171875,1.342993,238.602520,42.464379,1,0.1623,0.1825,0.0221,39.7354,0.026
4866,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.6113,0.0972,42.7883,0.019
4903,151.171875,2.238686,237.619933,42.994783,1,nan,0.4125,0.0223,41.7635,0.024
4910,1.723404,-45.981140,325.117958,-69.180825,1,nan,0.6558,0.0453,42.9739,0.010
4934,52.031250,-26.443335,220.963669,-55.168557,1,nan,1.2051,0.0888,44.6015,0.014
4937,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.5072,0.0257,42.2985,0.010
4948,151.171875,2.537361,237.288526,43.169764,1,nan,1.0614,0.0814,44.2604,0.024
4986,32.871094,-4.780192,166.959493,-60.615132,1,0.5554,0.4904,0.4394,42.2108,0.017
4990,149.414062,1.940072,236.565366,41.393323,1,nan,0.6203,0.0205,42.8268,0.018
4994,150.468750,1.641510,237.714575,42.075234,1,nan,0.8252,0.0451,43.5856,0.017
5024,352.711273,-63.823658,316.922299,-51.059403,1,0.3343,0.5197,0.1503,42.3618,0.024
5034,346.276581,-64.011238,320.448031,-49.344136,1,nan,0.2533,0.8554,40.5348,0.019
5057,150.468750,3.732834,235.392208,43.283244,1,0.8187,0.7582,0.0593,43.3595,0.020
5061,150.292969,2.686724,236.427488,42.541447,1,0.4812,0.4838,0.0111,42.1754,0.016
5084,52.207031,-26.610098,221.298836,-55.042928,1,nan,1.0849,0.0363,44.3190,0.014
5088,349.160583,-64.760857,318.219706,-49.458924,1,0.7677,0.7828,0.0261,43.4447,0.020
5094,2.071130,-45.191612,325.606223,-69.989264,1,nan,0.3655,0.0077,41.4542,0.011
5099,1.753247,-46.768478,324.030235,-68.498041,1,nan,0.3400,0.2409,41.2708,0.014
5135,33.925781,-5.979157,170.179895,-60.866303,1,0.7377,0.7425,0.0053,43.3035,0.022
5145,150.468750,1.641510,237.714575,42.075234,1,nan,1.1364,0.0385,44.4437,0.017
5153,149.414062,2.238686,236.239766,41.565558,1,0.7765,0.7695,0.0155,43.3989,0.017
5158,359.415588,-46.768478,327.729895,-67.686097,1,0.4004,0.4266,0.0302,41.8498,0.009
5162,52.558594,-27.279613,222.538937,-54.845107,1,nan,0.5872,0.0094,42.6823,0.008
5166,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7222,0.0331,43.2299,0.008
5176,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.8147,0.0386,43.5513,0.009
5204,33.574219,-4.780192,168.064587,-60.175886,1,0.4879,0.5068,0.0211,42.2965,0.019
5210,33.574219,-4.780192,168.064587,-60.175886,1,0.4971,0.4954,0.0328,42.2371,0.019
5217,53.261719,-27.615883,223.280041,-54.281374,1,nan,1.6018,0.0726,45.3649,0.006
5234,51.328125,-27.784405,223.130589,-55.999499,1,0.7616,0.8177,0.0589,43.5609,0.013
5236,35.332031,-5.979157,172.286722,-59.931743,1,0.3651,0.4588,0.1365,42.0379,0.022
5249,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.9021,0.0339,43.8239,0.018
5264,347.617462,-62.508568,321.121462,-50.904708,1,nan,0.6893,0.0103,43.1059,0.019
5278,53.789062,-27.784405,223.685697,-53.845803,1,0.4394,0.3893,0.1926,41.6151,0.009
5280,351.299988,-62.320400,319.038597,-52.026867,1,nan,1.2309,0.2137,44.6583,0.018
5283,347.013428,-62.508568,321.472056,-50.735330,1,nan,0.6296,0.1513,42.8661,0.018
5286,347.846710,-64.760857,318.929827,-49.143596,1,0.5384,0.4991,0.1472,42.2564,0.019
5307,2.457983,-45.389202,324.632685,-69.945696,1,0.4932,0.4967,0.0234,42.2437,0.011
5313,349.891296,-64.573555,317.972107,-49.786192,1,0.3924,0.4004,0.3110,41.6868,0.023
5315,359.805206,-46.768478,327.135979,-67.829903,1,nan,0.7107,0.0354,43.1872,0.011
5317,33.574219,-5.379379,168.838090,-60.637536,1,0.2383,0.2830,0.1436,40.8096,0.017
5319,148.710938,2.836105,235.050801,41.328739,1,nan,0.5440,0.0154,42.4816,0.031
5322,0.929752,-44.597992,328.531426,-70.083244,1,nan,1.0052,0.0257,44.1143,0.011
5335,348.908447,-63.823658,319.169886,-50.176186,1,0.2108,0.2283,0.1361,40.2787,0.018
5338,152.050781,3.284369,237.157374,44.318466,1,nan,0.8439,0.0429,43.6452,0.019
5354,34.980469,-6.279288,172.180075,-60.389399,1,0.2692,0.3663,0.1994,41.4598,0.023
5365,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.9173,0.0221,43.8686,0.018
5370,358.636353,-46.768478,328.890146,-67.388837,1,0.7437,0.6297,0.0526,42.8666,0.008
5372,348.908447,-63.823658,319.169886,-50.176186,1,0.5219,0.5300,0.0576,42.4134,0.018
5386,34.453125,-5.229529,169.987075,-59.956185,1,nan,0.6316,0.0102,42.8746,0.019
5403,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.9827,0.0233,44.0533,0.009
5412,151.347656,4.181528,235.568369,44.259942,1,0.3435,0.3761,0.2378,41.5268,0.016
5417,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.7730,0.0296,43.4109,0.017
5419,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.5571,0.0159,42.5440,0.008
5422,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.1195,0.1674,44.4033,0.005
5433,53.085938,-28.122234,224.100909,-54.509752,1,nan,1.1700,0.1749,44.5221,0.007
5453,150.117188,3.732834,235.120533,42.993809,1,nan,0.5035,0.3843,42.2794,0.020
5473,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.5390,0.0204,42.4575,0.008
5478,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.9349,0.0204,43.9195,0.018
5520,152.050781,2.985506,237.495952,44.143927,1,0.2992,0.2421,0.0411,40.4233,0.019
5521,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.9909,0.0426,44.0758,0.013
5535,53.261719,-27.615883,223.280041,-54.281374,1,nan,0.9054,0.0151,43.8337,0.006
5543,33.398438,-3.732834,166.492280,-59.466614,1,nan,0.8296,0.1240,43.5996,0.022
5550,151.171875,1.342993,238.602520,42.464379,1,nan,0.5898,0.0156,42.6937,0.026
5562,351.734680,-62.884678,318.284128,-51.651217,1,0.3430,0.4407,0.1538,41.9338,0.019
5564,351.321442,-64.198746,317.458993,-50.429931,1,nan,0.6240,0.0197,42.8426,0.023
5565,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.7586,0.0376,43.3607,0.011
5566,347.013428,-62.508568,321.472056,-50.735330,1,nan,0.9684,0.0328,44.0142,0.018
5569,33.574219,-5.079716,168.448505,-60.407218,1,0.2696,0.2658,0.6279,40.6536,0.016
5570,347.861847,-61.943836,321.519104,-51.424048,1,nan,0.6613,0.0352,42.9962,0.017
5576,0.589520,-47.161343,325.385896,-67.769893,1,nan,1.4255,0.1860,45.0525,0.009
5591,350.230255,-61.943836,320.053946,-52.070537,1,nan,0.7811,0.0143,43.4387,0.017
5623,1.753247,-46.768478,324.030235,-68.498041,1,0.3915,0.4107,1.0343,41.7522,0.014
5635,359.415588,-46.768478,327.729895,-67.686097,1,nan,1.0468,0.1246,44.2231,0.009
5655,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.4730,0.2564,45.1403,0.014
5656,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.9835,0.0168,44.0555,0.011
5661,35.683594,-5.379379,171.992947,-59.253501,1,nan,0.2668,0.2983,40.6628,0.020
5696,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.8087,0.0795,43.5316,0.018
5704,350.230255,-61.943836,320.053946,-52.070537,1,nan,0.5054,0.0636,42.2892,0.017
5723,351.321442,-64.198746,317.458993,-50.429931,1,nan,0.8582,0.1911,43.6902,0.023
5729,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.3930,0.0637,44.9906,0.007
5758,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.9711,0.0522,44.0216,0.010
5761,150.468750,1.641510,237.714575,42.075234,1,nan,0.9866,0.2871,44.0641,0.017
5763,349.891296,-64.573555,317.972107,-49.786192,1,nan,1.0138,0.2387,44.1370,0.023
5794,351.299988,-62.320400,319.038597,-52.026867,1,nan,1.3563,0.0554,44.9189,0.018
5798,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.8854,0.0513,43.7737,0.013
5801,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.8968,0.0208,43.8080,0.006
5817,34.277344,-5.079716,169.526841,-59.956640,1,0.5298,0.5217,0.0200,42.3721,0.019
5834,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.8078,0.0389,43.5284,0.020
5844,2.097458,-45.783966,324.737840,-69.478613,1,nan,1.1469,0.0339,44.4683,0.011
5852,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.8192,0.0304,43.5659,0.005
5864,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.3871,0.1040,44.9791,0.009
5877,151.347656,4.181528,235.568369,44.259942,1,0.3096,0.3166,0.7338,41.0910,0.016
5881,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.5213,0.0134,42.3702,0.018
5895,348.908447,-63.823658,319.169886,-50.176186,1,nan,0.8744,0.0299,43.7403,0.018
5911,33.750000,-4.630479,168.146242,-59.949072,1,0.2117,0.5363,0.4038,42.4441,0.019
5922,1.694561,-45.191612,326.278557,-69.858253,1,0.1845,0.2240,0.3763,40.2324,0.011
5924,51.855469,-28.630989,224.733260,-55.649872,1,0.7987,0.8068,0.0181,43.5253,0.009
5930,149.589844,3.583322,234.885369,42.474696,1,nan,1.3364,0.0830,44.8793,0.024
5937,2.457983,-45.389202,324.632685,-69.945696,1,nan,1.2012,0.0366,44.5926,0.011
5954,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.1290,0.0618,44.4261,0.019
5956,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.8904,0.0380,43.7889,0.009
5957,150.468750,3.732834,235.392208,43.283244,1,nan,0.7308,0.0756,43.2613,0.020
5978,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.5152,0.0154,42.3393,0.019
5982,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.4710,0.0617,42.1060,0.022
5995,359.058563,-45.191612,330.695783,-68.844915,1,nan,1.5541,0.1950,45.2840,0.011
6000,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.2101,0.0361,44.6126,0.006
6001,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.8961,0.0542,43.8060,0.008
6003,349.891296,-64.573555,317.972107,-49.786192,1,0.6069,0.5395,0.0894,42.4600,0.023
6017,151.347656,4.181528,235.568369,44.259942,1,nan,0.5050,0.5080,42.2869,0.016
6021,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.9940,0.0435,44.0841,0.008
6022,352.711273,-63.823658,316.922299,-51.059403,1,nan,0.2460,0.2146,40.4626,0.024
6023,348.595886,-63.072620,320.023289,-50.713060,1,nan,0.4699,0.0341,42.0998,0.021
6028,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.7000,0.0060,43.1467,0.009
6030,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7964,0.0420,43.4906,0.008
6076,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.2728,0.0758,44.7483,0.005
6087,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.8998,0.0507,43.8170,0.010
6091,359.058563,-45.191612,330.695783,-68.844915,1,0.1812,0.1971,0.0181,39.9205,0.011
6092,348.595886,-63.072620,320.023289,-50.713060,1,nan,1.2574,0.2139,44.7155,0.021
6096,351.953644,-62.132156,318.777388,-52.347124,1,nan,1.1012,0.0271,44.3592,0.019
6106,150.117188,2.836105,236.124718,42.483719,1,nan,0.5103,0.0193,42.3145,0.016
6119,1.708861,-45.586655,325.688716,-69.520253,1,0.3975,0.4588,0.5979,42.0377,0.011
6120,346.655182,-63.260487,320.952196,-50.040935,1,nan,1.1674,0.4165,44.5161,0.019
6126,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.7902,0.0393,43.4696,0.007
6135,348.586945,-64.573555,318.693903,-49.477869,1,0.8477,0.8451,0.1229,43.6492,0.018
6151,150.292969,2.686724,236.427488,42.541447,1,nan,0.8598,0.0520,43.6952,0.016
6162,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.3977,0.0957,44.9997,0.011
6164,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.5785,0.3260,45.3258,0.023
6173,150.117188,2.836105,236.124718,42.483719,1,0.3881,0.3821,0.0142,41.5673,0.016
6179,348.586945,-64.573555,318.693903,-49.477869,1,0.5991,0.5030,0.0291,42.2767,0.018
6187,349.891296,-64.573555,317.972107,-49.786192,1,nan,1.1064,0.0831,44.3719,0.023
6191,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.2328,0.0166,40.3267,0.006
6197,151.523438,3.134927,236.900695,43.803170,1,nan,1.0306,0.0174,44.1812,0.019
6205,33.574219,-6.579593,170.455585,-61.548219,1,0.5161,0.5873,0.0270,42.6825,0.021
6211,0.949367,-45.586655,326.991548,-69.251686,1,0.5940,0.5967,0.0095,42.7247,0.013
6217,35.683594,-5.379379,171.992947,-59.253501,1,nan,0.6309,0.0313,42.8717,0.020
6223,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.2937,0.0805,40.9020,0.017
6287,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.6517,0.0263,42.9572,0.009
6289,52.207031,-26.610098,221.298836,-55.042928,1,0.2186,0.3975,0.2967,41.6682,0.014
6293,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.8837,0.0179,43.7686,0.022
6297,53.964844,-28.630989,225.142950,-53.813613,1,0.5674,0.6449,0.0377,42.9295,0.009
6308,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.9701,0.0203,44.0187,0.021
6313,349.966217,-62.696659,319.542989,-51.376556,1,nan,0.3825,0.0215,41.5700,0.021
6343,33.398438,-4.331149,167.226341,-59.936551,1,nan,1.0126,0.0334,44.1339,0.018
6345,0.190678,-45.783966,327.956322,-68.803772,1,nan,1.0255,0.0178,44.1678,0.005
6347,149.414062,3.433834,234.919132,42.245550,1,nan,0.2243,0.0261,40.2360,0.027
6350,33.574219,-5.379379,168.838090,-60.637536,1,nan,1.2142,0.1253,44.6216,0.017
6352,150.996094,4.181528,235.291975,43.970869,1,nan,1.4079,0.0591,45.0191,0.015
6354,34.804688,-5.829153,171.307861,-60.174401,1,0.3701,0.4966,0.3999,42.2433,0.023
6368,351.299988,-62.320400,319.038597,-52.026867,1,nan,0.9252,0.0390,43.8918,0.018
6369,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.5996,0.0136,42.7374,0.017
6372,359.805206,-46.768478,327.135979,-67.829903,1,0.7964,0.7850,0.0422,43.4520,0.011
6376,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.7511,0.0307,43.3345,0.018
6378,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.6002,0.0209,42.7397,0.016
6390,0.589520,-47.161343,325.385896,-67.769893,1,0.5421,0.4998,0.2593,42.2602,0.009
6391,34.101562,-5.829153,170.247753,-60.638325,1,nan,1.1253,0.1662,44.4173,0.019
6402,51.679688,-27.447618,222.618229,-55.642263,1,0.4580,0.4038,0.0453,41.7087,0.010
6405,34.804688,-5.829153,171.307861,-60.174401,1,nan,1.3943,0.2658,44.9931,0.023
6436,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.4164,0.3756,41.7878,0.006
6437,33.925781,-5.979157,170.179895,-60.866303,1,0.4279,0.4344,0.0322,41.8968,0.022
6450,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.7339,0.0231,43.2726,0.005
6458,352.711273,-63.823658,316.922299,-51.059403,1,1.1032,1.0971,0.1861,44.3491,0.024
6460,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.3954,0.0255,41.6549,0.010
6470,358.636353,-46.768478,328.890146,-67.388837,1,nan,1.0362,0.0170,44.1958,0.008
6474,51.855469,-27.953188,223.543603,-55.561470,1,nan,0.7091,0.1441,43.1812,0.008
6498,348.586945,-64.573555,318.693903,-49.477869,1,nan,1.1153,0.1670,44.3933,0.018
6514,2.071130,-45.191612,325.606223,-69.989264,1,0.3119,0.3565,0.4554,41.3906,0.011
6515,34.804688,-5.829153,171.307861,-60.174401,1,nan,1.1225,0.0927,44.4106,0.023
6537,152.050781,2.985506,237.495952,44.143927,1,nan,1.2730,0.1522,44.7487,0.019
6546,53.085938,-28.122234,224.100909,-54.509752,1,0.2509,0.3826,0.2720,41.5706,0.007
6548,148.710938,2.836105,235.050801,41.328739,1,nan,1.1286,0.1579,44.4252,0.031
6555,151.699219,3.583322,236.533224,44.205648,1,0.2053,0.2236,0.0177,40.2274,0.016
6560,53.613281,-27.953188,223.929533,-54.024772,1,nan,0.7824,0.0662,43.4432,0.007
6567,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.7820,0.0137,43.4419,0.017
6568,51.679688,-27.447618,222.618229,-55.642263,1,nan,1.0385,0.0242,44.2017,0.010
6575,33.398438,-3.732834,166.492280,-59.466614,1,0.5197,0.5580,0.0155,42.5483,0.022
6598,0.589520,-47.161343,325.385896,-67.769893,1,nan,0.7918,0.0238,43.4750,0.009
6606,151.523438,3.134927,236.900695,43.803170,1,nan,0.5715,0.0474,42.6108,0.019
6620,351.259003,-64.386185,317.344860,-50.255113,1,nan,0.9861,0.0153,44.0628,0.020
6623,33.750000,-4.630479,168.146242,-59.949072,1,nan,0.8838,0.0439,43.7691,0.019
6633,349.891296,-64.573555,317.972107,-49.786192,1,0.5312,0.5073,0.1220,42.2989,0.023
6654,150.468750,3.732834,235.392208,43.283244,1,0.8058,0.8088,0.0111,43.5318,0.020
6655,0.929752,-44.597992,328.531426,-70.083244,1,0.6631,0.6768,0.0337,43.0574,0.011
6663,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.4449,0.3362,41.9584,0.008
6673,150.820312,3.732834,235.666318,43.572109,1,0.5028,0.5238,0.0312,42.3823,0.016
6682,352.711273,-63.823658,316.922299,-51.059403,1,0.2960,0.5328,0.7393,42.4271,0.024
6692,1.694561,-45.191612,326.278557,-69.858253,1,nan,0.4504,0.0611,41.9897,0.011
6704,348.586945,-64.573555,318.693903,-49.477869,1,0.3131,0.2939,0.0390,40.9044,0.018
6723,349.160583,-64.760857,318.219706,-49.458924,1,0.2401,0.2742,0.1134,40.7312,0.020
6729,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.9192,0.0320,43.8743,0.011
6742,349.615387,-63.636005,318.927246,-50.506542,1,nan,1.7240,0.2208,45.5614,0.018
6756,359.816315,-44.003082,331.451340,-70.123054,1,nan,1.0236,0.0293,44.1629,0.013
6814,152.050781,3.284369,237.157374,44.318466,1,0.6160,0.6226,0.6151,42.8365,0.019
6821,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.3834,0.0765,41.5757,0.011
6827,358.636353,-46.768478,328.890146,-67.388837,1,nan,1.1052,0.1737,44.3688,0.008
6833,150.996094,2.388015,237.313912,42.939977,1,nan,1.4382,0.0724,45.0762,0.021
6837,32.871094,-4.780192,166.959493,-60.615132,1,nan,0.8994,0.0420,43.8158,0.017
6847,33.574219,-5.079716,168.448505,-60.407218,1,nan,0.6653,0.0303,43.0120,0.016
6855,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.9313,0.0387,43.9092,0.020
6856,1.363636,-46.768478,324.669342,-68.371416,1,0.3068,0.2935,0.0090,40.9003,0.008
6876,150.644531,3.583322,235.698235,43.342784,1,0.3246,0.3216,0.4661,41.1300,0.018
6878,32.871094,-4.780192,166.959493,-60.615132,1,0.5919,0.5848,0.1811,42.6713,0.017
6884,52.910156,-27.279613,222.625192,-54.536648,1,0.2612,0.2825,1.1744,40.8053,0.007
6897,359.814819,-44.399834,330.775011,-69.801007,1,nan,0.6130,0.0239,42.7954,0.009
6907,151.171875,1.342993,238.602520,42.464379,1,0.4791,0.5900,0.3591,42.6946,0.026
6911,0.929752,-44.597992,328.531426,-70.083244,1,0.3125,0.3006,0.0066,40.9604,0.011
6919,346.655182,-63.260487,320.952196,-50.040935,1,nan,1.6301,0.2474,45.4117,0.019
6927,151.171875,2.537361,237.288526,43.169764,1,nan,0.5389,0.2528,42.4568,0.024
6930,52.207031,-26.610098,221.298836,-55.042928,1,0.6478,0.5687,0.2240,42.5980,0.014
6932,52.910156,-27.953188,223.774083,-54.639214,1,nan,1.3042,0.0324,44.8137,0.007
6937,0.965665,-46.375080,325.845907,-68.579427,1,nan,0.8820,0.0370,43.7636,0.007
6949,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.8422,0.0195,43.6399,0.020
6951,0.589520,-47.161343,325.385896,-67.769893,1,0.5372,0.5224,0.0077,42.3758,0.009
6952,150.117188,2.238686,236.784618,42.139082,1,nan,1.7307,0.2954,45.5719,0.016
6964,359.816315,-44.003082,331.451340,-70.123054,1,nan,1.1443,0.2764,44.4622,0.013
6982,358.312500,-44.993881,332.185785,-68.685906,1,0.3227,0.3189,0.0263,41.1089,0.009
6988,150.996094,2.985506,236.647967,43.287350,1,nan,0.8042,0.0261,43.5165,0.020
7013,34.277344,-5.679190,170.314930,-60.410322,1,nan,0.5236,0.0416,42.3817,0.020
7017,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.3210,0.0095,41.1251,0.020
7021,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.9959,0.0260,44.0892,0.009
7032,51.328125,-27.784405,223.130589,-55.999499,1,nan,0.8584,0.0214,43.6910,0.013
7055,34.277344,-5.679190,170.314930,-60.410322,1,nan,0.9347,0.0439,43.9190,0.020
7060,52.207031,-28.291550,224.208534,-55.300157,1,nan,0.6635,0.0179,43.0048,0.007
7065,52.910156,-26.276812,220.926149,-54.363918,1,nan,1.5901,0.1813,45.3453,0.008
7129,51.328125,-27.447618,222.535046,-55.950727,1,nan,1.1651,0.0362,44.5107,0.013
7140,150.820312,3.134927,236.341348,43.230123,1,nan,0.9363,0.0122,43.9237,0.016
7161,1.363636,-46.768478,324.669342,-68.371416,1,0.3784,0.3433,0.1675,41.2947,0.008
7162,52.910156,-27.953188,223.774083,-54.639214,1,0.3593,0.4691,0.9910,42.0956,0.007
7163,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.1234,0.1962,44.4129,0.007
7172,151.171875,1.342993,238.602520,42.464379,1,nan,0.6339,0.0129,42.8842,0.026
7175,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.0926,0.1310,44.3382,0.019
7224,53.964844,-28.630989,225.142950,-53.813613,1,nan,1.2376,0.0743,44.6730,0.009
7226,52.207031,-26.610098,221.298836,-55.042928,1,0.3847,0.3273,0.8615,41.1744,0.014
7233,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.2494,0.2608,44.6985,0.020
7241,150.820312,3.732834,235.666318,43.572109,1,nan,0.7200,0.0438,43.2218,0.016
7246,151.347656,4.181528,235.568369,44.259942,1,nan,0.6039,0.0147,42.7560,0.016
7265,151.171875,2.238686,237.619933,42.994783,1,nan,0.7073,0.0647,43.1743,0.024
7275,1.708861,-45.586655,325.688716,-69.520253,1,0.4268,0.4094,0.0089,41.7441,0.011
7282,51.855469,-28.630989,224.733260,-55.649872,1,nan,1.4386,0.0998,45.0769,0.009
7292,33.574219,-5.379379,168.838090,-60.637536,1,nan,0.7780,0.0880,43.4282,0.017
7297,0.574468,-45.981140,327.041068,-68.778764,1,0.5290,0.5535,0.4055,42.5268,0.006
7326,351.299988,-62.320400,319.038597,-52.026867,1,0.2394,0.2364,0.0161,40.3644,0.018
7344,33.398438,-4.331149,167.226341,-59.936551,1,0.5794,0.5951,0.0111,42.7176,0.018
7378,347.846710,-64.760857,318.929827,-49.143596,1,nan,1.0318,0.1438,44.1843,0.019
7381,52.031250,-26.443335,220.963669,-55.168557,1,0.3373,0.5151,0.2782,42.3389,0.014
7385,52.910156,-26.276812,220.926149,-54.363918,1,0.7536,0.7669,0.0410,43.3898,0.008
7389,359.446716,-44.201530,331.730015,-69.805709,1,0.6180,0.6621,0.1010,42.9994,0.010
7447,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.6276,0.0506,42.8577,0.005
7451,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.7431,0.0243,43.3058,0.019
7464,346.500000,-62.320400,321.951129,-50.736054,1,0.5846,0.6343,0.0145,42.8857,0.020
7479,359.058563,-45.191612,330.695783,-68.844915,1,0.3899,0.3252,0.7307,41.1585,0.011
7496,0.190678,-45.783966,327.956322,-68.803772,1,nan,0.6583,0.0121,42.9840,0.005
7508,34.980469,-6.279288,172.180075,-60.389399,1,nan,1.1198,0.0531,44.4042,0.023
7515,150.820312,3.134927,236.341348,43.230123,1,nan,1.0715,0.0116,44.2857,0.016
7535,33.222656,-4.780192,167.515653,-60.396584,1,nan,1.4866,0.1394,45.1649,0.018
7556,53.085938,-27.784405,223.525509,-54.460748,1,0.4003,0.4870,0.4078,42.1925,0.007
7590,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.7702,0.0277,43.4013,0.010
7593,53.085938,-27.784405,223.525509,-54.460748,1,nan,1.1395,0.0880,44.4511,0.007
7596,358.312500,-44.993881,332.185785,-68.685906,1,nan,1.0214,0.2333,44.1572,0.009
7597,152.050781,2.985506,237.495952,44.143927,1,0.7873,0.8649,0.0311,43.7112,0.019
7599,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.6460,0.0576,42.9341,0.020
7616,151.347656,3.583322,236.252362,43.918627,1,nan,0.6020,0.0567,42.7478,0.015
7651,2.097458,-45.783966,324.737840,-69.478613,1,0.4739,0.4618,0.0214,42.0550,0.011
7655,150.117188,2.836105,236.124718,42.483719,1,nan,0.7487,0.0442,43.3259,0.016
7657,352.398651,-62.696659,318.017427,-51.967966,1,nan,1.1347,0.0316,44.4397,0.020
7667,53.613281,-27.953188,223.929533,-54.024772,1,nan,0.6882,0.0180,43.1017,0.007
7693,349.615387,-63.636005,318.927246,-50.506542,1,0.5467,0.5108,0.1636,42.3168,0.018
7705,347.013428,-62.508568,321.472056,-50.735330,1,0.6672,0.6780,0.0682,43.0621,0.018
7723,33.398438,-4.331149,167.226341,-59.936551,1,nan,0.7023,0.0297,43.1555,0.018
7764,349.429535,-62.508568,320.039643,-51.393745,1,nan,0.5974,0.0081,42.7277,0.020
7771,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.5341,0.0966,42.4333,0.017
7775,349.615387,-63.636005,318.927246,-50.506542,1,0.3416,0.3512,0.0280,41.3526,0.018
7788,346.130127,-63.072620,321.423103,-50.042305,1,nan,1.3320,0.0596,44.8704,0.020
7793,33.925781,-5.979157,170.179895,-60.866303,1,nan,1.1112,0.0841,44.3835,0.022
7806,152.050781,2.985506,237.495952,44.143927,1,0.1438,1.9587,0.6082,45.9018,0.019
7809,358.648071,-46.375080,329.462659,-67.716008,1,nan,0.6914,0.0120,43.1142,0.009
7820,148.710938,2.836105,235.050801,41.328739,1,0.3550,0.2846,0.0753,40.8234,0.031
7830,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.6696,0.0570,43.0291,0.011
7858,1.363636,-46.768478,324.669342,-68.371416,1,0.4673,0.4927,0.0100,42.2231,0.008
7867,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.6117,0.0088,42.7899,0.010
7873,150.468750,1.641510,237.714575,42.075234,1,0.5708,0.6019,0.0233,42.7474,0.017
7920,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.8342,0.0228,43.6144,0.009
7928,151.171875,1.342993,238.602520,42.464379,1,nan,0.8590,0.0235,43.6927,0.026
7931,33.574219,-6.579593,170.455585,-61.548219,1,nan,0.5225,0.4053,42.3762,0.021
7940,52.910156,-25.944481,220.366350,-54.301439,1,0.6969,0.7104,0.0284,43.1859,0.010
7942,0.190678,-45.783966,327.956322,-68.803772,1,0.2814,0.3115,0.5654,41.0497,0.005
7957,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.9375,0.0504,43.9271,0.011
7977,33.398438,-3.732834,166.492280,-59.466614,1,nan,1.0780,0.0578,44.3020,0.022
7980,358.312500,-44.993881,332.185785,-68.685906,1,nan,1.2456,0.1373,44.6902,0.009
7982,346.276581,-64.011238,320.448031,-49.344136,1,0.4520,0.4716,0.0369,42.1091,0.019
7983,0.949367,-45.586655,326.991548,-69.251686,1,nan,1.0778,0.0319,44.3015,0.013
7984,34.453125,-5.229529,169.987075,-59.956185,1,nan,1.4448,0.2401,45.0886,0.019
7990,1.723404,-45.981140,325.117958,-69.180825,1,nan,0.8379,0.0368,43.6263,0.010
8019,33.574219,-6.579593,170.455585,-61.548219,1,0.2036,0.2032,0.0104,39.9944,0.021
8034,52.207031,-26.610098,221.298836,-55.042928,1,nan,1.5348,0.1021,45.2506,0.014
8036,51.328125,-27.447618,222.535046,-55.950727,1,0.3381,0.5336,0.3006,42.4311,0.013
8037,0.189873,-45.586655,328.254458,-68.969298,1,nan,0.8038,0.0994,43.5152,0.007
8054,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.8669,0.0250,43.7173,0.010
8057,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.7677,0.0225,43.3927,0.011
8058,52.031250,-26.443335,220.963669,-55.168557,1,nan,1.0909,0.0215,44.3339,0.014
8063,150.468750,3.732834,235.392208,43.283244,1,nan,1.0023,0.2042,44.1065,0.020
8094,149.414062,2.238686,236.239766,41.565558,1,nan,0.5647,0.0343,42.5793,0.017
8102,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.1741,0.0373,44.5314,0.011
8129,359.814819,-44.399834,330.775011,-69.801007,1,0.4569,0.4721,0.2974,42.1119,0.009
8135,0.190678,-45.783966,327.956322,-68.803772,1,0.2615,0.2935,0.0129,40.9008,0.005
8153,350.230255,-61.943836,320.053946,-52.070537,1,0.2924,0.3076,0.0101,41.0184,0.017
8165,33.750000,-4.630479,168.146242,-59.949072,1,nan,1.0355,0.1353,44.1940,0.019
8166,150.292969,2.686724,236.427488,42.541447,1,nan,1.2423,0.1331,44.6832,0.016
8179,52.207031,-28.630989,224.800211,-55.343637,1,nan,0.6201,0.0376,42.8258,0.009
8181,0.965665,-46.375080,325.845907,-68.579427,1,0.6159,0.6147,0.0327,42.8026,0.007
8182,348.586945,-64.573555,318.693903,-49.477869,1,nan,1.0126,0.0872,44.1337,0.018
8196,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.6223,0.0130,42.8354,0.007
8197,33.750000,-4.630479,168.146242,-59.949072,1,nan,0.8678,0.0526,43.7202,0.019
8201,53.261719,-27.615883,223.280041,-54.281374,1,nan,1.0299,0.0450,44.1792,0.006
8242,150.468750,1.641510,237.714575,42.075234,1,nan,0.9643,0.0352,44.0026,0.017
8250,51.855469,-27.953188,223.543603,-55.561470,1,nan,1.3103,0.1902,44.8263,0.008
8252,359.446716,-44.201530,331.730015,-69.805709,1,nan,0.8891,0.1482,43.7851,0.010
8263,32.695312,-4.929937,166.868469,-60.841230,1,0.7023,0.7653,0.0398,43.3844,0.018
8271,53.964844,-28.630989,225.142950,-53.813613,1,nan,1.2974,0.0994,44.7998,0.009
8283,349.160583,-64.760857,318.219706,-49.458924,1,nan,1.1567,0.0332,44.4912,0.020
8284,35.859375,-4.630479,171.270769,-58.580806,1,0.4045,0.4502,0.0352,41.9889,0.022
8315,152.050781,3.284369,237.157374,44.318466,1,0.7654,0.7712,0.0372,43.4049,0.019
8322,150.117188,2.836105,236.124718,42.483719,1,0.0718,0.0803,0.1025,37.8083,0.016
8329,52.031250,-26.443335,220.963669,-55.168557,1,nan,1.5343,0.1326,45.2496,0.014
8331,33.574219,-5.079716,168.448505,-60.407218,1,0.6347,0.6283,0.0176,42.8605,0.016
8332,35.859375,-4.630479,171.270769,-58.580806,1,0.5166,0.5551,0.0652,42.5347,0.022
8345,359.058563,-45.191612,330.695783,-68.844915,1,0.5651,0.5479,0.0133,42.5004,0.011
8361,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.9078,0.0244,43.8409,0.009
8386,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.9630,0.0958,43.9990,0.010
8388,33.750000,-4.630479,168.146242,-59.949072,1,nan,1.0142,0.0893,44.1381,0.019
8399,1.363636,-46.768478,324.669342,-68.371416,1,0.6489,0.6460,0.0193,42.9339,0.008
8401,152.050781,2.985506,237.495952,44.143927,1,0.5695,0.5583,0.0070,42.5495,0.019
8404,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.9683,0.0275,44.0136,0.010
8413,349.966217,-62.696659,319.542989,-51.376556,1,nan,0.9296,0.2134,43.9042,0.021
8420,359.811707,-45.191612,329.485675,-69.150905,1,nan,1.2904,0.0367,44.7852,0.010
8430,51.855469,-26.276812,220.627031,-55.293792,1,nan,1.3057,0.0606,44.8168,0.014
8441,32.695312,-4.929937,166.868469,-60.841230,1,nan,0.6746,0.0423,43.0487,0.018
8442,351.953644,-62.132156,318.777388,-52.347124,1,nan,0.0000,0.0000,nan,0.019
8455,151.347656,3.583322,236.252362,43.918627,1,nan,1.0052,0.0667,44.1141,0.015
8463,150.644531,3.583322,235.698235,43.342784,1,nan,0.9599,0.0304,43.9903,0.018
8469,348.586945,-64.573555,318.693903,-49.477869,1,nan,0.7521,0.0183,43.3379,0.018
8487,53.613281,-26.944359,222.237403,-53.863858,1,nan,0.6150,0.0095,42.8042,0.009
8492,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.9894,0.1246,44.0715,0.006
8512,52.558594,-27.279613,222.538937,-54.845107,1,0.8384,0.8562,0.0216,43.6841,0.008
8513,33.574219,-5.079716,168.448505,-60.407218,1,0.3943,0.4243,0.0320,41.8362,0.016
8521,33.750000,-4.630479,168.146242,-59.949072,1,0.4536,0.4521,0.0212,41.9996,0.019
8534,349.285706,-62.884678,319.786163,-51.046461,1,nan,0.8932,0.0144,43.7973,0.018
8548,151.347656,3.583322,236.252362,43.918627,1,0.4318,0.5090,0.0311,42.3075,0.015
8549,358.636353,-46.768478,328.890146,-67.388837,1,nan,0.7646,0.0436,43.3818,0.008
8553,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.4066,1.0187,41.7265,0.021
8565,359.058563,-45.191612,330.695783,-68.844915,1,0.6011,0.5779,0.1187,42.6400,0.011
8592,53.613281,-27.953188,223.929533,-54.024772,1,nan,0.8871,0.0176,43.7788,0.007
8619,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.4192,0.1271,45.0406,0.009
8625,351.734680,-62.884678,318.284128,-51.651217,1,0.3067,0.3063,0.0081,41.0077,0.019
8644,52.207031,-28.630989,224.800211,-55.343637,1,0.6866,0.7308,0.0554,43.2613,0.009
8646,1.694561,-45.191612,326.278557,-69.858253,1,nan,1.3300,0.1125,44.8664,0.011
8661,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.9744,0.0308,44.0307,0.017
8664,52.558594,-27.279613,222.538937,-54.845107,1,nan,0.5444,0.0180,42.4836,0.008
8665,151.171875,2.537361,237.288526,43.169764,1,nan,1.2262,0.0861,44.6480,0.024
8672,35.683594,-5.379379,171.992947,-59.253501,1,nan,1.1123,0.0767,44.3861,0.020
8674,53.789062,-27.784405,223.685697,-53.845803,1,0.9353,0.7665,0.0891,43.3883,0.009
8682,349.046051,-61.943836,320.796530,-51.753706,1,nan,1.0543,0.0550,44.2423,0.017
8683,52.910156,-27.953188,223.774083,-54.639214,1,0.6644,0.6624,0.0227,43.0003,0.007
8684,51.855469,-27.953188,223.543603,-55.561470,1,nan,0.8518,0.0200,43.6702,0.008
8691,351.734680,-62.884678,318.284128,-51.651217,1,nan,0.6565,0.0183,42.9769,0.019
8697,351.734680,-62.884678,318.284128,-51.651217,1,nan,1.4889,0.0787,45.1691,0.019
8701,347.812500,-63.448284,320.128971,-50.202348,1,0.2506,0.2395,0.0127,40.3969,0.021
8702,1.753247,-46.768478,324.030235,-68.498041,1,nan,1.2922,0.0134,44.7890,0.014
8705,150.820312,3.732834,235.666318,43.572109,1,nan,0.9669,0.0557,44.0099,0.016
8717,52.207031,-28.291550,224.208534,-55.300157,1,nan,1.4656,0.3245,45.1269,0.007
8724,2.071130,-45.191612,325.606223,-69.989264,1,0.2852,0.2994,0.0072,40.9502,0.011
8730,348.586945,-64.573555,318.693903,-49.477869,1,0.4718,2.9626,1.3143,46.9933,0.018
8734,348.586945,-64.573555,318.693903,-49.477869,1,0.4341,0.4524,0.0446,42.0014,0.018
8738,52.910156,-26.276812,220.926149,-54.363918,1,nan,1.0394,0.1965,44.2041,0.008
8739,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.1723,0.2076,44.5273,0.009
8767,150.820312,3.134927,236.341348,43.230123,1,nan,0.6681,0.0088,43.0230,0.016
8803,2.097458,-45.783966,324.737840,-69.478613,1,nan,1.0540,0.2342,44.2414,0.011
8805,0.965665,-46.375080,325.845907,-68.579427,1,nan,1.1091,0.1745,44.3785,0.007
8806,352.398651,-62.696659,318.017427,-51.967966,1,nan,1.3680,0.2494,44.9420,0.020
8812,151.347656,4.181528,235.568369,44.259942,1,nan,1.2164,0.1126,44.6264,0.016
8816,149.414062,3.433834,234.919132,42.245550,1,nan,0.8846,0.0380,43.7713,0.027
8821,35.859375,-4.630479,171.270769,-58.580806,1,nan,0.7988,0.0369,43.4987,0.022
8837,351.299988,-62.320400,319.038597,-52.026867,1,nan,0.5320,0.0097,42.4234,0.018
8850,346.562500,-63.448284,320.824720,-49.866957,1,nan,1.0271,0.0826,44.1720,0.021
8857,150.117188,2.238686,236.784618,42.139082,1,nan,1.0907,0.1270,44.3334,0.016
8861,351.321442,-64.198746,317.458993,-50.429931,1,nan,0.3476,1.1210,41.3264,0.023
8871,32.695312,-4.929937,166.868469,-60.841230,1,nan,1.8695,0.2279,45.7776,0.018
8883,359.446716,-44.201530,331.730015,-69.805709,1,nan,1.3015,0.2137,44.8082,0.010
8902,151.171875,2.238686,237.619933,42.994783,1,nan,0.4347,0.0093,41.8984,0.024
8904,347.812500,-63.448284,320.128971,-50.202348,1,nan,1.1116,0.0337,44.3843,0.021
8917,34.101562,-5.829153,170.247753,-60.638325,1,0.5976,0.5969,0.0088,42.7253,0.019
8919,51.679688,-27.447618,222.618229,-55.642263,1,nan,0.9152,0.0404,43.8626,0.010
8921,150.644531,3.583322,235.698235,43.342784,1,0.5296,0.5248,0.0115,42.3874,0.018
8933,349.046051,-61.943836,320.796530,-51.753706,1,nan,0.0000,0.0000,nan,0.017
8947,33.574219,-5.379379,168.838090,-60.637536,1,0.3757,2.9135,0.9540,46.9496,0.017
8962,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.2366,0.1848,44.6708,0.011
8978,348.529419,-61.755440,321.293980,-51.763351,1,nan,0.5523,0.1275,42.5212,0.016
8983,53.085938,-28.122234,224.100909,-54.509752,1,0.4014,0.3910,0.0331,41.6261,0.007
8987,0.574468,-45.981140,327.041068,-68.778764,1,nan,1.0629,0.0133,44.2641,0.006
9033,0.190678,-45.783966,327.956322,-68.803772,1,0.3132,0.2593,0.0293,40.5923,0.005
9041,347.617462,-62.508568,321.121462,-50.904708,1,nan,1.0424,0.0183,44.2118,0.019
9051,358.665253,-45.783966,330.353593,-68.203652,1,nan,0.9828,0.0093,44.0536,0.009
9053,1.708861,-45.586655,325.688716,-69.520253,1,nan,0.5661,0.0926,42.5860,0.011
9080,33.925781,-5.979157,170.179895,-60.866303,1,nan,0.8653,0.0487,43.7124,0.022
9084,52.910156,-26.276812,220.926149,-54.363918,1,nan,0.8458,0.0301,43.6512,0.008
9099,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.5841,0.0141,42.6684,0.020
9107,150.468750,1.641510,237.714575,42.075234,1,0.4687,0.5719,0.2556,42.6127,0.017
9110,0.929752,-44.597992,328.531426,-70.083244,1,0.2531,0.2978,0.0321,40.9371,0.011
9115,53.261719,-27.615883,223.280041,-54.281374,1,nan,0.5434,0.6581,42.4785,0.006
9124,34.277344,-5.679190,170.314930,-60.410322,1,nan,1.5382,0.2744,45.2564,0.020
9138,352.132874,-63.636005,317.424173,-51.095855,1,0.2703,0.2829,0.0150,40.8088,0.021
9165,349.429535,-62.508568,320.039643,-51.393745,1,nan,1.1058,0.0670,44.3703,0.020
9167,346.130127,-63.072620,321.423103,-50.042305,1,nan,1.1829,0.0376,44.5516,0.020
9170,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.5880,0.0130,42.6856,0.020
9187,35.859375,-4.630479,171.270769,-58.580806,1,nan,0.3500,0.0275,41.3440,0.022
9197,2.071130,-45.191612,325.606223,-69.989264,1,nan,1.3572,0.2236,44.9207,0.011
9209,150.644531,3.583322,235.698235,43.342784,1,nan,0.5872,0.0397,42.6820,0.018
9210,359.415588,-46.768478,327.729895,-67.686097,1,nan,1.2507,0.0994,44.7013,0.009
9216,359.811707,-45.191612,329.485675,-69.150905,1,0.9399,1.5923,0.1936,45.3491,0.010
9220,52.207031,-28.630989,224.800211,-55.343637,1,nan,0.6692,0.0244,43.0276,0.009
9229,52.910156,-27.953188,223.774083,-54.639214,1,0.3075,0.3138,0.0076,41.0683,0.007
9237,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.5799,0.2003,42.6494,0.011
9277,149.414062,1.940072,236.565366,41.393323,1,nan,0.7257,0.0452,43.2426,0.018
9293,151.171875,2.238686,237.619933,42.994783,1,0.5639,0.5209,0.5082,42.3682,0.024
9302,351.321442,-64.198746,317.458993,-50.429931,1,nan,1.3310,0.0582,44.8683,0.023
9303,34.453125,-5.229529,169.987075,-59.956185,1,nan,1.0260,0.0163,44.1691,0.019
9316,33.574219,-5.379379,168.838090,-60.637536,1,nan,0.6163,0.0159,42.8095,0.017
9322,33.574219,-5.079716,168.448505,-60.407218,1,nan,1.3730,0.1469,44.9518,0.016
9346,351.321442,-64.198746,317.458993,-50.429931,1,nan,1.3099,0.0995,44.8254,0.023
9362,53.437500,-29.142223,225.908120,-54.336118,1,nan,0.2692,0.0170,40.6858,0.008
9365,34.277344,-5.679190,170.314930,-60.410322,1,0.2935,0.2933,0.0217,40.8992,0.020
9374,53.085938,-27.111860,222.384291,-54.355086,1,nan,0.7189,0.0162,43.2179,0.007
9380,346.655182,-63.260487,320.952196,-50.040935,1,nan,0.7849,0.0428,43.4518,0.019
9408,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.9664,0.0147,44.0085,0.020
9423,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.5558,0.0329,42.5379,0.023
9444,349.046051,-61.943836,320.796530,-51.753706,1,nan,1.1578,0.0579,44.4939,0.017
9451,351.382965,-64.011238,317.574052,-50.604657,1,nan,0.9855,0.0096,44.0612,0.023
9461,358.665253,-45.783966,330.353593,-68.203652,1,nan,1.1498,0.0929,44.4752,0.009
9468,51.855469,-26.276812,220.627031,-55.293792,1,nan,0.7969,0.0460,43.4923,0.014
9469,32.871094,-4.780192,166.959493,-60.615132,1,0.8180,0.7700,0.0220,43.4006,0.017
9482,53.085938,-28.122234,224.100909,-54.509752,1,nan,0.7314,0.0368,43.2634,0.007
9497,51.328125,-27.784405,223.130589,-55.999499,1,0.4664,0.4234,0.0109,41.8306,0.013
9507,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.7375,0.0178,43.2857,0.010
9550,346.130127,-63.072620,321.423103,-50.042305,1,0.6995,0.7070,0.0097,43.1732,0.020
9554,51.679688,-27.447618,222.618229,-55.642263,1,0.6646,0.6800,0.0238,43.0701,0.010
9562,349.160583,-64.760857,318.219706,-49.458924,1,nan,0.3891,0.0546,41.6136,0.020
9566,52.910156,-25.944481,220.366350,-54.301439,1,nan,0.8552,0.0311,43.6808,0.010
9588,51.855469,-28.630989,224.733260,-55.649872,1,nan,0.8782,0.0321,43.7520,0.009
9597,53.964844,-28.630989,225.142950,-53.813613,1,nan,0.8452,0.2344,43.6493,0.009
9603,349.046051,-61.943836,320.796530,-51.753706,1,nan,1.6743,0.3261,45.4832,0.017
9626,150.644531,3.583322,235.698235,43.342784,1,nan,1.0665,0.0241,44.2730,0.018
9652,53.613281,-28.630989,225.073365,-54.119461,1,nan,0.6624,0.0296,43.0006,0.006
9662,53.613281,-26.944359,222.237403,-53.863858,1,nan,1.4334,0.0635,45.0673,0.009
9676,1.753247,-46.768478,324.030235,-68.498041,1,0.6764,0.7221,0.0336,43.2295,0.014
9678,151.523438,3.134927,236.900695,43.803170,1,nan,1.1508,0.7055,44.4777,0.019
9699,2.097458,-45.783966,324.737840,-69.478613,1,nan,0.6400,0.1023,42.9095,0.011
9705,352.711273,-63.823658,316.922299,-51.059403,1,0.5977,0.5685,0.0384,42.5972,0.024
9720,351.734680,-62.884678,318.284128,-51.651217,1,0.6071,0.5353,0.0256,42.4395,0.019
9725,350.230255,-61.943836,320.053946,-52.070537,1,nan,1.2775,0.0723,44.7581,0.017
9726,358.312500,-44.993881,332.185785,-68.685906,1,nan,0.7580,0.0243,43.3588,0.009
9761,52.207031,-28.291550,224.208534,-55.300157,1,nan,1.4229,0.1718,45.0476,0.007
9772,53.613281,-26.944359,222.237403,-53.863858,1,nan,0.5526,0.0238,42.5224,0.009
9776,351.953644,-62.132156,318.777388,-52.347124,1,0.4229,0.4114,0.0259,41.7568,0.019
9782,149.589844,3.583322,234.885369,42.474696,1,nan,1.1776,0.1586,44.5394,0.024
9799,346.276581,-64.011238,320.448031,-49.344136,1,nan,0.9878,0.3118,44.0673,0.019
9806,0.965665,-46.375080,325.845907,-68.579427,1,0.4619,0.5359,0.0762,42.4424,0.007
9809,347.812500,-63.448284,320.128971,-50.202348,1,nan,0.6301,0.0342,42.8682,0.021
9839,53.613281,-27.953188,223.929533,-54.024772,1,nan,1.5628,0.1334,45.2989,0.007
9864,0.929752,-44.597992,328.531426,-70.083244,1,nan,0.7718,0.0734,43.4067,0.011
9865,150.117188,3.732834,235.120533,42.993809,1,0.4651,2.3854,0.8502,46.4240,0.020
9868,54.667969,-27.615883,223.610785,-53.050840,1,nan,0.7611,0.0247,43.3696,0.009
9884,51.328125,-27.447618,222.535046,-55.950727,1,0.6206,0.6001,0.0161,42.7395,0.013
9893,33.750000,-4.630479,168.146242,-59.949072,1,0.4883,0.5053,0.0956,42.2888,0.019
9916,349.891296,-64.573555,317.972107,-49.786192,1,nan,0.6122,0.0358,42.7920,0.023
9924,349.285706,-62.884678,319.786163,-51.046461,1,0.7986,0.8711,0.0680,43.7303,0.018
9950,51.328125,-27.447618,222.535046,-55.950727,1,nan,0.5462,0.0962,42.4921,0.013
9951,346.276581,-64.011238,320.448031,-49.344136,1,0.8417,0.7427,0.0547,43.3044,0.019
9961,53.789062,-27.784405,223.685697,-53.845803,1,nan,1.0492,0.2680,44.2292,0.009
9967,150.292969,2.686724,236.427488,42.541447,1,nan,1.4395,0.2623,45.0786,0.016
9989,346.130127,-63.072620,321.423103,-50.042305,1,nan,0.9517,0.2190,43.9673,0.020
9998,2.457983,-45.389202,324.632685,-69.945696,1,nan,0.5127,0.0376,42.3265,0.011
10021,52.207031,-28.630989,224.800211,-55.343637,1,nan,1.4337,0.0978,45.0679,0.009
10056,52.558594,-27.279613,222.538937,-54.845107,1,0.4511,0.3564,0.0517,41.3900,0.008
10071,0.949367,-45.586655,326.991548,-69.251686,1,0.3219,0.4590,0.0597,42.0387,0.013
10102,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.7450,0.0248,43.3128,0.011
10103,359.805206,-46.768478,327.135979,-67.829903,1,nan,0.8130,0.0375,43.5455,0.011
10110,54.667969,-27.615883,223.610785,-53.050840,1,0.7638,0.8007,0.0367,43.5050,0.009
10119,351.953644,-62.132156,318.777388,-52.347124,1,0.6209,0.6434,0.0311,42.9234,0.019
10133,352.398651,-62.696659,318.017427,-51.967966,1,nan,0.9030,0.0216,43.8266,0.020
10139,350.230255,-61.943836,320.053946,-52.070537,1,nan,1.3210,0.0562,44.8481,0.017
10146,346.500000,-62.320400,321.951129,-50.736054,1,nan,0.5522,0.0066,42.5205,0.020
10150,359.814819,-44.399834,330.775011,-69.801007,1,0.4395,2.7379,0.4044,46.7868,0.009
10155,0.589520,-47.161343,325.385896,-67.769893,1,nan,1.0100,0.2301,44.1270,0.009
10180,359.811707,-45.191612,329.485675,-69.150905,1,nan,0.5829,0.0112,42.6631,0.010
10213,359.058563,-45.191612,330.695783,-68.844915,1,nan,0.5634,0.0192,42.5732,0.011
10218,52.558594,-27.279613,222.538937,-54.845107,1,0.3131,0.3414,0.0364,41.2809,0.008
10228,151.171875,1.342993,238.602520,42.464379,1,nan,0.7524,0.0449,43.3390,0.026
10244,53.789062,-27.784405,223.685697,-53.845803,1,nan,0.7687,0.0207,43.3962,0.009
================================================
FILE: examples/data/plasticc_training_set_1k.csv
================================================
object_id,mjd,passband,flux,flux_err,detected
615,59750.4229,2,-544.810303,3.622952,1
615,59750.4306,1,-816.434326,5.553370,1
615,59750.4383,3,-471.385529,3.801213,1
615,59750.4450,4,-388.984985,11.395031,1
615,59752.4070,2,-681.858887,4.041204,1
615,59752.4147,1,-1061.457031,6.472994,1
615,59752.4224,3,-524.954590,3.552751,1
615,59752.4334,4,-393.480225,3.599346,1
615,59752.4435,5,-355.886780,10.421921,1
615,59767.2968,2,-548.013550,3.462291,1
615,59767.3045,1,-815.188599,5.293019,1
615,59767.3122,3,-475.516052,3.340643,1
615,59767.3233,4,-405.663818,3.496113,1
615,59767.3343,5,-421.199066,6.377517,1
615,59770.2179,2,-554.903198,3.927843,1
615,59770.2256,1,-820.042786,5.875329,1
615,59770.2334,3,-477.004730,3.736262,1
615,59770.2445,4,-400.270386,3.834955,1
615,59770.2557,5,-415.286896,7.435979,1
615,59779.3188,2,-630.523682,4.333287,1
615,59779.3265,1,-921.002502,6.306800,1
615,59779.3342,3,-518.533997,3.915225,1
615,59779.3452,4,-422.184509,4.089213,1
615,59779.3562,5,-422.815094,8.124096,1
615,59782.1897,2,-280.039520,2.819228,1
615,59782.1974,1,-449.095612,4.028310,1
615,59782.2051,3,-316.704865,3.491153,1
615,59782.2162,4,-332.885437,4.021619,1
615,59782.2274,5,-365.075775,8.514805,1
615,59797.2861,2,391.399231,3.098059,1
615,59797.2938,1,35.511822,3.163646,1
615,59797.3015,3,330.623901,3.388776,1
615,59797.3126,4,360.397858,3.980607,1
615,59797.3237,5,369.439667,8.207490,1
615,59800.3168,2,168.739899,3.128495,1
615,59800.3244,1,129.541901,4.358776,1
615,59800.3320,3,30.120724,3.396606,1
615,59800.3429,4,-60.942333,3.704243,1
615,59800.3539,5,-128.920334,7.495701,1
615,59807.1738,2,-256.660980,2.781354,1
615,59807.1815,1,-420.796417,4.037735,1
615,59807.1892,3,-298.936859,3.041390,1
615,59807.2003,4,-311.977783,3.318007,1
615,59807.2114,5,-344.536072,6.367201,1
615,59810.1045,2,-342.819763,2.808321,1
615,59810.1122,1,-527.020325,4.204173,1
615,59810.1200,3,-363.282532,3.237536,1
615,59810.1311,4,-348.628662,3.774855,1
615,59810.1422,5,-391.271271,7.657067,1
615,59813.1044,2,-678.045715,4.032819,1
615,59813.1122,1,-1100.440063,6.709106,1
615,59813.1199,3,-506.687408,3.547398,1
615,59813.1310,4,-304.049713,3.590496,1
615,59813.1422,5,-187.285919,6.984019,1
615,59819.1532,0,6.878784,3.633152,0
615,59820.1047,0,39.364853,3.775619,1
615,59821.1026,0,-10.422381,4.172683,0
615,59822.1105,0,-65.485130,4.362876,1
615,59823.1505,0,-113.349159,4.069051,1
615,59835.0600,2,-54.949490,2.575779,1
615,59835.0678,1,-178.149399,3.809858,1
615,59835.0755,3,-140.818436,3.149077,1
615,59835.0866,4,-200.294128,3.578005,1
615,59835.0978,5,-263.578430,6.954262,1
615,59839.0306,2,-639.035950,3.928531,1
615,59839.0384,1,-953.883728,6.035410,1
615,59839.0461,3,-518.293274,3.595869,1
615,59839.0573,4,-418.723907,3.536483,1
615,59839.0684,5,-418.799927,6.580595,1
615,59842.0207,2,-502.215332,3.348443,1
615,59842.0285,1,-1003.971497,6.256784,1
615,59842.0362,3,-233.167755,2.872840,1
615,59842.0473,4,111.507675,3.229112,1
615,59842.0585,5,206.425323,6.615822,1
615,59851.1114,0,-68.502457,3.338555,1
615,59854.0796,2,459.452667,3.336711,1
615,59854.0873,1,217.894211,3.332742,1
615,59854.0950,3,361.023438,3.237847,1
615,59854.1061,4,374.446442,3.622074,1
615,59854.1172,5,370.346283,6.789766,1
615,59857.0453,2,599.812195,4.121032,1
615,59857.0531,1,646.523193,5.291624,1
615,59857.0608,3,354.961365,3.407785,1
615,59857.0719,4,293.879608,3.581862,1
615,59857.0830,5,232.535995,6.761845,1
615,59864.0162,2,-637.105347,3.818432,1
615,59864.0239,1,-942.167908,5.916004,1
615,59864.0316,3,-524.586548,3.538931,1
615,59864.0428,4,-414.447723,3.635253,1
615,59864.0539,5,-408.089233,7.119730,1
615,59867.0178,2,-332.763123,2.872951,1
615,59867.0255,1,-910.677734,5.852032,1
615,59867.0332,3,-62.065010,3.073413,1
615,59867.0443,4,202.288223,4.026751,1
615,59867.0554,5,270.584869,10.321785,1
615,59870.0194,2,604.344543,3.901750,1
615,59870.0272,1,659.486694,4.991051,1
615,59870.0349,3,373.986511,3.294667,1
615,59870.0459,4,322.604034,3.716555,1
615,59870.0571,5,263.481476,7.193131,1
615,59873.0212,2,4.656033,2.130510,0
615,59873.0289,1,-98.796974,3.034533,1
615,59873.0366,3,-93.732880,2.499724,1
615,59873.0477,4,-165.793457,2.906058,1
615,59873.0588,5,-233.501724,6.087882,1
615,59874.0599,0,-97.353195,3.133990,1
615,59875.0311,0,-97.523880,2.963075,1
615,59876.0231,0,-108.672577,3.449714,1
615,59877.0238,0,-116.913223,3.097836,1
615,59878.0246,0,-102.768921,3.135772,1
615,59879.0248,0,-52.407089,3.261559,1
615,59880.0258,0,55.567715,3.355268,1
615,59884.0823,2,-274.711029,2.572093,1
615,59884.0900,1,-437.425110,3.831595,1
615,59884.0976,3,-310.010925,2.957125,1
615,59884.1085,4,-317.630920,3.213168,1
615,59884.1195,5,-351.278198,6.231324,1
615,59887.0298,2,-491.146423,3.592675,1
615,59887.0375,1,-743.267212,5.624708,1
615,59887.0451,3,-449.714752,3.728483,1
615,59887.0562,4,-393.971649,3.532816,1
615,59887.0673,5,-406.549103,6.648589,1
615,60118.4163,0,-107.080536,3.102513,1
615,60124.2541,2,-588.397949,4.176047,1
615,60124.2618,1,-878.043396,6.210247,1
615,60124.2695,3,-495.472015,4.011444,1
615,60124.2807,4,-417.145325,4.094360,1
615,60124.2918,5,-413.673431,7.793959,1
615,60140.2290,0,-88.981155,3.468430,1
615,60141.2225,0,-50.179337,4.734193,1
615,60142.2202,0,50.008640,4.636651,1
615,60143.2212,0,110.753555,4.380840,1
615,60144.2186,0,120.867218,4.103332,1
615,60145.2123,0,111.464226,4.367030,1
615,60153.2274,2,-322.420471,2.833071,1
615,60153.2351,1,-917.875488,5.951387,1
615,60153.2428,3,-52.056461,2.698249,1
615,60153.2539,4,205.180893,3.169676,1
615,60153.2650,5,269.709167,6.682271,1
615,60162.1477,2,31.499735,2.753767,1
615,60162.1554,1,-62.120552,3.818642,1
615,60162.1631,3,-72.958771,3.515574,1
615,60162.1742,4,-151.126511,3.838288,1
615,60162.1853,5,-216.914032,8.832489,1
615,60165.1369,2,-568.408875,3.553168,1
615,60165.1446,1,-836.233154,5.405759,1
615,60165.1524,3,-483.071381,3.387615,1
615,60165.1635,4,-409.470642,3.541994,1
615,60165.1746,5,-412.820221,6.907444,1
615,60168.1260,2,-628.321350,3.804775,1
615,60168.1337,1,-1077.347900,6.591075,1
615,60168.1414,3,-421.859406,3.336656,1
615,60168.1525,4,-93.729095,3.211201,1
615,60168.1637,5,31.207939,6.615005,0
615,60176.1332,0,-49.905262,3.874426,1
615,60177.1370,0,-87.160583,4.078375,1
615,60181.3147,2,-180.729568,2.498579,1
615,60181.3223,1,-339.875153,3.626661,1
615,60181.3299,3,-249.205673,3.280824,1
615,60181.3409,4,-275.762329,4.694962,1
615,60181.3518,5,-330.891327,9.580047,1
615,60184.3625,2,-555.853943,3.979171,1
615,60184.3701,1,-1028.441528,6.719577,1
615,60184.3777,3,-306.200500,3.557627,1
615,60184.3887,4,49.555847,3.627351,1
615,60184.3996,5,154.876785,7.988054,1
615,60194.1575,2,469.654999,3.386857,1
615,60194.1652,1,276.757751,3.594162,1
615,60194.1729,3,374.669556,3.437137,1
615,60194.1839,4,374.948822,4.267094,1
615,60194.1926,5,363.130493,12.845472,1
615,60197.1181,2,607.786804,3.960346,1
615,60197.1258,1,650.984314,4.970811,1
615,60197.1335,3,365.408752,3.197298,1
615,60197.1446,4,305.330750,3.360043,1
615,60197.1557,5,256.966217,6.443069,1
615,60198.1077,0,100.129280,4.266314,1
615,60199.0914,0,86.776741,4.679742,1
615,60200.0650,0,82.078186,4.342434,1
615,60201.0680,0,41.947815,4.467065,1
615,60202.0552,0,9.061676,3.831397,0
615,60206.1107,0,-83.072884,3.130236,1
615,60207.1469,0,108.483109,4.458607,1
615,60208.0229,2,-672.681335,4.138056,1
615,60208.0307,1,-1094.027588,6.717340,1
615,60208.0384,3,-503.870422,3.665424,1
615,60208.0495,4,-284.747498,3.666287,1
615,60208.0606,5,-176.409851,8.308295,1
615,60211.0124,2,208.281052,3.229784,1
615,60211.0202,1,-370.189575,4.525907,1
615,60211.0279,3,269.200806,3.688238,1
615,60211.0390,4,326.272308,4.424663,1
615,60211.0502,5,358.320099,10.152412,1
615,60221.0153,2,-648.682312,3.906177,1
615,60221.0230,1,-1086.777710,6.620100,1
615,60221.0308,3,-455.588196,3.295532,1
615,60221.0419,4,-145.305023,3.087424,1
615,60221.0530,5,2.939076,6.798505,0
615,60224.0140,2,491.748383,3.509416,1
615,60224.0217,1,346.335083,3.835582,1
615,60224.0294,3,384.185303,3.529593,1
615,60224.0405,4,381.953735,3.885009,1
615,60224.0516,5,378.118225,7.311360,1
615,60227.0151,2,341.057709,2.940147,1
615,60227.0228,1,356.632690,3.856145,1
615,60227.0305,3,153.004929,2.847803,1
615,60227.0416,4,52.912033,3.109148,1
615,60227.0527,5,-19.384567,6.713308,0
615,60228.0187,0,6.768485,4.174600,0
615,60229.0162,0,-35.149330,4.086384,1
615,60234.0265,0,-52.922794,3.681808,1
615,60237.2206,2,-676.669189,4.009161,1
615,60237.2283,1,-1098.651489,6.689435,1
615,60237.2359,3,-511.148254,3.546333,1
615,60237.2468,4,-347.090027,3.624199,1
615,60237.2578,5,-240.316895,8.035271,1
615,60240.0223,2,85.162651,2.829378,1
615,60240.0300,1,14.526012,3.783879,0
615,60240.0377,3,-24.350578,3.397041,1
615,60240.0488,4,-111.062698,3.698180,1
615,60240.0598,5,-180.234787,6.894514,1
615,60249.0338,2,611.984558,3.908728,1
615,60249.0415,1,660.626343,4.961018,1
615,60249.0492,3,386.311920,3.240422,1
615,60249.0602,4,325.401184,3.454910,1
615,60249.0712,5,280.721069,6.623785,1
615,60260.0423,0,108.020546,4.337497,1
615,60261.0361,0,125.182808,3.909554,1
615,60262.0367,0,107.649780,3.796611,1
615,60263.0373,0,61.068066,3.877589,1
615,60264.0465,0,-9.100937,3.518127,0
615,60490.2647,2,-408.570984,3.169784,1
615,60490.2725,1,-624.518799,4.704853,1
615,60490.2802,3,-405.614258,3.513195,1
615,60490.2913,4,-371.286377,3.985296,1
615,60490.3024,5,-395.406128,8.139952,1
615,60493.2372,2,-680.489441,4.065931,1
615,60493.2450,1,-1031.102905,6.378702,1
615,60493.2527,3,-530.644592,3.672556,1
615,60493.2639,4,-406.733521,3.772714,1
615,60493.2750,5,-358.876160,7.310321,1
615,60499.2467,0,106.447296,4.481476,1
615,60500.2437,0,67.234062,4.020935,1
615,60501.2385,0,24.868933,4.027500,1
615,60502.2355,0,-15.392517,4.142292,0
615,60508.2638,2,365.607056,3.696270,1
615,60508.2715,1,-32.986282,4.440859,0
615,60508.2792,3,319.249847,3.828632,1
615,60508.2903,4,360.507599,4.336362,1
615,60508.3014,5,370.305267,8.601955,1
615,60524.2390,0,89.070496,3.901179,1
615,60525.1736,0,118.935989,4.737393,1
615,60532.3489,2,510.690094,3.489832,1
615,60532.3565,1,566.281433,4.607503,1
615,60532.3641,3,271.663910,3.007311,1
615,60532.3751,4,204.409866,3.353202,1
615,60532.3860,5,128.521851,7.134325,1
615,60535.1253,2,-664.729675,4.282414,1
615,60535.1330,1,-1084.891113,6.952323,1
615,60535.1408,3,-488.010925,3.928481,1
615,60535.1519,4,-222.254257,4.034600,1
615,60535.1630,5,-85.524307,8.625449,1
615,60538.2351,2,113.021248,2.712380,1
615,60538.2428,1,51.060081,3.762334,1
615,60538.2505,3,-4.268328,2.895656,0
615,60538.2615,4,-96.020035,3.141703,1
615,60538.2725,5,-175.912643,6.308159,1
615,60546.3406,2,178.624359,2.905459,1
615,60546.3482,1,142.089966,4.065646,1
615,60546.3558,3,41.418739,3.163731,1
615,60546.3668,4,-52.460590,3.784039,1
615,60546.3777,5,-112.286079,8.527776,1
615,60549.0879,2,-629.010254,3.867215,1
615,60549.0956,1,-1076.652100,6.604701,1
615,60549.1034,3,-435.558533,3.417534,1
615,60549.1145,4,-111.499573,3.424588,1
615,60549.1256,5,30.267401,7.478198,0
615,60554.0964,0,82.168922,4.318140,1
615,60555.0951,0,49.886921,3.917516,1
615,60556.0879,0,9.075453,4.103900,0
615,60557.0831,0,-30.764908,3.555157,1
615,60558.1093,0,-101.419899,3.653430,1
615,60559.1097,0,-110.688477,3.426444,1
615,60560.1065,0,-114.774445,4.013463,1
615,60567.2821,2,-447.681580,3.368270,1
615,60567.2897,1,-972.201111,6.347886,1
615,60567.2973,3,-176.163651,2.960412,1
615,60567.3083,4,140.860611,3.302721,1
615,60567.3192,5,228.033112,6.797573,1
615,60574.1118,2,-143.843872,2.547544,1
615,60574.1195,1,-812.792908,5.570116,1
615,60574.1272,3,86.606873,2.966459,1
615,60574.1383,4,257.570221,3.575394,1
615,60574.1493,5,302.167328,7.100554,1
615,60577.0186,2,-425.988464,3.239578,1
615,60577.0263,1,-963.216980,6.134610,1
615,60577.0340,3,-148.178238,3.247297,1
615,60577.0451,4,161.872543,4.228243,1
615,60577.0563,5,238.576889,9.461221,1
615,60580.0095,2,586.178345,3.996895,1
615,60580.0173,1,655.284058,5.148244,1
615,60580.0250,3,445.737061,3.952905,1
615,60580.0361,4,361.595764,4.508256,1
615,60580.0472,5,328.836731,9.460338,1
615,60582.0840,0,-51.614189,3.517908,1
615,60583.0169,0,20.364273,4.460314,0
615,60584.0117,0,-24.682575,3.866380,1
615,60585.0117,0,-63.546600,3.497667,1
615,60586.0123,0,-101.819290,3.383004,1
615,60587.0127,0,-110.978699,3.555624,1
615,60588.0131,0,-113.588432,3.241369,1
615,60593.0636,2,226.696259,2.514855,1
615,60593.0713,1,205.029755,3.258004,1
615,60593.0790,3,73.384720,2.584785,1
615,60593.0901,4,-19.212976,2.899512,0
615,60593.1012,5,-83.394951,6.073453,1
615,60596.0304,2,-224.917938,3.388916,1
615,60596.0381,1,-388.231476,4.931039,1
615,60596.0458,3,-274.108429,3.720238,1
615,60596.0569,4,-292.558990,4.188871,1
615,60596.0680,5,-354.074280,8.392479,1
615,60603.0208,2,404.391388,3.043772,1
615,60603.0286,1,70.494507,3.060846,1
615,60603.0363,3,338.994537,3.051842,1
615,60603.0473,4,362.888550,3.381572,1
615,60603.0584,5,378.188141,6.295821,1
615,60606.0225,2,422.610779,3.198191,1
615,60606.0303,1,457.502197,4.173640,1
615,60606.0379,3,205.937546,2.957614,1
615,60606.0490,4,123.048210,3.402847,1
615,60606.0601,5,33.726837,7.368811,0
615,60609.0247,2,-355.611389,2.720825,1
615,60609.0323,1,-537.169312,4.090708,1
615,60609.0400,3,-372.485565,2.985755,1
615,60609.0510,4,-350.518677,3.225662,1
615,60609.0621,5,-371.873230,6.150734,1
615,60612.0266,0,-110.649872,2.844200,1
615,60613.0269,0,-89.973892,2.937887,1
615,60614.0276,0,-10.015225,3.212408,0
615,60615.0375,0,99.438087,3.662484,1
615,60616.0290,0,120.849113,3.776495,1
615,60617.0295,0,121.411896,3.569777,1
615,60621.1734,2,56.559818,2.259825,1
615,60621.1810,1,-607.040771,4.452463,1
615,60621.1886,3,208.770279,2.959783,1
615,60621.1996,4,297.624725,3.718585,1
615,60621.2105,5,332.919006,8.157172,1
615,60624.1760,2,552.150269,3.917989,1
615,60624.1836,1,607.047668,5.140991,1
615,60624.1913,3,296.946533,3.475000,1
615,60624.2022,4,235.489929,3.926538,1
615,60624.2132,5,157.080200,8.453112,1
713,59825.2600,2,9.110147,1.013889,0
713,59825.2676,1,7.615042,1.160329,1
713,59825.2752,3,6.673631,1.932316,1
713,59825.2862,4,5.214194,3.018003,0
713,59825.2971,5,12.060948,7.163382,0
713,59839.2161,2,4.953065,1.196956,0
713,59839.2236,1,3.131028,1.351706,0
713,59839.2313,3,6.108739,1.789895,0
713,59839.2422,4,8.283792,2.527953,0
713,59839.2532,5,9.686500,5.882469,0
713,59842.1987,2,6.472355,1.110572,0
713,59842.2064,1,5.914848,1.134476,1
713,59842.2140,3,5.311658,1.683777,0
713,59842.2250,4,4.680908,2.349234,0
713,59842.2359,5,6.921503,5.573885,0
713,59851.2006,0,7.267655,2.866838,0
713,59854.2089,2,3.945918,0.922779,0
713,59854.2165,1,2.956484,0.953529,0
713,59854.2242,3,4.768611,1.422675,0
713,59854.2351,4,4.065430,1.943197,0
713,59854.2461,5,1.290383,3.980583,0
713,59857.1879,2,3.937931,2.088610,0
713,59857.1956,1,-2.223347,2.861396,0
713,59857.2032,3,-0.476698,2.280299,0
713,59857.2141,4,8.054095,2.712542,0
713,59857.2251,5,-9.332252,5.174713,0
713,59867.1600,2,1.380378,1.105223,0
713,59867.1676,1,1.569406,1.369900,0
713,59867.1753,3,1.433712,1.730335,0
713,59867.1862,4,5.299760,2.530279,0
713,59867.1971,5,2.355590,5.574841,0
713,59870.1521,2,3.404463,0.889940,0
713,59870.1597,1,2.961649,0.946383,0
713,59870.1673,3,2.594970,1.322884,0
713,59870.1782,4,2.805032,1.753258,0
713,59870.1892,5,3.101222,4.041600,0
713,59873.1442,2,3.869869,0.763644,0
713,59873.1519,1,4.354049,0.790372,1
713,59873.1595,3,2.949366,1.101570,0
713,59873.1704,4,3.117238,1.623630,0
713,59873.1814,5,4.010789,3.832515,0
713,59874.1612,0,4.171277,2.180456,0
713,59875.1175,0,2.671449,2.536783,0
713,59876.1160,0,-0.354117,2.566333,0
713,59877.1178,0,2.332870,2.217193,0
713,59878.1127,0,3.797837,2.270967,0
713,59879.1104,0,4.900619,2.339577,0
713,59880.1181,0,1.331082,1.865762,0
713,59884.1292,2,3.091794,0.995209,0
713,59884.1368,1,1.712878,1.129837,0
713,59884.1444,3,2.552051,1.229161,0
713,59884.1554,4,2.099711,1.737169,0
713,59884.1663,5,5.788035,3.869600,0
713,59887.0951,2,5.011691,1.668699,0
713,59887.1027,1,3.719429,2.314904,0
713,59887.1103,3,10.036420,1.852814,0
713,59887.1213,4,5.964674,2.277389,0
713,59887.1322,5,8.822542,4.767565,0
713,59896.0839,2,5.064992,0.999215,0
713,59896.0915,1,5.780192,1.083338,0
713,59896.0992,3,3.587355,1.642954,1
713,59896.1101,4,6.185760,2.277758,0
713,59896.1211,5,-4.762829,5.042903,0
713,59899.0854,2,4.823127,1.244829,0
713,59899.0930,1,6.899071,1.246326,0
713,59899.1007,3,3.249064,2.133093,0
713,59899.1116,4,7.382133,3.388385,0
713,59899.1226,5,-4.677240,9.115748,0
713,59902.0445,2,9.166100,1.426165,0
713,59902.0522,1,8.076466,1.395627,0
713,59902.0598,3,11.330316,2.051576,1
713,59902.0707,4,9.245844,2.876306,0
713,59902.0817,5,0.942024,7.248375,0
713,59904.0584,0,3.223553,2.679078,0
713,59905.0468,0,14.509829,3.098125,0
713,59906.0474,0,5.995616,2.589032,0
713,59907.0480,0,5.440472,2.469325,0
713,59908.0487,0,5.961231,3.348282,0
713,59909.0494,0,10.137896,2.151001,0
713,59910.0590,0,8.248549,2.160179,0
713,59913.2446,2,5.475236,0.822163,1
713,59913.2522,1,6.833441,0.969664,0
713,59913.2599,3,6.275328,1.430679,0
713,59913.2708,4,4.298039,2.311868,0
713,59913.2818,5,3.143612,5.875287,0
713,59916.0544,2,10.529041,1.787002,0
713,59916.0621,1,9.129021,2.415574,0
713,59916.0697,3,5.509865,2.141148,0
713,59916.0806,4,9.827934,2.274502,0
713,59916.0915,5,2.627945,4.551546,0
713,59924.0589,2,5.190053,0.786980,1
713,59924.0665,1,6.531730,0.851491,1
713,59924.0742,3,9.141804,1.210878,1
713,59924.0851,4,9.810373,1.739901,0
713,59924.0961,5,2.349317,4.040898,0
713,59927.0604,2,5.366942,0.863455,0
713,59927.0680,1,4.619713,0.947374,0
713,59927.0756,3,6.296741,1.472587,0
713,59927.0866,4,2.465199,2.073566,0
713,59927.0975,5,-0.702472,6.396966,0
713,59930.0619,2,2.780317,0.759708,1
713,59930.0695,1,4.959312,0.809846,0
713,59930.0771,3,4.033259,1.196190,0
713,59930.0881,4,4.485665,1.901773,0
713,59930.1063,5,6.218721,4.791905,0
713,59933.0632,2,1.830853,0.639458,0
713,59933.0709,1,1.716145,0.707228,0
713,59933.0785,3,4.893567,0.968482,0
713,59933.0944,4,3.197614,1.429430,0
713,59933.1150,5,3.335699,3.523145,0
713,59935.0739,0,0.554208,1.573855,0
713,59936.0735,0,2.584441,1.804314,0
713,59937.0743,0,6.470248,1.848658,0
713,59938.0754,0,0.724684,2.076312,0
713,59939.0808,0,2.375108,2.243821,0
713,59942.0746,2,-0.148046,0.800387,0
713,59942.0889,1,0.648101,0.878962,0
713,59942.0965,3,0.328905,1.080046,0
713,59942.1074,4,-4.550706,1.669870,0
713,59942.1184,5,2.364145,4.108390,0
713,59945.0770,2,-3.002108,1.474453,0
713,59945.0846,1,-1.725136,2.059556,0
713,59945.0922,3,-1.422123,1.846779,0
713,59945.1032,4,-6.208874,2.036851,0
713,59945.1141,5,-2.945050,4.744831,0
713,60192.2930,2,7.250862,1.972519,0
713,60192.3006,1,7.834616,2.653803,0
713,60192.3082,3,6.543319,2.216304,0
713,60192.3192,4,7.960829,2.695718,0
713,60192.3301,5,3.404367,5.894906,0
713,60195.2343,2,4.326025,1.101823,0
713,60195.2419,1,3.302556,1.100982,1
713,60195.2496,3,5.533146,1.655807,0
713,60195.2605,4,5.854890,2.367182,0
713,60195.2715,5,3.825871,5.774144,0
713,60198.2332,0,2.208139,3.192551,0
713,60199.2358,0,8.620851,2.547614,0
713,60200.2314,0,3.770694,2.643626,0
713,60201.2281,0,6.634655,2.975509,0
713,60202.2255,0,9.813441,2.750465,0
713,60209.2281,2,3.984369,1.161990,0
713,60209.2357,1,1.381281,1.139709,0
713,60209.2433,3,1.715379,1.724909,0
713,60209.2543,4,0.545876,2.622813,0
713,60209.2652,5,6.504875,6.293293,0
713,60212.2147,2,2.873843,1.790648,0
713,60212.2223,1,1.546698,2.668681,0
713,60212.2300,3,5.084908,2.077699,0
713,60212.2409,4,3.087726,2.654123,0
713,60212.2519,5,-2.272981,5.844298,0
713,60223.1948,2,0.405613,1.246678,0
713,60223.2024,1,1.120193,1.277229,0
713,60223.2100,3,1.011539,1.974625,0
713,60223.2210,4,3.507817,2.882992,0
713,60223.2319,5,14.770886,6.656366,0
713,60226.2721,2,1.071414,0.746168,0
713,60226.2797,1,1.648819,0.776689,0
713,60226.2931,3,1.727918,1.133994,0
713,60226.3129,4,-0.916487,1.736045,0
713,60226.3238,5,3.996732,4.304620,0
713,60236.1862,0,4.067199,2.328237,0
713,60238.2696,2,-0.087907,0.758784,0
713,60238.2803,1,-0.829578,0.881391,0
713,60238.2879,3,-0.576265,1.265385,0
713,60238.2988,4,0.329135,1.983817,0
713,60238.3098,5,-4.923808,4.832184,0
713,60241.1342,2,2.333379,1.636026,0
713,60241.1418,1,-0.293893,2.097461,0
713,60241.1495,3,3.571144,1.816384,0
713,60241.1604,4,1.146531,2.471305,0
713,60241.1713,5,-7.436915,5.122927,0
713,60260.0773,0,1.232121,3.040076,0
713,60261.0632,0,2.412768,2.796987,0
713,60262.0637,0,-1.678317,2.631186,0
713,60263.0643,0,-0.390618,2.617705,0
713,60264.0716,0,1.131548,2.522264,0
713,60265.0867,0,-2.189290,2.832789,0
713,60267.0443,2,-2.785007,1.335972,0
713,60267.0519,1,-1.797494,1.524965,0
713,60267.0595,3,-4.881196,1.920069,0
713,60267.0705,4,3.231234,2.734247,0
713,60267.0814,5,-12.699218,5.767424,0
713,60270.1082,2,0.393975,2.344530,0
713,60270.1158,1,2.370688,3.545214,0
713,60270.1234,3,2.489378,2.970400,0
713,60270.1344,4,-7.822262,3.554679,0
713,60270.1453,5,-10.877887,6.829591,0
713,60278.0525,2,-0.607012,0.969379,0
713,60278.0601,1,-0.027766,1.022582,0
713,60278.0677,3,2.024312,1.532588,0
713,60278.0786,4,-2.256550,2.154194,0
713,60278.0896,5,-1.112494,5.843420,0
713,60281.0552,2,-1.695972,1.034572,0
713,60281.0629,1,-1.833499,1.044974,0
713,60281.0705,3,-0.919016,1.470630,0
713,60281.0814,4,-1.391540,2.003621,0
713,60281.0924,5,-3.945375,4.607381,0
713,60284.0557,2,-2.572076,0.783297,0
713,60284.0633,1,-2.387862,0.801296,0
713,60284.0709,3,-5.832908,1.240340,0
713,60284.0819,4,-1.649157,1.857165,0
713,60284.0928,5,8.627832,4.482957,0
713,60287.0577,2,-0.764727,0.906658,0
713,60287.0653,1,-1.477176,0.966182,0
713,60287.0729,3,-0.536819,1.463827,0
713,60287.0839,4,-0.667864,2.361719,0
713,60287.0948,5,-5.286497,6.810267,0
713,60290.0641,0,-1.816348,2.038470,0
713,60291.0599,0,-5.132619,1.989085,0
713,60292.0607,0,-5.080487,1.908693,0
713,60293.0607,0,-5.075594,1.939040,0
713,60294.0616,0,-0.566193,1.833754,0
713,60295.0621,0,-3.857503,2.114682,0
713,60297.1169,2,-4.830737,0.921245,0
713,60297.1245,1,-4.334117,1.068175,0
713,60297.1321,3,-4.767125,1.234546,0
713,60297.1431,4,-4.473659,1.723665,0
713,60297.1540,5,-10.414721,4.258311,0
713,60300.0641,2,-5.492156,1.607434,0
713,60300.0717,1,-3.599649,2.103016,0
713,60300.0793,3,-3.476922,1.725975,0
713,60300.1031,4,-2.745461,2.269754,0
713,60300.1160,5,7.738044,4.946638,0
713,60554.2916,0,-9.100129,2.393532,0
713,60555.2620,0,-8.218450,2.308315,1
713,60556.2548,0,-5.576579,2.770439,0
713,60557.2501,0,-9.173389,2.218352,0
713,60558.2534,0,-13.083604,2.663738,0
713,60559.2490,0,-9.237353,2.428750,0
713,60560.2424,0,-10.050170,3.275514,0
713,60567.2231,2,-8.265152,1.515329,0
713,60567.2308,1,-8.954789,2.105672,1
713,60567.2384,3,-8.418892,1.630414,0
713,60567.2493,4,-12.286801,1.978125,1
713,60567.2603,5,-11.054881,4.445991,0
713,60578.2746,2,-4.951467,1.229683,0
713,60578.2822,1,-7.403615,1.305094,0
713,60578.2898,3,-5.050255,1.839125,1
713,60578.3008,4,-7.385537,2.667687,0
713,60578.3117,5,-6.356452,6.387929,0
713,60581.1779,2,-5.760825,1.288651,1
713,60581.1855,1,-7.428378,1.275975,0
713,60581.1931,3,-6.902376,1.927237,0
713,60581.2041,4,-9.594004,2.818656,0
713,60581.2150,5,-14.211164,6.624023,0
713,60582.2087,0,-11.829331,2.358846,0
713,60583.1842,0,-9.363182,3.042286,0
713,60584.1807,0,-9.220502,2.544668,0
713,60585.1757,0,-3.587870,2.280919,0
713,60586.1765,0,-9.129416,2.146863,0
713,60587.1702,0,-5.876253,2.481174,0
713,60588.1666,0,-9.116284,2.157747,0
713,60593.1682,2,-9.569608,0.985850,1
713,60593.1758,1,-8.809836,1.078624,1
713,60593.1834,3,-9.553467,1.347112,1
713,60593.1944,4,-9.193518,1.914358,1
713,60593.2053,5,-8.280509,4.493694,0
713,60596.1820,2,-8.760753,2.037911,0
713,60596.1896,1,-4.396494,2.863201,0
713,60596.1972,3,-11.907238,2.252078,1
713,60596.2081,4,-4.786119,2.784098,0
713,60596.2191,5,-5.489277,6.255779,0
713,60605.1380,2,-9.696579,0.873996,1
713,60605.1456,1,-11.159884,0.883977,1
713,60605.1532,3,-12.394593,1.305202,1
713,60605.1642,4,-9.511388,1.906236,0
713,60605.1751,5,-6.906372,4.587698,0
713,60608.1308,2,-9.163915,0.800012,1
713,60608.1384,1,-11.715749,0.823976,1
713,60608.1460,3,-11.449253,1.202452,0
713,60608.1569,4,-12.221146,1.749559,1
713,60608.1679,5,-2.633516,4.132709,0
713,60611.1227,2,-10.067919,0.717739,1
713,60611.1303,1,-9.289042,0.761477,1
713,60611.1380,3,-10.801243,1.080986,1
713,60611.1489,4,-11.623042,1.560488,1
713,60611.1599,5,-7.861447,3.710802,0
713,60612.1183,0,-11.605895,1.778605,1
713,60613.1019,0,-11.340659,1.930082,1
713,60614.0960,0,-10.934606,2.143276,1
713,60615.0917,0,-14.735178,2.326417,0
713,60616.0927,0,-12.353376,2.357691,1
713,60617.0896,0,-6.599936,2.023456,0
713,60620.1350,0,-6.110061,2.056073,0
713,60621.1263,2,-5.537477,0.829998,1
713,60621.1339,1,-7.972793,0.877838,1
713,60621.1416,3,-7.565215,1.233034,1
713,60621.1525,4,-6.638791,1.780862,0
713,60621.1635,5,-9.333499,4.242186,0
713,60624.0821,2,-4.490414,1.314625,0
713,60624.0897,1,-5.545699,1.695655,0
713,60624.0974,3,-7.286825,1.600662,0
713,60624.1083,4,-6.478677,2.086655,0
713,60624.1193,5,-7.099849,4.810002,0
713,60627.2801,2,-6.219934,2.530638,0
713,60627.2877,1,-5.039655,3.228468,0
713,60627.2954,3,-0.950650,2.782719,0
713,60627.3063,4,-4.143107,3.379841,0
713,60627.3173,5,0.243241,7.639313,0
713,60632.0400,2,-5.855491,1.492071,0
713,60632.0476,1,-6.847743,1.453141,0
713,60632.0552,3,-9.374930,2.217679,0
713,60632.0662,4,-10.557325,3.237312,0
713,60632.0771,5,-2.921649,7.857955,0
713,60635.0469,2,-6.480945,1.091159,1
713,60635.0545,1,-6.966879,1.087843,1
713,60635.0621,3,-6.185159,1.629475,1
713,60635.0731,4,-5.490345,2.377301,0
713,60635.0840,5,-2.292507,5.646507,0
713,60640.0504,2,-7.312206,0.974580,1
713,60640.0580,1,-7.250492,0.991461,0
713,60640.0656,3,-10.161006,1.479101,1
713,60640.0766,4,-6.631466,2.145102,0
713,60640.0875,5,-10.591419,5.138685,0
713,60643.0609,0,-9.289350,1.992813,1
713,60644.0533,0,-8.482151,2.118450,0
713,60645.0537,0,-2.605739,2.197297,1
713,60646.0548,0,-8.104684,2.135281,0
713,60647.0546,0,-7.506279,2.275638,0
713,60648.0553,0,-10.602926,1.838902,1
713,60649.0561,0,-12.232555,1.708795,0
713,60651.1265,2,-9.331477,0.865811,1
713,60651.1451,1,-10.061421,0.932510,1
713,60651.1527,3,-9.335849,1.315029,1
713,60651.1637,4,-6.167844,1.952829,0
713,60651.1746,5,-10.171921,4.815349,0
713,60654.0597,2,-9.607999,1.647062,1
713,60654.0673,1,-6.258916,2.222855,0
713,60654.0749,3,-9.524345,1.901351,0
713,60654.0859,4,-9.513783,2.389906,0
713,60654.0968,5,-9.744430,5.404162,0
713,60662.1451,2,-5.698765,1.334831,0
713,60662.1527,1,-4.317381,2.017339,0
713,60662.1603,3,-6.093997,1.675434,0
713,60662.1713,4,-5.760686,2.093239,0
713,60662.1822,5,-5.400730,4.721159,0
713,60665.0637,2,-6.826318,0.676851,1
713,60665.0713,1,-6.498077,0.740772,1
713,60665.0789,3,-5.938825,1.048616,1
713,60665.1017,4,-6.440791,1.543503,1
713,60665.1156,5,-13.727009,3.770338,0
713,60668.0647,2,-6.938087,0.920544,1
713,60668.0723,1,-8.995543,0.954973,1
713,60668.0893,3,-10.263328,1.437371,1
713,60668.1055,4,-5.455149,2.164149,0
713,60668.1165,5,-9.138229,5.354884,0
713,60671.0655,0,-10.165054,1.726118,1
713,60672.0693,0,-10.828177,1.470152,1
713,60673.0745,0,-12.148479,2.243120,0
713,60674.0798,0,-8.669188,2.216094,0
730,59798.3205,2,1.177371,1.364300,0
730,59798.3281,1,2.320849,1.159247,0
730,59798.3357,3,2.939447,1.771328,0
730,59798.3466,4,2.128097,2.610659,0
730,59798.3576,5,-12.809639,5.380097,0
730,59801.3553,2,0.111235,2.460576,0
730,59801.3629,1,-3.393080,3.564052,0
730,59801.3705,3,-1.899219,2.292693,0
730,59801.3815,4,2.284906,2.523534,0
730,59801.3924,5,5.203419,5.395980,0
730,59818.2740,0,-2.342200,1.801066,0
730,59819.2541,0,3.380978,2.469600,0
730,59820.2522,0,-2.230815,1.915426,0
730,59821.2478,0,1.159034,2.461736,0
730,59822.2433,0,5.942166,2.901580,0
730,59823.2659,0,-0.180970,2.714361,0
730,59826.3105,2,0.521923,0.925337,0
730,59826.3181,1,-1.421768,0.929596,0
730,59826.3258,3,0.972355,1.513987,0
730,59826.3367,4,-0.570261,2.162375,0
730,59826.3477,5,-2.301237,5.548611,0
730,59842.2456,2,0.156290,0.853800,0
730,59842.2532,1,-0.567360,0.819375,0
730,59842.2608,3,-0.251899,1.325633,0
730,59842.2718,4,2.019500,2.173066,0
730,59842.2827,5,10.142254,6.086383,0
730,59851.1792,0,-1.472170,2.597541,0
730,59854.1485,2,0.368931,1.230250,0
730,59854.1563,1,0.664051,1.345911,0
730,59854.1640,3,3.201455,1.909905,0
730,59854.1750,4,3.012713,2.778862,0
730,59854.1860,5,3.750187,5.803461,0
730,59857.1408,2,1.076537,2.141015,0
730,59857.1485,1,5.693109,2.937809,0
730,59857.1563,3,-2.640246,2.100464,0
730,59857.1673,4,0.402461,2.684283,0
730,59857.1782,5,-4.509360,6.643411,0
730,59867.1112,2,-0.449365,1.088300,0
730,59867.1189,1,0.282022,0.981426,0
730,59867.1267,3,0.026595,1.544194,0
730,59867.1377,4,0.956947,2.364042,0
730,59867.1487,5,2.724518,5.845339,0
730,59870.1049,2,1.070328,0.976301,0
730,59870.1126,1,0.511964,0.828288,0
730,59870.1204,3,-0.505236,1.377689,0
730,59870.1314,4,0.251195,2.289763,0
730,59870.1424,5,4.119082,5.293428,0
730,59873.0971,2,-0.211154,1.045822,0
730,59873.1049,1,-1.287062,1.048773,0
730,59873.1126,3,-1.557674,1.446841,0
730,59873.1236,4,-0.739414,2.074561,0
730,59873.1346,5,5.151175,4.601235,0
730,59874.1461,0,0.412505,1.627923,0
730,59875.0995,0,-2.200486,2.037783,0
730,59876.0980,0,-2.931559,2.450620,0
730,59877.0976,0,2.024089,1.789397,0
730,59878.0964,0,-1.250103,2.029308,0
730,59879.0895,0,-0.671039,1.877854,0
730,59880.1017,0,0.189355,1.384724,0
730,59884.1760,2,-0.108323,0.771566,0
730,59884.1836,1,-1.113737,0.892852,0
730,59884.1913,3,-0.427802,1.142666,0
730,59884.2022,4,1.402694,1.614300,0
730,59884.2132,5,-1.060647,4.220271,0
730,59887.2856,2,1.474370,1.835391,0
730,59887.2933,1,-3.203188,2.221069,0
730,59887.3009,3,-5.435799,2.359130,0
730,59887.3118,4,0.192088,3.154000,0
730,59887.3228,5,-1.082339,7.966248,0
730,59896.1307,2,1.145174,0.826742,0
730,59896.1384,1,-0.032153,0.707979,0
730,59896.1460,3,-0.357363,1.398256,0
730,59896.1569,4,-2.788487,2.198583,0
730,59896.1679,5,2.883538,5.962979,0
730,59899.1519,2,-1.456884,1.371527,0
730,59899.1595,1,-0.707794,1.396877,0
730,59899.1672,3,5.298447,2.388603,0
730,59899.1781,4,6.412822,3.720956,0
730,59899.1891,5,1.091714,7.924479,0
730,59902.1384,2,-0.887660,1.191683,0
730,59902.1460,1,0.168580,1.085883,0
730,59902.1537,3,-1.369444,1.950019,0
730,59902.1646,4,0.303218,2.770533,0
730,59902.1755,5,11.777126,7.044582,0
730,59904.1053,0,-0.620050,2.301550,0
730,59905.0555,0,0.922903,2.527480,0
730,59906.0562,0,-1.153271,2.043133,0
730,59907.0567,0,0.449173,1.764913,0
730,59908.0681,0,0.837362,2.710272,0
730,59909.0582,0,-0.985495,1.850359,0
730,59910.0503,0,-0.355463,1.880359,0
730,59914.0526,2,-0.833646,1.445693,0
730,59914.0602,1,0.370377,1.970406,0
730,59914.0678,3,-1.719942,1.692403,0
730,59914.0788,4,1.004354,2.274112,0
730,59914.0897,5,0.138586,5.367689,0
730,59924.1060,2,-0.787230,1.137160,0
730,59924.1136,1,-1.572903,1.584968,0
730,59924.1212,3,0.555294,1.735223,0
730,59924.1322,4,-2.475216,2.533980,0
730,59924.1431,5,-0.816748,5.644574,0
730,59927.1074,2,-0.474538,1.196533,0
730,59927.1151,1,0.973025,1.142775,0
730,59927.1227,3,-2.039601,1.598035,0
730,59927.1336,4,-1.036243,2.305239,0
730,59927.1446,5,-3.438392,5.903537,0
730,59930.1236,2,-0.114812,0.945627,0
730,59930.1312,1,0.475511,0.835235,0
730,59930.1388,3,0.226621,1.287869,0
730,59930.1498,4,-3.755495,2.037717,0
730,59930.1607,5,2.542647,5.343603,0
730,59933.1249,2,0.603719,0.695106,0
730,59933.1325,1,-0.226574,0.698751,0
730,59933.1401,3,0.106692,1.273440,0
730,59933.1511,4,0.993756,1.919590,0
730,59933.1620,5,5.318815,5.441072,0
730,59934.0638,0,1.190260,1.159169,0
730,59935.0646,0,-0.320948,1.132809,0
730,59936.0642,0,-1.230814,1.533033,0
730,59937.0650,0,-0.751357,1.654440,0
730,59938.0647,0,-3.109122,2.015928,0
730,59939.0650,0,1.571790,2.219707,0
730,60165.3032,2,-0.502432,1.200698,0
730,60165.3109,1,-2.832010,1.356671,0
730,60165.3186,3,1.843434,1.524752,0
730,60165.3295,4,3.196369,2.029726,0
730,60165.3405,5,-8.889149,4.793297,0
730,60168.2892,2,-0.176546,0.837958,0
730,60168.2970,1,0.201754,0.793672,0
730,60168.3047,3,0.369397,1.305260,0
730,60168.3157,4,-2.235131,2.071596,0
730,60168.3267,5,-5.047883,4.827778,0
730,60176.2820,0,1.753881,2.391554,0
730,60177.2726,0,-1.260694,2.790846,0
730,60181.4088,2,1.103341,0.929294,0
730,60181.4164,1,1.391831,1.272189,0
730,60181.4232,3,-4.766650,3.249354,0
730,60183.2660,2,-2.848838,1.924783,0
730,60183.2736,1,-3.133007,2.955767,0
730,60183.2812,3,2.159384,2.557742,0
730,60183.2922,4,-5.836310,3.003132,0
730,60183.3031,5,-8.716421,6.192660,0
730,60195.2812,2,-0.162802,0.937681,0
730,60195.2888,1,0.864197,1.028533,0
730,60195.2964,3,-2.294667,1.631539,0
730,60195.3073,4,-2.693345,2.345676,0
730,60195.3183,5,-0.570636,6.289552,0
730,60198.2690,0,1.026459,2.192766,0
730,60199.2186,0,0.527036,2.176080,0
730,60200.2139,0,-1.516695,2.611164,0
730,60201.2072,0,0.150572,2.513518,0
730,60202.2089,0,-1.982165,2.505382,0
730,60209.1811,2,-0.462435,1.572888,0
730,60209.1888,1,2.153122,1.421764,0
730,60209.1965,3,0.765511,1.838664,0
730,60209.2075,4,-1.591831,2.917069,0
730,60209.2184,5,0.939232,6.712256,0
730,60212.1675,2,-2.669531,1.944909,0
730,60212.1753,1,3.706729,2.772595,0
730,60212.1830,3,-1.340184,2.303333,0
730,60212.1941,4,-3.525083,2.598093,0
730,60212.2050,5,7.831807,5.678200,0
730,60223.2416,2,1.328195,1.188578,0
730,60223.2493,1,-0.298775,1.249490,0
730,60223.2569,3,2.987647,1.656098,0
730,60223.2678,4,4.241424,2.481011,0
730,60223.2788,5,9.333996,5.811805,0
730,60226.3337,2,-0.679090,0.949060,0
730,60226.3413,1,1.282680,0.952563,0
730,60226.3489,3,0.970081,1.800158,0
730,60226.3599,4,1.366870,3.079182,0
730,60226.3708,5,-4.552550,8.128254,0
730,60238.3197,2,-0.599053,1.058323,0
730,60238.3273,1,0.734427,1.161131,0
730,60238.3349,3,2.553997,2.179708,0
730,60238.3459,4,1.377842,3.840058,0
730,60238.3568,5,-19.159811,11.281384,0
730,60241.0870,2,0.823192,2.008905,0
730,60241.0948,1,3.386674,3.088520,0
730,60241.1025,3,0.043122,2.453789,0
730,60241.1136,4,-1.052531,2.925313,0
730,60241.1245,5,-8.036972,5.953956,0
730,60250.1708,2,-0.962673,2.112349,0
730,60250.1957,1,0.580816,2.829899,0
730,60250.2034,3,6.845217,2.940232,0
730,60250.2143,4,0.204509,3.994097,0
730,60250.2253,5,1.290714,7.954757,0
730,60261.1296,0,1.237353,2.094631,0
730,60262.0550,0,3.469973,2.338792,0
730,60263.0556,0,2.352035,1.998888,0
730,60264.0559,0,-2.396658,2.192123,0
730,60265.0780,0,3.070599,2.439756,0
730,60268.0449,2,-0.465229,1.410433,0
730,60268.0525,1,0.174091,1.861911,0
730,60268.0601,3,0.538344,2.137292,0
730,60268.0711,4,-3.556071,2.741589,0
730,60268.0820,5,-3.639747,6.760314,0
730,60278.0993,2,2.521567,1.652593,0
730,60278.1069,1,-2.468382,2.289480,0
730,60278.1145,3,-1.407348,2.177464,0
730,60278.1255,4,3.475310,3.068326,0
730,60278.1364,5,8.474236,7.658961,0
730,60281.1023,2,-1.139811,0.801878,0
730,60281.1099,1,-1.247972,0.800422,0
730,60281.1175,3,-1.347594,1.374244,0
730,60281.1285,4,-0.890039,1.996277,0
730,60281.1394,5,2.285095,5.189152,0
730,60284.1027,2,-0.679968,0.840813,0
730,60284.1104,1,-0.530991,0.766401,0
730,60284.1180,3,-1.148911,1.164351,0
730,60284.1289,4,0.493227,1.707686,0
730,60284.1399,5,-4.683412,4.445528,0
730,60287.1047,2,0.288175,1.256500,0
730,60287.1123,1,-2.067724,1.090506,0
730,60287.1200,3,-0.846692,1.802521,0
730,60287.1309,4,-5.051833,2.972183,0
730,60287.1418,5,-13.252449,7.029711,0
730,60290.0761,0,-3.000368,1.929932,0
730,60291.0689,0,1.081815,1.681175,0
730,60292.0699,0,2.211185,1.882060,0
730,60293.0699,0,0.182480,1.575780,0
730,60294.0708,0,1.605139,1.418435,0
730,60532.3019,2,20.994711,1.047298,1
730,60532.3097,1,1.504146,0.960956,0
730,60532.3173,3,31.523088,1.569497,1
730,60532.3282,4,41.159981,2.310168,1
730,60532.3392,5,46.795868,5.458707,1
730,60535.2802,2,20.880348,1.426747,0
730,60535.2879,1,2.271271,1.348233,0
730,60535.2957,3,30.361010,2.107024,1
730,60535.3068,4,40.715591,3.043571,1
730,60535.3177,5,47.310059,7.197146,1
730,60538.2826,2,19.450977,1.865142,1
730,60538.2903,1,3.462672,2.695356,0
730,60538.2980,3,33.572102,1.944897,1
730,60538.3089,4,38.518837,2.334413,1
730,60538.3199,5,40.146099,5.039364,1
730,60554.2651,0,0.190944,2.266587,0
730,60555.2411,0,0.098122,2.049620,0
730,60556.2370,0,-0.253067,2.551228,0
730,60557.2322,0,-2.200897,1.848830,0
730,60558.2332,0,-3.459960,2.511074,0
730,60559.2274,0,0.328893,2.224590,0
730,60560.2268,0,2.453341,3.110694,0
730,60567.3291,2,15.044784,0.951184,1
730,60567.3368,1,-0.142653,1.050350,0
730,60567.3444,3,18.416132,1.262663,1
730,60567.3553,4,28.234451,1.676854,1
730,60567.3663,5,31.623583,4.281011,1
730,60580.1736,2,12.164557,1.463993,1
730,60580.1813,1,2.065962,1.402610,0
730,60580.1889,3,10.053763,2.203885,1
730,60580.1999,4,19.975168,3.213686,1
730,60580.2108,5,24.093925,7.662856,0
730,60582.1681,0,-0.473370,2.422541,0
730,60583.1640,0,-3.070249,3.006098,0
730,60584.1591,0,0.970706,2.362254,0
730,60585.1601,0,-0.533032,1.881978,0
730,60586.1564,0,-0.049936,1.830623,0
730,60587.1540,0,-2.202578,2.138732,0
730,60588.1461,0,1.361049,1.798501,0
730,60593.1209,2,6.307311,1.155241,1
730,60593.1287,1,0.462838,1.313489,0
730,60593.1365,3,8.789671,1.545997,1
730,60593.1476,4,10.031554,2.179338,0
730,60593.1585,5,10.850924,5.118365,0
730,60596.1351,2,8.231540,2.483539,0
730,60596.1427,1,1.623348,3.539990,0
730,60596.1504,3,9.673650,2.658536,0
730,60596.1613,4,8.778720,3.238315,0
730,60596.1723,5,10.870938,7.199404,0
730,60605.0908,2,5.607800,0.980471,1
730,60605.0986,1,-0.587054,0.918425,0
730,60605.1063,3,6.155015,1.461859,1
730,60605.1174,4,7.274523,2.152366,0
730,60605.1283,5,2.950838,5.145659,0
730,60608.0836,2,5.938226,0.889949,0
730,60608.0913,1,0.898013,0.825604,0
730,60608.0991,3,5.509429,1.330889,0
730,60608.1101,4,9.166319,1.962560,0
730,60608.1211,5,3.346682,4.667600,0
730,60611.0756,2,2.112415,0.773398,1
730,60611.0833,1,0.247475,0.704158,0
730,60611.0911,3,1.898379,1.172223,0
730,60611.1021,4,4.244992,1.745410,0
730,60611.1130,5,6.172510,4.175368,0
730,60612.0813,0,1.228119,1.461220,0
730,60613.0818,0,1.540095,1.541647,0
730,60614.0803,0,1.231758,1.758784,0
730,60615.0761,0,-0.502854,1.998764,0
730,60616.0769,0,-2.247711,2.024976,0
730,60617.0737,0,-1.035569,1.574103,0
730,60620.1444,0,-1.018565,1.722706,0
730,60621.2673,2,2.330264,0.944892,0
730,60621.2749,1,-0.153496,0.950369,0
730,60621.2825,3,3.589653,1.651967,0
730,60621.2934,4,1.950011,2.746167,0
730,60621.3044,5,6.258384,7.527862,0
================================================
FILE: examples/data/plasticc_training_set_metadata_1k.csv
================================================
object_id,ra,decl,gal_l,gal_b,ddf,hostgal_specz,hostgal_photoz,hostgal_photoz_err,distmod,mwebv,target
615,349.046051,-61.943836,320.796530,-51.753706,1,0.0000,0.0000,0.0000,nan,0.017,92
713,53.085938,-27.784405,223.525509,-54.460748,1,1.8181,1.6267,0.2552,45.4063,0.007,88
730,33.574219,-6.579593,170.455585,-61.548219,1,0.2320,0.2262,0.0157,40.2561,0.021,42
745,0.189873,-45.586655,328.254458,-68.969298,1,0.3037,0.2813,1.1523,40.7951,0.007,90
1124,352.711273,-63.823658,316.922299,-51.059403,1,0.1934,0.2415,0.0176,40.4166,0.024,90
1227,35.683594,-5.379379,171.992947,-59.253501,1,0.0000,0.0000,0.0000,nan,0.020,65
1598,347.846710,-64.760857,318.929827,-49.143596,1,0.1352,0.1820,0.0304,39.7279,0.019,90
1632,348.595886,-63.072620,320.023289,-50.713060,1,0.6857,0.7014,0.0100,43.1524,0.021,42
1920,149.414062,3.433834,234.919132,42.245550,1,0.3088,0.3229,0.3360,41.1401,0.027,90
1926,149.414062,1.940072,236.565366,41.393323,1,0.0000,0.0000,0.0000,nan,0.018,65
2072,0.965665,-46.375080,325.845907,-68.579427,1,0.1516,0.1900,0.0104,39.8317,0.007,90
2103,346.500000,-62.320400,321.951129,-50.736054,1,0.1695,0.5409,0.2283,42.4667,0.020,42
2300,359.446716,-44.201530,331.730015,-69.805709,1,0.2360,2.7474,0.5335,46.7959,0.010,42
2330,359.805206,-46.768478,327.135979,-67.829903,1,0.4541,0.5736,0.2827,42.6207,0.011,90
2624,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65
2677,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,16
2922,352.398651,-62.696659,318.017427,-51.967966,1,0.1539,0.1469,0.0094,39.2171,0.020,67
3041,346.130127,-63.072620,321.423103,-50.042305,1,0.1069,0.1274,0.0198,38.8800,0.020,67
3285,150.820312,1.641510,237.994507,42.358984,1,0.1610,0.1818,0.0079,39.7258,0.020,42
3423,349.615387,-63.636005,318.927246,-50.506542,1,1.9876,1.1213,0.1591,44.4078,0.018,95
3489,150.117188,2.836105,236.124718,42.483719,1,1.1330,1.4377,0.2168,45.0753,0.016,88
3910,0.589520,-47.161343,325.385896,-67.769893,1,0.1969,2.6766,0.5926,46.7274,0.009,62
4088,0.965665,-46.375080,325.845907,-68.579427,1,0.4833,0.4644,0.0321,42.0691,0.007,88
4132,359.811707,-45.191612,329.485675,-69.150905,1,0.0561,0.0556,0.0301,36.9750,0.010,42
4171,2.097458,-45.783966,324.737840,-69.478613,1,0.0000,0.0000,0.0000,nan,0.011,16
4173,152.050781,3.284369,237.157374,44.318466,1,0.5149,0.5512,0.0221,42.5158,0.019,15
4220,358.648071,-46.375080,329.462659,-67.716008,1,0.1197,0.1322,0.3351,38.9679,0.009,42
4389,151.699219,3.583322,236.533224,44.205648,1,0.2333,0.2205,0.9667,40.1939,0.016,90
4595,349.615387,-63.636005,318.927246,-50.506542,1,0.5919,0.5995,0.0127,42.7370,0.018,90
4819,35.332031,-5.979157,172.286722,-59.931743,1,0.3053,0.2870,0.0076,40.8445,0.022,90
5527,347.861847,-61.943836,321.519104,-51.424048,1,0.1315,0.2487,0.8604,40.4896,0.017,42
6180,33.222656,-4.780192,167.515653,-60.396584,1,0.3201,0.2685,0.5211,40.6793,0.018,90
6266,0.929752,-44.597992,328.531426,-70.083244,1,0.0000,0.0000,0.0000,nan,0.011,65
6762,348.595886,-63.072620,320.023289,-50.713060,1,0.3863,0.3983,0.0132,41.6735,0.021,90
6947,34.277344,-5.679190,170.314930,-60.410322,1,0.5680,0.5667,0.0181,42.5888,0.020,90
7033,52.207031,-28.291550,224.208534,-55.300157,1,0.0826,0.0850,0.0073,37.9414,0.007,42
7164,347.861847,-61.943836,321.519104,-51.424048,1,0.4299,0.4245,0.0288,41.8371,0.017,90
7315,2.071130,-45.191612,325.606223,-69.989264,1,0.1330,0.1337,0.0171,38.9942,0.011,88
7409,352.398651,-62.696659,318.017427,-51.967966,1,3.4451,0.5176,1.2609,42.3516,0.020,88
7566,359.446716,-44.201530,331.730015,-69.805709,1,0.0000,0.0000,0.0000,nan,0.010,16
7698,347.013428,-62.508568,321.472056,-50.735330,1,0.2628,0.1876,0.0216,39.8011,0.018,90
7703,53.085938,-28.122234,224.100909,-54.509752,1,0.0830,0.0820,0.2257,37.8568,0.007,62
7756,149.414062,2.238686,236.239766,41.565558,1,0.0000,0.0000,0.0000,nan,0.017,16
8328,1.694561,-45.191612,326.278557,-69.858253,1,0.3779,0.4808,0.2970,42.1592,0.011,90
8688,32.695312,-4.929937,166.868469,-60.841230,1,0.0000,0.0000,0.0000,nan,0.018,65
8745,349.966217,-62.696659,319.542989,-51.376556,1,0.6276,0.6136,0.0129,42.7983,0.021,90
8784,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,16
9006,34.277344,-5.079716,169.526841,-59.956640,1,0.0000,0.0000,0.0000,nan,0.019,65
9172,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65
9184,0.949367,-45.586655,326.991548,-69.251686,1,1.4031,1.2719,0.4971,44.7463,0.013,88
9203,51.855469,-27.953188,223.543603,-55.561470,1,0.2138,0.1111,0.0626,38.5591,0.008,90
9543,352.132874,-63.636005,317.424173,-51.095855,1,0.0000,0.0000,0.0000,nan,0.021,65
9936,32.871094,-4.780192,166.959493,-60.615132,1,0.1633,0.0719,0.0389,37.5580,0.017,42
9985,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,65
10321,358.312500,-44.993881,332.185785,-68.685906,1,1.0833,1.1162,0.1020,44.3954,0.009,95
10337,54.667969,-27.615883,223.610785,-53.050840,1,0.6830,0.6725,0.0089,43.0404,0.009,90
10349,34.980469,-6.279288,172.180075,-60.389399,1,0.0000,0.0000,0.0000,nan,0.023,65
10478,52.910156,-27.953188,223.774083,-54.639214,1,0.5552,0.2233,0.2002,40.2248,0.007,90
10586,358.636353,-46.768478,328.890146,-67.388837,1,0.6052,0.6017,0.0153,42.7467,0.008,88
10757,52.910156,-26.276812,220.926149,-54.363918,1,0.1699,0.1711,0.0185,39.5801,0.008,52
10796,52.910156,-25.944481,220.366350,-54.301439,1,0.0000,0.0000,0.0000,nan,0.010,65
10798,351.299988,-62.320400,319.038597,-52.026867,1,0.1778,0.1872,0.0121,39.7959,0.018,42
11165,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16
11359,349.966217,-62.696659,319.542989,-51.376556,1,0.1529,0.1415,0.0072,39.1281,0.021,42
11507,53.085938,-28.122234,224.100909,-54.509752,1,0.3312,0.5095,0.0718,42.3102,0.007,90
11770,346.130127,-63.072620,321.423103,-50.042305,1,0.1415,0.2171,0.4350,40.1560,0.020,62
11773,150.644531,3.583322,235.698235,43.342784,1,0.2207,0.5279,0.1679,42.4027,0.018,52
11931,149.589844,3.583322,234.885369,42.474696,1,0.0000,0.0000,0.0000,nan,0.024,65
11978,358.648071,-46.375080,329.462659,-67.716008,1,0.4920,0.4605,0.0179,42.0472,0.009,90
12695,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,92
12872,347.861847,-61.943836,321.519104,-51.424048,1,0.0000,0.0000,0.0000,nan,0.017,65
13079,151.699219,3.583322,236.533224,44.205648,1,0.2019,2.4470,1.0434,46.4913,0.016,90
13138,346.655182,-63.260487,320.952196,-50.040935,1,0.0756,0.5192,0.2158,42.3596,0.019,52
13194,53.789062,-27.784405,223.685697,-53.845803,1,0.5195,0.5624,0.2843,42.5685,0.009,90
13459,150.117188,2.836105,236.124718,42.483719,1,0.3495,0.3449,0.6556,41.3068,0.016,90
13482,33.750000,-4.630479,168.146242,-59.949072,1,0.2929,0.3115,0.0205,41.0501,0.019,90
13504,1.363636,-46.768478,324.669342,-68.371416,1,0.4469,0.3816,0.0766,41.5643,0.008,90
14080,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,65
14156,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,65
14279,54.667969,-27.615883,223.610785,-53.050840,1,0.3434,0.5728,0.4518,42.6167,0.009,52
14398,2.071130,-45.191612,325.606223,-69.989264,1,0.2812,0.2634,1.0581,40.6310,0.011,90
14539,150.644531,3.583322,235.698235,43.342784,1,0.2882,0.2359,0.0434,40.3590,0.018,88
14553,359.805206,-46.768478,327.135979,-67.829903,1,1.1897,1.1667,0.1717,44.5143,0.011,95
14601,32.695312,-4.929937,166.868469,-60.841230,1,0.3837,0.3653,0.2005,41.4527,0.018,90
14674,33.750000,-4.630479,168.146242,-59.949072,1,0.2012,0.0567,0.4176,37.0171,0.019,90
14983,349.615387,-63.636005,318.927246,-50.506542,1,0.3391,0.3238,0.0255,41.1476,0.018,90
15002,349.046051,-61.943836,320.796530,-51.753706,1,0.3409,0.3512,0.0531,41.3530,0.017,90
15251,32.871094,-4.780192,166.959493,-60.615132,1,0.4653,2.3270,0.6097,46.3585,0.017,90
15475,351.382965,-64.011238,317.574052,-50.604657,1,0.0000,0.0000,0.0000,nan,0.023,65
15626,346.130127,-63.072620,321.423103,-50.042305,1,0.0000,0.0000,0.0000,nan,0.020,16
15674,0.965665,-46.375080,325.845907,-68.579427,1,0.2927,0.2727,0.3286,40.7172,0.007,90
15700,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,16
15718,51.855469,-27.953188,223.543603,-55.561470,1,0.1193,2.3179,0.7672,46.3482,0.008,52
15845,53.789062,-27.784405,223.685697,-53.845803,1,0.3174,0.3471,0.8216,41.3232,0.009,90
15968,149.414062,2.238686,236.239766,41.565558,1,0.3509,0.4729,0.4544,42.1164,0.017,90
16339,51.328125,-27.447618,222.535046,-55.950727,1,0.0000,0.0000,0.0000,nan,0.013,16
16349,150.820312,3.134927,236.341348,43.230123,1,0.0000,0.0000,0.0000,nan,0.016,16
16463,151.699219,3.583322,236.533224,44.205648,1,0.2023,0.1805,0.0254,39.7082,0.016,90
16496,359.415588,-46.768478,327.729895,-67.686097,1,0.3391,0.3895,0.2635,41.6162,0.009,52
16802,53.437500,-29.142223,225.908120,-54.336118,1,0.3145,0.3319,0.0234,41.2094,0.008,90
16983,150.117188,3.732834,235.120533,42.993809,1,0.2899,0.2762,0.1879,40.7495,0.020,90
17094,52.207031,-28.291550,224.208534,-55.300157,1,0.0000,0.0000,0.0000,nan,0.007,16
17172,53.437500,-29.142223,225.908120,-54.336118,1,0.0000,0.0000,0.0000,nan,0.008,16
17285,148.710938,2.836105,235.050801,41.328739,1,0.3073,0.3057,0.0484,41.0025,0.031,90
17366,349.285706,-62.884678,319.786163,-51.046461,1,0.2387,0.2024,0.0247,39.9853,0.018,90
17370,0.949367,-45.586655,326.991548,-69.251686,1,0.3138,0.3391,0.4176,41.2636,0.013,62
17515,52.207031,-28.630989,224.800211,-55.343637,1,0.3577,0.3487,0.0073,41.3345,0.009,90
18029,359.415588,-46.768478,327.729895,-67.686097,1,0.3525,0.3609,0.0112,41.4219,0.009,90
18507,352.711273,-63.823658,316.922299,-51.059403,1,0.3755,0.3457,0.0230,41.3125,0.024,88
18556,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,6
18645,358.636353,-46.768478,328.890146,-67.388837,1,0.1640,2.3025,1.1022,46.3306,0.008,62
18706,34.277344,-5.679190,170.314930,-60.410322,1,0.1706,0.1766,0.0158,39.6556,0.020,62
18937,348.595886,-63.072620,320.023289,-50.713060,1,0.2142,0.2222,0.0102,40.2123,0.021,90
18952,151.699219,3.583322,236.533224,44.205648,1,0.2800,0.2658,1.1944,40.6541,0.016,90
19154,351.382965,-64.011238,317.574052,-50.604657,1,0.2354,2.4138,0.5022,46.4553,0.023,67
19213,1.753247,-46.768478,324.030235,-68.498041,1,0.1254,0.1484,0.0086,39.2403,0.014,62
19866,359.814819,-44.399834,330.775011,-69.801007,1,0.2608,0.2877,0.0235,40.8505,0.009,90
20567,351.259003,-64.386185,317.344860,-50.255113,1,0.1549,0.1481,0.2206,39.2350,0.020,62
20934,348.908447,-63.823658,319.169886,-50.176186,1,0.0999,2.5704,1.2137,46.6209,0.018,42
21335,33.574219,-5.379379,168.838090,-60.637536,1,0.1542,0.2082,0.4220,40.0542,0.017,90
22184,358.312500,-44.993881,332.185785,-68.685906,1,0.3508,0.3850,0.6064,41.5869,0.009,90
22574,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16
22901,151.171875,1.342993,238.602520,42.464379,1,0.2581,0.2502,0.0061,40.5039,0.026,90
23116,53.261719,-27.615883,223.280041,-54.281374,1,0.8237,0.7520,0.0300,43.3376,0.006,15
23127,149.414062,3.433834,234.919132,42.245550,1,0.3221,0.4025,0.7933,41.7004,0.027,52
23299,33.222656,-4.780192,167.515653,-60.396584,1,0.5869,0.5400,0.0151,42.4624,0.018,88
23373,150.117188,3.732834,235.120533,42.993809,1,0.5442,0.5636,0.2043,42.5744,0.020,88
23396,359.811707,-45.191612,329.485675,-69.150905,1,0.5667,0.6192,0.1193,42.8220,0.010,90
23409,348.595886,-63.072620,320.023289,-50.713060,1,0.1407,0.1392,0.0136,39.0882,0.021,52
23539,34.277344,-5.079716,169.526841,-59.956640,1,0.4550,0.2524,0.3112,40.5254,0.019,95
23795,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,65
23822,2.457983,-45.389202,324.632685,-69.945696,1,0.2411,0.2420,0.9270,40.4218,0.011,52
23848,33.925781,-5.979157,170.179895,-60.866303,1,0.3316,0.3185,1.0181,41.1057,0.022,90
23857,151.699219,3.583322,236.533224,44.205648,1,0.2988,0.4769,0.0894,42.1379,0.016,90
23931,32.695312,-4.929937,166.868469,-60.841230,1,0.6282,0.6337,0.0073,42.8832,0.018,88
24193,152.050781,2.985506,237.495952,44.143927,1,2.0958,1.3937,0.2518,44.9919,0.019,88
24236,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65
24592,349.966217,-62.696659,319.542989,-51.376556,1,0.2901,0.2846,0.0249,40.8234,0.021,90
24849,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,16
24903,52.031250,-26.443335,220.963669,-55.168557,1,0.0000,0.0000,0.0000,nan,0.014,65
24947,150.117188,2.238686,236.784618,42.139082,1,0.4723,0.4521,0.0193,41.9998,0.016,90
24989,34.804688,-5.829153,171.307861,-60.174401,1,0.4468,0.4763,0.0117,42.1349,0.023,90
25003,359.814819,-44.399834,330.775011,-69.801007,1,0.3137,0.2996,0.0218,40.9523,0.009,90
25039,346.562500,-63.448284,320.824720,-49.866957,1,0.3161,0.2675,1.1577,40.6696,0.021,90
25474,151.523438,3.134927,236.900695,43.803170,1,0.5236,0.5626,0.0155,42.5697,0.019,90
25529,358.312500,-44.993881,332.185785,-68.685906,1,0.2835,0.5789,0.2180,42.6448,0.009,90
25577,348.529419,-61.755440,321.293980,-51.763351,1,0.4028,0.3918,0.0170,41.6314,0.016,90
25783,150.820312,3.134927,236.341348,43.230123,1,0.1040,0.1439,0.0116,39.1669,0.016,42
25920,150.644531,3.583322,235.698235,43.342784,1,0.0000,0.0000,0.0000,nan,0.018,16
25925,35.332031,-5.979157,172.286722,-59.931743,1,1.7327,1.7075,0.1320,45.5358,0.022,88
26161,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,92
26338,151.171875,2.537361,237.288526,43.169764,1,0.1892,0.2250,0.0141,40.2436,0.024,62
26352,1.708861,-45.586655,325.688716,-69.520253,1,0.0000,0.0000,0.0000,nan,0.011,65
26401,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,16
26531,351.259003,-64.386185,317.344860,-50.255113,1,2.5314,2.4324,0.2792,46.4755,0.020,88
26660,347.846710,-64.760857,318.929827,-49.143596,1,0.0000,0.0000,0.0000,nan,0.019,65
26783,150.820312,1.641510,237.994507,42.358984,1,0.0000,0.0000,0.0000,nan,0.020,92
27124,351.299988,-62.320400,319.038597,-52.026867,1,0.0000,0.0000,0.0000,nan,0.018,16
27339,51.855469,-26.276812,220.627031,-55.293792,1,0.1432,0.1625,0.0226,39.4561,0.014,90
27941,149.414062,1.940072,236.565366,41.393323,1,0.3632,0.3746,0.0319,41.5166,0.018,90
28220,1.694561,-45.191612,326.278557,-69.858253,1,0.2985,0.3605,0.3149,41.4193,0.011,90
28301,0.189873,-45.586655,328.254458,-68.969298,1,0.3606,0.2852,1.3620,40.8288,0.007,90
28391,351.953644,-62.132156,318.777388,-52.347124,1,0.0000,0.0000,0.0000,nan,0.019,92
28636,51.855469,-28.630989,224.733260,-55.649872,1,0.1743,0.4412,0.3366,41.9364,0.009,67
28843,151.171875,2.537361,237.288526,43.169764,1,0.3664,0.3611,0.0225,41.4234,0.024,90
28915,53.789062,-27.784405,223.685697,-53.845803,1,0.0000,0.0000,0.0000,nan,0.009,16
29088,52.558594,-27.279613,222.538937,-54.845107,1,0.3037,0.3244,0.0203,41.1521,0.008,90
29252,51.855469,-28.630989,224.733260,-55.649872,1,0.1439,0.1421,0.0233,39.1376,0.009,42
29416,1.694561,-45.191612,326.278557,-69.858253,1,0.2168,0.1921,0.0349,39.8588,0.011,90
29420,2.097458,-45.783966,324.737840,-69.478613,1,0.5849,0.5559,0.0102,42.5385,0.011,90
29576,346.655182,-63.260487,320.952196,-50.040935,1,0.2362,2.5224,1.0484,46.5713,0.019,90
29668,151.699219,3.583322,236.533224,44.205648,1,0.1461,0.1584,0.0175,39.3960,0.016,42
29670,1.694561,-45.191612,326.278557,-69.858253,1,0.1135,0.1208,0.0198,38.7544,0.011,62
30066,351.259003,-64.386185,317.344860,-50.255113,1,0.0000,0.0000,0.0000,nan,0.020,65
30172,33.574219,-5.379379,168.838090,-60.637536,1,0.5444,0.5455,0.0094,42.4889,0.017,90
30191,150.117188,2.238686,236.784618,42.139082,1,1.5405,1.3073,0.1521,44.8201,0.016,88
30505,151.171875,2.238686,237.619933,42.994783,1,0.0000,0.0000,0.0000,nan,0.024,16
30545,2.071130,-45.191612,325.606223,-69.989264,1,0.2160,0.2221,0.0338,40.2113,0.011,90
30576,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65
30673,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,16
30895,349.429535,-62.508568,320.039643,-51.393745,1,0.0000,0.0000,0.0000,nan,0.020,16
31033,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,65
31100,1.694561,-45.191612,326.278557,-69.858253,1,0.3614,0.3471,1.2589,41.3231,0.011,90
31310,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,92
31569,346.655182,-63.260487,320.952196,-50.040935,1,1.0133,1.0031,0.0118,44.1084,0.019,95
31605,150.292969,2.686724,236.427488,42.541447,1,0.0000,0.0000,0.0000,nan,0.016,65
31824,352.398651,-62.696659,318.017427,-51.967966,1,0.1838,0.0844,0.3311,37.9246,0.020,42
32238,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,16
32309,34.804688,-5.829153,171.307861,-60.174401,1,0.2251,0.2258,0.9011,40.2521,0.023,42
32375,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,65
32695,358.636353,-46.768478,328.890146,-67.388837,1,0.7689,0.7806,0.0113,43.4371,0.008,90
33088,351.259003,-64.386185,317.344860,-50.255113,1,0.3437,0.3495,0.0200,41.3400,0.020,90
33179,51.855469,-27.953188,223.543603,-55.561470,1,0.4407,0.4765,0.4079,42.1357,0.008,90
33191,151.171875,2.238686,237.619933,42.994783,1,0.4030,0.4039,0.0174,41.7094,0.024,42
33409,33.222656,-4.780192,167.515653,-60.396584,1,0.0000,0.0000,0.0000,nan,0.018,65
33419,150.820312,3.732834,235.666318,43.572109,1,0.7462,0.7461,0.0356,43.3165,0.016,90
33422,33.574219,-6.579593,170.455585,-61.548219,1,1.1111,1.1054,0.0101,44.3693,0.021,88
34012,35.683594,-5.379379,171.992947,-59.253501,1,0.0853,0.0793,0.0210,37.7805,0.020,52
34166,0.189873,-45.586655,328.254458,-68.969298,1,0.0873,0.0909,0.0246,38.0959,0.007,42
34243,34.101562,-5.829153,170.247753,-60.638325,1,0.1416,0.5642,0.3631,42.5771,0.019,88
34299,346.276581,-64.011238,320.448031,-49.344136,1,0.1901,0.2016,0.0086,39.9759,0.019,62
34437,152.050781,2.985506,237.495952,44.143927,1,0.2657,0.2629,1.0604,40.6263,0.019,67
35197,51.679688,-27.447618,222.618229,-55.642263,1,0.2509,0.2366,0.1269,40.3670,0.010,42
35315,150.468750,3.732834,235.392208,43.283244,1,1.8476,1.5239,0.2256,45.2314,0.020,95
35555,359.805206,-46.768478,327.135979,-67.829903,1,0.0000,0.0000,0.0000,nan,0.011,65
35743,34.277344,-5.679190,170.314930,-60.410322,1,0.0781,0.0752,0.0197,37.6598,0.020,42
35772,150.117188,2.836105,236.124718,42.483719,1,0.2385,0.2588,0.0217,40.5879,0.016,90
35855,0.929752,-44.597992,328.531426,-70.083244,1,0.3815,0.4401,0.5094,41.9300,0.011,90
36085,352.398651,-62.696659,318.017427,-51.967966,1,0.1689,0.1759,0.5357,39.6465,0.020,42
36153,150.468750,1.641510,237.714575,42.075234,1,0.2547,0.2589,0.0256,40.5887,0.017,52
36337,52.558594,-27.279613,222.538937,-54.845107,1,0.0000,0.0000,0.0000,nan,0.008,65
36362,53.085938,-27.784405,223.525509,-54.460748,1,0.0000,0.0000,0.0000,nan,0.007,65
36671,149.589844,3.583322,234.885369,42.474696,1,0.0000,0.0000,0.0000,nan,0.024,65
36783,349.966217,-62.696659,319.542989,-51.376556,1,0.1287,0.1431,0.0129,39.1539,0.021,90
37149,359.816315,-44.003082,331.451340,-70.123054,1,0.9435,0.9017,0.0524,43.8228,0.013,90
37168,53.613281,-27.953188,223.929533,-54.024772,1,0.0000,0.0000,0.0000,nan,0.007,65
37661,32.871094,-4.780192,166.959493,-60.615132,1,0.1226,0.0973,0.0168,38.2528,0.017,52
37776,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,16
37865,151.171875,2.238686,237.619933,42.994783,1,0.2263,0.2221,0.0172,40.2112,0.024,90
37872,150.820312,3.134927,236.341348,43.230123,1,0.2517,0.2448,0.0217,40.4506,0.016,67
38174,1.694561,-45.191612,326.278557,-69.858253,1,1.6152,1.7388,0.1564,45.5843,0.011,88
38205,33.750000,-4.630479,168.146242,-59.949072,1,0.2945,0.2311,1.2272,40.3089,0.019,42
38244,346.655182,-63.260487,320.952196,-50.040935,1,0.0000,0.0000,0.0000,nan,0.019,65
38690,33.222656,-4.780192,167.515653,-60.396584,1,0.1801,0.1274,0.0307,38.8795,0.018,90
38730,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,16
38754,33.574219,-6.579593,170.455585,-61.548219,1,0.2646,0.2656,0.0093,40.6515,0.021,90
38899,1.666667,-44.399834,327.519190,-70.529554,1,0.4828,0.4754,0.0332,42.1297,0.009,90
39223,150.996094,2.388015,237.313912,42.939977,1,0.0000,0.0000,0.0000,nan,0.021,65
39305,346.562500,-63.448284,320.824720,-49.866957,1,0.4045,0.2986,0.1602,40.9435,0.021,90
39398,51.679688,-27.447618,222.618229,-55.642263,1,0.3347,0.5555,0.5802,42.5365,0.010,90
39597,53.085938,-28.122234,224.100909,-54.509752,1,0.1280,0.1327,0.0064,38.9763,0.007,62
39626,149.414062,2.238686,236.239766,41.565558,1,0.5197,0.4293,0.7344,41.8662,0.017,90
39846,351.382965,-64.011238,317.574052,-50.604657,1,0.1886,0.2780,0.6915,40.7654,0.023,62
40290,35.859375,-4.630479,171.270769,-58.580806,1,0.3153,0.5118,0.8469,42.3221,0.022,42
41515,358.648071,-46.375080,329.462659,-67.716008,1,0.5720,0.5797,0.0188,42.6484,0.009,90
41738,150.117188,3.732834,235.120533,42.993809,1,0.1206,0.1277,0.0222,38.8865,0.020,42
42118,0.574468,-45.981140,327.041068,-68.778764,1,0.1801,0.1977,0.0131,39.9287,0.006,62
42224,51.328125,-27.784405,223.130589,-55.999499,1,0.1119,0.0888,0.1482,38.0396,0.013,42
42288,359.415588,-46.768478,327.729895,-67.686097,1,0.3487,0.3864,0.0241,41.5962,0.009,90
42333,346.562500,-63.448284,320.824720,-49.866957,1,0.1921,0.2046,0.0101,40.0116,0.021,67
42469,2.071130,-45.191612,325.606223,-69.989264,1,1.5989,1.4913,0.1216,45.1735,0.011,95
42689,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65
42776,152.050781,3.284369,237.157374,44.318466,1,0.0000,0.0000,0.0000,nan,0.019,16
42852,351.321442,-64.198746,317.458993,-50.429931,1,0.6771,0.6680,0.0223,43.0226,0.023,88
43028,51.679688,-27.447618,222.618229,-55.642263,1,0.1366,0.1364,0.0092,39.0408,0.010,42
43151,34.980469,-6.279288,172.180075,-60.389399,1,0.1096,0.1352,0.0222,39.0199,0.023,52
43211,34.980469,-6.279288,172.180075,-60.389399,1,0.3321,0.3275,0.0164,41.1762,0.023,90
43337,51.328125,-27.447618,222.535046,-55.950727,1,0.1775,0.2488,0.0180,40.4902,0.013,90
43413,348.595886,-63.072620,320.023289,-50.713060,1,0.0000,0.0000,0.0000,nan,0.021,16
43509,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,92
43812,150.820312,3.134927,236.341348,43.230123,1,0.2783,0.2821,0.0219,40.8022,0.016,90
43962,347.861847,-61.943836,321.519104,-51.424048,1,0.6595,0.6813,0.0340,43.0750,0.017,90
44102,152.050781,3.284369,237.157374,44.318466,1,0.2450,0.2640,0.2125,40.6366,0.019,42
44217,51.855469,-27.953188,223.543603,-55.561470,1,0.4288,0.1772,0.6584,39.6645,0.008,90
44309,34.980469,-6.279288,172.180075,-60.389399,1,0.3131,0.3059,1.2366,41.0045,0.023,90
44480,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,16
44836,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,65
45060,346.276581,-64.011238,320.448031,-49.344136,1,0.3613,0.3300,0.1387,41.1948,0.019,62
45115,2.097458,-45.783966,324.737840,-69.478613,1,0.0000,0.0000,0.0000,nan,0.011,65
45127,35.859375,-4.630479,171.270769,-58.580806,1,0.1378,0.1359,0.0125,39.0331,0.022,90
45203,150.820312,3.134927,236.341348,43.230123,1,0.0000,0.0000,0.0000,nan,0.016,16
45319,348.595886,-63.072620,320.023289,-50.713060,1,0.1270,0.0737,0.0272,37.6138,0.021,42
45349,32.695312,-4.929937,166.868469,-60.841230,1,0.2821,2.6404,0.9837,46.6916,0.018,67
45549,52.207031,-28.291550,224.208534,-55.300157,1,0.6733,0.7639,0.0513,43.3795,0.007,42
46210,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65
46567,149.414062,3.433834,234.919132,42.245550,1,0.0000,0.0000,0.0000,nan,0.027,16
46804,352.398651,-62.696659,318.017427,-51.967966,1,0.2933,2.3248,0.6338,46.3561,0.020,62
46958,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,65
47148,53.085938,-27.784405,223.525509,-54.460748,1,0.3153,0.3234,0.0167,41.1439,0.007,90
47725,351.259003,-64.386185,317.344860,-50.255113,1,0.3615,0.2950,0.7526,40.9133,0.020,90
48187,52.207031,-26.610098,221.298836,-55.042928,1,0.5466,0.5379,0.1063,42.4521,0.014,90
48260,52.207031,-28.291550,224.208534,-55.300157,1,0.1677,0.1746,0.4651,39.6283,0.007,42
48426,351.734680,-62.884678,318.284128,-51.651217,1,0.0000,0.0000,0.0000,nan,0.019,16
48473,51.855469,-27.953188,223.543603,-55.561470,1,0.0809,0.0825,0.0165,37.8722,0.008,42
48575,32.695312,-4.929937,166.868469,-60.841230,1,0.7796,0.7698,0.0437,43.4000,0.018,90
48687,347.812500,-63.448284,320.128971,-50.202348,1,0.7916,0.8371,0.0551,43.6237,0.021,88
48725,53.613281,-28.630989,225.073365,-54.119461,1,0.3658,0.3599,0.1804,41.4145,0.006,67
48749,348.529419,-61.755440,321.293980,-51.763351,1,1.6645,1.5782,0.0657,45.3252,0.016,88
48817,348.586945,-64.573555,318.693903,-49.477869,1,0.0962,0.1446,0.6309,39.1788,0.018,90
48981,358.648071,-46.375080,329.462659,-67.716008,1,0.0000,0.0000,0.0000,nan,0.009,16
49219,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,65
49389,349.285706,-62.884678,319.786163,-51.046461,1,0.0000,0.0000,0.0000,nan,0.018,92
49529,53.613281,-27.953188,223.929533,-54.024772,1,0.1974,0.2117,0.0123,40.0947,0.007,62
49783,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,65
49937,151.347656,3.583322,236.252362,43.918627,1,0.0947,0.1091,0.0218,38.5175,0.015,52
50277,349.046051,-61.943836,320.796530,-51.753706,1,0.3009,0.2478,1.3214,40.4807,0.017,67
50395,349.891296,-64.573555,317.972107,-49.786192,1,0.1552,0.1574,0.3022,39.3801,0.023,90
51178,52.558594,-27.279613,222.538937,-54.845107,1,0.9668,0.9534,0.0188,43.9721,0.008,88
51279,1.708861,-45.586655,325.688716,-69.520253,1,0.1760,0.1824,0.3979,39.7343,0.011,90
51318,34.277344,-5.679190,170.314930,-60.410322,1,0.2987,0.3038,0.0273,40.9868,0.020,62
51490,0.574468,-45.981140,327.041068,-68.778764,1,0.3707,0.4717,0.3810,42.1099,0.006,90
51987,352.711273,-63.823658,316.922299,-51.059403,1,0.0000,0.0000,0.0000,nan,0.024,92
52150,52.207031,-26.610098,221.298836,-55.042928,1,0.2746,0.3069,0.0656,41.0122,0.014,90
52320,52.910156,-25.944481,220.366350,-54.301439,1,0.1656,2.0324,0.6458,46.0000,0.010,88
52370,352.711273,-63.823658,316.922299,-51.059403,1,0.0000,0.0000,0.0000,nan,0.024,16
52425,52.910156,-26.276812,220.926149,-54.363918,1,0.1587,0.1182,0.6266,38.7042,0.008,42
52740,53.613281,-27.953188,223.929533,-54.024772,1,0.3653,0.4721,0.8955,42.1118,0.007,90
52854,149.414062,1.940072,236.565366,41.393323,1,0.4383,0.4343,0.0335,41.8957,0.018,90
53025,358.636353,-46.768478,328.890146,-67.388837,1,0.0000,0.0000,0.0000,nan,0.008,65
53249,349.285706,-62.884678,319.786163,-51.046461,1,0.3941,0.4211,0.4203,41.8166,0.018,90
53354,34.980469,-6.279288,172.180075,-60.389399,1,0.2231,0.2134,0.0125,40.1138,0.023,62
53525,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,16
53574,0.574468,-45.981140,327.041068,-68.778764,1,0.4176,0.4427,0.0420,41.9454,0.006,90
53782,53.261719,-27.615883,223.280041,-54.281374,1,0.3798,0.3714,0.0189,41.4948,0.006,90
53938,53.437500,-29.142223,225.908120,-54.336118,1,0.0000,0.0000,0.0000,nan,0.008,16
54416,347.812500,-63.448284,320.128971,-50.202348,1,0.3708,0.3734,0.0162,41.5087,0.021,42
54883,347.617462,-62.508568,321.121462,-50.904708,1,0.7844,0.8578,0.0392,43.6891,0.019,88
54915,148.886719,2.686724,235.347248,41.389003,1,0.0000,0.0000,0.0000,nan,0.028,65
55002,52.207031,-28.291550,224.208534,-55.300157,1,0.4345,0.4175,0.0286,41.7946,0.007,90
55018,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,65
55033,151.171875,1.342993,238.602520,42.464379,1,0.4920,0.5345,0.0204,42.4355,0.026,42
55060,51.855469,-28.630989,224.733260,-55.649872,1,0.0234,0.0824,0.0202,37.8680,0.009,42
55141,2.097458,-45.783966,324.737840,-69.478613,1,0.0000,0.0000,0.0000,nan,0.011,65
55155,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,65
55354,350.230255,-61.943836,320.053946,-52.070537,1,0.1958,2.3870,0.2904,46.4258,0.017,90
55419,150.820312,3.732834,235.666318,43.572109,1,0.0967,0.1092,0.0137,38.5190,0.016,62
55946,347.617462,-62.508568,321.121462,-50.904708,1,0.0000,0.0000,0.0000,nan,0.019,65
56053,150.292969,2.686724,236.427488,42.541447,1,0.8507,0.8283,0.0406,43.5954,0.016,90
56245,1.708861,-45.586655,325.688716,-69.520253,1,0.0000,0.0000,0.0000,nan,0.011,16
56334,358.312500,-44.993881,332.185785,-68.685906,1,0.1955,0.1948,0.0100,39.8925,0.009,62
56349,1.694561,-45.191612,326.278557,-69.858253,1,0.1646,0.2000,0.1961,39.9565,0.011,52
56461,347.812500,-63.448284,320.128971,-50.202348,1,0.1760,2.4036,1.0097,46.4440,0.021,90
56769,33.574219,-5.079716,168.448505,-60.407218,1,0.0000,0.0000,0.0000,nan,0.016,65
56821,52.207031,-26.610098,221.298836,-55.042928,1,0.0775,0.0884,0.0181,38.0309,0.014,62
56893,150.117188,2.836105,236.124718,42.483719,1,0.0000,0.0000,0.0000,nan,0.016,65
56987,33.574219,-6.579593,170.455585,-61.548219,1,0.2910,0.3152,0.0132,41.0794,0.021,62
57205,359.816315,-44.003082,331.451340,-70.123054,1,0.5891,0.6057,0.0306,42.7639,0.013,90
57237,52.207031,-26.610098,221.298836,-55.042928,1,0.2544,0.3807,0.6581,41.5580,0.014,42
57263,351.299988,-62.320400,319.038597,-52.026867,1,0.0000,0.0000,0.0000,nan,0.018,65
57561,152.050781,3.284369,237.157374,44.318466,1,0.0000,0.0000,0.0000,nan,0.019,65
57666,1.666667,-44.399834,327.519190,-70.529554,1,0.4399,0.4648,0.0183,42.0716,0.009,90
57784,348.529419,-61.755440,321.293980,-51.763351,1,0.0000,0.0000,0.0000,nan,0.016,65
58174,348.529419,-61.755440,321.293980,-51.763351,1,1.1032,1.1018,0.1226,44.3607,0.016,42
58265,0.190678,-45.783966,327.956322,-68.803772,1,0.0650,0.1027,1.1126,38.3778,0.005,67
58323,349.615387,-63.636005,318.927246,-50.506542,1,1.0155,1.0187,0.0471,44.1500,0.018,88
59068,34.277344,-5.079716,169.526841,-59.956640,1,0.5716,0.5429,0.1404,42.4762,0.019,42
59128,348.529419,-61.755440,321.293980,-51.763351,1,0.6794,0.4288,0.2152,41.8629,0.016,90
59163,150.644531,3.583322,235.698235,43.342784,1,0.0000,0.0000,0.0000,nan,0.018,65
59427,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,65
59463,52.031250,-26.443335,220.963669,-55.168557,1,0.1439,0.1438,0.0169,39.1654,0.014,42
59580,1.753247,-46.768478,324.030235,-68.498041,1,0.4178,0.5020,0.7485,42.2716,0.014,42
59644,34.980469,-6.279288,172.180075,-60.389399,1,0.4238,0.3347,0.1863,41.2308,0.023,90
59732,347.812500,-63.448284,320.128971,-50.202348,1,0.2489,2.7125,0.9243,46.7623,0.021,90
60023,359.814819,-44.399834,330.775011,-69.801007,1,0.6857,0.6858,0.0077,43.0925,0.009,90
60098,53.261719,-27.615883,223.280041,-54.281374,1,0.2529,0.2478,0.0697,40.4804,0.006,90
60340,148.710938,2.836105,235.050801,41.328739,1,0.3521,0.3472,0.0096,41.3237,0.031,67
60350,52.558594,-27.279613,222.538937,-54.845107,1,0.5295,0.5982,0.1284,42.7310,0.008,62
60376,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,16
60407,346.130127,-63.072620,321.423103,-50.042305,1,0.0000,0.0000,0.0000,nan,0.020,65
60554,347.013428,-62.508568,321.472056,-50.735330,1,0.2356,0.2869,0.0805,40.8438,0.018,67
60742,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65
60976,358.648071,-46.375080,329.462659,-67.716008,1,0.0615,0.0821,0.0078,37.8606,0.009,52
61101,151.171875,2.537361,237.288526,43.169764,1,0.0000,0.0000,0.0000,nan,0.024,16
61165,53.789062,-27.784405,223.685697,-53.845803,1,0.1089,0.1122,0.0140,38.5832,0.009,62
61407,53.613281,-27.953188,223.929533,-54.024772,1,0.0000,0.0000,0.0000,nan,0.007,92
61763,52.207031,-28.291550,224.208534,-55.300157,1,0.5749,0.5237,0.0263,42.3819,0.007,90
62078,0.965665,-46.375080,325.845907,-68.579427,1,0.0000,0.0000,0.0000,nan,0.007,65
62187,33.398438,-3.732834,166.492280,-59.466614,1,0.1391,0.1301,1.1896,38.9300,0.022,64
62230,53.085938,-27.784405,223.525509,-54.460748,1,0.0000,0.0000,0.0000,nan,0.007,16
62253,51.328125,-27.447618,222.535046,-55.950727,1,0.7671,0.7610,0.0367,43.3693,0.013,90
62254,53.085938,-27.111860,222.384291,-54.355086,1,0.2738,0.2895,0.0180,40.8666,0.007,90
62384,351.321442,-64.198746,317.458993,-50.429931,1,0.0000,0.0000,0.0000,nan,0.023,65
62541,347.013428,-62.508568,321.472056,-50.735330,1,0.5490,0.3280,0.8012,41.1797,0.018,90
62908,150.820312,3.134927,236.341348,43.230123,1,0.2372,0.2111,0.0189,40.0878,0.016,62
63561,359.814819,-44.399834,330.775011,-69.801007,1,0.7386,0.7247,0.0129,43.2390,0.009,90
63718,0.965665,-46.375080,325.845907,-68.579427,1,0.2891,0.4200,0.0613,41.8100,0.007,90
63860,351.382965,-64.011238,317.574052,-50.604657,1,0.1617,0.1988,0.0128,39.9412,0.023,42
64248,2.097458,-45.783966,324.737840,-69.478613,1,0.1653,0.1710,0.0196,39.5787,0.011,52
64485,348.586945,-64.573555,318.693903,-49.477869,1,0.0000,0.0000,0.0000,nan,0.018,16
64854,1.723404,-45.981140,325.117958,-69.180825,1,0.1163,0.0705,0.0098,37.5117,0.010,62
64888,52.031250,-26.443335,220.963669,-55.168557,1,0.3802,0.4086,0.7097,41.7390,0.014,90
64896,347.846710,-64.760857,318.929827,-49.143596,1,0.0991,0.1224,1.1874,38.7854,0.019,42
64911,150.820312,1.641510,237.994507,42.358984,1,0.2375,2.2621,1.0009,46.2837,0.020,90
65745,53.085938,-28.122234,224.100909,-54.509752,1,0.0680,0.0552,1.1581,36.9555,0.007,90
65749,33.398438,-4.331149,167.226341,-59.936551,1,0.0000,0.0000,0.0000,nan,0.018,65
65877,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,65
66126,348.908447,-63.823658,319.169886,-50.176186,1,0.0601,0.0718,0.0163,37.5531,0.018,42
66325,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,65
66536,150.996094,2.388015,237.313912,42.939977,1,0.3144,0.3150,0.2975,41.0779,0.021,90
66548,347.617462,-62.508568,321.121462,-50.904708,1,0.3713,0.4282,1.3857,41.8598,0.019,90
66852,2.071130,-45.191612,325.606223,-69.989264,1,0.5802,0.5679,0.0103,42.5945,0.011,42
66904,349.285706,-62.884678,319.786163,-51.046461,1,0.0000,0.0000,0.0000,nan,0.018,16
66967,150.996094,2.388015,237.313912,42.939977,1,0.4580,0.4551,0.0163,42.0168,0.021,88
66999,33.222656,-4.780192,167.515653,-60.396584,1,0.3635,0.4147,0.0332,41.7770,0.018,90
67245,150.292969,2.686724,236.427488,42.541447,1,0.0482,0.0598,0.0147,37.1379,0.016,42
67480,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,65
67514,34.101562,-5.829153,170.247753,-60.638325,1,0.2756,0.5256,0.1081,42.3913,0.019,90
67686,359.415588,-46.768478,327.729895,-67.686097,1,0.1785,0.2040,0.0109,40.0042,0.009,90
67730,52.910156,-25.944481,220.366350,-54.301439,1,0.3927,0.3926,0.0321,41.6368,0.010,90
67898,347.812500,-63.448284,320.128971,-50.202348,1,0.3824,0.3802,0.0081,41.5546,0.021,42
67981,151.523438,3.134927,236.900695,43.803170,1,0.0000,0.0000,0.0000,nan,0.019,65
68003,150.820312,3.732834,235.666318,43.572109,1,0.4201,0.4099,0.0162,41.7474,0.016,90
68276,33.398438,-3.732834,166.492280,-59.466614,1,0.0251,0.0342,0.0167,35.8818,0.022,42
68298,349.429535,-62.508568,320.039643,-51.393745,1,0.6264,0.6544,0.1479,42.9683,0.020,90
68667,348.595886,-63.072620,320.023289,-50.713060,1,0.0000,0.0000,0.0000,nan,0.021,16
68835,348.908447,-63.823658,319.169886,-50.176186,1,0.0177,0.0504,0.0318,36.7530,0.018,42
68886,2.457983,-45.389202,324.632685,-69.945696,1,0.3991,0.4463,0.3629,41.9666,0.011,52
69271,148.710938,2.836105,235.050801,41.328739,1,0.4269,0.4491,0.8217,41.9826,0.031,90
69490,51.855469,-26.276812,220.627031,-55.293792,1,0.2919,0.4825,0.1693,42.1687,0.014,42
69767,1.694561,-45.191612,326.278557,-69.858253,1,0.1603,2.4521,1.2066,46.4968,0.011,62
70046,351.321442,-64.198746,317.458993,-50.429931,1,0.3441,0.3129,0.0280,41.0610,0.023,90
70135,0.949367,-45.586655,326.991548,-69.251686,1,0.4500,0.4728,0.0167,42.1160,0.013,42
70171,53.789062,-27.784405,223.685697,-53.845803,1,0.4708,0.5059,0.0358,42.2916,0.009,90
70272,34.277344,-5.079716,169.526841,-59.956640,1,0.3088,0.3314,0.0357,41.2058,0.019,88
70276,151.171875,1.342993,238.602520,42.464379,1,0.8102,0.7948,0.0774,43.4853,0.026,88
70430,150.820312,1.641510,237.994507,42.358984,1,0.0451,0.0858,0.0255,37.9624,0.020,42
70571,149.414062,1.940072,236.565366,41.393323,1,0.3902,0.4137,1.1078,41.7706,0.018,90
70816,349.429535,-62.508568,320.039643,-51.393745,1,0.0000,0.0000,0.0000,nan,0.020,65
70898,151.171875,2.238686,237.619933,42.994783,1,0.5346,0.5478,0.0157,42.4996,0.024,90
70977,51.328125,-27.784405,223.130589,-55.999499,1,0.3428,0.3404,0.0099,41.2737,0.013,90
71068,34.453125,-5.229529,169.987075,-59.956185,1,0.2726,0.2679,1.0038,40.6730,0.019,88
71080,53.964844,-28.630989,225.142950,-53.813613,1,0.0000,0.0000,0.0000,nan,0.009,92
71084,52.558594,-27.279613,222.538937,-54.845107,1,0.1522,0.1472,0.0208,39.2216,0.008,42
71126,53.964844,-28.630989,225.142950,-53.813613,1,0.3021,0.5146,0.5741,42.3363,0.009,62
71438,52.558594,-27.279613,222.538937,-54.845107,1,0.0000,0.0000,0.0000,nan,0.008,65
71676,53.437500,-29.142223,225.908120,-54.336118,1,0.4396,0.4103,0.0209,41.7496,0.008,88
71890,348.595886,-63.072620,320.023289,-50.713060,1,0.7036,0.7024,0.0078,43.1558,0.021,88
71954,1.666667,-44.399834,327.519190,-70.529554,1,0.2483,0.2571,0.6448,40.5711,0.009,90
72053,53.613281,-27.953188,223.929533,-54.024772,1,0.2832,0.2616,0.0190,40.6141,0.007,88
72256,358.636353,-46.768478,328.890146,-67.388837,1,0.0000,0.0000,0.0000,nan,0.008,65
72337,34.277344,-5.079716,169.526841,-59.956640,1,0.2449,0.2107,0.1165,40.0824,0.019,90
72385,150.117188,2.836105,236.124718,42.483719,1,0.3029,0.2983,0.3068,40.9409,0.016,67
72426,51.679688,-27.447618,222.618229,-55.642263,1,0.5166,0.3846,0.3149,41.5839,0.010,90
72428,33.574219,-5.379379,168.838090,-60.637536,1,0.2214,2.4663,0.4396,46.5120,0.017,90
72489,53.613281,-26.944359,222.237403,-53.863858,1,0.8358,0.8312,0.0197,43.6047,0.009,90
72525,34.101562,-5.829153,170.247753,-60.638325,1,0.2407,0.2580,0.0145,40.5802,0.019,90
72735,151.699219,3.583322,236.533224,44.205648,1,0.2648,0.2313,0.0265,40.3104,0.016,67
73031,34.277344,-5.079716,169.526841,-59.956640,1,0.3151,0.5833,0.2442,42.6648,0.019,52
73236,33.398438,-3.732834,166.492280,-59.466614,1,0.1398,0.5280,0.4528,42.4036,0.022,90
73339,351.299988,-62.320400,319.038597,-52.026867,1,0.5183,0.2010,0.3771,39.9686,0.018,90
73433,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,65
73509,34.453125,-5.229529,169.987075,-59.956185,1,0.2953,0.2879,0.0259,40.8522,0.019,42
73610,34.277344,-5.079716,169.526841,-59.956640,1,0.8134,1.5079,0.3508,45.2031,0.019,95
74093,351.259003,-64.386185,317.344860,-50.255113,1,0.3940,0.4643,1.0625,42.0686,0.020,90
75116,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,16
75223,351.382965,-64.011238,317.574052,-50.604657,1,0.5652,0.5648,0.0157,42.5800,0.023,90
75562,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65
75598,1.723404,-45.981140,325.117958,-69.180825,1,0.1652,0.1477,0.0151,39.2287,0.010,42
75646,348.529419,-61.755440,321.293980,-51.763351,1,0.2103,0.2089,0.0083,40.0620,0.016,67
75754,151.347656,3.583322,236.252362,43.918627,1,0.0000,0.0000,0.0000,nan,0.015,16
75792,148.886719,2.686724,235.347248,41.389003,1,0.3699,0.3602,0.0451,41.4171,0.028,90
75886,358.636353,-46.768478,328.890146,-67.388837,1,0.3985,0.3735,0.0375,41.5091,0.008,90
75987,54.667969,-27.615883,223.610785,-53.050840,1,0.9014,0.8506,0.0414,43.6664,0.009,88
76242,152.050781,2.985506,237.495952,44.143927,1,0.4916,0.5572,0.3767,42.5443,0.019,90
76304,348.595886,-63.072620,320.023289,-50.713060,1,2.4303,2.6811,1.0262,46.7317,0.021,95
76305,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,16
76639,346.500000,-62.320400,321.951129,-50.736054,1,0.5322,0.5077,0.0148,42.3012,0.020,90
77010,34.453125,-5.229529,169.987075,-59.956185,1,0.0000,0.0000,0.0000,nan,0.019,65
77041,346.276581,-64.011238,320.448031,-49.344136,1,0.3793,0.4070,0.6742,41.7290,0.019,90
77157,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,6
77192,151.347656,4.181528,235.568369,44.259942,1,0.3227,0.2764,0.0329,40.7513,0.016,62
77222,151.171875,1.342993,238.602520,42.464379,1,0.4342,0.3228,0.3114,41.1398,0.026,90
77292,34.277344,-5.079716,169.526841,-59.956640,1,0.1250,0.1890,0.2309,39.8194,0.019,62
77306,148.710938,2.836105,235.050801,41.328739,1,0.7167,0.6876,0.0165,43.0996,0.031,90
77340,346.500000,-62.320400,321.951129,-50.736054,1,0.8207,0.8217,0.0401,43.5741,0.020,88
77391,346.130127,-63.072620,321.423103,-50.042305,1,1.3214,1.4667,0.1459,45.1288,0.020,95
77518,53.437500,-29.142223,225.908120,-54.336118,1,0.3531,0.3522,0.0125,41.3602,0.008,67
77623,0.190678,-45.783966,327.956322,-68.803772,1,0.2765,0.2994,0.0135,40.9507,0.005,42
77825,349.046051,-61.943836,320.796530,-51.753706,1,0.1071,0.1065,0.0160,38.4603,0.017,42
77906,359.811707,-45.191612,329.485675,-69.150905,1,0.1265,0.0738,0.0297,37.6154,0.010,90
77952,358.665253,-45.783966,330.353593,-68.203652,1,0.1258,0.0858,0.0192,37.9614,0.009,90
78095,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,65
78233,148.710938,2.836105,235.050801,41.328739,1,0.2391,0.2060,0.0706,40.0283,0.031,90
78677,53.437500,-29.142223,225.908120,-54.336118,1,0.0000,0.0000,0.0000,nan,0.008,65
78702,349.891296,-64.573555,317.972107,-49.786192,1,0.0000,0.0000,0.0000,nan,0.023,92
78705,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,92
78727,51.679688,-27.447618,222.618229,-55.642263,1,0.1592,0.1479,0.0203,39.2318,0.010,90
78974,152.050781,3.284369,237.157374,44.318466,1,0.6592,0.6572,0.0085,42.9797,0.019,90
79002,2.097458,-45.783966,324.737840,-69.478613,1,0.4446,0.2938,0.8441,40.9035,0.011,90
79155,1.666667,-44.399834,327.519190,-70.529554,1,0.4097,0.4154,0.0160,41.7815,0.009,42
79235,52.031250,-26.443335,220.963669,-55.168557,1,0.1639,0.1669,0.0564,39.5206,0.014,42
79428,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,65
79515,53.964844,-28.630989,225.142950,-53.813613,1,0.2207,0.2188,1.1526,40.1744,0.009,90
79743,32.695312,-4.929937,166.868469,-60.841230,1,0.6266,0.6062,0.0094,42.7661,0.018,90
79819,150.820312,1.641510,237.994507,42.358984,1,0.1742,0.1725,0.0134,39.5994,0.020,42
79921,352.132874,-63.636005,317.424173,-51.095855,1,0.1638,0.1709,0.0219,39.5766,0.021,42
80155,53.085938,-28.122234,224.100909,-54.509752,1,0.3628,0.3633,0.0418,41.4386,0.007,90
80205,33.925781,-5.979157,170.179895,-60.866303,1,0.8134,0.7508,0.1176,43.3333,0.022,95
80780,152.050781,3.284369,237.157374,44.318466,1,0.1684,0.1902,0.0581,39.8341,0.019,42
80832,150.117188,2.238686,236.784618,42.139082,1,0.3950,0.3665,0.0146,41.4614,0.016,90
80852,151.347656,3.583322,236.252362,43.918627,1,0.2569,0.2659,0.0082,40.6545,0.015,62
80903,52.910156,-26.276812,220.926149,-54.363918,1,0.4140,0.4279,0.0171,41.8576,0.008,90
81000,150.996094,2.985506,236.647967,43.287350,1,0.2989,0.3106,0.0100,41.0422,0.020,42
81252,33.574219,-4.780192,168.064587,-60.175886,1,0.1494,0.1737,0.6985,39.6157,0.019,62
81464,149.414062,3.433834,234.919132,42.245550,1,0.2854,0.2818,0.3207,40.7987,0.027,42
81665,53.789062,-27.784405,223.685697,-53.845803,1,0.0000,0.0000,0.0000,nan,0.009,65
82302,346.276581,-64.011238,320.448031,-49.344136,1,0.0000,0.0000,0.0000,nan,0.019,65
82401,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,16
82409,348.595886,-63.072620,320.023289,-50.713060,1,0.1147,2.6274,1.1973,46.6786,0.021,42
82702,35.683594,-5.379379,171.992947,-59.253501,1,0.3194,0.2986,0.0204,40.9437,0.020,90
82740,349.615387,-63.636005,318.927246,-50.506542,1,1.0263,1.0228,0.2611,44.1609,0.018,88
83348,349.429535,-62.508568,320.039643,-51.393745,1,0.0000,0.0000,0.0000,nan,0.020,6
83410,51.855469,-26.276812,220.627031,-55.293792,1,0.5138,0.5071,0.0093,42.2980,0.014,90
83462,150.820312,3.134927,236.341348,43.230123,1,0.3424,0.3239,0.0179,41.1483,0.016,90
83634,349.285706,-62.884678,319.786163,-51.046461,1,2.1107,1.3813,0.4158,44.9680,0.018,95
83821,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,65
83872,149.589844,3.583322,234.885369,42.474696,1,0.2160,0.2054,0.2819,40.0211,0.024,90
83954,346.276581,-64.011238,320.448031,-49.344136,1,0.4390,0.4535,0.0428,42.0079,0.019,90
83961,1.753247,-46.768478,324.030235,-68.498041,1,0.2177,0.2211,0.0151,40.2004,0.014,90
84306,151.171875,2.537361,237.288526,43.169764,1,0.0000,0.0000,0.0000,nan,0.024,16
84716,151.523438,3.134927,236.900695,43.803170,1,0.4303,0.2925,1.0152,40.8923,0.019,90
84758,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,65
85125,53.085938,-28.122234,224.100909,-54.509752,1,0.0000,0.0000,0.0000,nan,0.007,65
85470,33.398438,-4.331149,167.226341,-59.936551,1,0.0000,0.0000,0.0000,nan,0.018,65
85490,348.908447,-63.823658,319.169886,-50.176186,1,0.2537,0.5506,0.2880,42.5130,0.018,90
85789,53.437500,-29.142223,225.908120,-54.336118,1,0.2990,0.2893,0.0367,40.8644,0.008,62
86456,33.574219,-5.379379,168.838090,-60.637536,1,0.2127,0.2191,0.0141,40.1785,0.017,42
86487,0.574468,-45.981140,327.041068,-68.778764,1,0.3850,0.3847,0.0333,41.5849,0.006,90
86759,348.529419,-61.755440,321.293980,-51.763351,1,0.0000,0.0000,0.0000,nan,0.016,65
86834,149.589844,3.583322,234.885369,42.474696,1,0.3160,0.4622,0.4742,42.0568,0.024,90
87180,150.820312,1.641510,237.994507,42.358984,1,0.1782,0.1820,0.0103,39.7288,0.020,62
87467,150.820312,3.134927,236.341348,43.230123,1,0.0000,0.0000,0.0000,nan,0.016,65
87498,152.050781,2.985506,237.495952,44.143927,1,0.0000,0.0000,0.0000,nan,0.019,16
87608,358.665253,-45.783966,330.353593,-68.203652,1,0.1894,0.2101,0.0179,40.0763,0.009,90
87685,347.861847,-61.943836,321.519104,-51.424048,1,0.3743,0.3612,0.0219,41.4237,0.017,90
87703,51.855469,-26.276812,220.627031,-55.293792,1,0.3246,0.5019,0.3014,42.2711,0.014,90
88073,347.013428,-62.508568,321.472056,-50.735330,1,0.0000,0.0000,0.0000,nan,0.018,92
88180,149.414062,1.940072,236.565366,41.393323,1,0.2996,2.6936,1.3051,46.7439,0.018,67
88195,53.964844,-28.630989,225.142950,-53.813613,1,0.3403,0.3448,0.0304,41.3063,0.009,90
88511,53.613281,-28.630989,225.073365,-54.119461,1,0.0000,0.0000,0.0000,nan,0.006,65
88587,352.711273,-63.823658,316.922299,-51.059403,1,0.7839,0.7789,0.0120,43.4314,0.024,90
88600,351.321442,-64.198746,317.458993,-50.429931,1,0.3303,0.3150,0.3295,41.0778,0.023,90
88627,32.871094,-4.780192,166.959493,-60.615132,1,0.3934,0.4406,0.0433,41.9329,0.017,67
88980,351.382965,-64.011238,317.574052,-50.604657,1,0.0000,0.0000,0.0000,nan,0.023,16
89157,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,92
89298,359.811707,-45.191612,329.485675,-69.150905,1,0.0000,0.0000,0.0000,nan,0.010,16
89387,346.130127,-63.072620,321.423103,-50.042305,1,0.6210,0.4739,0.0648,42.1217,0.020,90
89455,0.189873,-45.586655,328.254458,-68.969298,1,0.2603,0.2485,0.0142,40.4873,0.007,90
89709,52.910156,-25.944481,220.366350,-54.301439,1,0.0000,0.0000,0.0000,nan,0.010,92
89999,149.238281,3.882372,234.283829,42.351155,1,0.2640,0.3194,0.0576,41.1125,0.033,90
90399,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,65
90534,152.050781,3.284369,237.157374,44.318466,1,0.1759,0.1913,0.0131,39.8481,0.019,62
90645,51.855469,-28.630989,224.733260,-55.649872,1,0.4496,0.4486,0.0173,41.9797,0.009,90
90814,348.595886,-63.072620,320.023289,-50.713060,1,0.2252,0.2208,0.0172,40.1971,0.021,62
90892,152.050781,3.284369,237.157374,44.318466,1,0.0322,0.0365,0.0161,36.0320,0.019,52
91219,150.820312,3.134927,236.341348,43.230123,1,0.2921,0.5086,0.1231,42.3058,0.016,90
91291,352.711273,-63.823658,316.922299,-51.059403,1,0.1826,0.1746,0.0132,39.6288,0.024,90
91335,151.699219,3.583322,236.533224,44.205648,1,1.0655,1.4889,0.2658,45.1692,0.016,88
91337,53.613281,-27.953188,223.929533,-54.024772,1,0.0000,0.0000,0.0000,nan,0.007,65
91460,53.613281,-28.630989,225.073365,-54.119461,1,0.2782,0.5294,0.2771,42.4103,0.006,90
91610,346.130127,-63.072620,321.423103,-50.042305,1,0.2326,0.2302,0.0070,40.2992,0.020,42
91644,349.891296,-64.573555,317.972107,-49.786192,1,0.1893,0.1839,0.1322,39.7535,0.023,90
91917,152.050781,2.985506,237.495952,44.143927,1,0.2448,0.2740,0.0117,40.7289,0.019,90
91988,150.468750,1.641510,237.714575,42.075234,1,0.0000,0.0000,0.0000,nan,0.017,65
92334,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,65
92354,51.328125,-27.784405,223.130589,-55.999499,1,0.5449,0.5807,0.0374,42.6530,0.013,88
92566,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,16
92577,351.734680,-62.884678,318.284128,-51.651217,1,0.4426,0.3854,0.6831,41.5893,0.019,90
92904,347.812500,-63.448284,320.128971,-50.202348,1,0.2536,0.2814,0.0325,40.7958,0.021,62
92929,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,65
93333,151.171875,1.342993,238.602520,42.464379,1,0.4029,0.3217,0.4659,41.1310,0.026,90
93362,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,92
93509,51.855469,-26.276812,220.627031,-55.293792,1,0.0587,0.0644,0.0151,37.3066,0.014,42
93663,53.964844,-28.630989,225.142950,-53.813613,1,0.4196,0.4099,0.0367,41.7471,0.009,90
94004,52.910156,-27.953188,223.774083,-54.639214,1,0.2354,0.2257,0.0162,40.2510,0.007,62
94107,34.980469,-6.279288,172.180075,-60.389399,1,0.0000,0.0000,0.0000,nan,0.023,16
94229,52.207031,-28.291550,224.208534,-55.300157,1,0.3853,0.4477,0.0268,41.9745,0.007,90
94613,2.457983,-45.389202,324.632685,-69.945696,1,0.0739,0.0616,0.0188,37.2044,0.011,62
94704,349.966217,-62.696659,319.542989,-51.376556,1,0.2722,0.2658,0.0080,40.6537,0.021,90
95127,351.299988,-62.320400,319.038597,-52.026867,1,1.8136,1.6691,0.2276,45.4751,0.018,95
95147,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,65
95369,53.613281,-28.630989,225.073365,-54.119461,1,0.1313,0.5834,0.7684,42.6652,0.006,62
95455,351.259003,-64.386185,317.344860,-50.255113,1,0.4942,0.5590,0.1899,42.5528,0.020,90
95483,150.820312,3.732834,235.666318,43.572109,1,0.8512,0.8164,0.0436,43.5568,0.016,88
95508,149.414062,2.238686,236.239766,41.565558,1,0.6107,0.5480,0.0306,42.5009,0.017,88
95566,33.574219,-5.079716,168.448505,-60.407218,1,0.3880,0.4580,0.0992,42.0331,0.016,90
95580,34.101562,-5.829153,170.247753,-60.638325,1,0.4200,0.4311,0.0132,41.8770,0.019,90
95690,351.734680,-62.884678,318.284128,-51.651217,1,0.3442,0.3507,0.0052,41.3494,0.019,42
95741,35.332031,-5.979157,172.286722,-59.931743,1,0.5064,0.5368,0.0209,42.4469,0.022,52
95864,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,92
96284,152.050781,3.284369,237.157374,44.318466,1,0.1593,2.4014,0.4125,46.4417,0.019,42
97053,150.117188,3.732834,235.120533,42.993809,1,0.0000,0.0000,0.0000,nan,0.020,65
97406,347.846710,-64.760857,318.929827,-49.143596,1,0.1163,0.0897,0.8592,38.0649,0.019,15
97687,346.655182,-63.260487,320.952196,-50.040935,1,0.4603,0.4626,0.0271,42.0594,0.019,90
97850,351.259003,-64.386185,317.344860,-50.255113,1,0.3090,0.3005,0.0119,40.9599,0.020,90
97920,150.820312,3.134927,236.341348,43.230123,1,0.3588,0.3524,0.0238,41.3611,0.016,67
97957,1.723404,-45.981140,325.117958,-69.180825,1,0.2947,0.2906,0.0093,40.8754,0.010,90
98533,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,65
98570,1.708861,-45.586655,325.688716,-69.520253,1,0.6197,0.5296,0.0328,42.4113,0.011,42
98749,33.750000,-4.630479,168.146242,-59.949072,1,0.1473,0.0973,0.6573,38.2516,0.019,67
99013,350.230255,-61.943836,320.053946,-52.070537,1,0.5255,0.5303,0.0220,42.4150,0.017,90
99050,52.207031,-26.610098,221.298836,-55.042928,1,0.2972,0.3036,0.1492,40.9851,0.014,42
99261,53.613281,-27.953188,223.929533,-54.024772,1,0.1411,0.0857,0.0282,37.9579,0.007,90
99280,359.811707,-45.191612,329.485675,-69.150905,1,0.2037,0.2150,0.0079,40.1324,0.010,62
99293,347.846710,-64.760857,318.929827,-49.143596,1,0.3106,0.3644,0.0251,41.4467,0.019,90
99294,348.529419,-61.755440,321.293980,-51.763351,1,0.5552,0.5204,0.0185,42.3656,0.016,90
99452,352.711273,-63.823658,316.922299,-51.059403,1,0.8420,0.8479,0.0465,43.6581,0.024,88
99642,347.846710,-64.760857,318.929827,-49.143596,1,0.2320,0.2369,1.1397,40.3699,0.019,90
99862,52.207031,-28.291550,224.208534,-55.300157,1,0.5810,0.5733,0.0141,42.6194,0.007,42
99932,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,65
100057,346.130127,-63.072620,321.423103,-50.042305,1,0.8320,0.8006,0.0197,43.5045,0.020,90
100097,348.529419,-61.755440,321.293980,-51.763351,1,0.1566,0.1745,0.0186,39.6271,0.016,42
100133,346.655182,-63.260487,320.952196,-50.040935,1,0.2309,0.2215,0.0190,40.2046,0.019,42
100331,52.207031,-26.610098,221.298836,-55.042928,1,0.1684,0.1999,0.0173,39.9556,0.014,42
101050,32.695312,-4.929937,166.868469,-60.841230,1,0.2222,0.2529,0.0906,40.5303,0.018,90
101298,34.277344,-5.079716,169.526841,-59.956640,1,0.1848,0.1983,0.0069,39.9355,0.019,90
101374,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,16
101489,359.816315,-44.003082,331.451340,-70.123054,1,0.0721,0.0695,0.0025,37.4803,0.013,42
101508,358.648071,-46.375080,329.462659,-67.716008,1,0.0656,0.0249,0.0130,35.1822,0.009,90
101890,346.276581,-64.011238,320.448031,-49.344136,1,0.0000,0.0000,0.0000,nan,0.019,65
102036,53.789062,-27.784405,223.685697,-53.845803,1,0.2331,0.2269,0.0197,40.2635,0.009,42
102330,148.886719,2.686724,235.347248,41.389003,1,0.0000,0.0000,0.0000,nan,0.028,16
102343,51.328125,-27.447618,222.535046,-55.950727,1,0.1081,0.1422,0.0164,39.1394,0.013,67
102363,349.160583,-64.760857,318.219706,-49.458924,1,0.8046,0.7835,0.0167,43.4470,0.020,90
102745,349.615387,-63.636005,318.927246,-50.506542,1,0.2460,0.2333,1.0359,40.3315,0.018,90
102823,347.861847,-61.943836,321.519104,-51.424048,1,0.0000,0.0000,0.0000,nan,0.017,65
102864,0.574468,-45.981140,327.041068,-68.778764,1,0.2123,0.2254,0.0097,40.2473,0.006,42
103023,359.811707,-45.191612,329.485675,-69.150905,1,0.3221,0.2994,0.0125,40.9500,0.010,90
103026,348.529419,-61.755440,321.293980,-51.763351,1,0.1949,0.2006,0.0174,39.9637,0.016,42
103100,34.101562,-5.829153,170.247753,-60.638325,1,0.5057,0.4350,0.0240,41.9001,0.019,90
103145,349.160583,-64.760857,318.219706,-49.458924,1,0.6603,0.6380,0.0433,42.9010,0.020,90
103162,350.230255,-61.943836,320.053946,-52.070537,1,0.1101,0.1207,0.0096,38.7539,0.017,90
103171,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,92
103350,359.446716,-44.201530,331.730015,-69.805709,1,0.6777,0.6935,0.0507,43.1220,0.010,90
103354,150.117188,2.836105,236.124718,42.483719,1,2.7124,2.7655,0.0801,46.8131,0.016,88
103572,52.207031,-28.291550,224.208534,-55.300157,1,0.0000,0.0000,0.0000,nan,0.007,65
103927,150.820312,1.641510,237.994507,42.358984,1,0.2130,0.2282,0.0218,40.2779,0.020,90
103948,33.574219,-4.780192,168.064587,-60.175886,1,0.2257,0.2319,0.1182,40.3169,0.019,52
103967,0.190678,-45.783966,327.956322,-68.803772,1,0.5094,0.4789,0.0608,42.1493,0.005,42
104212,53.789062,-27.784405,223.685697,-53.845803,1,0.3895,0.3850,1.0136,41.5869,0.009,90
104397,349.966217,-62.696659,319.542989,-51.376556,1,0.3220,0.3038,0.0187,40.9870,0.021,90
104476,51.855469,-28.630989,224.733260,-55.649872,1,1.4820,1.4819,0.1602,45.1565,0.009,95
104498,149.238281,3.882372,234.283829,42.351155,1,0.1312,0.1100,0.0094,38.5371,0.033,67
104523,152.050781,3.284369,237.157374,44.318466,1,0.0000,0.0000,0.0000,nan,0.019,16
104526,349.615387,-63.636005,318.927246,-50.506542,1,0.2353,0.2389,0.0148,40.3902,0.018,42
104701,352.132874,-63.636005,317.424173,-51.095855,1,0.0000,0.0000,0.0000,nan,0.021,65
105744,348.908447,-63.823658,319.169886,-50.176186,1,0.3087,0.3318,1.0382,41.2090,0.018,90
106177,358.312500,-44.993881,332.185785,-68.685906,1,0.6593,0.5956,0.1033,42.7194,0.009,90
106429,152.050781,2.985506,237.495952,44.143927,1,0.0000,0.0000,0.0000,nan,0.019,16
106434,33.574219,-4.780192,168.064587,-60.175886,1,0.2861,0.2795,0.0098,40.7789,0.019,62
106594,0.589520,-47.161343,325.385896,-67.769893,1,0.2403,0.2370,1.0591,40.3710,0.009,88
106730,347.013428,-62.508568,321.472056,-50.735330,1,0.1576,2.2437,0.3333,46.2621,0.018,52
106743,0.574468,-45.981140,327.041068,-68.778764,1,0.2154,0.2539,0.0150,40.5405,0.006,90
106818,348.595886,-63.072620,320.023289,-50.713060,1,0.0000,0.0000,0.0000,nan,0.021,92
106937,53.085938,-28.122234,224.100909,-54.509752,1,0.3656,0.3882,0.8841,41.6079,0.007,52
107193,359.058563,-45.191612,330.695783,-68.844915,1,0.2109,0.1761,0.0082,39.6492,0.011,90
107439,150.468750,3.732834,235.392208,43.283244,1,0.9187,1.5247,0.3061,45.2328,0.020,88
107451,347.861847,-61.943836,321.519104,-51.424048,1,0.0000,0.0000,0.0000,nan,0.017,65
107568,33.750000,-4.630479,168.146242,-59.949072,1,0.1042,0.0802,0.0143,37.8074,0.019,67
107615,150.117188,2.836105,236.124718,42.483719,1,0.1323,2.3872,0.9055,46.4260,0.016,42
107712,53.261719,-27.615883,223.280041,-54.281374,1,0.1282,0.1450,0.0286,39.1863,0.006,90
107901,359.058563,-45.191612,330.695783,-68.844915,1,0.0000,0.0000,0.0000,nan,0.011,65
108021,150.996094,4.181528,235.291975,43.970869,1,0.2815,0.3756,0.8142,41.5232,0.015,62
108141,53.789062,-27.784405,223.685697,-53.845803,1,0.2179,0.2239,0.0151,40.2317,0.009,90
108229,351.321442,-64.198746,317.458993,-50.429931,1,0.1858,0.2022,1.1539,39.9826,0.023,90
108358,349.891296,-64.573555,317.972107,-49.786192,1,0.0769,0.2433,1.0869,40.4350,0.023,52
108487,359.816315,-44.003082,331.451340,-70.123054,1,0.1787,0.2401,0.6645,40.4026,0.013,42
108554,33.222656,-4.780192,167.515653,-60.396584,1,0.4478,0.4542,0.0258,42.0120,0.018,90
108693,0.574468,-45.981140,327.041068,-68.778764,1,0.4868,0.4183,0.0364,41.7991,0.006,90
108739,53.085938,-28.122234,224.100909,-54.509752,1,0.4453,0.4388,0.0272,41.9226,0.007,90
108888,358.648071,-46.375080,329.462659,-67.716008,1,0.6247,0.5951,0.0599,42.7172,0.009,90
109036,1.753247,-46.768478,324.030235,-68.498041,1,0.1924,0.1862,0.2684,39.7838,0.014,90
109057,348.595886,-63.072620,320.023289,-50.713060,1,0.4499,0.3307,1.0499,41.2005,0.021,90
109294,359.814819,-44.399834,330.775011,-69.801007,1,0.3014,0.3643,0.8070,41.4461,0.009,90
109516,1.753247,-46.768478,324.030235,-68.498041,1,0.3828,0.3941,0.0095,41.6466,0.014,95
109654,347.013428,-62.508568,321.472056,-50.735330,1,0.1395,0.1231,0.0131,38.8003,0.018,90
109860,0.929752,-44.597992,328.531426,-70.083244,1,0.2708,0.2639,0.0184,40.6357,0.011,42
109903,150.996094,2.388015,237.313912,42.939977,1,0.3892,0.3579,0.0137,41.4006,0.021,62
109937,149.414062,3.433834,234.919132,42.245550,1,0.6620,0.6423,0.0188,42.9190,0.027,88
110241,34.453125,-5.229529,169.987075,-59.956185,1,0.1072,0.0924,0.4007,38.1327,0.019,42
110257,148.886719,2.686724,235.347248,41.389003,1,0.9318,0.8495,0.0379,43.6631,0.028,88
110270,54.667969,-27.615883,223.610785,-53.050840,1,0.8259,0.8541,0.0446,43.6776,0.009,90
110304,53.261719,-27.615883,223.280041,-54.281374,1,0.2475,0.5457,0.2676,42.4899,0.006,62
110387,151.347656,4.181528,235.568369,44.259942,1,0.4318,0.4218,0.6713,41.8205,0.016,90
110551,51.328125,-27.447618,222.535046,-55.950727,1,0.4861,0.3027,0.3931,40.9783,0.013,88
110768,351.734680,-62.884678,318.284128,-51.651217,1,2.9378,2.8626,1.1139,46.9035,0.019,88
110958,349.615387,-63.636005,318.927246,-50.506542,1,0.4630,0.4742,0.5031,42.1236,0.018,90
111281,0.589520,-47.161343,325.385896,-67.769893,1,1.4967,1.6014,0.0895,45.3643,0.009,42
111283,150.468750,1.641510,237.714575,42.075234,1,0.6565,0.6363,0.0252,42.8941,0.017,90
111448,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65
111650,351.734680,-62.884678,318.284128,-51.651217,1,0.0000,0.0000,0.0000,nan,0.019,65
111795,34.453125,-5.229529,169.987075,-59.956185,1,0.4070,0.3077,1.0584,41.0187,0.019,90
111799,52.558594,-27.279613,222.538937,-54.845107,1,0.2813,0.5741,0.4749,42.6227,0.008,42
112151,349.966217,-62.696659,319.542989,-51.376556,1,0.5341,0.5419,0.0204,42.4716,0.021,90
112462,33.574219,-4.780192,168.064587,-60.175886,1,0.4045,0.4356,0.0299,41.9040,0.019,90
112629,346.276581,-64.011238,320.448031,-49.344136,1,0.0000,0.0000,0.0000,nan,0.019,16
112717,35.683594,-5.379379,171.992947,-59.253501,1,0.2867,0.3138,0.8216,41.0680,0.020,67
112764,347.812500,-63.448284,320.128971,-50.202348,1,0.3208,0.2995,0.0255,40.9517,0.021,90
112782,32.871094,-4.780192,166.959493,-60.615132,1,0.4211,0.4038,0.0224,41.7088,0.017,90
112886,351.953644,-62.132156,318.777388,-52.347124,1,0.0000,0.0000,0.0000,nan,0.019,65
113028,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,65
113206,52.207031,-28.291550,224.208534,-55.300157,1,0.2446,0.2208,0.0078,40.1969,0.007,42
113335,358.636353,-46.768478,328.890146,-67.388837,1,0.0000,0.0000,0.0000,nan,0.008,65
113625,149.589844,3.583322,234.885369,42.474696,1,0.2615,0.2381,0.0160,40.3824,0.024,90
113669,351.299988,-62.320400,319.038597,-52.026867,1,0.2914,0.2905,0.0061,40.8748,0.018,15
113982,359.805206,-46.768478,327.135979,-67.829903,1,0.7438,0.9216,0.1482,43.8813,0.011,90
114191,33.574219,-6.579593,170.455585,-61.548219,1,0.0000,0.0000,0.0000,nan,0.021,65
114341,151.699219,3.583322,236.533224,44.205648,1,0.1709,2.3232,0.5051,46.3542,0.016,42
114626,2.071130,-45.191612,325.606223,-69.989264,1,0.5702,0.5408,0.0178,42.4659,0.011,90
114670,150.996094,2.388015,237.313912,42.939977,1,0.2930,0.2607,0.0670,40.6057,0.021,90
114715,0.965665,-46.375080,325.845907,-68.579427,1,0.2141,0.2020,0.0362,39.9807,0.007,67
114808,52.207031,-26.610098,221.298836,-55.042928,1,0.3037,0.2944,0.0104,40.9082,0.014,90
115053,33.925781,-5.979157,170.179895,-60.866303,1,0.1649,0.1424,0.2329,39.1427,0.022,42
115079,1.694561,-45.191612,326.278557,-69.858253,1,0.2207,0.2325,0.0072,40.3232,0.011,90
115157,2.457983,-45.389202,324.632685,-69.945696,1,0.6382,0.5628,0.0442,42.5706,0.011,90
115336,351.734680,-62.884678,318.284128,-51.651217,1,1.7312,1.7123,0.0766,45.5432,0.019,95
115638,358.665253,-45.783966,330.353593,-68.203652,1,0.6365,0.6239,0.0079,42.8420,0.009,90
115670,151.347656,3.583322,236.252362,43.918627,1,0.0000,0.0000,0.0000,nan,0.015,16
115792,352.711273,-63.823658,316.922299,-51.059403,1,0.2389,0.2859,0.0505,40.8350,0.024,90
115859,148.886719,2.686724,235.347248,41.389003,1,1.4248,1.2871,0.1757,44.7782,0.028,88
115937,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,65
116132,150.292969,2.686724,236.427488,42.541447,1,0.0878,0.0965,0.0207,38.2336,0.016,62
116212,150.644531,3.583322,235.698235,43.342784,1,0.2344,0.2757,1.1756,40.7443,0.018,90
116570,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,16
116720,34.980469,-6.279288,172.180075,-60.389399,1,0.4081,0.3212,0.0367,41.1269,0.023,90
116818,359.805206,-46.768478,327.135979,-67.829903,1,0.6528,0.7091,0.0211,43.1811,0.011,88
117016,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,92
117104,346.130127,-63.072620,321.423103,-50.042305,1,0.0000,0.0000,0.0000,nan,0.020,92
117184,352.132874,-63.636005,317.424173,-51.095855,1,0.0000,0.0000,0.0000,nan,0.021,16
117393,52.910156,-25.944481,220.366350,-54.301439,1,0.8259,0.8295,0.0204,43.5993,0.010,90
117461,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,92
117513,151.347656,3.583322,236.252362,43.918627,1,0.8515,0.8845,0.0148,43.7712,0.015,90
117774,346.276581,-64.011238,320.448031,-49.344136,1,0.5589,0.5608,0.1532,42.5611,0.019,90
118211,151.699219,3.583322,236.533224,44.205648,1,0.2420,0.2139,1.1778,40.1199,0.016,90
118422,149.589844,3.583322,234.885369,42.474696,1,0.0000,0.0000,0.0000,nan,0.024,65
118455,347.861847,-61.943836,321.519104,-51.424048,1,0.1346,0.1396,0.3139,39.0951,0.017,90
118770,53.085938,-28.122234,224.100909,-54.509752,1,0.2642,0.6375,0.0267,42.8991,0.007,90
118868,350.230255,-61.943836,320.053946,-52.070537,1,0.0000,0.0000,0.0000,nan,0.017,65
118979,2.457983,-45.389202,324.632685,-69.945696,1,0.2374,0.5280,0.1420,42.4036,0.011,67
119215,52.031250,-26.443335,220.963669,-55.168557,1,0.2490,0.2486,0.0069,40.4887,0.014,90
119383,0.949367,-45.586655,326.991548,-69.251686,1,0.3501,0.2602,0.6723,40.6012,0.013,42
119494,150.117188,2.836105,236.124718,42.483719,1,0.2113,0.2256,0.0219,40.2502,0.016,90
119647,151.523438,3.134927,236.900695,43.803170,1,0.1844,0.1571,0.0210,39.3762,0.019,62
119811,348.586945,-64.573555,318.693903,-49.477869,1,0.7287,0.7966,0.0308,43.4912,0.018,90
119882,358.312500,-44.993881,332.185785,-68.685906,1,0.0000,0.0000,0.0000,nan,0.009,65
120356,32.695312,-4.929937,166.868469,-60.841230,1,0.0582,0.0543,0.0185,36.9203,0.018,42
120927,152.050781,3.284369,237.157374,44.318466,1,0.4807,0.4897,0.0334,42.2071,0.019,90
121107,35.332031,-5.979157,172.286722,-59.931743,1,0.5440,0.2209,0.2405,40.1982,0.022,90
121182,53.085938,-28.122234,224.100909,-54.509752,1,0.0000,0.0000,0.0000,nan,0.007,65
121224,359.811707,-45.191612,329.485675,-69.150905,1,0.1244,2.3005,0.4304,46.3283,0.010,42
121266,151.347656,3.583322,236.252362,43.918627,1,0.4062,0.5002,0.0481,42.2623,0.015,42
121301,52.207031,-28.630989,224.800211,-55.343637,1,0.1459,0.1449,1.1250,39.1845,0.009,62
121440,53.613281,-28.630989,225.073365,-54.119461,1,0.2230,0.2219,0.1530,40.2089,0.006,42
121447,0.189873,-45.586655,328.254458,-68.969298,1,0.2571,0.2465,0.0106,40.4674,0.007,90
121704,359.805206,-46.768478,327.135979,-67.829903,1,1.5357,1.5245,0.1053,45.2324,0.011,88
121705,352.132874,-63.636005,317.424173,-51.095855,1,0.1125,0.1219,0.0187,38.7767,0.021,42
121783,35.332031,-5.979157,172.286722,-59.931743,1,0.1324,0.0956,0.0132,38.2109,0.022,90
121803,348.586945,-64.573555,318.693903,-49.477869,1,0.2563,0.3703,1.1769,41.4870,0.018,90
121883,150.117188,2.836105,236.124718,42.483719,1,0.0000,0.0000,0.0000,nan,0.016,65
122235,53.085938,-28.122234,224.100909,-54.509752,1,0.0000,0.0000,0.0000,nan,0.007,65
122275,54.667969,-27.615883,223.610785,-53.050840,1,0.3172,0.5059,0.1684,42.2916,0.009,90
122716,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,65
122965,53.085938,-28.122234,224.100909,-54.509752,1,0.7135,0.7416,0.0359,43.3006,0.007,90
123035,150.468750,3.732834,235.392208,43.283244,1,0.2740,0.3135,0.5471,41.0660,0.020,52
123151,52.207031,-28.291550,224.208534,-55.300157,1,0.2548,0.2657,0.0184,40.6528,0.007,62
123211,150.996094,2.985506,236.647967,43.287350,1,1.6177,1.6654,0.0205,45.4691,0.020,88
123244,1.363636,-46.768478,324.669342,-68.371416,1,0.4582,0.3682,0.3963,41.4728,0.008,42
123437,348.595886,-63.072620,320.023289,-50.713060,1,0.1987,0.2137,0.0164,40.1178,0.021,52
123493,34.804688,-5.829153,171.307861,-60.174401,1,1.0395,0.9736,0.2348,44.0284,0.023,95
123743,152.050781,2.985506,237.495952,44.143927,1,0.2016,0.2129,0.0188,40.1084,0.019,90
123926,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,16
123927,150.468750,3.732834,235.392208,43.283244,1,0.0000,0.0000,0.0000,nan,0.020,16
124006,2.457983,-45.389202,324.632685,-69.945696,1,0.3574,0.3496,0.7517,41.3410,0.011,90
124183,33.574219,-6.579593,170.455585,-61.548219,1,0.0000,0.0000,0.0000,nan,0.021,65
124188,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,65
124361,51.855469,-27.953188,223.543603,-55.561470,1,0.3256,0.3420,0.8779,41.2856,0.008,88
124394,52.207031,-26.610098,221.298836,-55.042928,1,0.9198,1.4761,0.2825,45.1459,0.014,90
124679,346.276581,-64.011238,320.448031,-49.344136,1,0.3602,0.3409,0.0345,41.2771,0.019,52
124762,347.861847,-61.943836,321.519104,-51.424048,1,0.1442,2.8043,0.6276,46.8496,0.017,42
125095,352.711273,-63.823658,316.922299,-51.059403,1,0.0000,0.0000,0.0000,nan,0.024,65
125242,349.891296,-64.573555,317.972107,-49.786192,1,0.2721,0.5662,0.2621,42.5866,0.023,90
125258,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,65
125426,349.429535,-62.508568,320.039643,-51.393745,1,1.1327,1.1093,0.0315,44.3789,0.020,88
125470,34.277344,-5.079716,169.526841,-59.956640,1,0.5731,0.5848,0.0092,42.6713,0.019,90
125518,359.805206,-46.768478,327.135979,-67.829903,1,0.0000,0.0000,0.0000,nan,0.011,65
125743,347.013428,-62.508568,321.472056,-50.735330,1,0.9950,1.0046,0.0133,44.1126,0.018,42
125762,1.753247,-46.768478,324.030235,-68.498041,1,0.0000,0.0000,0.0000,nan,0.014,65
126061,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,65
126084,149.238281,3.882372,234.283829,42.351155,1,0.1250,0.1223,0.4578,38.7846,0.033,90
126970,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16
127056,34.453125,-5.229529,169.987075,-59.956185,1,0.1629,0.1385,0.0373,39.0772,0.019,42
127488,348.908447,-63.823658,319.169886,-50.176186,1,0.0000,0.0000,0.0000,nan,0.018,65
127773,53.085938,-27.111860,222.384291,-54.355086,1,0.0000,0.0000,0.0000,nan,0.007,65
127942,348.595886,-63.072620,320.023289,-50.713060,1,0.3899,0.5026,0.2876,42.2748,0.021,90
127996,346.562500,-63.448284,320.824720,-49.866957,1,0.6993,0.6759,0.0121,43.0539,0.021,95
128339,0.965665,-46.375080,325.845907,-68.579427,1,0.2192,2.1719,1.1420,46.1760,0.007,67
128405,352.398651,-62.696659,318.017427,-51.967966,1,0.0000,0.0000,0.0000,nan,0.020,65
128488,150.468750,1.641510,237.714575,42.075234,1,0.1333,0.1194,0.0257,38.7277,0.017,42
128518,34.101562,-5.829153,170.247753,-60.638325,1,0.5644,0.5449,0.0420,42.4859,0.019,90
128564,150.117188,2.238686,236.784618,42.139082,1,0.4541,2.8846,1.1987,46.9235,0.016,90
128737,52.558594,-27.279613,222.538937,-54.845107,1,0.4204,0.4270,0.0598,41.8523,0.008,90
128746,152.050781,2.985506,237.495952,44.143927,1,0.3408,0.3671,0.5585,41.4654,0.019,52
128967,150.644531,3.583322,235.698235,43.342784,1,0.3145,0.3149,0.0153,41.0770,0.018,90
129179,346.130127,-63.072620,321.423103,-50.042305,1,0.1422,0.1600,0.0245,39.4198,0.020,52
129490,34.453125,-5.229529,169.987075,-59.956185,1,0.3803,0.4821,0.2384,42.1664,0.019,42
129503,150.292969,2.686724,236.427488,42.541447,1,0.1939,0.1882,0.0140,39.8092,0.016,52
129637,33.925781,-5.979157,170.179895,-60.866303,1,0.0000,0.0000,0.0000,nan,0.022,92
129648,348.529419,-61.755440,321.293980,-51.763351,1,0.3613,2.9518,0.4926,46.9838,0.016,62
129861,351.321442,-64.198746,317.458993,-50.429931,1,0.5180,0.5215,0.0294,42.3710,0.023,90
130220,346.276581,-64.011238,320.448031,-49.344136,1,0.5002,0.4635,0.0365,42.0643,0.019,42
130404,150.820312,3.732834,235.666318,43.572109,1,0.2531,0.2629,0.0072,40.6262,0.016,42
130502,35.683594,-5.379379,171.992947,-59.253501,1,0.2141,0.1909,0.0935,39.8439,0.020,90
130625,1.708861,-45.586655,325.688716,-69.520253,1,0.0000,0.0000,0.0000,nan,0.011,16
130750,359.816315,-44.003082,331.451340,-70.123054,1,0.2536,0.2917,0.7396,40.8850,0.013,90
131075,52.910156,-25.944481,220.366350,-54.301439,1,0.1709,0.1776,0.0174,39.6689,0.010,42
131181,34.101562,-5.829153,170.247753,-60.638325,1,0.0000,0.0000,0.0000,nan,0.019,16
131305,351.382965,-64.011238,317.574052,-50.604657,1,0.2846,0.3113,0.0407,41.0484,0.023,90
131368,150.996094,2.388015,237.313912,42.939977,1,0.0000,0.0000,0.0000,nan,0.021,65
131488,0.965665,-46.375080,325.845907,-68.579427,1,0.2691,0.2621,0.0183,40.6187,0.007,90
131492,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,16
131629,33.574219,-5.079716,168.448505,-60.407218,1,1.1119,1.0817,0.0329,44.3113,0.016,95
131814,151.699219,3.583322,236.533224,44.205648,1,0.2823,0.5387,0.2038,42.4557,0.016,90
131815,32.695312,-4.929937,166.868469,-60.841230,1,0.3521,0.3520,0.0254,41.3584,0.018,90
132021,359.058563,-45.191612,330.695783,-68.844915,1,0.2525,0.6394,0.1855,42.9069,0.011,90
132278,359.415588,-46.768478,327.729895,-67.686097,1,0.0000,0.0000,0.0000,nan,0.009,92
133074,347.617462,-62.508568,321.121462,-50.904708,1,0.2610,0.2672,0.8507,40.6666,0.019,90
133191,346.276581,-64.011238,320.448031,-49.344136,1,0.4648,0.4423,0.0247,41.9430,0.019,90
133234,151.523438,3.134927,236.900695,43.803170,1,0.2288,0.2866,0.0891,40.8416,0.019,15
133354,358.665253,-45.783966,330.353593,-68.203652,1,0.4990,0.5076,0.0086,42.3007,0.009,90
133513,34.980469,-6.279288,172.180075,-60.389399,1,0.4337,0.4221,0.0715,41.8224,0.023,90
133773,149.414062,3.433834,234.919132,42.245550,1,0.0000,0.0000,0.0000,nan,0.027,53
134380,150.996094,2.985506,236.647967,43.287350,1,0.3943,0.3670,0.0425,41.4645,0.020,90
134824,351.734680,-62.884678,318.284128,-51.651217,1,0.1196,0.1143,0.0103,38.6270,0.019,42
135054,350.230255,-61.943836,320.053946,-52.070537,1,0.3893,0.4001,0.3373,41.6849,0.017,90
135067,148.886719,2.686724,235.347248,41.389003,1,0.5285,0.5059,0.6484,42.2917,0.028,90
135097,151.171875,1.342993,238.602520,42.464379,1,0.2066,0.3032,0.0238,40.9819,0.026,42
135357,51.855469,-28.630989,224.733260,-55.649872,1,0.0000,0.0000,0.0000,nan,0.009,65
135588,151.523438,3.134927,236.900695,43.803170,1,0.0000,0.0000,0.0000,nan,0.019,16
135790,150.117188,2.238686,236.784618,42.139082,1,0.4136,0.3284,0.7703,41.1829,0.016,90
135813,0.589520,-47.161343,325.385896,-67.769893,1,0.2577,0.2445,0.0139,40.4473,0.009,52
136110,53.261719,-27.615883,223.280041,-54.281374,1,0.3785,0.4183,0.7565,41.7995,0.006,90
136352,51.328125,-27.784405,223.130589,-55.999499,1,0.0000,0.0000,0.0000,nan,0.013,65
136407,33.574219,-4.780192,168.064587,-60.175886,1,0.1110,0.0954,0.0100,38.2070,0.019,42
136704,150.820312,3.732834,235.666318,43.572109,1,0.2725,1.2302,1.1813,44.6568,0.016,62
136931,52.558594,-27.279613,222.538937,-54.845107,1,0.3304,0.3783,0.1057,41.5421,0.008,52
136949,351.321442,-64.198746,317.458993,-50.429931,1,0.0000,0.0000,0.0000,nan,0.023,65
137510,346.562500,-63.448284,320.824720,-49.866957,1,0.0000,0.0000,0.0000,nan,0.021,65
137645,51.855469,-26.276812,220.627031,-55.293792,1,0.1590,0.1754,0.0130,39.6390,0.014,42
138010,52.207031,-28.630989,224.800211,-55.343637,1,0.2411,0.2709,0.0169,40.7009,0.009,42
138068,150.292969,2.686724,236.427488,42.541447,1,0.2270,2.7496,0.8355,46.7979,0.016,42
138263,51.328125,-27.784405,223.130589,-55.999499,1,0.4085,0.4242,0.0398,41.8353,0.013,90
138415,349.429535,-62.508568,320.039643,-51.393745,1,0.2761,0.2500,0.0203,40.5026,0.020,42
138553,347.812500,-63.448284,320.128971,-50.202348,1,0.3010,0.2608,0.2763,40.6065,0.021,90
138947,349.160583,-64.760857,318.219706,-49.458924,1,0.5965,0.5575,0.2970,42.5456,0.020,42
139016,151.523438,3.134927,236.900695,43.803170,1,0.0000,0.0000,0.0000,nan,0.019,16
139329,151.699219,3.583322,236.533224,44.205648,1,0.5029,0.5454,0.2843,42.4885,0.016,90
139362,149.238281,3.882372,234.283829,42.351155,1,0.0809,0.0779,0.0097,37.7408,0.033,64
139405,1.666667,-44.399834,327.519190,-70.529554,1,0.3701,0.4458,0.8403,41.9636,0.009,90
139637,359.816315,-44.003082,331.451340,-70.123054,1,0.4142,0.4028,0.3993,41.7023,0.013,90
140096,2.457983,-45.389202,324.632685,-69.945696,1,0.2211,0.2342,0.0540,40.3413,0.011,90
140472,52.558594,-27.279613,222.538937,-54.845107,1,0.0000,0.0000,0.0000,nan,0.008,65
140948,1.666667,-44.399834,327.519190,-70.529554,1,0.2048,0.1926,0.0117,39.8645,0.009,62
141212,53.613281,-28.630989,225.073365,-54.119461,1,0.1256,2.3091,0.5229,46.3382,0.006,90
141302,150.292969,2.686724,236.427488,42.541447,1,0.4316,0.4620,0.3119,42.0560,0.016,42
141334,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,65
141686,53.085938,-28.122234,224.100909,-54.509752,1,0.3431,0.3179,0.2942,41.1008,0.007,67
141937,151.523438,3.134927,236.900695,43.803170,1,0.4487,0.4658,0.0138,42.0770,0.019,90
142099,52.207031,-26.610098,221.298836,-55.042928,1,0.0000,0.0000,0.0000,nan,0.014,65
142254,52.910156,-25.944481,220.366350,-54.301439,1,0.4858,0.5210,0.3055,42.3685,0.010,90
142368,347.013428,-62.508568,321.472056,-50.735330,1,0.2586,0.0878,0.7050,38.0148,0.018,90
142866,150.996094,2.388015,237.313912,42.939977,1,0.6327,0.6476,0.0215,42.9408,0.021,42
142867,349.966217,-62.696659,319.542989,-51.376556,1,0.1548,0.1591,0.0118,39.4056,0.021,42
142885,52.207031,-28.291550,224.208534,-55.300157,1,0.4057,0.3852,0.0218,41.5882,0.007,90
143066,0.189873,-45.586655,328.254458,-68.969298,1,0.1937,0.5270,0.1240,42.3983,0.007,42
143275,2.071130,-45.191612,325.606223,-69.989264,1,0.1114,0.1030,0.0117,38.3830,0.011,42
143651,34.101562,-5.829153,170.247753,-60.638325,1,0.3590,0.3440,0.0396,41.3003,0.019,90
143865,33.925781,-5.979157,170.179895,-60.866303,1,0.0000,0.0000,0.0000,nan,0.022,16
144204,53.085938,-27.111860,222.384291,-54.355086,1,0.4825,0.5378,0.3853,42.4513,0.007,90
144244,33.222656,-4.780192,167.515653,-60.396584,1,0.2120,0.6014,0.1268,42.7453,0.018,90
145107,34.980469,-6.279288,172.180075,-60.389399,1,0.0000,0.0000,0.0000,nan,0.023,16
145160,0.189873,-45.586655,328.254458,-68.969298,1,0.0000,0.0000,0.0000,nan,0.007,65
145257,52.207031,-28.291550,224.208534,-55.300157,1,0.2570,0.2708,0.2312,40.7001,0.007,90
145675,51.855469,-27.953188,223.543603,-55.561470,1,0.8655,0.8841,0.0293,43.7698,0.008,90
145859,34.980469,-6.279288,172.180075,-60.389399,1,0.2213,0.2411,0.0230,40.4126,0.023,90
145926,152.050781,3.284369,237.157374,44.318466,1,0.2061,0.2269,0.0092,40.2640,0.019,42
145990,33.222656,-4.780192,167.515653,-60.396584,1,0.4648,0.4366,0.0319,41.9096,0.018,90
146187,33.398438,-3.732834,166.492280,-59.466614,1,0.0000,0.0000,0.0000,nan,0.022,92
146410,359.805206,-46.768478,327.135979,-67.829903,1,0.2555,0.4043,0.6020,41.7117,0.011,90
146429,150.117188,2.238686,236.784618,42.139082,1,0.1260,0.1220,0.0168,38.7782,0.016,62
147214,51.855469,-27.953188,223.543603,-55.561470,1,0.1677,0.1843,0.0127,39.7584,0.008,90
147571,0.190678,-45.783966,327.956322,-68.803772,1,0.1885,0.2575,0.1606,40.5748,0.005,90
147642,52.910156,-26.276812,220.926149,-54.363918,1,0.2453,0.2263,0.0090,40.2576,0.008,62
147752,151.523438,3.134927,236.900695,43.803170,1,0.2042,0.1823,0.0158,39.7324,0.019,90
147816,34.277344,-5.079716,169.526841,-59.956640,1,0.0000,0.0000,0.0000,nan,0.019,92
148204,151.171875,2.238686,237.619933,42.994783,1,0.0000,0.0000,0.0000,nan,0.024,65
148466,151.171875,2.238686,237.619933,42.994783,1,0.5915,0.5684,0.0143,42.5965,0.024,90
148535,34.277344,-5.679190,170.314930,-60.410322,1,0.1966,0.1835,0.0107,39.7482,0.020,42
148543,349.046051,-61.943836,320.796530,-51.753706,1,0.4717,0.5098,0.0652,42.3120,0.017,90
148976,53.964844,-28.630989,225.142950,-53.813613,1,0.6993,0.6715,0.0353,43.0365,0.009,90
148996,51.855469,-26.276812,220.627031,-55.293792,1,0.0258,0.0954,0.0390,38.2066,0.014,15
149129,0.589520,-47.161343,325.385896,-67.769893,1,0.1440,2.5349,0.7043,46.5843,0.009,90
149130,151.171875,2.537361,237.288526,43.169764,1,0.1925,0.1847,0.0130,39.7633,0.024,90
149478,2.071130,-45.191612,325.606223,-69.989264,1,0.1744,0.5308,0.1019,42.4174,0.011,42
149673,33.398438,-4.331149,167.226341,-59.936551,1,0.3411,0.3499,0.0077,41.3436,0.018,90
150266,349.615387,-63.636005,318.927246,-50.506542,1,0.0000,0.0000,0.0000,nan,0.018,16
150344,349.966217,-62.696659,319.542989,-51.376556,1,0.0000,0.0000,0.0000,nan,0.021,65
150561,53.261719,-27.615883,223.280041,-54.281374,1,0.2265,0.2307,0.0085,40.3046,0.006,42
150765,51.855469,-27.953188,223.543603,-55.561470,1,0.0000,0.0000,0.0000,nan,0.008,65
150818,53.613281,-26.944359,222.237403,-53.863858,1,0.1401,0.1231,0.0150,38.8002,0.009,90
150880,351.382965,-64.011238,317.574052,-50.604657,1,0.1168,0.0932,0.0179,38.1533,0.023,42
151356,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,16
151427,0.574468,-45.981140,327.041068,-68.778764,1,0.0370,0.0313,0.0117,35.6883,0.006,42
151458,53.261719,-27.615883,223.280041,-54.281374,1,0.0000,0.0000,0.0000,nan,0.006,65
151462,33.574219,-4.780192,168.064587,-60.175886,1,0.2901,0.3886,0.4161,41.6102,0.019,42
151498,359.446716,-44.201530,331.730015,-69.805709,1,0.4319,0.4487,0.5711,41.9804,0.010,90
151694,54.667969,-27.615883,223.610785,-53.050840,1,0.0000,0.0000,0.0000,nan,0.009,92
151704,150.292969,2.686724,236.427488,42.541447,1,0.1558,0.1295,0.6463,38.9183,0.016,52
151973,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,92
152079,33.574219,-5.379379,168.838090,-60.637536,1,0.3503,0.3429,0.0254,41.2924,0.017,90
152083,150.468750,3.732834,235.392208,43.283244,1,0.5001,0.5041,0.0165,42.2827,0.020,90
152300,33.574219,-6.579593,170.455585,-61.548219,1,0.7306,0.6981,0.0380,43.1398,0.021,42
152425,150.820312,3.732834,235.666318,43.572109,1,0.0000,0.0000,0.0000,nan,0.016,65
152453,52.910156,-27.953188,223.774083,-54.639214,1,0.2741,0.5064,0.2047,42.2944,0.007,90
152567,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,16
152618,33.574219,-5.079716,168.448505,-60.407218,1,0.0000,0.0000,0.0000,nan,0.016,16
152640,33.574219,-4.780192,168.064587,-60.175886,1,0.0000,0.0000,0.0000,nan,0.019,65
152682,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,16
152756,150.996094,4.181528,235.291975,43.970869,1,0.1326,0.1837,0.4894,39.7513,0.015,62
152787,351.299988,-62.320400,319.038597,-52.026867,1,0.0000,0.0000,0.0000,nan,0.018,65
152812,32.695312,-4.929937,166.868469,-60.841230,1,2.2378,2.2813,0.0673,46.3061,0.018,88
153089,35.332031,-5.979157,172.286722,-59.931743,1,0.0000,0.0000,0.0000,nan,0.022,16
153539,54.667969,-27.615883,223.610785,-53.050840,1,0.3371,0.3157,0.0155,41.0832,0.009,88
153880,51.679688,-27.447618,222.618229,-55.642263,1,0.3501,0.3152,0.0731,41.0793,0.010,90
154053,34.101562,-5.829153,170.247753,-60.638325,1,0.1526,0.1479,0.0167,39.2327,0.019,42
154402,151.171875,2.238686,237.619933,42.994783,1,0.0294,0.0619,0.0172,37.2157,0.024,90
154631,358.648071,-46.375080,329.462659,-67.716008,1,0.9576,0.8148,0.0688,43.5515,0.009,90
154648,359.811707,-45.191612,329.485675,-69.150905,1,1.4174,1.4408,0.0548,45.0811,0.010,88
154762,35.859375,-4.630479,171.270769,-58.580806,1,0.3187,0.2788,1.0862,40.7724,0.022,90
154986,150.996094,2.985506,236.647967,43.287350,1,0.0000,0.0000,0.0000,nan,0.020,16
155110,148.886719,2.686724,235.347248,41.389003,1,0.0000,0.0000,0.0000,nan,0.028,65
155380,150.996094,2.388015,237.313912,42.939977,1,0.2147,0.2288,0.0127,40.2844,0.021,88
155468,150.996094,4.181528,235.291975,43.970869,1,0.0000,0.0000,0.0000,nan,0.015,65
155541,151.347656,4.181528,235.568369,44.259942,1,0.5115,0.4788,0.0312,42.1482,0.016,90
155613,148.710938,2.836105,235.050801,41.328739,1,0.3585,0.3694,0.0090,41.4811,0.031,90
155778,53.085938,-27.111860,222.384291,-54.355086,1,0.2234,0.5912,0.2372,42.6999,0.007,42
156386,151.523438,3.134927,236.900695,43.803170,1,1.4308,1.2425,0.1528,44.6834,0.019,88
156537,352.132874,-63.636005,317.424173,-51.095855,1,2.0260,2.3090,0.0769,46.3380,0.021,88
156739,351.321442,-64.198746,317.458993,-50.429931,1,0.3405,0.4842,0.8467,42.1775,0.023,90
157120,51.328125,-27.447618,222.535046,-55.950727,1,0.2126,0.2539,0.5364,40.5402,0.013,42
157299,34.453125,-5.229529,169.987075,-59.956185,1,0.5504,0.6351,0.0335,42.8890,0.019,42
157477,359.805206,-46.768478,327.135979,-67.829903,1,0.9586,0.7920,0.0736,43.4756,0.011,95
157746,149.414062,1.940072,236.565366,41.393323,1,0.2167,0.5603,0.2926,42.5588,0.018,52
158042,1.666667,-44.399834,327.519190,-70.529554,1,0.1965,0.2093,0.0089,40.0667,0.009,90
158241,347.013428,-62.508568,321.472056,-50.735330,1,0.0424,0.0344,0.0149,35.8982,0.018,62
158507,351.299988,-62.320400,319.038597,-52.026867,1,0.6109,0.7017,0.0936,43.1535,0.018,90
158515,52.910156,-26.276812,220.926149,-54.363918,1,0.3265,0.3196,0.0097,41.1146,0.008,42
158573,33.574219,-4.780192,168.064587,-60.175886,1,0.2736,0.5337,0.1790,42.4315,0.019,90
158697,35.332031,-5.979157,172.286722,-59.931743,1,0.3830,0.4049,0.0394,41.7159,0.022,90
158731,347.846710,-64.760857,318.929827,-49.143596,1,0.0000,0.0000,0.0000,nan,0.019,65
158813,150.820312,3.134927,236.341348,43.230123,1,0.1966,0.1962,0.0134,39.9093,0.016,62
158904,2.097458,-45.783966,324.737840,-69.478613,1,1.8591,1.6874,0.1207,45.5041,0.011,88
159277,359.805206,-46.768478,327.135979,-67.829903,1,0.6452,0.5890,0.0356,42.6903,0.011,90
159316,151.699219,3.583322,236.533224,44.205648,1,0.0000,0.0000,0.0000,nan,0.016,65
159491,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,16
159665,33.574219,-4.780192,168.064587,-60.175886,1,0.3253,0.3336,0.0122,41.2221,0.019,90
159925,150.820312,3.134927,236.341348,43.230123,1,1.6095,1.6763,0.0691,45.4866,0.016,88
160048,51.855469,-28.630989,224.733260,-55.649872,1,0.3638,0.4409,0.3427,41.9350,0.009,90
160426,351.321442,-64.198746,317.458993,-50.429931,1,0.3511,0.3353,0.0637,41.2356,0.023,90
160527,2.097458,-45.783966,324.737840,-69.478613,1,0.3162,0.2266,0.6723,40.2609,0.011,90
160737,52.207031,-28.291550,224.208534,-55.300157,1,0.3593,0.5374,0.2042,42.4496,0.007,88
160921,349.160583,-64.760857,318.219706,-49.458924,1,0.0000,0.0000,0.0000,nan,0.020,65
161135,52.207031,-28.291550,224.208534,-55.300157,1,0.4293,0.4326,0.0209,41.8860,0.007,90
161411,150.468750,3.732834,235.392208,43.283244,1,0.0000,0.0000,0.0000,nan,0.020,65
161432,149.414062,2.238686,236.239766,41.565558,1,0.0000,0.0000,0.0000,nan,0.017,16
161521,346.130127,-63.072620,321.423103,-50.042305,1,0.1147,0.1640,0.9394,39.4785,0.020,90
161591,150.996094,4.181528,235.291975,43.970869,1,0.0761,0.1416,0.0294,39.1295,0.015,62
161877,150.468750,1.641510,237.714575,42.075234,1,0.0000,0.0000,0.0000,nan,0.017,65
161988,51.679688,-27.447618,222.618229,-55.642263,1,0.5641,0.5875,0.0180,42.6834,0.010,90
162093,51.855469,-28.630989,224.733260,-55.649872,1,0.1580,0.1822,0.0192,39.7311,0.009,62
162139,150.820312,3.134927,236.341348,43.230123,1,0.1036,0.0926,0.0114,38.1367,0.016,42
162152,348.529419,-61.755440,321.293980,-51.763351,1,0.0000,0.0000,0.0000,nan,0.016,16
162531,347.861847,-61.943836,321.519104,-51.424048,1,0.5290,0.4535,0.0438,42.0076,0.017,90
162538,150.996094,4.181528,235.291975,43.970869,1,0.2795,0.5154,0.2561,42.3402,0.015,90
162994,53.085938,-28.122234,224.100909,-54.509752,1,0.5827,0.5520,0.0825,42.5199,0.007,90
163208,33.398438,-4.331149,167.226341,-59.936551,1,0.1546,0.1227,0.0022,38.7928,0.018,42
163680,0.574468,-45.981140,327.041068,-68.778764,1,0.3407,0.3516,0.0201,41.3557,0.006,90
163894,52.207031,-28.630989,224.800211,-55.343637,1,0.3027,0.3162,0.0110,41.0874,0.009,88
164582,32.871094,-4.780192,166.959493,-60.615132,1,0.6341,0.6230,0.0125,42.8383,0.017,90
164805,51.679688,-27.447618,222.618229,-55.642263,1,0.0000,0.0000,0.0000,nan,0.010,92
165406,351.321442,-64.198746,317.458993,-50.429931,1,0.0000,0.0000,0.0000,nan,0.023,65
165494,349.615387,-63.636005,318.927246,-50.506542,1,0.2616,0.2665,0.0118,40.6604,0.018,90
165507,151.171875,1.342993,238.602520,42.464379,1,0.2323,0.2251,0.0187,40.2445,0.026,67
165821,149.414062,2.238686,236.239766,41.565558,1,0.2458,1.0072,0.4491,44.1195,0.017,42
165985,51.679688,-27.447618,222.618229,-55.642263,1,0.4551,0.4933,0.2305,42.2259,0.010,88
166103,0.589520,-47.161343,325.385896,-67.769893,1,0.2871,0.2079,0.4627,40.0507,0.009,90
166165,150.117188,2.836105,236.124718,42.483719,1,0.0000,0.0000,0.0000,nan,0.016,16
166186,51.679688,-27.447618,222.618229,-55.642263,1,0.5156,0.5329,0.0107,42.4278,0.010,42
166195,149.238281,3.882372,234.283829,42.351155,1,0.0000,0.0000,0.0000,nan,0.033,16
166330,149.414062,1.940072,236.565366,41.393323,1,0.0000,0.0000,0.0000,nan,0.018,65
166697,53.613281,-28.630989,225.073365,-54.119461,1,1.1664,1.1160,0.0368,44.3951,0.006,95
166727,34.453125,-5.229529,169.987075,-59.956185,1,0.2659,0.2946,0.0220,40.9098,0.019,62
166956,359.811707,-45.191612,329.485675,-69.150905,1,0.0000,0.0000,0.0000,nan,0.010,65
167123,149.414062,2.238686,236.239766,41.565558,1,0.0000,0.0000,0.0000,nan,0.017,65
167220,359.446716,-44.201530,331.730015,-69.805709,1,0.7136,0.7235,0.0416,43.2348,0.010,90
167260,2.071130,-45.191612,325.606223,-69.989264,1,0.3449,0.3554,0.0193,41.3826,0.011,62
167310,349.615387,-63.636005,318.927246,-50.506542,1,0.3079,0.3246,0.0181,41.1538,0.018,42
167417,349.046051,-61.943836,320.796530,-51.753706,1,0.5774,0.5558,0.0410,42.5377,0.017,90
167436,350.230255,-61.943836,320.053946,-52.070537,1,0.1918,0.2263,0.0138,40.2576,0.017,90
167488,348.586945,-64.573555,318.693903,-49.477869,1,0.3928,0.3939,0.0171,41.6449,0.018,90
167910,348.908447,-63.823658,319.169886,-50.176186,1,0.4761,0.4646,0.0434,42.0701,0.018,90
168146,51.855469,-26.276812,220.627031,-55.293792,1,0.0000,0.0000,0.0000,nan,0.014,65
168465,149.414062,1.940072,236.565366,41.393323,1,2.1492,2.4337,0.1988,46.4769,0.018,95
168659,53.613281,-27.953188,223.929533,-54.024772,1,0.2663,0.3135,0.2502,41.0656,0.007,67
168952,358.312500,-44.993881,332.185785,-68.685906,1,0.0000,0.0000,0.0000,nan,0.009,65
168957,53.085938,-27.784405,223.525509,-54.460748,1,0.0000,0.0000,0.0000,nan,0.007,65
168967,347.812500,-63.448284,320.128971,-50.202348,1,0.1845,0.2142,0.1608,40.1234,0.021,90
168989,151.171875,1.342993,238.602520,42.464379,1,0.0000,0.0000,0.0000,nan,0.026,16
169133,347.013428,-62.508568,321.472056,-50.735330,1,0.4656,0.4799,0.2303,42.1546,0.018,90
169203,347.861847,-61.943836,321.519104,-51.424048,1,0.1833,0.1791,0.1849,39.6900,0.017,90
169282,149.414062,3.433834,234.919132,42.245550,1,0.3181,0.3458,0.3165,41.3133,0.027,90
================================================
FILE: examples/docker/modin-ray/Dockerfile
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
# Build image from this dockerfile like this:
# docker build -t modin-ray:latest .
FROM ubuntu:20.04
# Proxy settings
ENV http_proxy=${http_proxy}
ENV https_proxy=${https_proxy}
ENV no_proxy=${no_proxy}
RUN apt-get update --yes \
&& apt-get install wget --yes \
&& rm -rf /var/lib/apt/lists/*
ENV USER modin
ENV UID 1000
ENV HOME /home/$USER
RUN adduser --disabled-password \
--gecos "Non-root user" \
--uid $UID \
--home $HOME \
$USER
# Conda settings
ENV CONDA_DIR=${HOME}/miniconda
ENV CONDA_ENV_NAME=modin-ray
ENV PATH="${CONDA_DIR}/bin:${PATH}"
RUN wget -nv https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /tmp/miniconda3.sh \
&& bash /tmp/miniconda3.sh -b -p "${CONDA_DIR}" -f -u \
&& "${CONDA_DIR}/bin/conda" init bash \
&& rm -f /tmp/miniconda3.sh
RUN conda update -n base -c defaults conda -y \
&& conda create -n ${CONDA_ENV_NAME} --yes -c conda-forge --strict-channel-priority \
modin-ray \
ray-dashboard \
scikit-learn \
scikit-learn-intelex \
xgboost \
&& conda clean --all --yes
# Activate ${CONDA_ENV_NAME} for interactive shells
RUN echo "source ${CONDA_DIR}/bin/activate ${CONDA_ENV_NAME}" >> "${HOME}/.bashrc"
# Activate ${CONDA_ENV_NAME} for non-interactive shells
# The following line comments out line that prevents ~/.bashrc execution in
# non-interactive mode.
RUN sed -e 's,\(^[[:space:]]\+[*]) return;;$\),# \1,' -i "${HOME}/.bashrc"
ENV BASH_ENV="${HOME}/.bashrc"
# Set up benchmark scripts
COPY nyc-taxi.py "${HOME}"
COPY census.py "${HOME}"
COPY plasticc.py "${HOME}"
RUN mkdir /dataset
WORKDIR ${HOME}
# Clean up proxy settings to publish on Docker Hub
ENV http_proxy=
ENV https_proxy=
ENV no_proxy=
# Set entrypoint with arguments expansion
ENTRYPOINT ["/bin/bash", "-c", "exec $0 $*"]
================================================
FILE: examples/docker/modin-ray/build-docker-image.sh
================================================
#!/bin/bash -e
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
cd "`dirname \"$0\"`"
docker build -t modin-ray .
echo -e '\nNYC TAXI BENCHMARK
User is responsible for preparing the dataset.
It Can be generated by following the instructions on the link:
https://github.com/toddwschneider/nyc-taxi-data#instructions
To run the benchmark execute:
\tdocker run --rm -v /path/to/dataset:/dataset modin-ray python nyc-taxi.py
CENSUS BENCHMARK
User is responsible for preparing the dataset.
It can be downloaded from the following link:
https://rapidsai-data.s3.us-east-2.amazonaws.com/datasets/ipums_education2income_1970-2010.csv.gz
To run the benchmark execute:
\tdocker run --rm -v /path/to/dataset:/dataset modin-ray python census.py
PLASTICC BENCHMARK
User is responsible for preparing the datasets.
The datasets must include four files: training set, test set,
training set metadata and test set metadata.
To run the benchmark execute:
\tdocker run --rm -v /path/to/dataset:/dataset modin-ray python plasticc.py \n'
================================================
FILE: examples/docker/modin-ray/census.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import sys
import time
import sklearnex
from sklearn import config_context
import modin.pandas as pd
sklearnex.patch_sklearn()
import numpy as np
import sklearn.linear_model as lm
from sklearn.model_selection import train_test_split
def read(filename):
columns_names = [
"YEAR0",
"DATANUM",
"SERIAL",
"CBSERIAL",
"HHWT",
"CPI99",
"GQ",
"QGQ",
"PERNUM",
"PERWT",
"SEX",
"AGE",
"EDUC",
"EDUCD",
"INCTOT",
"SEX_HEAD",
"SEX_MOM",
"SEX_POP",
"SEX_SP",
"SEX_MOM2",
"SEX_POP2",
"AGE_HEAD",
"AGE_MOM",
"AGE_POP",
"AGE_SP",
"AGE_MOM2",
"AGE_POP2",
"EDUC_HEAD",
"EDUC_MOM",
"EDUC_POP",
"EDUC_SP",
"EDUC_MOM2",
"EDUC_POP2",
"EDUCD_HEAD",
"EDUCD_MOM",
"EDUCD_POP",
"EDUCD_SP",
"EDUCD_MOM2",
"EDUCD_POP2",
"INCTOT_HEAD",
"INCTOT_MOM",
"INCTOT_POP",
"INCTOT_SP",
"INCTOT_MOM2",
"INCTOT_POP2",
]
columns_types = [
"int64",
"int64",
"int64",
"float64",
"int64",
"float64",
"int64",
"float64",
"int64",
"int64",
"int64",
"int64",
"int64",
"int64",
"int64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
"float64",
]
dtypes = {columns_names[i]: columns_types[i] for i in range(len(columns_names))}
df = pd.read_csv(
filename,
names=columns_names,
dtype=dtypes,
skiprows=1,
)
return df
def etl(df):
keep_cols = [
"YEAR0",
"DATANUM",
"SERIAL",
"CBSERIAL",
"HHWT",
"CPI99",
"GQ",
"PERNUM",
"SEX",
"AGE",
"INCTOT",
"EDUC",
"EDUCD",
"EDUC_HEAD",
"EDUC_POP",
"EDUC_MOM",
"EDUCD_MOM2",
"EDUCD_POP2",
"INCTOT_MOM",
"INCTOT_POP",
"INCTOT_MOM2",
"INCTOT_POP2",
"INCTOT_HEAD",
"SEX_HEAD",
]
df = df[keep_cols]
df = df[df["INCTOT"] != 9999999]
df = df[df["EDUC"] != -1]
df = df[df["EDUCD"] != -1]
df["INCTOT"] = df["INCTOT"] * df["CPI99"]
for column in keep_cols:
df[column] = df[column].fillna(-1)
df[column] = df[column].astype("float64")
y = df["EDUC"]
X = df.drop(columns=["EDUC", "CPI99"])
return (df, X, y)
def mse(y_test, y_pred):
return ((y_test - y_pred) ** 2).mean()
def cod(y_test, y_pred):
y_bar = y_test.mean()
total = ((y_test - y_bar) ** 2).sum()
residuals = ((y_test - y_pred) ** 2).sum()
return 1 - (residuals / total)
def ml(X, y, random_state, n_runs, test_size):
clf = lm.Ridge()
X = np.ascontiguousarray(X, dtype=np.float64)
y = np.ascontiguousarray(y, dtype=np.float64)
mse_values, cod_values = [], []
ml_scores = {}
print("ML runs: ", n_runs)
for i in range(n_runs):
(X_train, X_test, y_train, y_test) = train_test_split(
X, y, test_size=test_size, random_state=random_state
)
random_state += 777
with config_context(assume_finite=True):
model = clf.fit(X_train, y_train)
y_pred = model.predict(X_test)
mse_values.append(mse(y_test, y_pred))
cod_values.append(cod(y_test, y_pred))
ml_scores["mse_mean"] = sum(mse_values) / len(mse_values)
ml_scores["cod_mean"] = sum(cod_values) / len(cod_values)
ml_scores["mse_dev"] = pow(
sum([(mse_value - ml_scores["mse_mean"]) ** 2 for mse_value in mse_values])
/ (len(mse_values) - 1),
0.5,
)
ml_scores["cod_dev"] = pow(
sum([(cod_value - ml_scores["cod_mean"]) ** 2 for cod_value in cod_values])
/ (len(cod_values) - 1),
0.5,
)
return ml_scores
def measure(name, func, *args, **kw):
t0 = time.time()
res = func(*args, **kw)
t1 = time.time()
print(f"{name}: {t1 - t0} sec")
return res
def main():
if len(sys.argv) != 2:
print(
f"USAGE: docker run --rm -v /path/to/dataset:/dataset python census.py "
)
return
# ML specific
N_RUNS = 50
TEST_SIZE = 0.1
RANDOM_STATE = 777
df = measure("Reading", read, sys.argv[1])
_, X, y = measure("ETL", etl, df)
measure(
"ML", ml, X, y, random_state=RANDOM_STATE, n_runs=N_RUNS, test_size=TEST_SIZE
)
if __name__ == "__main__":
main()
================================================
FILE: examples/docker/modin-ray/nyc-taxi.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import sys
import time
import modin.pandas as pd
def read(filename):
columns_names = [
"trip_id",
"vendor_id",
"pickup_datetime",
"dropoff_datetime",
"store_and_fwd_flag",
"rate_code_id",
"pickup_longitude",
"pickup_latitude",
"dropoff_longitude",
"dropoff_latitude",
"passenger_count",
"trip_distance",
"fare_amount",
"extra",
"mta_tax",
"tip_amount",
"tolls_amount",
"ehail_fee",
"improvement_surcharge",
"total_amount",
"payment_type",
"trip_type",
"pickup",
"dropoff",
"cab_type",
"precipitation",
"snow_depth",
"snowfall",
"max_temperature",
"min_temperature",
"average_wind_speed",
"pickup_nyct2010_gid",
"pickup_ctlabel",
"pickup_borocode",
"pickup_boroname",
"pickup_ct2010",
"pickup_boroct2010",
"pickup_cdeligibil",
"pickup_ntacode",
"pickup_ntaname",
"pickup_puma",
"dropoff_nyct2010_gid",
"dropoff_ctlabel",
"dropoff_borocode",
"dropoff_boroname",
"dropoff_ct2010",
"dropoff_boroct2010",
"dropoff_cdeligibil",
"dropoff_ntacode",
"dropoff_ntaname",
"dropoff_puma",
]
parse_dates = ["pickup_datetime", "dropoff_datetime"]
return pd.read_csv(
filename, names=columns_names, header=None, parse_dates=parse_dates
)
def q1(df):
return df.groupby("cab_type")["cab_type"].count()
def q2(df):
return df.groupby("passenger_count", as_index=False).mean()[
["passenger_count", "total_amount"]
]
def q3(df):
transformed = pd.DataFrame(
{
"pickup_datetime": df["pickup_datetime"].dt.year,
"passenger_count": df["passenger_count"],
}
)
return transformed.groupby(
["pickup_datetime", "passenger_count"], as_index=False
).size()
def q4(df):
transformed = pd.DataFrame(
{
"passenger_count": df["passenger_count"],
"pickup_datetime": df["pickup_datetime"].dt.year,
"trip_distance": df["trip_distance"].astype("int64"),
}
)
return (
transformed.groupby(
["passenger_count", "pickup_datetime", "trip_distance"], as_index=False
)
.size()
.sort_values(by=["pickup_datetime", "size"], ascending=[True, False])
)
def measure(name, func, *args, **kw):
t0 = time.time()
res = func(*args, **kw)
t1 = time.time()
print(f"{name}: {t1 - t0} sec")
return res
def main():
if len(sys.argv) != 2:
print(
f"USAGE: docker run --rm -v /path/to/dataset:/dataset python nyc-taxi.py "
)
return
df = measure("Reading", read, sys.argv[1])
measure("Q1", q1, df)
measure("Q2", q2, df)
measure("Q3", q3, df)
measure("Q4", q4, df)
if __name__ == "__main__":
main()
================================================
FILE: examples/docker/modin-ray/plasticc.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import sys
import time
from functools import partial
import numpy as np
import sklearnex
import xgboost as xgb
import modin.pandas as pd
sklearnex.patch_sklearn()
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
################ helper functions ###############################
def create_dtypes():
dtypes = dict(
[
("object_id", "int32"),
("mjd", "float32"),
("passband", "int32"),
("flux", "float32"),
("flux_err", "float32"),
("detected", "int32"),
]
)
# load metadata
columns_names = [
"object_id",
"ra",
"decl",
"gal_l",
"gal_b",
"ddf",
"hostgal_specz",
"hostgal_photoz",
"hostgal_photoz_err",
"distmod",
"mwebv",
"target",
]
meta_dtypes = ["int32"] + ["float32"] * 4 + ["int32"] + ["float32"] * 5 + ["int32"]
meta_dtypes = dict(
[(columns_names[i], meta_dtypes[i]) for i in range(len(meta_dtypes))]
)
return dtypes, meta_dtypes
def ravel_column_names(cols):
d0 = cols.get_level_values(0)
d1 = cols.get_level_values(1)
return ["%s_%s" % (i, j) for i, j in zip(d0, d1)]
def measure(name, func, *args, **kw):
t0 = time.time()
res = func(*args, **kw)
t1 = time.time()
print(f"{name}: {t1 - t0} sec")
return res
def all_etl(train, train_meta, test, test_meta):
train_final = etl(train, train_meta)
test_final = etl(test, test_meta)
return (train_final, test_final)
def split_step(train_final, test_final):
X = train_final.drop(["object_id", "target"], axis=1).values
Xt = test_final.drop(["object_id"], axis=1).values
y = train_final["target"]
assert X.shape[1] == Xt.shape[1]
classes = sorted(y.unique())
class_weights = {c: 1 for c in classes}
class_weights.update({c: 2 for c in [64, 15]})
lbl = LabelEncoder()
y = lbl.fit_transform(y)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.1, stratify=y, random_state=126
)
return X_train, y_train, X_test, y_test, Xt, classes, class_weights
def multi_weighted_logloss(y_true, y_preds, classes, class_weights):
"""
refactor from
@author olivier https://www.kaggle.com/ogrellier
multi logloss for PLAsTiCC challenge
"""
y_p = y_preds.reshape(y_true.shape[0], len(classes), order="F")
y_ohe = pd.get_dummies(y_true)
y_p = np.clip(a=y_p, a_min=1e-15, a_max=1 - 1e-15)
y_p_log = np.log(y_p)
y_log_ones = np.sum(y_ohe.values * y_p_log, axis=0)
nb_pos = y_ohe.sum(axis=0).values.astype(float)
class_arr = np.array([class_weights[k] for k in sorted(class_weights.keys())])
y_w = y_log_ones * class_arr / nb_pos
loss = -np.sum(y_w) / np.sum(class_arr)
return loss
def xgb_multi_weighted_logloss(y_predicted, y_true, classes, class_weights):
loss = multi_weighted_logloss(
y_true.get_label(), y_predicted, classes, class_weights
)
return "wloss", loss
################ helper functions ###############################
def read(
training_set_filename,
test_set_filename,
training_set_metadata_filename,
test_set_metadata_filename,
dtypes,
meta_dtypes,
):
train = pd.read_csv(training_set_filename, dtype=dtypes)
test = pd.read_csv(
test_set_filename,
names=list(dtypes.keys()),
dtype=dtypes,
header=0,
)
train_meta = pd.read_csv(training_set_metadata_filename, dtype=meta_dtypes)
target = meta_dtypes.pop("target")
test_meta = pd.read_csv(test_set_metadata_filename, dtype=meta_dtypes)
meta_dtypes["target"] = target
dfs = (train, train_meta, test, test_meta)
return dfs
def etl(df, df_meta):
# workaround for Modin_on_ray. Eventually this should be fixed
df["flux_ratio_sq"] = (df["flux"] / df["flux_err"]) * (
df["flux"] / df["flux_err"]
) # np.power(df["flux"] / df["flux_err"], 2.0)
df["flux_by_flux_ratio_sq"] = df["flux"] * df["flux_ratio_sq"]
aggs = {
"passband": ["mean"],
"flux": ["min", "max", "mean", "skew"],
"flux_err": ["min", "max", "mean"],
"detected": ["mean"],
"mjd": ["max", "min"],
"flux_ratio_sq": ["sum"],
"flux_by_flux_ratio_sq": ["sum"],
}
agg_df = df.groupby("object_id", sort=False).agg(aggs)
agg_df.columns = ravel_column_names(agg_df.columns)
agg_df["flux_diff"] = agg_df["flux_max"] - agg_df["flux_min"]
agg_df["flux_dif2"] = agg_df["flux_diff"] / agg_df["flux_mean"]
agg_df["flux_w_mean"] = (
agg_df["flux_by_flux_ratio_sq_sum"] / agg_df["flux_ratio_sq_sum"]
)
agg_df["flux_dif3"] = agg_df["flux_diff"] / agg_df["flux_w_mean"]
agg_df["mjd_diff"] = agg_df["mjd_max"] - agg_df["mjd_min"]
agg_df = agg_df.drop(["mjd_max", "mjd_min"], axis=1)
agg_df = agg_df.reset_index()
df_meta = df_meta.drop(["ra", "decl", "gal_l", "gal_b"], axis=1)
df_meta = df_meta.merge(agg_df, on="object_id", how="left")
return df_meta
def ml(train_final, test_final):
X_train, y_train, X_test, y_test, Xt, classes, class_weights = split_step(
train_final, test_final
)
cpu_params = {
"objective": "multi:softprob",
"eval_metric": "merror",
"tree_method": "hist",
"nthread": 16,
"num_class": 14,
"max_depth": 7,
"verbosity": 1,
"subsample": 0.7,
"colsample_bytree": 0.7,
}
func_loss = partial(
xgb_multi_weighted_logloss, classes=classes, class_weights=class_weights
)
dtrain = xgb.DMatrix(data=X_train, label=y_train)
dvalid = xgb.DMatrix(data=X_test, label=y_test)
dtest = xgb.DMatrix(data=Xt)
watchlist = [(dvalid, "eval"), (dtrain, "train")]
clf = xgb.train(
cpu_params,
dtrain=dtrain,
num_boost_round=60,
evals=watchlist,
feval=func_loss,
early_stopping_rounds=10,
verbose_eval=None,
)
yp = clf.predict(dvalid)
cpu_loss = multi_weighted_logloss(y_test, yp, classes, class_weights)
ysub = clf.predict(dtest) # noqa: F841 (unused variable)
return cpu_loss
def main():
if len(sys.argv) != 5:
print(
f"USAGE: docker run --rm -v /path/to/dataset:/dataset python plasticc.py "
)
return
dtypes, meta_dtypes = create_dtypes()
train, train_meta, test, test_meta = measure(
"Reading",
read,
sys.argv[1],
sys.argv[2],
sys.argv[3],
sys.argv[4],
dtypes,
meta_dtypes,
)
train_final, test_final = measure(
"ETL", all_etl, train, train_meta, test, test_meta
)
cpu_loss = measure("ML", ml, train_final, test_final)
print("validation cpu_loss:", cpu_loss)
if __name__ == "__main__":
main()
================================================
FILE: examples/jupyter/Modin_Taxi.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "cc4bd9e9",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"# To run this notebook as done in the README GIFs, you must first locally download the 2015 NYC Taxi Trip Data.\n",
"import urllib.request\n",
"url_path = \"https://modin-datasets.intel.com/green-taxi/green_tripdata_2015-01.csv\"\n",
"urllib.request.urlretrieve(url_path, \"taxi.csv\")\n",
"\n",
"from modin.config import Engine\n",
"Engine.put(\"dask\")\n",
"from dask.distributed import Client\n",
"client = Client(n_workers=12)\n",
"\n",
"from modin.config import BenchmarkMode\n",
"BenchmarkMode.put(True)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "97b245e5",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"import modin.pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "b65b121c",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 1.57 s, sys: 683 ms, total: 2.26 s\n",
"Wall time: 14.2 s\n"
]
}
],
"source": [
"%time df = pd.read_csv(\"taxi.csv\", parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c48193b2",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 138 ms, sys: 27.3 ms, total: 166 ms\n",
"Wall time: 404 ms\n"
]
}
],
"source": [
"%time isnull = df.isnull()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "1d32ed7c",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 175 ms, sys: 28.4 ms, total: 203 ms\n",
"Wall time: 663 ms\n"
]
}
],
"source": [
"%time rounded_trip_distance = df[[\"pickup_longitude\"]].applymap(round)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ef271dc",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: examples/jupyter/Pandas_Taxi.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "5d674ce8",
"metadata": {},
"outputs": [],
"source": [
"# To run this notebook as done in the README GIFs, you must first locally download the 2015 NYC Taxi Trip Data.\n",
"import urllib.request\n",
"url_path = \"https://modin-datasets.intel.com/green-taxi/green_tripdata_2015-01.csv\"\n",
"urllib.request.urlretrieve(url_path, \"taxi.csv\")\n",
"\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "27f7321c",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "8de98215",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 30.7 s, sys: 4.25 s, total: 35 s\n",
"Wall time: 35.3 s\n"
]
}
],
"source": [
"%time df = pd.read_csv(\"taxi.csv\", parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "14422c3f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 1.37 s, sys: 300 ms, total: 1.67 s\n",
"Wall time: 1.67 s\n"
]
}
],
"source": [
"%time isnull = df.isnull()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f8f87974",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 3.07 s, sys: 305 ms, total: 3.37 s\n",
"Wall time: 3.37 s\n"
]
}
],
"source": [
"%time rounded_trip_distance = df[[\"pickup_longitude\"]].applymap(round)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c7d62bf",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: examples/jupyter/integrations/NLTK.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating NLTK Modin Interoperability"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## All the examples in this section are taken / adapted from https://www.kirenz.com/post/2021-12-11-text-mining-and-sentiment-analysis-with-nltk-and-pandas-in-python/text-mining-and-sentiment-analysis-with-nltk-and-pandas-in-python/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import nltk\n",
"from nltk.tokenize import RegexpTokenizer\n",
"from nltk.corpus import stopwords"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Import some Tweets from Barack Obama \n",
"modin_df = pd.read_csv(\"https://raw.githubusercontent.com/kirenz/twitter-tweepy/main/tweets-obama.csv\")\n",
"modin_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df['text'] = modin_df['text'].astype(str).str.lower()\n",
"modin_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"regexp = RegexpTokenizer('\\w+')\n",
"\n",
"modin_df['text_token']=modin_df['text'].apply(regexp.tokenize)\n",
"modin_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nltk.download('stopwords')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Make a list of english stopwords\n",
"stopwords = nltk.corpus.stopwords.words(\"english\")\n",
"\n",
"# Extend the list with your own custom stopwords\n",
"my_stopwords = ['https']\n",
"stopwords.extend(my_stopwords)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Remove stopwords\n",
"modin_df['text_token'] = modin_df['text_token'].apply(lambda x: [item for item in x if item not in stopwords])\n",
"modin_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df['text_string'] = modin_df['text_token'].apply(lambda x: ' '.join([item for item in x if len(item)>2]))\n",
"modin_df[['text', 'text_token', 'text_string']].head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nltk.download('punkt')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_words = ' '.join([word for word in modin_df['text_string']])\n",
"tokenized_words = nltk.tokenize.word_tokenize(all_words)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nltk.probability import FreqDist\n",
"\n",
"fdist = FreqDist(tokenized_words)\n",
"fdist"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df['text_string_fdist'] = modin_df['text_token'].apply(lambda x: ' '.join([item for item in x if fdist[item] >= 1 ]))\n",
"modin_df[['text', 'text_token', 'text_string', 'text_string_fdist']].head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#lemmatization\n",
"nltk.download('wordnet')\n",
"nltk.download('omw-1.4')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nltk.stem import WordNetLemmatizer\n",
"\n",
"wordnet_lem = WordNetLemmatizer()\n",
"\n",
"modin_df['text_string_lem'] = modin_df['text_string_fdist'].apply(wordnet_lem.lemmatize)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# check if the columns are equal\n",
"modin_df['is_equal']= (modin_df['text_string_fdist']==modin_df['text_string_lem'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# show level count\n",
"modin_df.is_equal.value_counts()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_words_lem = ' '.join([word for word in modin_df['text_string_lem']])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"from wordcloud import WordCloud\n",
"\n",
"wordcloud = WordCloud(width=600, \n",
" height=400, \n",
" random_state=2, \n",
" max_font_size=100).generate(all_words_lem)\n",
"\n",
"plt.figure(figsize=(10, 7))\n",
"plt.imshow(wordcloud, interpolation='bilinear')\n",
"plt.axis('off');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Replicating NLTK workflow with pandas"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Import some Tweets from Barack Obama as pandas df\n",
"pandas_df = pandas.read_csv(\"https://raw.githubusercontent.com/kirenz/twitter-tweepy/main/tweets-obama.csv\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df['text'] = pandas_df['text'].astype(str).str.lower()\n",
"pandas_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"regexp = RegexpTokenizer('\\w+')\n",
"\n",
"pandas_df['text_token']=pandas_df['text'].apply(regexp.tokenize)\n",
"pandas_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Remove stopwords\n",
"pandas_df['text_token'] = pandas_df['text_token'].apply(lambda x: [item for item in x if item not in stopwords])\n",
"pandas_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df['text_string'] = pandas_df['text_token'].apply(lambda x: ' '.join([item for item in x if len(item)>2]))\n",
"pandas_df[['text', 'text_token', 'text_string']].head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_words = ' '.join([word for word in pandas_df['text_string']])\n",
"tokenized_words = nltk.tokenize.word_tokenize(all_words)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nltk.probability import FreqDist\n",
"\n",
"fdist = FreqDist(tokenized_words)\n",
"fdist"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df['text_string_fdist'] = pandas_df['text_token'].apply(lambda x: ' '.join([item for item in x if fdist[item] >= 1 ]))\n",
"pandas_df[['text', 'text_token', 'text_string', 'text_string_fdist']].head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nltk.stem import WordNetLemmatizer\n",
"\n",
"wordnet_lem = WordNetLemmatizer()\n",
"\n",
"pandas_df['text_string_lem'] = pandas_df['text_string_fdist'].apply(wordnet_lem.lemmatize)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# check if the columns are equal\n",
"pandas_df['is_equal']= (pandas_df['text_string_fdist']==pandas_df['text_string_lem'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# show level count\n",
"pandas_df.is_equal.value_counts()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"all_words_lem = ' '.join([word for word in pandas_df['text_string_lem']])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"from wordcloud import WordCloud\n",
"\n",
"wordcloud = WordCloud(width=600, \n",
" height=400, \n",
" random_state=2, \n",
" max_font_size=100).generate(all_words_lem)\n",
"\n",
"plt.figure(figsize=(10, 7))\n",
"plt.imshow(wordcloud, interpolation='bilinear')\n",
"plt.axis('off');"
]
}
],
"metadata": {
"interpreter": {
"hash": "9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f"
},
"kernelspec": {
"display_name": "Python 3.9.7 ('base')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/jupyter/integrations/altair.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating Altair Modin Interoperability\n",
"### Currently Altair is not interoperable with Modin. Each visualization is created with a Modin and then pandas dataframe for comparison."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import altair as alt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from vega_datasets import data\n",
"pandas_cars = data.cars()\n",
"modin_cars = pd.DataFrame(data.cars())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"alt.Chart(modin_cars).mark_point().encode(\n",
" x='Horsepower',\n",
" y='Miles_per_Gallon',\n",
" color='Origin',\n",
").interactive()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"alt.Chart(pandas_cars).mark_point().encode(\n",
" x='Horsepower',\n",
" y='Miles_per_Gallon',\n",
" color='Origin',\n",
").interactive()"
]
}
],
"metadata": {
"interpreter": {
"hash": "9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f"
},
"kernelspec": {
"display_name": "Python 3.9.7 ('base')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/jupyter/integrations/bokeh.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating Bokeh Modin Interoperability\n",
"### Currently Boken is not interoperable with Modin. Each visualization is created with a Modin and then pandas dataframe for comparison."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"from bokeh.plotting import figure\n",
"from bokeh.models import ColumnDataSource\n",
"from bokeh.io import show"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"UserWarning: `from_dict` is not currently supported by PandasOnRay, defaulting to pandas implementation.\n",
"Please refer to https://modin.readthedocs.io/en/stable/supported_apis/defaulting_to_pandas.html for explanation.\n",
"2023-04-06 12:14:58,510\tINFO worker.py:1544 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32m127.0.0.1:8265 \u001b[39m\u001b[22m\n",
"UserWarning: When using a pre-initialized Ray cluster, please ensure that the runtime env sets environment variable __MODIN_AUTOIMPORT_PANDAS__ to 1\n"
]
},
{
"ename": "ValueError",
"evalue": "expected a dict or pandas.DataFrame, got x_values y_values\n0 1 6\n1 2 7\n2 3 2\n3 4 3\n4 5 6",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/var/folders/qj/jybppsbd2jl75s8y2q8s2xx80000gn/T/ipykernel_5953/1336630338.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;31m# create a ColumnDataSource by passing the dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0msource\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mColumnDataSource\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodin_data\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfigure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/opt/anaconda3/lib/python3.9/site-packages/bokeh/models/sources.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 229\u001b[0m \u001b[0mraw_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_data_from_groupby\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mraw_data\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 231\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"expected a dict or pandas.DataFrame, got {raw_data}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 232\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 233\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mraw_data\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mValueError\u001b[0m: expected a dict or pandas.DataFrame, got x_values y_values\n0 1 6\n1 2 7\n2 3 2\n3 4 3\n4 5 6"
]
}
],
"source": [
"# Create a visualization with Modin df \n",
"modin_data = pd.DataFrame.from_dict({'x_values': [1, 2, 3, 4, 5], 'y_values': [6, 7, 2, 3, 6]})\n",
"\n",
"# create a ColumnDataSource by passing the dict\n",
"source = ColumnDataSource(modin_data)\n",
"\n",
"p = figure()\n",
"p.circle(x='x_values', y='y_values', source=source)\n",
"show(p)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df \n",
"pandas_data = pandas.DataFrame.from_dict({'x_values': [1, 2, 3, 4, 5], 'y_values': [6, 7, 2, 3, 6]})\n",
"\n",
"# create a ColumnDataSource by passing the dict\n",
"source = ColumnDataSource(pandas_data)\n",
"\n",
"p = figure()\n",
"p.circle(x='x_values', y='y_values', source=source)\n",
"show(p)"
]
}
],
"metadata": {
"interpreter": {
"hash": "9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f"
},
"kernelspec": {
"display_name": "Python 3.9.7 ('base')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/jupyter/integrations/huggingface.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating Hugging Face Modin Interoperability\n",
"## All the examples in this section are taken/ adapted from https://www.kaggle.com/code/satyampd/imdb-sentiment-analysis-using-bert-w-huggingface/notebook"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import numpy as np # linear algebra"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import sklearn\n",
"from tqdm import tqdm"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import urllib.request\n",
"url_path = \"https://modin-datasets.intel.com/testing/IMDB_Dataset.csv\"\n",
"urllib.request.urlretrieve(url_path, \"imdb.csv\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"modin_df = pd.read_csv(\"imdb.csv\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"type(modin_df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df.sample()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import BertTokenizer, TFBertForSequenceClassification"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Loading the BERT Classifier and Tokenizer along with Input module\n",
"from transformers import InputExample, InputFeatures\n",
"\n",
"model = TFBertForSequenceClassification.from_pretrained(\"bert-base-uncased\")\n",
"tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# changing positive and negative into numeric values\n",
"\n",
"def cat2num(value):\n",
" if value=='positive': \n",
" return 1\n",
" else: \n",
" return 0\n",
" \n",
"modin_df['sentiment'] = modin_df['sentiment'].apply(cat2num)\n",
"train = modin_df[:45000]\n",
"test = modin_df[45000:]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# But first see BERT tokenizer exmaples and other required stuff!\n",
"\n",
"example='In this Kaggle notebook, I will do sentiment analysis using BERT with Huggingface'\n",
"tokens=tokenizer.tokenize(example)\n",
"token_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
"print(tokens)\n",
"print(token_ids)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"type(train)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def convert_data_to_examples(train, test, review, sentiment): \n",
" train_InputExamples = train.apply(lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this case\n",
" text_a = x[review], \n",
" label = x[sentiment]), axis = 1)\n",
"\n",
" validation_InputExamples = test.apply(lambda x: InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this case\n",
" text_a = x[review], \n",
" label = x[sentiment]), axis = 1,)\n",
" \n",
" return train_InputExamples, validation_InputExamples\n",
"\n",
"train_InputExamples, validation_InputExamples = convert_data_to_examples(train, test, 'review', 'sentiment')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def convert_examples_to_tf_dataset(examples, tokenizer, max_length=128):\n",
" features = [] # -> will hold InputFeatures to be converted later\n",
"\n",
" for e in tqdm(examples):\n",
" input_dict = tokenizer.encode_plus(\n",
" e.text_a,\n",
" add_special_tokens=True, # Add 'CLS' and 'SEP'\n",
" max_length=max_length, # truncates if len(s) > max_length\n",
" return_token_type_ids=True,\n",
" return_attention_mask=True,\n",
" pad_to_max_length=True, # pads to the right by default # CHECK THIS for pad_to_max_length\n",
" truncation=True\n",
" )\n",
"\n",
" input_ids, token_type_ids, attention_mask = (input_dict[\"input_ids\"],input_dict[\"token_type_ids\"], input_dict['attention_mask'])\n",
" features.append(InputFeatures( input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, label=e.label) )\n",
"\n",
" def gen():\n",
" for f in features:\n",
" yield (\n",
" {\n",
" \"input_ids\": f.input_ids,\n",
" \"attention_mask\": f.attention_mask,\n",
" \"token_type_ids\": f.token_type_ids,\n",
" },\n",
" f.label,\n",
" )\n",
"\n",
" return tf.data.Dataset.from_generator(\n",
" gen,\n",
" ({\"input_ids\": tf.int32, \"attention_mask\": tf.int32, \"token_type_ids\": tf.int32}, tf.int64),\n",
" (\n",
" {\n",
" \"input_ids\": tf.TensorShape([None]),\n",
" \"attention_mask\": tf.TensorShape([None]),\n",
" \"token_type_ids\": tf.TensorShape([None]),\n",
" },\n",
" tf.TensorShape([]),\n",
" ),\n",
" )\n",
"\n",
"\n",
"DATA_COLUMN = 'review'\n",
"LABEL_COLUMN = 'sentiment'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_InputExamples"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_data = convert_examples_to_tf_dataset(list(train_InputExamples), tokenizer)\n",
"train_data = train_data.shuffle(100).batch(32).repeat(2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"validation_data = convert_examples_to_tf_dataset(list(validation_InputExamples), tokenizer)\n",
"validation_data = validation_data.batch(32)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0), \n",
" loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), \n",
" metrics=[tf.keras.metrics.SparseCategoricalAccuracy('accuracy')])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model.fit(train_data, epochs=2, validation_data=validation_data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pred_sentences = ['worst movie of my life, will never watch movies from this series', \n",
" 'Wow, blew my mind, what a movie by Marvel, animation and story is amazing']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tf_batch = tokenizer(pred_sentences, max_length=128, padding=True, truncation=True, return_tensors='tf') # we are tokenizing before sending into our trained model\n",
"tf_outputs = model(tf_batch) \n",
"tf_predictions = tf.nn.softmax(tf_outputs[0], axis=-1) # axis=-1, this means that the index that will be returned by argmax will be taken from the *last* axis.\n",
"labels = ['Negative','Positive']\n",
"label = tf.argmax(tf_predictions, axis=1)\n",
"label = label.numpy()\n",
"for i in range(len(pred_sentences)):\n",
" print(pred_sentences[i], \": \", labels[label[i]])"
]
}
],
"metadata": {
"interpreter": {
"hash": "9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f"
},
"kernelspec": {
"display_name": "Python 3.9.7 ('base')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/jupyter/integrations/matplotlib.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating Matplotlib Modin Interoperability"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"# Example modified from https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/xcorr_acorr_demo.html#sphx-glr-gallery-lines-bars-and-markers-xcorr-acorr-demo-py\n",
"\n",
"# Fixing random state for reproducibility\n",
"np.random.seed(19680801)\n",
"\n",
"x = pd.DataFrame(np.random.randn(100, 1),columns=[\"Col_1\"])\n",
"y = pd.DataFrame(np.random.randn(100, 1),columns=[\"Col_1\"])\n",
"\n",
"fig, [ax1, ax2] = plt.subplots(2, 1, sharex=True)\n",
"ax1.xcorr(x[\"Col_1\"], y[\"Col_1\"], usevlines=True, maxlags=50, normed=True, lw=2)\n",
"ax1.grid(True)\n",
"\n",
"ax2.acorr(x[\"Col_1\"], usevlines=True, normed=True, maxlags=50, lw=2)\n",
"ax2.grid(True)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"# Example modified from https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/xcorr_acorr_demo.html#sphx-glr-gallery-lines-bars-and-markers-xcorr-acorr-demo-py\n",
"\n",
"# Fixing random state for reproducibility\n",
"np.random.seed(19680801)\n",
"\n",
"x = pandas.DataFrame(np.random.randn(100, 1),columns=[\"Col_1\"])\n",
"y = pandas.DataFrame(np.random.randn(100, 1),columns=[\"Col_1\"])\n",
"\n",
"fig, [ax1, ax2] = plt.subplots(2, 1, sharex=True)\n",
"ax1.xcorr(x[\"Col_1\"], y[\"Col_1\"], usevlines=True, maxlags=50, normed=True, lw=2)\n",
"ax1.grid(True)\n",
"\n",
"ax2.acorr(x[\"Col_1\"], usevlines=True, normed=True, maxlags=50, lw=2)\n",
"ax2.grid(True)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\n",
"\n",
"names = ['group_a', 'group_b', 'group_c']\n",
"values = [1, 10, 100]\n",
"\n",
"modin_df = pd.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\n",
"\n",
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.bar(modin_df['names'], modin_df['values'])\n",
"plt.subplot(132)\n",
"#plt.scatter(df['names'], df['values'])\n",
"#plt.subplot(133)\n",
"plt.plot(modin_df['names'], modin_df['values'])\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\n",
"\n",
"names = ['group_a', 'group_b', 'group_c']\n",
"values = [1, 10, 100]\n",
"\n",
"pandas_df = pandas.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\n",
"\n",
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.bar(pandas_df['names'], pandas_df['values'])\n",
"plt.subplot(132)\n",
"\n",
"plt.plot(pandas_df['names'], pandas_df['values'])\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\n",
"\n",
"names = ['group_a', 'group_b', 'group_c']\n",
"values = [1, 10, 100]\n",
"\n",
"modin_df = pd.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\n",
"\n",
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.barh(modin_df['names'], modin_df['values'])\n",
"plt.subplot(132)\n",
"#plt.scatter(df['names'], df['values'])\n",
"#plt.subplot(133)\n",
"plt.plot(modin_df['names'], modin_df['values'])\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\n",
"\n",
"names = ['group_a', 'group_b', 'group_c']\n",
"values = [1, 10, 100]\n",
"\n",
"pandas_df = pandas.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\n",
"\n",
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.barh(pandas_df['names'], pandas_df['values'])\n",
"plt.subplot(132)\n",
"plt.plot(pandas_df['names'], pandas_df['values'])\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.hlines(pandas_df['values'], 1, 3)\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\n",
"\n",
"names = ['group_a', 'group_b', 'group_c']\n",
"values = [1, 10, 100]\n",
"\n",
"modin_df = pd.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\n",
"\n",
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.bar(modin_df['names'], modin_df['values'])\n",
"plt.subplot(132)\n",
"#plt.scatter(df['names'], df['values'])\n",
"#plt.subplot(133)\n",
"plt.plot(modin_df['names'], modin_df['values'])\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"# Example modified from https://matplotlib.org/stable/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py\n",
"\n",
"names = ['group_a', 'group_b', 'group_c']\n",
"values = [1, 10, 100]\n",
"\n",
"pandas_df = pandas.DataFrame({'names':['group_a', 'group_b', 'group_c'],'values':[1, 10, 100]})\n",
"\n",
"plt.figure(figsize=(9, 3))\n",
"\n",
"plt.subplot(131)\n",
"plt.bar(pandas_df['names'], pandas_df['values'])\n",
"plt.subplot(132)\n",
"#plt.scatter(df['names'], df['values'])\n",
"#plt.subplot(133)\n",
"plt.plot(pandas_df['names'], pandas_df['values'])\n",
"plt.suptitle('Categorical Plotting')\n",
"plt.show()"
]
}
],
"metadata": {
"interpreter": {
"hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
},
"kernelspec": {
"display_name": "Python 3.9.10 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/jupyter/integrations/plotly.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating Plotly Modin Interoperability"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Currently Plotly is not completely interoperable with Modin. Each visualization is created with a Modin and then pandas dataframe for comparison."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import numpy as np\n",
"import plotly.express as px\n",
"import plotly.io as pio\n",
"pio.renderers.default = \"notebook\"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df = pd.DataFrame(dict(a=[1,3,2,4], b=[3,2,1,0]))\n",
"pandas_df = pandas.DataFrame(dict(a=[1,3,2,4], b=[3,2,1,0]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig2 = px.bar(modin_df)\n",
"fig2.show()\n",
"# py.iplot(fig2 , filename='jupyter-basic_bar')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig2 = px.bar(pandas_df)\n",
"fig2.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig = px.line(modin_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig = px.line(pandas_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig = px.area(modin_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig = px.area(pandas_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig = px.area(modin_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig = px.area(pandas_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig = px.violin(modin_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig = px.violin(pandas_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig = px.box(modin_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig = px.box(pandas_df)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"fig = px.histogram(modin_df, opacity=0.5, orientation='h', nbins=5)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"fig = px.histogram(pandas_df, opacity=0.5, orientation='h', nbins=5)\n",
"fig.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with Modin df\n",
"# Example from https://plotly.com/python/mapbox-county-choropleth/#choropleth-map-using-plotlyexpress-and-carto-base-map-no-token-needed\n",
"from urllib.request import urlopen\n",
"import json\n",
"with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response:\n",
" counties = json.load(response)\n",
"import modin.pandas as pd\n",
"modin_df = pd.read_csv(\"https://raw.githubusercontent.com/plotly/datasets/master/fips-unemp-16.csv\",\n",
" dtype={\"fips\": str})\n",
"fig = px.choropleth(modin_df, geojson=counties, locations='fips', color='unemp',\n",
" color_continuous_scale=\"Viridis\",\n",
" range_color=(0, 12),\n",
" scope=\"usa\",\n",
" labels={'unemp':'unemployment rate'}\n",
" )\n",
"fig.update_layout(margin={\"r\":0,\"t\":0,\"l\":0,\"b\":0})\n",
"fig.show()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create a visualization with pandas df\n",
"# Example from https://plotly.com/python/mapbox-county-choropleth/#choropleth-map-using-plotlyexpress-and-carto-base-map-no-token-needed\n",
"from urllib.request import urlopen\n",
"import json\n",
"with urlopen('https://raw.githubusercontent.com/plotly/datasets/master/geojson-counties-fips.json') as response:\n",
" counties = json.load(response)\n",
"import pandas\n",
"pandas_df = pandas.read_csv(\"https://raw.githubusercontent.com/plotly/datasets/master/fips-unemp-16.csv\",\n",
" dtype={\"fips\": str})\n",
"\n",
"fig = px.choropleth(pandas_df, geojson=counties, locations='fips', color='unemp',\n",
" color_continuous_scale=\"Viridis\",\n",
" range_color=(0, 12),\n",
" scope=\"usa\",\n",
" labels={'unemp':'unemployment rate'}\n",
" )\n",
"fig.update_layout(margin={\"r\":0,\"t\":0,\"l\":0,\"b\":0})\n",
"fig.show()\n"
]
}
],
"metadata": {
"interpreter": {
"hash": "9752fa87da8bf164654ccc33a595e9110c8fc9bb15d763374a7037fd32519b1f"
},
"kernelspec": {
"display_name": "Python 3.9.7 ('base')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/jupyter/integrations/seaborn.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstrating Seaborn Modin Interoperability"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### All the examples in this section are taken / adapted from https://seaborn.pydata.org/tutorial/introduction.html. Each visualization is created with a Modin and then pandas dataframe for comparison."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"import seaborn as sns\n",
"import modin.pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"UserWarning: Distributing object. This may take some time.\n"
]
},
{
"data": {
"text/plain": [
"modin.pandas.dataframe.DataFrame"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Apply the default theme\n",
"sns.set_theme()\n",
"\n",
"# Load an example dataset\n",
"pandas_tips = sns.load_dataset(\"tips\")\n",
"modin_tips = pd.DataFrame(pandas_tips)\n",
"\n",
"type(modin_tips)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
total_bill
\n",
"
tip
\n",
"
sex
\n",
"
smoker
\n",
"
day
\n",
"
time
\n",
"
size
\n",
"
\n",
" \n",
" \n",
"
\n",
"
0
\n",
"
16.99
\n",
"
1.01
\n",
"
Female
\n",
"
No
\n",
"
Sun
\n",
"
Dinner
\n",
"
2
\n",
"
\n",
"
\n",
"
1
\n",
"
10.34
\n",
"
1.66
\n",
"
Male
\n",
"
No
\n",
"
Sun
\n",
"
Dinner
\n",
"
3
\n",
"
\n",
"
\n",
"
2
\n",
"
21.01
\n",
"
3.50
\n",
"
Male
\n",
"
No
\n",
"
Sun
\n",
"
Dinner
\n",
"
3
\n",
"
\n",
"
\n",
"
3
\n",
"
23.68
\n",
"
3.31
\n",
"
Male
\n",
"
No
\n",
"
Sun
\n",
"
Dinner
\n",
"
2
\n",
"
\n",
"
\n",
"
4
\n",
"
24.59
\n",
"
3.61
\n",
"
Female
\n",
"
No
\n",
"
Sun
\n",
"
Dinner
\n",
"
4
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" total_bill tip sex smoker day time size\n",
"0 16.99 1.01 Female No Sun Dinner 2\n",
"1 10.34 1.66 Male No Sun Dinner 3\n",
"2 21.01 3.50 Male No Sun Dinner 3\n",
"3 23.68 3.31 Male No Sun Dinner 2\n",
"4 24.59 3.61 Female No Sun Dinner 4"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"modin_tips.head()"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAwEAAAFcCAYAAACQkLIVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAACZvElEQVR4nOzdZ4BU1dnA8f+902dntvdCWXrvKEXBggXEAhp7iRo1+qoxMUaNpvhGo4mvRGOKJiZGYxcQsKNYqErvvSxsYXubXu59PywsLLsLy/bdeX5fYO7MnHvO7O6Z+9xzznMUXdd1hBBCCCGEEBFD7egKCCGEEEIIIdqXBAFCCCGEEEJEGAkChBBCCCGEiDASBAghhBBCCBFhJAgQQgghhBAiwkgQIIQQQgghRISRIEB0GbfeeitlZWUA/OhHP2LPnj0dWp+HH36YV155pc3Pc+ONN/Lpp5+2+XmEEN1TZ+w7zzrrLC677DIuu+wypk+fzq9+9SuKi4sBKCws5JprrunQOgoRCYwdXQEhmmr58uW1///HP/7RgTURQoiuozP2nbfccgu33XYbALqu89JLL3H77bczb948UlJSePvttzu4hkJ0fxIEiC7hkUceAeDmm2/m5Zdf5vrrr+f555/H4/Hw3HPPkZaWxv79+7HZbNxxxx28/vrr7N+/nwsuuIBHH30UgCVLlvC3v/2NYDCI1WrlF7/4BaNGjapznj179vCzn/2s3vlvuukmZs+e3aS65ubmMnPmTNavX1/v8bx581i8eDGqqpKTk4PVauWZZ56hT58+FBcX8+tf/5p9+/ahqirXXHMNN910EwBffvklr7zyCiUlJUyYMIHf/e53qKoM5AkhTq4r9J2KonDXXXcxf/58li9fTnZ2dm2f+ec//5m8vDyKi4vJy8sjJSWFP/7xjyQnJ3PuuedyxRVXsHLlSgoKCrjsssv4yU9+ctI6//nPf2bDhg0UFRUxYMAAnn322Vb4lIXoonQhuoj+/fvrpaWluq7r+jnnnKNv2rRJX7VqlT5o0CB969atuq7r+m233aZfffXVut/v10tLS/UhQ4bohw8f1vfv369fcsklellZma7rur5r1y590qRJutvtbnZ9fvGLX+j//Oc/6x0/dOiQPnLkyAYfz507Vx8zZoxeUFCg67quP/HEE/pDDz2k67qu33PPPfozzzyj67quV1VV6TNmzNAPHDig33DDDfqPf/xjPRQK6R6PR580aZK+evXqZtdbCBFZukrfee+99+r/+Mc/6vSZL7zwgn7eeefp1dXVuq7r+p133qk///zztW15+umndV3X9cOHD+vDhg3TDx48eNI6v/DCC/qFF16oB4PBZtdfiO5CRgJEl5eZmcngwYMB6NGjB06nE7PZTHx8PFFRUVRWVrJ69WqKioq45ZZbat+nKAoHDx5k4MCBtcdaYyTgVIYMGUJqaioAgwcPZvHixQCsWLGCn//85wA4nU4+/PDD2vdMnz4dg8GAzWajV69elJaWtkpdhBCRq7P1nYqiYLPZ6h0fP348DocDqOkzKysra58777zzAEhJSSEhIYHKyko2btzYaJ0BRo4cidEolz9CyF+B6PLMZnOdxw117pqmMWHCBP70pz/VHisoKCA5ObnO6/r27cuCBQtaVB9FUdB1vfZxMBis87zVam3wtUajEUVRap87dOgQcXFxtc81Vr4QQjRHZ+o7dV1n69at3HDDDfWea6zPBLBYLPWeO1mdFy9ejN1ub3Y9hehOZFKx6DIMBgOhUKhZ750wYQLLly9n7969AHzzzTdceuml+Hy+1qwiANHR0QSDwdoMHB999FGT6zh37lwAqqurufnmmzlw4ECr108IEVk6e98ZDof5y1/+QlxcHOPGjWtxee3Z3wvRlclIgOgyLrroIm688Ub+/Oc/n/Z7+/btyxNPPMFPf/pTdF3HaDTyt7/9jaioqBbVac6cObz44ou1j8855xyee+45fv7zn/OjH/2I+Ph4LrrooiaV9atf/Yrf/OY3zJw5E13XufPOOxk6dGiL6ieEEJ2x73z11VdZuHAhiqIQDocZNmwYL7/8covKbOs6C9HdKLrMKxBCCCGEECKiyHQgIYQQQgghIowEAUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEIIEWEkCBBCCCGEECLCdLkUoaWlLjSt+yY0iouzU17u6ehqtLtIbHckthmk3a0hKcnZrPd19/4TIvP3KxLbDJHZ7khsM7R+u5vbh3Y3MhLQyRiNho6uQoeIxHZHYptB2i3aViR+zpHYZojMdkdimyFy293WJAgQQgghhBAiwkgQIIQQQgghRISRIEAIIYQQQogII0GAEEIIIYQQEUaCACGEEEIIISKMBAFCCCGEEEJEGAkChBBCCCGEiDBtGgS4XC4uueQScnNzAVixYgUzZ87kggsuYM6cOW15aiGEEEII0Y0oCpg1d+1jix55G6e1pjYLAjZu3Mi1117LgQMHAPD5fDz66KP89a9/5eOPP2bLli188803bXV6IYQQQgjRTSgKmMr2UP3Rc1j8pVjceVR+8Hss7vyOrlqX1WZBwLvvvsuvf/1rkpOTAdi0aRM9e/YkKysLo9HIzJkz+fTTT9vq9EIIIYQQoptQtRDBgl0ECvZSsfBZyuY9Q7D4EFp5PorS0bXrmoxtVfCTTz5Z53FRURFJSUm1j5OTkyksLGyr0wshhBBCiG4irBgxDjqfKFc57g2LAYg55ybCmaPQ9Q6uXBfVZkHAiTRNQzkuVNN1vc7jpkpIcLRmtTqlpCRnR1ehQ0RiuyOxzSDt7iiR0H9Cx3/OHSES2wyR2e5IbDPUtNtfsJfKXd/XHvNu+ZrkAWMxJ2R0YM26rnYLAlJTUykuLq59XFxcXDtV6HSUlrrQtO4b8iUlOSkuru7oarS7SGx3JLYZpN2tVVZzdPf+EyLz9ysS2wyR2e5IbDPUtLu8pJzgxq/QPJXETrudcHUZ1avm4d6zkSo9+rRGAyI1kDpRuwUBI0aMYP/+/eTk5JCZmcmHH37I7Nmz2+v0QgghhBCiiwrpRswjZpDUZwyB2N4Y9TBJWQMJxfSU6UDN1G5BgMVi4emnn+bee+/F7/czZcoULrroovY6vRBCCCGE6MKCRgfB2L4AaIqBUFz/Dq5R19bmQcCSJUtq/z9hwgQWLlzY1qcUQgghhBBCnITsGCyEEEIIIUSEkSBACCGEEEKICCNBgBBCCCGEEBFGggAhhBBCCCEijAQBQgghhBBCRBgJAoQQQgghhIgwEgQIIYQQQggRYSQIEEIIIYQQIsJIECCEEEIIIUSEkSBACCGEEEKICCNBgBBCCCGEEBFGggAhhBBCCCEijAQBQgghhBBCRBgJAoQQQgghhIgwEgQIIYQQQggRYSQIEEIIIYQQIsJIECCEEEIIIUSEkSBACCGEEEKICCNBgBBCCCGEEBFGggAhhBBCCNGuFKWjayCMHV0BIYQQQggRGTyBMPsLqjlwuIqEaCv9s2KJd5g7uloRSYIAIYQQQgjR5jz+MC8v2sqmPSW1x6KsRn55y3hSY60dWLPIJNOBhBBCCCFEm9uTX1knAABw+0J8uHw/egfVKZJJECCEEEIIIdqUoijsOlTR4HOrtxfiDYTbt0JCggAhhBBCCNG2dF0nKdbW4HNJsTZMRrkkbW/yiQshhBBCiDY3uFc8FpOh3vEfnNcPkyrpgtqbBAFCCCGEEKLNJcdYeOyH4xjeLxGApFgr9101gsE9Yzu2YhFKsgMJIYQQQog2p+uQEW/nvlnD8QTCmI0qZoOMAHQUCQKEEEIIIUS7URVwWOpPCxLtS6YDCSGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAgjQYAQQgghhBARRoIAIYQQQgghIowEAUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAgjQYAQQgghhBARRoIAIYQQQgghIowEAUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAjTIUHAggULmDFjBjNmzOCZZ57piCoIIYQQQggRsdo9CPB6vTz55JO8/vrrLFiwgDVr1rBixYr2roYQQgghhBARq92DgHA4jKZpeL1eQqEQoVAIi8XS3tUQQgghhBAiYhnb+4QOh4P777+fiy++GJvNxrhx4xg9enR7V0MIIYQQQoiIpei6rrfnCXfs2MHDDz/MK6+8gtPp5MEHH2T48OHcfvvt7VkNIYQQQgghIla7jwQsW7aMCRMmkJCQAMCsWbN48803mxwElJa60LR2jVvaVVKSk+Li6o6uRruLxHZHYptB2t1aZTVHd+8/ITJ/vyKxzRCZ7Y7ENkPrt7u5fWh30+5rAgYOHMiKFSvweDzous6SJUsYNmxYe1dDCCGEEEKIiNXuIwGTJ09m27ZtzJo1C5PJxLBhw7jjjjvauxpCCCGEEEJErHYPAgDuuOMOufAXQgghhBCig8iOwUIIIYQQQkQYCQKEEEIIIYSIMBIECCGEEEII0QkcTaPfHjpkTYAQQgghhGhYSNPJL/OQX+zGYjbQOy2aWLupo6sluhkJAoQQQgghOomQprN4zSHeW7Kn9pjNYuSXN4+T/PadiNvt5pFHHiEnJwdVVRkyZAgzZszgT3/6E2lpaezfvx+bzcYdd9zB66+/zv79+7ngggt49NFHAXjnnXd4/fXXUVWVxMREHn/8cXr37l3nHE899RQ7d+7kr3/9KyaTiWeffZbVq1cTDocZPHgwjz32GA6Hg3PPPZfhw4ezc+dOfvrTnzJt2rQmtUGmAwkhhBBCdBL5ZZ46AQCA1x/i3x9tw+0NdlCtxIkWL16M2+1mwYIFvP/++wDk5uayefNm7rjjDhYsWIDD4eDll1/mpZdeYt68ebz55psUFhaycuVK/vnPf/Laa6+xcOFCLrnkEu655x50vWYzR13XeeKJJ8jPz+cf//gHUVFRvPzyyxgMBubNm8fChQtJTk7m2Wefra1Pv379+OSTT5ocAICMBAghhBBCdBr5Je4Gj+/Nq6Sw3IPTJPdvO4MxY8YwZ84cbrzxRiZOnMjNN99MWVkZmZmZDB48GIAePXrgdDoxm83Ex8cTFRVFZWUlS5cuZfr06cTHxwMwa9YsnnzySXJzcwF49dVXKS0t5YMPPsBsNgPw9ddfU11dzYoVKwAIBoMkJCTU1mfs2LGn3QYJAoQQQgghOgmLydDgcaNBwWKUAKCzyMrKYvHixXz33XesWrWKH/7whzzxxBO1F+1HGY31L7U1Tat3TNd1QqEQAOPGjWP06NE88sgjvPPOO5hMJjRN49FHH2XKlClAzXQkv99f+3673X7abZDfJiGEEEKITqJ3WjQ2S/0Lx4sn9iIt0dEBNRINefPNN3nkkUeYPHkyP//5z5k8eTLbtm1r0nvPOussPv74Y8rKygCYO3cusbGx9OzZE4ChQ4dyww034HQ6efHFFwGYPHkyb7zxBoFAAE3TePzxx3nuueda1AYJAoQQQgghOolYu4lf3jKOvpmxQM0IwMyzejNtbBaqqnRs5UStyy+/nHA4zPTp05k1axbV1dUMGDCgSe+dNGkSt9xyCzfffDMzZszggw8+4KWXXkJVj12WK4rCU089xZtvvsm6deu4++67ycjI4IorrmD69Onous7DDz/cojYo+tFVCF1EaakLTetSVT4tSUlOiourO7oa7S4S2x2JbQZpd2uV1Rzdvf+EyPz9isQ2Q/dvd0jTqXAHMBlVYuwm0Lt/mxvT2u2WLEs1ZE2AEEIIIUQnY1QVEp2WmgfdO3YXHUSmAwkhhBBCCBFhJAgQQgghhBAiwkgQIIQQQgghRISRIEAIIYQQQogII0GAEEIIIYQQEUaCACGEEEIIISKMBAFCCCGEEEJ0Erm5uQwYMIDly5fXOX7uueeSm5vbaueRIEAIIYQQQohOxGQy8fjjj+NyudrsHLJZmBBCCCGEEKfh67WHeO2T7ZSUe0mMs3HTxYOYOiar1cpPTk5m4sSJPPPMM/zv//5vnef+/ve/s3DhQgwGA5MmTeLnP/85BoPhtM8hIwFCCCGEEEI00ddrD/HiexspLveiA8XlXl58byNfrz3Uqud5+OGHWbZsWZ1pQd9++y1Llixh7ty5zJ8/n5ycHN5+++1mlS9BgBBCCCGEEE302ifb8QfDdY75g2Fe+2R7q57H4XDwv//7v3WmBa1atYoZM2Zgs9kwGo3Mnj2blStXNqt8CQKEEEIIIYRoopJy72kdb4nJkyfXTgsC0DSt3mtCoVCzypYgQAghhBBCiCZKjLOd1vGWOjotqKioiDPPPJOPPvoIn89HKBRi7ty5nHnmmc0qV4IAIYQQQgghmuimiwdhMdVdiGsxGbjp4kFtcr6j04KCwSBTp05l6tSpzJ49mxkzZpCens4NN9zQrHIlO5AQQgghhBBNdDQLUFtlB8rMzGTJkiV1jk2ePJmdO3cCcPfdd3P33Xe3+DwSBAjRzRkMCgDhsN7BNRFCCCFAA0IhDbNJhS761TR1TFarpgTtCBIECNFNufwh9uZVsXp7IaqqMG5QCtlp0URZTj+XsBBCCNFSYV1nV24lC5buo6jMy5lDUzl3TCaJTktHVy0iSRAgRDdU5Q3ywvsb2ZdXVXts2cZ8BvWK567Lh+K0yp++EEKI9rXtQAVz3llf+/jTVTl8t+Uwv7p1PDF2UwfWLDLJwmAhuhlFUVi2qaBOAHDU9gNlrNtVjKJ0QMWEEEJErEBY47+f7ah3vNzlZ09eZQfUSEgQIEQ34/aH+HD5/kafX7h0H75Q/TzDQgghRFvxBzWKKxrJr1/Z+vn1xalJECBENxMK6/gC4Uafr/YEZJGwEEKIdmUzG+ibGdvgcxmJjvatjAAkCBCi27FbDPTJiGn0+aF9ErGa5U9fCCFE+zGqCtdd0B+DWnc+6sCecfROc3ZQrTqfJ554gvvuu6/OsWXLlnHeeefhcrla9VxyJSBEN2NUFa48t2+jz18yqTcqsihACHGMMeTGGD42JcMarmrXtUOyTiky9Ep28Nvbz2TW1D5MGp7G3bOGc/cVw4iySLKKo372s5+xZcsWvvzySwA8Hg+/+c1veOqpp3A4WnfERD51IbqhvunR3HfVCP790TaqPUEAYh0Wbps5mN4pMuwqhDjGFHLjW/0+itGCZezlqK4iyhbNIW7GfQRie6O34exBRQFT0VYwmgkl9MdUeQDNVUYobSS63KzoltLjbWRM6IWqdt39a/RwkMPvPg1AyuwHKZz7LACpP3gYxdCyLEdRUVH87ne/49FHH2XChAm88MILnHvuudhsNq699lp8Ph9xcXH89re/JSsri3//+9/Mnz8fVVUZPnw4TzzxRJPPJUGAEN2QQVEY1TeBfndO5HC5B1WBlDg7drPsESCEqMvgL8OzbSmEQ2ieCvwHNqH5PXg2fYll8o2ElLbL4W4NlFK86HkA4s7/IaVLXkMP+km+6Wm8luQ2O6/oWLquE2586Vqnd/jdp/Ed3AbAwRfuQA+Hao+nXft4i8ufOHEikydP5pFHHmHfvn28+eabXH/99fz9738nPT2dpUuX8vjjj/PKK6/w0ksvsXTpUgwGA7/85S8pLCwkJSWlSeeRIECIbkrXIcpioE+qzLUUQjTO7+hBwuUPUjr/Wbw7VwFg6zsW64QfEGzDAADAa0og/tIHKP3gWco+fQlQiJ95Pz5LUpueV4jWoIcC6KEAAIrR3KplP/zww0ydOpW//OUvFBQUcOjQIX784x/XPu9yuTAYDIwaNYorr7yS8847jx/+8IdNDgBAggAhhBAioum6jmK2o6gq+pHswWpULLraPps3KRY7iqKiEwZFQbU5auYJdc2ZIiICpMx+sGYE4EgAAKAYjKTM/nmrncPhcBAdHU1GRgYul4vMzEwWLFgAQDgcpqSkBIC//vWvbNiwgW+//Zbbb7+dZ599lvHjxzfpHLIwWAghhIhgVnceZfOeRg8FMCX3BtWIe+MXBNcvwqD72/bcgVJK5z2DroVxnnk5KCql8/6A1VvYpucVoiUK5z5bOwXoKD0conDuH9vkfNnZ2VRWVrJmzRoA5s6dy4MPPkhZWRnTp0+nf//+3H///UyaNImdO3c2uVwZCRBCCCEimGZ2YErri2q0EDX1ZvTyXMoWPIc5axAh1dymd+QD5lhiz7kJxWRFyxhBQvoAwuWHCVgT2u6kQrQSxWhGMRjrBQStzWw28/zzz/Pkk0/i9/txOBw888wzxMfHc/XVV3PllVdis9no3bs3s2fPbnK5iq635br/1lda6kLTulSVT0tSkpPi4uqOrka7i8R2R2KbQdrdWmU1R3fvPyEyf79ao83mQAW6ohI0RaMoCtZACT5zPLre9hl6FMKAWpsNSCWMxqmTGMjPOnK0drub24ce1ZbZgdqTjAQIIYQQES5gjq39v67reE0J7TYnXz/hgr8pAYAQHUkxmOpkAWqNjEAdQdYECCGEEEIIEWEkCBBCCCGEECLCSBAghBBCCCFEhJEgQAghhBBCiAgjQYAQQgghhBARRoIAIYQQQkQERQGjUUVV2z71qRCdXYcEAUuWLGHWrFlcfPHF/O53v+uIKgghhBAigpS6Any2JpenXl/L3G/3UVDuRZFYQLSizZs3c99993V0NZqs3fcJOHToEL/+9a957733SEhI4Oabb+abb75hypQp7V0VIYQQQkSAMleAp15dTbnLD8COnHI+W5XD47eOJyPe3sG1E93FsGHDeOGFFzq6Gk3W7kHA4sWLmT59OqmpqQDMmTMHi8XS3tUQQgghRCdX6Q1iNKg4LEZ0vXm7lykKbNpbUhsAHBUIaXz+/UF+ePHAdtsYTXQfbrebRx55hJycHFRVZciQIcyYMYMnn3ySDz/8kNtuu42SkhIAPB4Phw4d4tNPPyU9PZ1nn32W1atXEw6HGTx4MI899hgOh6Pd29Du04FycnIIh8PcddddXHbZZbz55pvExMS0dzWEEEII0UlVeIK88vF2fvrCUh76y3K+3VRAMKw1qyxVVdl2oLzB57buKyMQal65IrItXrwYt9vNggULeP/99wHIzc2tff6VV15hwYIFvPfee6SkpPDTn/6UXr168fLLL2MwGJg3bx4LFy4kOTmZZ599tkPa0O4jAeFwmDVr1vD6669jt9v58Y9/zPz585k1a1aT3p+Q0P6RUntLSnJ2dBU6RCS2OxLbDNLujhIJ/Sd0/OfcEbpTmwOhMP/4aC0rNxcA4PWH+PdH20hOsDN5REad1za13QN6xrFme2H94z3iSE50YDIaWl7xdtKdftano7O1e8yYMcyZM4cbb7yRiRMncvPNN1NWVlbnNZqm8eCDD5Kdnc0dd9wBwNdff011dTUrVqwAIBgMkpCQ0O71hw4IAhITE5kwYQLx8fEAnH/++WzatKnJQUBpqQtN677jdklJToqLqzu6Gu2uPdtdXOVnT14FBoNK/8xYYu2mdjnvieRnHVlas93N/TLs7v0nRObvV3drc7knWBsAHO+bdbkMzoolfGRE4HTaPaJPAvOtRjy+UO0xVVW48MweVJR7Wqfi7aC7/aybqrXb3RoBRVZWFosXL+a7775j1apV/PCHP+SJJ56o85onn3wSr9fLnDlzao9pmsajjz5auxbW7Xbj99edqtZe2j0IOOecc/jFL35BVVUVUVFRLF26lPPOO6+9qyEiVEm1n9+88h1ef80XQXKcncduGYvD0u5/CkIIIRpgVBUsJgP+YLjO8VinpdnrAhKdFn71w/F8uzGfDbuK6ZMRw/nje5CVaJf1AKJZ3nzzTdauXcuzzz7LWWedRWlpKdu2bat9/uWXX2b9+vW8/vrrGAzHRpomT57MG2+8wYQJEzAajTz++OPY7fYOyZbZ7lc+I0aM4Pbbb+e6664jGAwyadIkZs+e3d7VEBFqf0FVbQAAUFTuIbfIzcAsWZcihBCdQbTNyJXn9OWNz3fWHjOoChOHprVoJCs5xspVU7KZdXY2BkWpCSgkABDNdPnll/P9998zffp0bDYbaWlpDBgwgE8//ZTCwkKee+45evfuzQ033ICm1Yxe3Xfffdx9990888wzXHHFFYTDYQYNGsTDDz/cIW3okNufV155JVdeeWVHnFpEOJOx/lp4k0n2zBNCiM5C1+Gs4Wkkxtr4dkMeCTFWzhqZQWYr3LXX9ZqMKM0dURDiKLvdzp/+9Kd6x6dNmwbAjh07Gn3vr3/967aq1mmRORAiovRJj6FPRgx78yoBmDA0lczEqA6ulRBCiOOZjSojsuMZ3S8RXddrRgDkul2IViVBgIgoTquRn10zkrwSNwaDSnqCHbNBRgKEEAJAOTpNhpr8+h19wzzczLSgQohTkyBARByryUCftOiOroYQQnQq5mAleuEu9IzhGDzF6D43gYQBHV2tRgVDYUqq/fj8IRKirdjMXSfNpxCdgQQBQgghhECpzKP0478QNXQq3j1rMMam4LzkAQJq59tfwuUP8dbcjXyx+hC6DrEOC3ddMYz+mdEybUiIJpJ5EEIIIYQglDiA6AmzcG/5Gj3oJ/biuztlAKAo8NW6PBZ/f6h2ulKFy88f3lhLYbmvYysnRBfS5CCgsrISl8vVlnURQgghRAcxugpwrf8c1WJHDwfxbluKQet8F9XVvhAfLd9f77im6ewrqOyAGgnRNZ0yCNi3bx+zZ89mwoQJnHHGGdxwww3k5+e3R92EEEII0U4UdExJWSRc9ztip90GWrhTTq3RNJ1gIwuGAyFZSCy6B5fLxSWXXEJubm6bneOUQcAjjzzCVVddxcaNG1m/fj0XXnghv/zlL9usQkIIIYRofz5HFo6L7sdnikfvPRHDiBmEDdaOrlY9TpuJCUPTGnyuV4qznWsjROvbuHEj1157LQcOHGjT85wyCPB6vVxzzTWYTCbMZjM33ngjJSUlbVopIYQQQrS/gFJz0a9hIKx2vgAAQAGumNKHlHh7nePXXtBf9n0R7SoUClFSUkIoFGrVct99911+/etfk5yc3KrlnuiU2YGys7NZt24do0ePBmDXrl1kZma2aaWEEEIIIRqT4DDz9D2T2bG/FI8/RFqCnaQYKwZF6eiqiQixceNG7r//fgKBAGazmeeff54RI0a0StlPPvlkq5RzKqcMAvLz87nxxhsZMGAARqORbdu2kZSUxMyZMwFYtGhRm1dSCCGEEOJ4ibE2BmTGdHQ1RAQKhULcf//9tQlzAoEA999/P19++SUGQ9fZr+KUQcCDDz7YHvUQQgghhBCi06uoqCAQCNQ5FggEKC8vJzExsYNqdfoaDQL27t1Lnz59iIpqeH7dkCFD2qxSQgghhBBCdEaxsbGYzeY6gYDZbCYuLq4Da3X6Gg0C/vCHP/DSSy9x1VVXkZaWhq4fyxPm9XpZuXJlu1RQCCGEEKIzUBTQO2HaVNG+jEYjzz//fL01AV1pKhCcJAh45plnqKiooE+fPrz++uvouo6iKASDQW644Yb2rKMQopWoqoKiKIQbybEthBCivgpPkN25FRSVe8lMdtAnPRqH5ZQzqkU3NmLECL788kvKy8uJi4trkwBgyZIlrV7m8Rr9Df7Zz37G8uXLURSFCRMm1B43GAxceOGFbVopIUTr0tE5WOxh1dbDFJV5OGNIKoN7xuGwypeYEKJ1WL2HCVpiCatWDJoPk78Cny211cpXFHD7wxRVeDEaVGxRllYr+2RKqv089Z/VVLiOTf3okxHDvVcNJ9pqapc6iM7JYDB0qTUAJ2r0CuCVV14BajYL+/3vf99uFRJCtL5duVX84Y21tcPY63cVM7h3PPdcMQybuWsNX4ruRVF0LMFKfMZYjJoXJRwgaJKML12NpTqHkrlP4xg5DevIi/Bt+oTKjV+SMOsX+J09W+EMOlsPVvKPBVuoPHIx3ivNyR2XDSM1tg33M1Dg63V5dQIAgL15lWw/UM4ZA9s2j7sQbemUm4VJACBE1+YPabz68bZ681i37S/jQJGrYyolxBHm0l2Uvv0rrK5DBNYuwPPNq5jD7o6uljgNigK6z4Ue9FP9/ULK3/kN1d8vQg/60X0uWiN1/8ESD//35rraAADgQEE1T/1nNVXeYMtP0IhASGPV1sMNPrd+VzEGwykvo4TotOS3V4huzuULUVjmbfC50sqGjwvRbqISMUTFUvzm47jXf4p94ERCBltH10qcBl2HYPJg4mf+BIBQZREA8TPvJ5g8uMULaVVVYcXmggbLcXmD7C+obtkJTsJkUOvtTHxUepIDTZNVwqLrkiBAiCYqdwdYv7eUFdsK2ZlXiTcY7ugqNYndYiDW0fDc2Zh2mlN7SgqUuQLklrip9oVQTnHr8PinFYVWudMoOkbYZMec2rfmgcGIIS4NTb6amkeBck/N31GFJ9iufxdq2E/w8O46x4IFe1DD/haXrSgK+/KqGn2+3NXyczR6buCys7LrHTeoCmMGJAMSBJwOX0hjZ14VK7YVsn5vKWXuwKnfJNqMrAoU4lQU2H6okj+9vZ5g6FhWnYwkBw9cPZJ4h7ndq2RQwoR1Q73/N8RuNnD9RQP4y/ub6hxPT4yid5qzTevZFC5/iI9X5fD5dwfRNB2bxcjV5/fjzMEpmBsYajcHK9EObUTpOYaQMQpT6U5AJZjQT1L3dUFqyV7cm5cQe/6tuDd+QcUnfyNm9qP4lYb3qBEN8wTCLFmXx8Kl+wiFNcxGlSum9OHskenYTHX7B0WpuRhToMG/sdNhUMLoihElZy3V3y8C1UD0GZdRteoDqr9fSFxcCmrvyS26Y67rOkOyE9iTW9Hg8zbzsUsZTdc5XOHjcKmHWIeZzKSoFrexT7qTB68bzZuf7yS/xE3/HrGcMyaLtxbvYNygVEb1SyLaJpdTp1LmDvCndzaQe9w0VJNR5SdXj2JQjxiJpzqA/NYKcQqFFT6ee3Md4RO+xPKKXfxj0VYevGYUhna842YKuQhu+gTLoLMA8G9fimn4xQSNjgZfr+swIjuBn18/mg+XH6CsysdZI9KZMDStw1Pc6cDcb/bxzbrc2mNef4hXP9qO1Wxk/ICkOq83aT7cy/6Lb/dq7MP2Y+s7ltKFc1BQSLz2N/jsGe3cAtFSWmJfkq77LQFHFjEZQyAckADgNGmazpJ1ucz7em/tsUBI450vd6OoKheNy6y9CA/rOut3l/LmZztRVYVbpg9icM841Gb0YRZfMf6tS7CMnI4am4IlcxCOsTMIpw4lIaUPrnUfo6YNJtTCKTOapjN2YDIfr9hf50YMQEq8HU3XUFUFTddYsbWIfy3aWvv8xOFp3DCtP1ZT8xMgGBSFwT1i+fWt4zlY7OLDZQd4+YPN6Dps21/Oyh4F3H/ViHrBljgmrOu8smhrnQAAIBjS+L+31vHknRNIiWnDBd6iQRIECHEKO3LK6wUAR+3MKSe/1ENWYsNzRtuCGvLg2b0az47lNQcMZuIGT4VGggAAo6owKCuWgdeOJBzWMRnUOhsAdpSiCl+dAOB4b3+xi2HZ8XW+WEMGK1EjL8S/fyOezV/h2fwVAPZRFxKydK2dGkWNkGol5KjJHuO3JHRwbbqmvGIXC5fub/C5+d/s4czBKbV3qg8Vu/nr3GOjgs+9vZ7/vWMCGfHNWIfhq6R6w2J8B7cSLM0jbur16Am9CGugpQzGMa0XAUPrBHQZCTZuu3QIi787yN68SlQFRg9MoVdaNAo1gUJJdYBXP9pW530rNhUweXg6AzNbnnEqENR44Z0NVHvqLkTedbCCg0UuBmRIVqvG5Jd52X6gvMHnNE1ne04ZKcPT27lWndeLL77IJ598AsCUKVN46KGH2uQ8MvFSiJNQVYX8kpNnKqn2tO+cRr81mfiL7iTsqiDsqiD+orvwW5NO/UZA0WsCgs4QAABUneSzq6j24wvUXXeh6xBK6EvM2dfUHlNtTuxjZxIytF8gJkRnUuUOEGpkA0B/IIzbd+yitbC8fjKA4gpPs84bjO9LzMQrCRYfBEXB1GsUQUPNzQhdV1otAKgpENLi7cQ4zFx6VjaXTM6mqNzDt+tyGdgzHoDyan+D045KK32tUgW3L1gvADiqvKrt1iV0B65TfE/mF7tRmzMc1Q2tWLGCZcuWMX/+fD744AO2bt3K4sWL2+RcEgQIcRKappOZ1PgddoDoqPZdE2DxFVH20YsYnIkYnImUffwiFl9Ru9ahtcQ4zI0uXoyPttaZ6ws1c5mNJTup+ObN2mOatxrPqnmYJK2kiFCxDjNmY8Nf5zaLkajj5qunnpDpRlEgJa55AbSpdDeVy9/DnNYHgOpv38AUaru0w1mJUcya0pdwWGP3oXLOHZPFL24cQ6y9ZsOueKcVQwMXkomxrZNtymEzNdrfx0fLVJaTcdpP/j2Zkdz1Mi2Vl5fzt7/9jXvuuYe///3vlJc3PNJxupKSknj44Ycxm82YTCb69OlDfn5+q5R9IpkOJMQpDOgRi9GgEArX76CGZMeT1pxh9BbQjFFEDZ2KuX/NTt6B3SvRTF1zDnVStJXzxmbxxepD9Z67dlp/rKa6FzbGsBfv5q8gHCJq1IXY+oyh9INn8exYjn3E+QTtXfNzEKIl0pMcXD6lD+9+ubvec1ee25cYm7l29C8z0c79PxjJm4t3YlJVrr9oIKlxzbyAtcfhHHcJ5mEXQHkumquMsKFtL4bT421cNbUPqqoQFxdFcfGx9KAJTjO3XzqElxdsqU0SMHV0Jj2TW6dfiLIYuHn6IP783sY6x4dkx5OVfPKbRZEuPd7GsD4JbN5bWu85o0FhUM+uNZ2zvLycq6++murqaoLBIOvXr2f+/Pm8/fbbxMW1rC39+vWr/f+BAwf45JNPeOutt1pa5QYpemeZF9BEpaWuLhctno6kJGedTi1SdPZ2786v4rm31+M/bnpKr7Ro7r1yOHHNHAloSZsNikZYV+v9vys4sd1uf5gv1h7io+UHCIU1HDYT110wgDEDkjA1cFfPFKpCz98KGSMIG+2YyvagqCqB2OxOnR2oNX/Hk5Kal9Wpu/ef0Pn7kraQlOTkYH4FSzcdZv43e/AHwtgsRq48ty8Th6RiaWCUIKjpKNRMD2yJo/2PogC6ht6OEwwa+lnr1CRzKCr3EOMwkx5vx9SKG3ppus6+w9UsWZtLeZWfs0amMyw7Aae1fe6pduXf73J3gBfnbmJ//rF0rxazgQeuGUX/9OiTvre1293cPvSov/3tb7z22msEg8emh5lMJm666SZ+/OMft7R6AOzevZs777yTe++9lyuuuKJVyjyRjAQI0QT9M6J5+scTOXi4Gk8gTEK0laykqAa/XNvD8Rf9XSkAaEiUxcBlk3oxZWQGXn+IaLsZh9XQ6AV90BiN0nNizZ1NvWZesq4j6eVERLOZDFw0LpMzB6fg9gVrpq7YTI2u/2kowG6Oo/1PzWk6vi9SgNRYK6mxbTMioSoKfdOiGXDZUADCjazFaAtGjl1wKgqoWpCwYmq387dUXJSZh64bTW6xm5IqHzazgZ4pTmKjTJ36Bk5DtmzZUicAAAgGg2zZsqVVyl+7di333Xcfjz76KDNmzGiVMhsiQYAQTaDrEGMzMax3fEdXpXvSIdZuqp3be6ovhOMvbLral4cQbUXTdKJtxtpMQG090K/pUFzpI7/UjcVkoEeKo8PTDreX9rz4BzAFK/Gvno9//HQUUwqm0l0EDm3BOOwiQmrX2WHbYlTpk+akz3F71HTFPnzYsGGsX7+eQODYgmez2cywYcNaXHZBQQH33HMPc+bMYcKECS0u72Qi469VCCGEEK1G03WWby3k1Y+21V7ExTksPHTjmIjL964oCsGwhtGgtMmIpKrohHYtxb3la7z71hN7zo2UffoSejhIQmo2SuqILnkh3ZVdffXVzJ8/n6qqKoLBIGazGafTydVXX93isl955RX8fj9PP/107bFrrrmGa6+9tsVln0iCACGEEEKclsPlPv79Yd2c/OUuP699up2f/WAkamNpv7qZSm+Q5ZsKWLoxn8xkBzMn96ZHYusmKNB0BXP/yVgObcd/cCtlH70IQNSI89ES+0oA0AHi4uJ4++23efvtt9myZQvDhg3j6quvbvGiYIDHHnuMxx57rBVqeWoSBAghhBDitBwqajgV6Pb95ZS5AiQ6Le1co/YX1nVe/3QH63YWA1BY5mHj7mKe+NGZpLZSWtKjguZYHGOm4z94bDdk+7Bz8LXmXgzitMTFxbXaIuCO0vGreIQQQgjRpZiMDd/pV1UFQytm4+nMiip8tQHAUaGwzq5DFa16HkWp2ZOhbOGfah4batZOlc77AxZP2+SPF5EhMv5ShRBCCNFqeqZGN7hB2TljMomzd52MNS3R2MLrcGun4dU1dG8VejiEc9Q0km7+A5YeQ9ADXgjU3wFaiKaS6UBCCCGEOC2JTjMP3zSWv83bTHGFF0WBScPTmTmxV0dXrVYwrJFX6sHtDZIUZyMp2kprrlRIjrUxsGccO3KO7RSrKtA/K7YVzwI6KqHMUSRd+2tsKVmUeww4zvsRqr8Kv7OHpEcWzSZBgBBCCCFOi65Dr2QHT9x+BsWVPiwmA4nRlla9yG6JSm+QfyzayrZ9ZUDNxfm1Fwxgyoj0Fm+QdpRRVfjRpUP57Psclm7IJz0ximum9Scj3t4q5R9P11V8zl44o5zgqSZgigVTrAQAokUkCBBCCCFEs1iMKpkJrX/R2xKKAl+vy6sNAKBmT4M3PttJr7Ro+qS2bLfY48VFmbj23H5cNjkbs1HF0FmiICGaQNYECCGEEKLb8ATCfPpdToPP7cwpp7Wzl+q6js0kAYBoPc8//zzTp09nxowZ/Pvf/26z88hIgBBCCCFaRFFAVVU0TevwvPUKCoZGpvyorTQVSAgAn89HSUkJiYmJWK2ts0ne999/z6pVq1i4cCGhUIjp06czZcoUsrOzW6X840kQIIQQQohmK6n2s2F3CXvzKhjQI57hfROIjzJ3WH1sZgMzJ/XmnS9313tuUK/4Dg9SRNcXCoV44YUXmDt3bm3wO3v2bO677z6MxpZdWo8fP57XXnsNo9FIYWEh4XAYu71tptxJECCEEEJEAFVV0HW9VS+Ci6v8PPGv73D7QgB8t7WQOIeFR28ZR4KjYwIBXdeZNCyNvBI3yzbW5NG3mAzcMmMwWYmda/2C6JpeeOEF5s2bh9/vrz02b948FEXhgQceaHH5JpOJF154gX/9619cdNFFpKSktLjMhsiaACGEEKKNmANlWMp2oipgqdyPxXu43esQ0nS251byn093sGR9PpWeYKuUaw6UUVFUQFjTeeTSNPql1Vxgl7v8rN1Z1Opz70+Hw2rklosG8NSdE/jlzWN5+scTOXNQMmpHVkp0Cz6fj7lz5+Lz+eodf//99+sdb6777ruPlStXUlBQwLvvvtsqZZ5IggAh2pE3qLE7v4otOeXsPlgu2d2E6MZUFcL711Iy9xnY9iml857Gs/ZDTErrXIQ31a7cSv7437V8tS6P1z/dwT8XbSPUwg2tzJoH11f/Jm3XPH4zI5q075/nx+M1HLaajcI27Czu8J2DVUUhNc5Gn7RoYuymRjf3EuJ0lJSUoKoN/26rqkpJSUmLyt+7dy/bt28HwGazccEFF7Bz584WldkYmQ4kRDtQFNhX6OLP722kotpfe+ycMZlcflY2Dov8KQrR3WgamPtNxpKzmcqlb2OMTsI+4Sr8evvtqGswqHyzIa/Osa37Symt9pMS0/yFjEGDHee4SymZ9wy23E1oSX3YWObA5a25AOrbIxattXfOFaITSExMRNO0Bp/TdZ3ExMQWlZ+bm8sLL7zAW2+9BcCXX37J7NmzW1RmY2QkQHQIgx7E5slDQcccrMBfeKCjq9SmCit9PPPamtoAAGo221myJpdPvztIp9lhRwjRahQFlOoCAnk7QTUSqi5FK9qP0o5jgJqmk5XsqHPMYjZgM7fsxoOuA0YTypHpNbrRgjtQc2FkMqpMGJIqQYDolqxWK7Nnz66XDaix46drypQpTJ06lcsvv5zZs2czatQoZsyY0aIyGyO3H0WHMJbtpWjeH4i76C5c276lsrqM2FmP4lcdp35zF6MosHlvKYFQw3cOPl2VwzmjMklwdlw2DSFEW9DR3BWYU/sSM+02XKsXEa4sQk0NEVbaZzRA13UmDk1j9fZCDhW6MBlV7rp8WIunx5h1L1VL38KUmo111AyqP/kzZ/WfTPnYLKaOziA93iZZeES3dd9996EoCu+//36d7ED33ntvq5R/7733tlpZJ6PoXWySXGmpq1vfXUhKclJcXN3R1WhzprAb38q38WxbCopK6nW/wuPo1S2/NAwGlefnbmTdjuJGX/PLm8fSJy26HWvVcSLld/xErdnupKTm7Xja3ftP6Hy/Xwo6prCbgMGBMeRGN5gIK60b8Delzf6QRkmljyibibgoU6v0tWZfCagqQUs8Vk8BbnMcqtFKew10dLafdXuIxDZD67e7uX3oidpin4D2JCMBokMoWpCw68iW7rqO5qmCKJ3uOC9G13XS4qOAxoMAm6X95ggLIdqPjkLAUDPCGTJGneR1OsVVfkorfaTE2Ulwmpt9oe4LahwqcWEyqKQn2DEbVCxGlYyEmuw9rXWzJWBNPFp5vLa0mvnF3TvGFKIOq9VKZmZmR1ej2SQIEB1CqS4iULCHxKsexbNjBZXfLcI5oy9+5WgOZx1FUTrlyIAOhDUds1Ft0l1VTdMZPSCZj1YcaPD5Qb3jSI3rencQhBCtQ1Fgy/4K5ryzHl2vmVP/6M3j6JnUeNBwIkv5bjBH4TLEoh3cxLaDFhasKeesEelcf0F/zB2cqUcI0fl0aK/wzDPP8PDDD3dkFcRpMFGT1k5RwKgHTvraoKbjD/ix+Q6jKDoKOjbvYVTCAATi+5F0/ZME4vtjHXcVCRfeRnDHt5hCLhRFx3R4M6aSHbWLzjqDkKaz41Alz769gV/+YxUfLDtAmevkn8NRPZKjuOGiAfWOx0db+OH0wZK7WohmMOjHUm0e7Z86+zmNR96j6iFUvWaDLW9Q45+Lttbe9AiGNN7/aneTb6oHq0qp/Pq/lM1/Bn3dXMJf/52ze2iYjSpLN+aTX+I5ZRlW72GMmhcAi68IU7DytNrlC4bZmVfJ9zuLySl2EW7kBomR9v+ZdUaBsMbew9V8v7OYfYerCYQbXjMmRFvqsJGAlStXMn/+fKZOndpRVRCnwRwow/vdXKLGz0L3VuDbvwHTiOmEVFu91+aXeZj/1S6u7V1EYO3bxF76U1Rdo2jRn4i74EfoPc9ER8FrTgQdFFWlbPEb+HI2Yy/Lw9prJGUf/wVUlaRrn8BnT++AFte39UA5z7+7ofbxB9/uZcXmAn5581ic1pP/KRkUhSnD0+mfGcfOQ+VUuQP0y4qjZ7IDp00G5IQ4XUbNT3j7YszJ2Shx6XhWvod93OVA68z1bfSc2xZjTslGiU3Hs6rmnH5rUpPeb3Hn4t28BNv42YQObqy5ydFjHGFNwesP1XltlSuApoOhCfcHTNEJxM24l6LXHsa/5Uu04Zfw8nfh2mQEgVD4pO+3unMpef8p7EPOJmrYuZQt+D+MCRnYp/6QoPHUn6c3GOZfH21n7Y6i2mPXTuvP+WMy60zwNIVcBNYtwDpoMhgteNZ9hG38bAKm2FM3shsJhjXe/3ovX6w+VHvsojN7csVZ2Zia8gMXopV0yNVHRUUFc+bM4a677mLHjh0dUQVxmpSQD1/OFvyHtqH53JgzBmAO+cFcNwjwBTX+9M4G3L4QroFJxBtNVCx4tqYMsxVDXBqhE+b9Bw024s66ksP5u/BsXYpn61IAnOMuJ2RrWb7d1uILabz+af3f1aJyD/vyqxiRHX/KMgyqQmaincxEO4qikJjoiMgFXkK0BkULECzJpWrlfAzOeDSvC/uoC9vhnIeoWnX8OS9q2nsV0L1VuLd8g//QVkLlhTjHz8SQFcJhsTJral/e+WJX7etnTe3bpAAAIOzz4N2xArRQTT+7+1suH/kj/ljgJi3BTmaSA1WtmV7ZUC4QzRSFKTEL97pPca/7FICoUdPQDE2bpnig0FUnAAB4+4tdjOibSPJxexEoYT/+w3txb1uGYjSjGM3Yx/ghwpZE5ZV66gQAUJMl7syhqfRIbPoUMCFaqkOCgF/96lc88MADFBQUnPZ7ExK6XwrJE7XWqvXWNQBl2i0UL3gegMTzbsSSllXvVfvyKimprNky+5XvAjw4/BJYNxeAuLOvwZk9hGil/iw0PWEwsZOvovyr/wJgiIohdtyFmKIT2qpBp+VQYTVlVQ1vBV7lCTT7Z9Y5f9ZtT9rdMbpX/+nEPuUacnd+R7iymIRpt+LsNRhoy8/ZiX3qteTuOnLOC27D2WsQ0U2czqfFjiIw4jyq13+OanUQO346ppiaUYRLz+5DdmYMRWUespKdDOmTgLWJufzD7ioIeki44HZsvYdSuPBFeiXbufeqEQzqlUCFy8+n3x/CZFSYNDyDAT3j6rXLMu0W8l99BABjfDoxI87B6Iht0vm/21k/6YGugycQPuFn4cR8wW0UvPZL9ICXlB88QlRW3yadozEd/TfVHJtzKho87vKGmtSertjm1hCp7W5L7R4EvPfee6SlpTFhwgTmzZt32u/v7inuOmv6L6s7l5KP/oY5rS+hymIKP3ie2MsexG+q+2ViNSj0THVS7fZx97gwrJqPYqxJh1f25WsojiQCyYM5PguQouhYS7ZR/vWbtcfC7kpKl/wX6xlXEzR2/IWLQdfJTHKQW+yq91xCtLVZP7PO+rNua9Lu1imrObpT/2kOVVH9yYsoZivmlN6Ufvkf1PgsnH2Ht9nv17Fz2mrO+cWrqHGZ+GJ6n/K9iqJgzFtL9frPsfYeiS9nM6VfvoF10g0E1Zq75f1SnfRLrfnZVld6aWorkpKiMY68lJDBTKVuJOaSn+FXbIwAiiq9PPbyKkJH5px/ujKH/73jTOKjjqUptXiLKF84BwDV6iBUlk/pt+9hHn0ZIUP9KZ8nSoiuP2JgNChE20x1fhYWfwnl8/8PQ3QiqiWKogXPk3jV4/jsaU1s6Ynt7pp9SVwje8LEOcynbE9XbXNLddYUoW3tmWeeoby8nKeffrpNym/3IODjjz+muLiYyy67jMrKSjweD0899RSPPvpoe1dFnIawOYboCbMw9p2AEnQTPryHUAMX5yaDwj2zh/PlmoMYDSWoZiuxl/8cFZ3SD56lofxxxrAX9/YVoGs4z5yFpccQSuf/Ad/edUSNnt4pggCTQeGm6QN5+rU1HH8NNbBnHL1Su0ZnIkR3EjLYiRoxDUNcKro9AeuuZWj2E+9wt8U5z8cQl3ba59R1HTU+i9ipN6D2P4uo4j2AQkBtncxgIYO99v9+5diF+8HC6toAAMDrD5Fb7CI+6rgpjAqg68RMvQFL9lgqPv87NX110wLGHklRXDKpNx8u3w+AqircecUw4h11L3ZDZifOsTMwZg1DV42E9q0mbI6M/VGOlx5r4/oLB/DGZzuBmqliN140iORYyRLXlbhcLj777DNycnLo2bMnF154IQ5H612vtMfa2Q7dLGzevHl8//33pxXhdKc7WQ3pzFG+qlB7AawoOrp+kiFwBcKahiNUidcUh6KAJVCG3xzX4PviLAHcOdsIpwxCUy2YK/aBwYTfUX/KUUc6VOLm+22FFJS5GT8olUE94065KLgxnfln3Zak3a1TVnN0t/5TQUc/Mqp4tH9q69+vhs55Ok71HgUN/bjEfQZFI6yfPJHfydq8/VAFf3xjXZ1jj90yjuwTbl5YAqWETNGEFRPmQDmawVInsGiwrke6cl2vSZtcWOGlwhUgOc5GgtOM0sC+L8e3vzmf3/G6cl+io1NY4aOsyk9CjJXkaCtNmVXWldvcEp1tJGDNmjU88MAD6LqOz+fDarWiKApz5sxh7NixLa5fRUUFd9xxB9OnT2fHjh3dZyRAdF3Hd9YnDQAAdDAoKt4j04V0HXym+EZvLBmjEwimjaxJkaeDPya7dSrdyrISo+g5tQ+KohCWlG5CdCj9uIvM9optWnrOk71HJYx6aC3+6EyqjIkkmP2Et3yGeej5zc6g0zvVydTRmXy9LheASyb1IrOB/Qf85mPrrwLmk49uhHWdQ8Vuth8oR9d1BvaMp0dyFOnxdtLjTx44HN/+bhSPnjYFhdRYG6mxp55uJToXl8vFAw88gNfrrT3m89WsGXzggQf45JNPWjwi0JK1s6ejQ4OAWbNmMWvWrI6sguhEOuPGYA2puZPaRSorhOhwvmAYtz9MjN2EUW38BoqlOpedVUb+9vEBCst3MLhHNDcOjCPxu7lYJ99EsBlpdKwmA9dP68eFZ2TVZCVzWjlJFU5Nge+2FfHPhVvrHL5p+iCmjkiFU90gEqKL++yzzxrMsgU10/4+++wzZs+e3ezyW7p29nTISIAQQgjRJnS2Hazknwu3UuHy0zPVyY8uHUp6fMN3f6ttqby8Mp/C8pq7itsOVvFVfCI3nzsCfwvyaBoUhZSY1rnjXO4K8OpH2+sd/++nOxjaO55Ep6VVztNZKGhYqg4SticRNEZhDlag+irxR/fsMjeuROvKycmpvfN/Ip/Px8GDB1tUfnuunZUgQAghhGgD+WU+/u+tdbUXizmHq3nmv2t48s4JOCz1v35dQZX8Em+dY9vy/fjMiQ3Mru8YxZW+OguNj9I0neIKb7cKAhR0jPkbKP7wz9iHTcUx7lIql7xCIHcn8bMeIhDXr6OrKDpAz549sVqtDQYCVquVHj16tKj8f//737X/P7p2tq2S55x8tZEQQgghmuVQUXW9u8XVniCHSz0Nvj7e6GNUn9g6x87ub8NatqvB17cXDQiENdyBMKFw47e/o2zdbNcvRUG1x6AYTXg2f0XRf36OP2cLisWOYpZNvSLVhRdeWLPbdwMUReHCC9t208LWJEGAEEII0QYsZkMjxxsehDf7K7mmTxkzxiTSM9XJjedmMTq0nkDeTgx6oC2r2gidPQXV/PGt9fzy5VV8uHw/+cVuRvRLqvfKkf0SSTvFouCuRtchENeH+EsfqDkQDgGQcMVD+KPSO7BmoiM5HA7mzJmDzWbDaq1J62q1WrHZbMyZM6dV04TOmjWrzTIDgUwHEkIIIdpE77Ro4qMtlFX5a4+NHpBEaiNrAnyOTDL7hbluiB2/NRmT5kUtDRKKzyasNLzBVFs6WOzh96+trh3N+Oy7g4wdlEL/HrEkxdlYvjEfXde5eEIvpozMwNSiFcedkylQgWvtR3WOuTd9gXXclQSNMhoQqcaOHcsnn3zCZ599xsGDB+nRo0er7xPQHiQIEEIIIdpAjM3EIzeNY+3OIvYcqmD0gCSG9E7AoCi4/GEsJrXehXPYmUEII+g6umrEnzi4fsFK7f5e9Z9SQNWC+HUjvqBGlClMTkmASpefuGgr6fH2k2YoOlaOwvrdxfXOsXZHIRlJUazZXsjkEenEx1iZPr5Ht0yZrCoQ2rMKf84WVHs0cefeRNmnL+PZ/BW2XsNR0kdF1OJgI0FCRxaoG7QgmsF46nTh3ZjD4WhRFqDOQIKAbk1H1cNoihEDIcJt8OP2BsPsL6ii3BUgK9lBVmJUp1nAJoQQHS3BYebCsZlcfORCubDSxz8WbqW4wsuFZ/ZEUSAh2krfjGhivLn4dq7EPGomatCNZ91H2MZdUbtHgKLAwRIPC5buw+0JctnZ2QzIOLbjrqKAuXwvnv0b+cw1iPE9jBgPLmdzYATzV5cDcNbIdK45tx92k8b+Ej95RS6yU+0kxDowG06YIXzcBa5BVZh2Rk8sJgOxTgtnj8zg8+9yGDsopdF0iV2dpoO1z3iiynIxJ2TgWr+Y2EmzCbur0FMGRlQAYAq7CWz4EGu/MwlHpxPetQRTfCbBlMERHQh0dRIEdFOKomMq3EbYU46l11gCmz/H0nskfkfLVq3XPQl8vDKHj1YcqD30yE1j6ZceedvACyFEY3QdwmENtz/M06+vodIV4LoLB/D6J9trLyTvubQvI4o/wbtzFTZ3JYHDewlXFmHLHoVyZCPF0uoAT/77ewKhmrvuf/jvWn5163iSk2v6XJPmpWrVPAIHt3JWnxIMS7cQ9FQx+ZwRfLheJRjSWLohn5F9Exjm/R6UDAweIzHb11HW6xzSsrJq66PrOiP7J7Jw2T50Ha6Y2pcvVh+korpmapPNYmT2uf3ITo/uVrtQn8hnjscx4SqK/vtLdJ8LS88hmMZcTkDvZougT8HgLcW17jPcm7/GPmgS7o1fYErMJOayX+A3tGz3XdFxZGFwN6WGgwQP76Xi839SueBpqlbOI1x+GEVpeWd9dFV8hSvAJysP1HnuoxUHULvhvFAhhGip/FI3la4AvdKi2bqvrM6d5L8u2oMy7geYMwfh3bmScGUR0WddSzhlSO3rCso8tQHAUXnFrtr/BxQb/rE3QFwmyt4VaJ4qvKOv4YWlPoLHve+rdflolSUkrvk7Q/b/F33vKnzVlbj94Tpl90iK4uEbxzJuUDKHy9y1AQCA1x9iw64ieiQ1PgfaoPux+IsBUPUQtkAxjSRV6bRMYTdVS99CD3ixZA2iasV8KNrV5drRUn5HFvEz/gc94MW98QsMzgRiZvxEAoAuToKAbiqsmrEMOx9jdBLBwgPY+o9D6TG6RcN2bl+I73cWM2/pPnbmVoKqYDxh+Nhpj6y7I0KIrqW02s/WnAoOV3jbfd9vo6Gm//UHwlhPyBxkNKiYtADhqpLaY6GKAtTwsaxA8Q3k4I+PrrvI2KoEUHxVtY9N7mLSY+v2y3abkcCg89EDPqg8THDcdbyywlPvBo6CQr/0aH506RD251Vxop055fiD4XrHAQyan/DmTymf+xRW32HUg99T/MZjmCv2Nfj6zipksGPrfyYJl/0Mx0X3ET35KpTo5IiaCgSg6kFCFYW1jzW/BwLuDqyRaA0SBHRTBj2Ab8MnhKqKsWQMwLtrNXrOmmaPBOjA3G/38ff5m1m0bD/P/HctBaVubpo+qPY1FrOBi87s2a2HhoUQXVdOkZtH/r6C/3trHY/+fSUb95W26x3d9IQo+mTEUFDqJjsjBttxG4Y9cs1g/KveJlxVTPTkq7H0GoFn89coRTtr65gSa+XHs4ZhsxgxqApXTO1DdtqxO7Ehn4vw8v+ge6swT7oeQ2pfjDu/5KrhKibjsa/7c0akYlj2T1SLHeIyMH7/X359eTI2U8OXBCZVYUS/xHrHB2cnYG3kPbpqxOCIJ+wqp+TNX1H+2csoJiuYrM345DqOjkIobQSBpMEEFBvKkIvwW+qnSO3uzK58qpa9gzE+nbgL70AP+qn4+EWsWnVHV61buvHGG5kxYwaXXXYZl112GRs3bmyT88iagG4qrJgx9xhCXEImao+RWLd/hSEhi1AzRwIqPEG+WZ9b59jCb/fz8+tH0TPVSaUrQFqCnXiHOeLukAghOj8deHfJ7jqbXf1r0TaeuXtSoxe/rc1iVLln9nBWbT3M2h1F3DZzCAZVITHGSlqcDVvSDdgHTEDLHI2j30TseZvRjluAqioK4wckM7hXHOGwTozdVPtcSbWfJ/61jjumziI9u5znvjNyz8W3EFO9j39t1AiGNCwmA9dc0J/+6U4M3omQ0p9qzYotZxmW6FiCx/fdCri8IVRVIcpi4NzRmazZVkhRRc2OxlE2E1ef1w+1kShKw4C5zwRsBzbi3bMGgPjpd+O3d738+vpx6S40PTLvnQYcmcRf/GPU5GwClkQSLnWgOOLxqZE5HSgUCrFw4ULeeustiouLSUpK4tprr+XSSy/FaGzZpbWu6xw4cICvvvqqxWWdigQB3VggYSBKgkYIFXXwBfhb0HmZDSp2ixG3L1R7LDXRjlFRyIi3k3FkkxgJAIQQnZGug8cfrHMsEAyjtXOnFWs3cfH4Hlx8Rg8UqB05VRRw6QlYeiWhhXXCphjU7LPqjazquk7Ukc3Gjq/6pr0luLwh/vRpEdEOCxXVXn75xh6eu/8sfpAR5uKzg0RHmYmLOhI4DDwPTVeJAgwJVxA87vvBH9L4cm0uH3y7F4vZyG0zBzMiO4HHbx3PoSIXYU0nKymK6JPsEKzqIcIHvq8JAAxGCIco//TvxF3+EH57Wmt9nKKdhBUjes8zCR75fQymjui2WaFOJRQKcd9997Fp0yZ8Ph8ALpeL5557ji+++IIXXnihRRfv+/bVTJm79dZbqaio4Ac/+AE33HBDq9T9RJEZ0kYQ/ciPuKV3L+JNPu65cjhGg8IV4xLonxnNRWc0PPUnpOnkl3nYk1+FO9DwfFEhhGhPqgKzp/atc2z2Of1wWBre1bfJ5arKaSdD0HUdXdNr+89qX4gPlh/gl/9YyWuf7aTcXbMO4HSmVhYfuUOv6dQu4A2GNFzuIHFRZnokRRF73MjB0e+EkKaTU+JjT0EVniP99b6CKt7/ag+hsI7bG+TP722ksMJLlNnAwMwYhvSIPWkAAKDoYcIVhaj2GJKu+19ip92G5nND0Nv0D0p0Ksf/PkZqAACwcOHCOgHAUT6fj02bNrFo0aIWlV9VVcWECRP4y1/+wquvvsrbb7/N8uXLW1RmY2QkQJySxVdM5cd/Zvg5N/HS/4zE/80rXD71HLQYS72FdZqus3hNLu8t2Q1AcqyNh24YQ7yj/Xe7FEKI4w3qEcuvbh1PXombhGgr2WnOFo1eFlb6WL6pALcvyNkjMuiZfPo7yCoKLFmXy4Jva+7+LVmTS4XLz92XD0Vt4q4rIVc5o/on8+36fC4eFc+8VUUA9Ex1Eh9dfzFx7fs0nYUrDvDhsv0AZCRF8bNrR1NSWffiRtehvNpPamzDOx03JKxaMAy/mIRBk/FZklB7J5OYPgC/NZl2X5EtRCt666236gUAR/l8Pt58802uuOKKZpc/atQoRo0aVfv4yiuv5JtvvmHSpEnNLrMxEgSIU/NUECzLp2z+HzDGJhMsySUYl4Y5fTAhte6XQkmVvzYAACiq8LJhTzHnjsxo71oLIUQdBkWhV7KDXsmNp7VsqkpvkN+9uhq3t2aK0Tfrcvntj86snRrZVIGwzopNBXWOrdtRjMcfxmFp/Cs6pOkUVXhJMLop2PAdg/qM5I9XJaGvnUv81Ev5fLufH88aXm9H4uMVVfhqAwCAvGI323PKyUysG8yYjSopcafXLoCwaiVsqVkIrGHAZ2n/AMAcrCRstBFWzFjDlfiNMTJtVbRIcXFxi54/lTVr1hAMBpkwYQJQM+rSVmsDZDqQOCV/fD8SLrkXPRQgWJKLpddwrONn1wsAAPzB+lvHuzzB2r0FhBCiO8gvcdcGAFAzDSfncE2mFIuvEEvlflQFLOW7MAfKGi3HZFAY3rdu5p3+PWKxmRv/0g+GdT5Yto9vN+bzyaZqfvt9InM+LeRwRQCjM55xg9N49KaxJJ9kFABoML2nxxeiR4qDn107iuyMaEb0S+SXt4wnwdn80VxNC1NZWcnGfaWUl5cTDLfPNFFzsBLXFy+jbfsCa3UOpW89jqUqp13OLbqvpKSTZ4c61fOnUl1dzR/+8Af8fj8ul4v58+czbdq0FpXZGBkJEKdkDlbgWv957eNA7g6oOIQS36/eHZWUOBuDe8Wz7UDNl56qKozsnxTR8weFEN2Pw1b/othpN2NSQnjWfYx3x0qiJ86iZOm7xEz+Aergi2hwir8O0yf0pNIdYM32Qvr3iOW2S4ZgOMl9k8PlXvblV2E1G9mwq+auY2EZbN6v8MRNV5Iek9Sku91pCTayM6LZd2QPAKNBYVDPOAyKwpCecQy6cSyqUlPH5nbhgWCQ0L7v0PN2oyRPRt37BZ6sURh7jyHK0rx9Zazew2gGC+DEoAcxewvx29PRTrivGTZFYe0zmsqvX4flYE7rh2apyWajoGMOVeM3RmMMe4CaPQGEOJVrr72W5557rsEpQVarleuuu65F5Z9zzjls3LiRyy+/HE3TuO666+pMD2pNEgSIU1Iq8/Ef2krUiPOx9h5F6cI5uNd/hn1qJsETRgPMBoU7LhvKrtwKXN4AfTNiyUy0yxxQIUSXp6hKbUaftHgb10wbwLtf7kLTdM4akU6f9GiCuoGoM2YTyN1B5bdvY+kxBOPAKQSO9IEuf4i9eVVUeQKkJUTRM9lBXJSZ+2b0JDQthbAtAXuglLAGIbXhO/m+QIgBPeJZ8O3eOsfDms7m3YVkx+l4LSmnbI/VaODeK0ew61AlHn+QfpmxpMcd69NVaHHffbDITbRPw7bnW/ocWo0W8OFOHcnuncWcPSL9tIMLi6eAsvnPYIhPxz7zHrSdyyn69i3ip9+NnjmmNhkGQBgj1rR+x97bYwhhkwNFAdPhLVR8+xbxl/4E94bPAQXzuFkNjnALcbxLL72UL774ot7iYKvVyvDhw5k5c2aLz/GTn/yEn/zkJy0u51QkCBCnFEwcQNI1vyIUlUrAYCfp6sfQrHEEGukso21Gxh6/sYwEAEKILkxHZ29+NYuWH8BuNTBjYm+yEu1MG5PBmAFJgE5slBmDoqCgEy4+QKiyGFQjgfzdKJX5KHF9qPKGmPPOBg4UHNt99+rz+jF9XCrB9QsJHN5H7Dk3UbroeZxjZxDMnopBVTlxWn96YhT7C+ru4BtlMzF6QDKpGfEErXFN7ndjbCbG9a+/EVhrUFWFZZsLyYxOZEJ8BlpZLmpqf74rcbJsxwHOGJKKxahiCVbgM8YCNXfoLaHK2seNCRzaRt6/foHmc0EjC6it4SrKFs3BnNYPa59RVC17l6SeQ/HF9kV1JqIHvBT95yFAIX7G/xBS228jM4OiET6SoUlFQ1GofXzi86JzMRqNvPDCCyxatIg333yzdp+A66+/nksuuaTNc/u3pq5TU9HuKj1B9uVX4QuG6ZGcSrrBhgL4nL2aXaaCjqoFajJHaD40gwW9kQ3MFEWh2hdE0+puilP/dTV1VVUFp9UkU4+EEK0qt8TLc++sZ8LQdJx2E+98sYtbZgzGZlY5UFBFhTtAdno0PeONOPCgqeAYdwlRQyYRLjqA7qsGXWdvfhUHC6uZPqk3FpNKOKzTK9FMiUcndfBZhN3lFL35K9ToRNzx/Vi4dB+KonDGkFSSYqzYTDXpTB0WI+MGpbJ1Xymb95ZyzphMLGYj3289zJa9JRRP7MWEgQnYbVaMmr/REYX20CfVxijXt2jl+SiDp6Ft+4JzU3dRmNYXg0HFVLqT0o9eJP6ynxGM6YUhfyNlS/5D/OUPErIlElItddrgt6eRMOthiv776JEAAOIuuI1w1hj0Ey6afYZo4i/9KZo5irA5muTMQQQdNZuVha2xWLIG4tmxEtVqx5DUk2ATszG1lMVXRGDXSkxDzidssGI8vIlQWT6WHsPxO7KweAsJ7FmFafA0go1MUTIRIIgZBQ1VCxLuwJ9xJDIajVxxxRUtygLUGUgQIBpU6vLz1H/WUH4k37SiwAPXjGJYr7hmzw1V0DEe3kgwfzfWURfjWbsIa8/hBJIH13ttWNf5fnsRr32ynVBI44opfThvTCYWY91O3h/SWLwmlwXf7sVoVLn54kGMH5jU6C6WQghxug6XebjynH58uHw/FdV+RvZPosLl58udRVQdyemfrpZh08NUbfoKa88hmKMT8G5dij9/L9FnX4uOQkW1j5mTs1m6IY+yKh+3T00ka+d8VOeVKEoVChA18EzcRXkcLguSmeJk0+4SfvPP74h1WLh79nCy0xys3l7Eqx9v58IzejK0TyI7c8pZt/PYju5vfLaTfQcTuGtmPwKr52HtM4ZA4qB2/9w0TSczNY5DhSOIn9iP1zcZuWVSH/ICUYwYlEpUqJKyT/+O5q2mdN7TRE+YTdm3b4GuUb30LaKGnY01fTCeNQuw9h5FIGkwBj2IL2cT6MeSULi3r8CZPoSAOa5eHXyOrNr/e6N7AzU3mMjbiGfHKmLPvZnq7xZS9cU/cUz/CQHadjTAormo/OgFgqW52N3lmNP6ogU8VC57D9XyEfEz7qV88SuEq0uItUShDjwf7YR8GxZPPq4V7+GcchPhor0ES/MwDL2QcDuOZIjuQcaaRD2KAmt3FNcGAFCzKOyVRVtbuPmXjh7wU73mI8refAz3+s/Rgz6UBsatC8q8/GPBFvyBMGFN5/2v9nDgSOaN4+0/XM28r/cQ1nT8gTAvL9hCQfmxzWg0HQ6VuPlyQz6L1+axv9BF6DQ24BFCiKQ4G/O/3lO7CdeGXcXsya1kxaZ8kuPsrNpcgMmgo3kq8eXuoGr1R5R98W/CrgrMKb3QLNEAZCU78QdClFXVzCO2mRS0inzCHz9NyQd/IuypRut7FoqrhFTPLipdAb7fVghAhcvPn95Zz+EKHy99UNM3Lly6D13XWbezqF6dV24vZf+mTbg3LUEP+hvsZ9tDz2QH1dZU/vh1iG05lTy5JESOL5oR2fH4jDHEXfYgqs2BHvBR+c0boGuYknriHDed0o/+WvNdseEL9FAABR1TZQ5V374JKMSfdxOGqFgCh7bhX/8RRqVp30+6rkPaEJKufhy931Tir3iI6Kk3t3kAABAwOIg574coRjOezV9T8fk/CZbmYes9Es3voWTeM4SrSzCn9cXQa3S9AABA0cL483ZS9u5vKPvoL2h+N4rewAuFOAUZCRD1qKrKnrzKescrXQG8/hB2U/N22NRRMWQNx5Tci2DRAcwZ/VEyhtZZyHXU8QHIUaVVPiCmzrGjX6Z13lvlJyPejqIorN9Twl/e31jn+ZsuHsSUEWntNPArhOjqTAYVty9U59ieQxX07xFHtSeApsOrK108frYRx/BzqF77CYrJgiUtG0NKX3ymWAB6pzlZvOZQbRmvLq/gN+dfgXHZyzXnGTeb+948zAPT/odd5Qb2l9Tthz2+EBXVgTrHTqzX8ar9Oqk9BqOkDULvoB5PVRXOGJjMwB5xVLj8RNmMJDgttWsWgo4MoideScWXr9a+J+bs69ATemBKzCJYcghL1pDaNoSiM3COm4kpqQcxw89GTRuEa+mbWEdehF9v+ndT0OgkGOMEHXz2tFZudeN0HUJxvYgacT6utR8D4MvfTcLFd+Pdv6H2dTFnXYvPHN9gGQFnBtHjL6Ny6VugGogaeSFeyWwkmkFGAkQ94bDG6P7189z2THUSbW9+rmiDEsa/8ROCRTlEDTmbQN4uQlu/QFXq36FKT7BjNNT99cxKdtZ7XWZS3U1/TEaVtCMb3VS4A/xz4ZZ67/nvp9spqaofZAghREMSYyxknNDXnDksjfPHZpEQbUVVFX56bjROc5jqtZ9g6z0CUHBvX4lv5wosnprNwFSo07f+/MJ4TCv/DakDMMQk4/78r9x6dgK/W1jI3OUFDOtTd8FuaoKd1HgbpuOmRaqKgqGBDcFUVSE5MRp/zhZCO7+t7WctngIs/hIURcPqysGoeeu9t7Xpuk60zUiPpCgSHMcCAAUdQ8FGKpa8Vuf1pYvmQP5WQpXFRA05G/+hrYS2f42qaIRUG4YRlxDOGoNiMOK3pxF14f/gt7TN4ubWphKGvStqAwAUFcegSZQseK7O60oXPofFdaiBEsBUupvKpW9j6zcexWSh4tO/YQ672rrqohuSkQDRoKG94xk3KIXV22uGoqOjzNxx2dCT7j55KmHdgKXfGZhSekPGcCy9RqDGpePXFSpcfg5XeHHazUSZDSRGW3jsh+NZuGwfPn+ISydn16QaPUFmYhS/uHEsC77di91m4rLJ2SQ6zeg6lFT58DcwfUnToajcS9IpNtIRQggAi0Hl/h+M4PPvD7Int5ILz+jB0F5xWIwqyXE2EmNt7CovJqVXInFTr0Pze3COuQjN66Jy3WdYhpxXW9aQXnGcOTSVVVsO881uH5ePvxpTz5FEmzUCh/eSm2umb2YMZwxNQ1F0brx4EEs35NE7PZqLz+xJgsPMY7eMZ8GyfXi8QYb3SSDGYea1j7fXqfO15/YirV9PFMs9qAlZ+HUFi+6h8rO/omsaztEXUbz4XyRc+hOUtBEdsouuOVRFxXFTgGoyI81B87pwb19B4qwHCcf1xtJzGIaELHxHFv6GVUud7EcBuk5fbgh58R7cDEDstNsxJmYSLthVM3UsrS+xF95F9bK38e5ZQ7g8H8WZWS95hu5MIW7arSg9xxLlLkb3VBA0RDV0OiFOStG7WCqV0lIXWjee052U5KS4uP7c944Q0nQOl3vxB8KkxtuJsjRvGtCJFEVB1/Xaf4ur/Tz/znrySzxER5l54JpR9Ew60qEpoKCcMuOPoiig6MevFeNwuZdHX1rZ4Ot/9cPx9EpxNPhce+lMP+v2JO1unbKao7v3n9C2v1+KqtRcLDfQHykqqIqKEvahBL2EzLGoeghjqBr/Cekuj+9b0xPttVl/1CM3WVz+EIeKXOw4UEZ8jJURfROJsZvRj/vZKSqg1/SNDqeNDbsK2bC7BHSdEf2SyE5zYlAUVFWp8zO3evIpefs36KEA9iFTsJx5dYdukmXxFuJe8S5Rk6/Db0nA4s7DvXoh9knXEjTHoet6vTYc1VX7ElOoGrUyj2BCP3TFgLnqIErAheZMJWCOxxSsQq0qIBjfB02pe6/2aJuP/0yOfpd2Z639s25uH3qivLy82hShGRkZrVImwJIlS3jxxRfxer1MmjSJxx57rNXKPp6MBHRnioJywoXx6TCqCpkJrf/lcLSz0nUdFPhw2X7yS2p2bKxyB/jb/E389rYzsBjUmp0qm7CgTdf1enmxk2KtTByWxorNBXWOD+gZS3qi3DURQpwe/SQBlK5BGA0wg8kMuk4YA+EG8t0f7VuDYR1/SEOH2k3IAOwmAwMyYhiUFVt77MRz1/TrNcdsViP9MmLonxkLul7ngvn4/yuKRri8AD0UBCBweA/2kLtDgwC/LQXbeXfip2aqqT8qA9vU2whgrg22ulvgGjQ6IWFgzQMd/M4edZ83RUNC9EnLOP4z6e4BQGe0bds2nnrqKfbv34/JZCIYDNK7d28effRRBg+un/HwdBw6dIhf//rXvPfeeyQkJHDzzTfzzTffMGXKlFaq/TESBHRDIU1n64FyPly+n5R4O5dM6k1qbNtlPWjsLk1TaDrsy6+76U1xuZdAUKsJAlrAoChcfV4/UhPsfLTiAOGwzrTxPZg2PguzQZYFCyEapygK3mAYg6Jgau3+QoE9+VW8+tEO8ktcjB6QxNXn9yfJWXdaS5P6VQXW7Szi/SW70TSdy8/Opk9qw3c5jSEv3l2riBp5PvahUyib/yx6dQmKNalDpgMdFcR80sdCdCbbtm3jjjvuqN0t2O+vWWO4Y8cO7rjjDl5++eUWBQKLFy9m+vTppKamAjBnzhwslraZ8iZBQDe0t6Ca59/dUPP/vEo27y3hyTsn4LC07o+7whNg455Siso9jB6QTO8Ux2nn5zcocMEZPfj3h9tqj501IgOntXXq6rQauXRiL84dnYkONVOa5KaJEOIkAmGNlVsLef+rPURHmfjhjCH0TXe2Wt9RUObl6dfX1l7kr9tZTG6xi9/eeka9vVBOpbDCx2//uaq2rO37S3nijgmkx9Xf0T1oiMI2+UZ01YjPYCf+6t/gN8Z0aAAgRFfz1FNP1QYAJ/L5fPz+97/n9ddfb3b5OTk5mEwm7rrrLgoKCpg6dSo/+clPml3eyUh2oG5GUWDXwfI6x6o9QQrLWzcDRCCs8dd5m/nPx9v5ZGUOT766mv2Fp5+dQNdh/MBk/ueqEYzqn8T1Fw5k1tTsVq2rpunYzQaizBIACCFObW9+Ff/5eDtub5CCEg9/+O8aSk9IzdkSOYer693lLyrzklfiPu2y8kvcdaf/6JBX3HhfHDRF107/8RljOyx1qBBdUV5eHvv37z/pa/bt20deXl6zzxEOh1m5ciVPPfUU77zzDps2bWL+/PnNLu9kJAjoZnSdeqnsDKpCrKN1h5KKK3zsya2bw3r19sLahW2nw2JUufDMXvzkqhGcPzqdaKuptaophBCnLf+Ei/FQWKe0svVupJyY/vioPbmV5JZ6TqusOGf9vj0+WnaOFaItFBcXYzKd/BrFZDJRXFzc7HMkJiYyYcIE4uPjsVqtnH/++WzatKnZ5Z2MBAHd0MAeMVxwRg9UBZx2E/dfPYoER+vOsbRajPVyUyfF2lq0QCkc1mRYWgjRbCGtZvfc73YWc7DY3aSkAg058UaK0aAS42i9C+vs9GjsJ0x5HNAjjt25FTzz37W4TrIB2ImyEqO4beYQLCYDJqPK9RcMoEdSx2Y+E6K7SkpKIhgMnvQ1wWCQpKT6ey011TnnnMOyZcuoqqoiHA6zdOlShgwZ0uzyTkbWBHRDdrORa87ty8Vn9sRkUImyGFr94jrBYeLWmUN4ZdFWNE2nZ6qT0f07dnFZR1MUsATL8BnjMehBjMFq/I3s+CiEaF2KovD9jkL+uXBr7bGfXjuaoT1jT7usvunRXHfhAD5ZcQCn3cyU0Zks35THrLNaZ6pivMPM47eMZ/Hqg+zLr2Jgzzh04LNVOQAUlHnol37y7DBHGVSFy6b0YUSfBHQg2mZs1X5Y03WqfSHsFmOL9okRojvIyMigd+/e7Nixo9HXZGdntyhd6IgRI7j99tu57rrrCAaDTJo0idmzZze7vJORIKCb8IU0yqp8OO3m2i+BGFvNkFXbXJgrTBiURL/MSXj9QZJjbae9oK27sbjyKHn/SeIuvptweQEVm74k9rKHJBAQoh14AiHe/HxnnWPvfrGLgbeOw3iaCQs0XeebdbkM65uI2xvkjc92oCpw0Rk9sZtbZ7+UlFgrYwcmU1ThZdnGfFzeY3cXDQaFjfvKCGs6WckOEqPNKCeZu68oCo4jIwut2d+XuQL8++NtbN1XRq+0aO66YijJMtWoW1MUMGh+QkrNNDMTAcnWdIJHH320Tnag41mtVh555JEWn+PKK6/kyiuvbHE5pyJBQDdQUu1nztsbKCh1Y7MY+ck1o+iX1nCKuLCms+9wNTmHq0mJt9EvIwarqfEvNVXRsFTloKhGQtZYjJ5iNJMdnzUVUEh0msEpHQRA2BKNtdcIyj54FoDos64hZJJheSHag6Io9ebam4wqiqKgoGNxHSRkT0bRNAy+UvzOrNqdWF3+EAVlHnIOV2NQFcZmR/PoxXHMWVzGpL52pvdP5L2NIaq8QVZuPUyc00L/zNjaC++m1Q9UVSUc1mofa8DhUnedAGDsoBQ+XZnDmh1FAKgK3HvVSEb0iW9SYoOQpuMNhLGZDRhbcudegUXL9rN1XxkABwqqeGXRNh66bjSSYbl7UhQwl+/Du/UbLGdchRqoxr3qfeyTrycgN7NqDR48mJdffpnf//737Nu3r3afgOzsbB555JEW7xPQniQI6OIUFT777iAFpTUL2bz+EH+du5Gn7pyIzVT/zvzmA2W88O7G2sdXTOnDpRN7Nnr3yOIppOitJ7APGA+Kgmf3GhKm3Yol04FflQvc42kGK6akLLw7a3YpNiZkElQkQBKiPdhMKrfMGFTbv6kKXHNBfwyA1VdI0Vu/JWrE+eihAJ5ty0i+8Sm8lmTCus6mvaW8v2QPFS4/qqow+soEQl88xz2jr0TN3YDhYBk/mflL7vnrCo5cwzNuUAp3XDoYQxNGGaq8QdbtKmbXoQrGDUphSM84jEaVd77YxcRh6QRDGkXlHjKTHWQmO/nL+8f6aE2Hv8zdyNM/nkT8SdZ26cDuvCre+HwHhwpd9Ex1cv2FA+mT5jjpKEJjAiGNzftK6xzbk1uBLxButd3jRediCPvxbPkKz7alhDwVhIoOEnaXYx8yBSU1PqKn+55o8ODBvP766222Y3B7kSCgy1M4UFB3s61KVwBfIFwvCNCBBd/uq3Ns0bJ9nDMqo9E7Wn5bCnHTbqV88SsAxE6ajZrQE58EAPWYPYcpWvYu0ROvIliaS/nHfyHx+ifxmeQOihBtTddhRO94nvmfyeQXVZMSH1W7SaLfmkzcBT+i/LOXAIif8T/4LDUL98qq/RRX+Khw1Wz4o2k6/13r50ejpuNf9z4Asdf8hv+uKa8NAKAmG9qsKX1IOcVGjLoO73y5m5VbDgOwasth7rpiGOMHJpGdHsPCpfuwWYzEOi3kF7soq/LXKyMU1skvcZ80CDhQ6OLp19fUPs45XM3vX1vNr249g55Jp79DusWocvaoDD74Zm/tsTOHpGKXAKDbCqkWrBN+QNhTiX9/TSAae+7NhFKGSADQiIyMjC558X9UZE/i7gZ0TeeCM+puOT5haCqx9voprBQF4k6Yz2m3mjCcZGdeQ9hLIG9X7eNgWQG6qwRVb3r2ikjht6eRdN0TqEOmYZt0HYlX/VLWAwjRjhRFYXDvBEZkJ9TZJd0Q8uLPP9aP+fN3Y9Bq5vOajAbUE7rAoekWwoV7ah+HS3LISKibitOgKpiasA6q0husDQCO+uy7HEBhxsReZCQ58PpDuL1BbrlkCKu2FDRYjvkk0zbDms7n3x+sd1zX4dsNeRiaMX9H1+HcURlceW4/0hOjmD6xF1ed2092FejmDH4XoeJDtY/9uTsxhE4vba3oOmQkoBsY0SeB+68eyfdbC+mXFcPo/kk0OEKtw+wpfdhxoAxfIIyqwI8uG4rdrDYa5RsC1XgPbCT2rB+g2qKpXPkBlqxBGMJeNGPD6w4ilaYY8TlqArKw0UzQ2Ho7jAohms8QdOHft574S+5HDwepWvYOtmHnE7LaiI0y0TstmhH9kti4uxhVVRiRYUDbl0/SNb/Gd2Aznh0rOWPKMOY5LZRX19ypv+GigcQ7Taf8G7eYVGIdltqRBoDs9BgUBRKdFh6/eSyFlTUjERt2F9MnI4YdOXU3fIx3Wsg4yd38UEijoJGNxnKLXIBCczojh9XIjDN6cOG4LAxqs4oQXYiJANUr3yPsLif2nJvx5+3Au2sV9gHjUdJHy2hAN6ToLUns3gFKS131dlrsTpKSnBQXV5/0NRo6Fa4gFpMBh/VY+k+DQUXTTp5rX1Gg3BOkqNxDnMN6yqwTANZQBaAQNEdj8ZcRxEjQFHN6DTuFprS7u4nENoO0u7XKao7u3n9C45+zNVSB3xiDgo45VIXPGFv7nK7rFFcHKKn0YjKq9Ex2EK3XvMaoB1CDHgLm2JoFxKUeHDYTKbFW1CZmHdpzuJo/vb0ejy9EWkIUD1wzksQjm3y5/SH++OY6Dh7Zcf2KqX3ZfaicHQfKuXhiL2xmA3arkd5p0WQmRjXYWyclOfnvJ9t454vd9Z676eKBnDMyvVtewEViX9LWbbYEytDLDhFKHYoh6EEp3kU4dQhhtWOzQrV2u5vbh3Y3MhLQxVT7Qrz9xS5WbjlMlM3Ej68YxuAesQC1WSdORtch1mYi1tb0i3ivMYZ9h118uHwzFpOB6RN70TOprVKPCiFE6zt60a+j1AkAoGYaUXK0heToY1N+fNS8JqSYwVwzF99hMTY5f//x+qU5+f1dE6n2BkhwWuukU95fUF0bAADM/3oPA3vFce9VI3hl0Vaq3IHa5x64ZhTDe8c12PeOH5TCN+vyOFx2bOpGRpKDkf0ie/8WcXr85niUtJpFwJrJiZIxRn5/ujEJArqYTXtLaueXur1Bnn93A8/cPanBNQCtJb/My1P/WV3bEazbWcxTd02ovZMlhBDdgaIoLdr1vE5ZKuSXesk5XI3ZZCA7PZr0OHu915VV1881nl/sZvO+0joBAMC/PtzaaOa3uCgzj9w4ltxSF8GQTpTNRFqcrdX2NRCR4/g/AQkAujcJAroQVVXIOXxsOExVICPRRrUnQKz96I+y9Zdt5Ra563QEobBGfom7UwUBqqqgKEqTRkOEEOJE5kA5evE+9IwRGCprFtkGYrNPOb3SH9IwqirHp+QvrvaxN6+KfyzYUvv+mCgzj94yjqQT+s34BjbfinNaKCqrvxiz0hXA4w9hMzWcJchsUtmTW8nCb/dhMqrcPH0Q4wYkNXnakhAiskh2oC5E03RG9qtJa6cqcN+FSfxs6GF6xqmYCjdjKtyMorR+2B7XwMV+bCcJABQF8su9/HfxLn732hqWbz2M2y+Zi4QQTacoCnrRHso++jOB79+mdN4zuFYvwqjVv0t/VCCk8e2mAn758ir+750N5B25aHcHwny1Lp/3l+ypE0BUugN8t/UwygkX5L1To+mTUXeKUX6Jm9EDkuuds09GDDEnGfXdV1DN/K/3EtZ0fIEwL32whcPl3qZ8BEKICCQjAV1M/8xo7v/BSJZuzKO/00fwy/n4inbgz9uJffBkLEn9CKm2Vj1nrxQHs6b2ZcG3e1FVhesuGEB6fP1h7Y5QWOHjf//1Pf5gGIDdhyqYMiqDGy8YQEs2yxRCRA5d19GzRhE1/DzcG75AtTqIPucWvErjiyF3HKrg3x9tB6C00sfTr63hqbsm1i4uLquqH0DsOliBYbJCKHQsOoiyGPifK0ewaU8pyzblkxpvZ+roTBJjrew8mMbKzTUpQ+Ojrdw2c8hJdwFu6Jzl1f5O018LIToXCQK6GKOqMiI7nlH9EjCgESi7ANf6z1GtDqImXImvlQMAALNR5ZIJPThreBqKohBjN3aaeYI7D1bUBgBHfbM+j4sn9CS5gWF2IYRoiKHiIJ7ty1CtUWg+F/69qzH0m0JYqX/nXVUVdh6sqHPM7QtRWuXDaTeTX+yid3o0+/PrbuQ4fnAKoVD9KYsxNhNnD09l6sh04FiSh1unD2TGxF74A2FS4+0NrgU4XmZy3TSiZqNKWsLpbxQmhIgMEgR0UbqmoxRtw7XhC0yJmQRLcvGs+xjT6MvbJpWXTu0wdGcJAAC8jUz9aeiLVgghGqXrWDIHEX3OLfh2r0IPh1AaSYyvaTp9M+pmWLOYDSSavCSYPEwclkaFy095tZ+KI/sKjOyfxIi+iSc7fb01TQZFIT2u6Td2shId/OLGsSz4di8Om4mZZ2WT4DR3qj77lBSo9oYwGBSizMZWW6gthKhPgoAuStdBiYrDMepCrGNmouVtBSCsNr6tfHMEwzrBsEaUxdApv0gGHEmPerzs9GiSYlt/REQI0X0F4voQdd6deFUbhoHnAhBWGu9PB/aM5cpz+rBw6X4SYyzcfkEWykf/i3/wZMaNvYpyT5hhfRKpdPkxG1VS4u0EQxpBTcfUSnMVPYEwhuPKUhUYkBHNL64bDYqOrrXspo2igKqF0FQjqh5CUWrWQvg1A1aTodXTUFR5Q3y86gBfrD6E1Wzg6vP6ccbgFMyN7GofKM1DJQqNmgxI1mAZflMcuuxrLESTdEgQ8OKLL/LJJ58AMGXKFB566KGOqEbrU6DCFUCnJl3b6TLrXvSQFT3gxR8MYrcY0U02dL2mQ9OAclcAk0Ehxm7Cb0/HNG42pT4IxwwjyezCEPJgNULQ48ZliCHW4MFrjEdRwBT24FdtlLuCGAwKBlXFFwgT5zBhUBQUNMxlu9EdSWgGC5TsY68vga15AUb3spOUnFgnv3VnkJXk4KfXjOI/H2+ntMrHuEEp/OC8fq32JSuE6CQUqHAH0TSdOIf5pJd5iqJjDHsJqnaMmhdNNddeKJ7sBKUBE/6AnzinhUp3AE33owDJdo3ygJFgWCfFoaOHdaJMJmac2ZOZY+Jxbf2W0OJXMCRmYBk5A7+m4LQacaLj8YXYlVvBlv1lhMM663cVMWtKHwZlxaIe108Zwj50gwkNA2bdS0Bp/EZGSXWAT1YdYOmGPKJsJmZN6Utqgh2zSSUjIQpjE3f3VRQwaTXnMug16UiPBj6KAqbSnYQK92POGIhWeRh/4QFM6QMorAjzVX4054/vcVrTLlU9hKKFCBusmDQvIYO19vsNBT5edYDPv6vJzOTxhfj3R9uJdlgY0Tv+WBlomF25KEYThz/8M/Hn3oAWnQGeckrm/4G4C+4glDZCAgEhmqDdg4AVK1awbNky5s+fj6Io3H777SxevJhp06a1d1VaVTCss2R9Hu8v2Y2u68w8K5uLx/fA3MSLZouvmOov/4l6xqUYq8sJ7F1POGMAhtgUSB+O2x/mva/38c26XMxGlR9eMpjxA1PYfNDNX+Zu5Ipx8Zx5+F3iz5hO+foviBpwJk6bneq963GOmU44FMa1ehGHsi/jqbkHMKgqF0/sxeY9JaQnRnHNef1INFRRsuh5TAmZGGJT8W5bysDLH8QXYyJhw7uYxs2CpL5t/EmeHlWBob3i+N2dZxIIakRZjZLySohuRtN0lm09zH8/3UlY05g2vgeXTe6NzVT/wl7XNYwFm/Ht/g7HxKvxrl2EMSUbQ+ZQDL5K/I7Mmqk3mk5RpY9gWCM13s6OnDJe+mALg3rFE+s089XaXAyqwvUX9MOgBXn9yxyCIY3bz09nAhswJmRhzhhI1eKXcPYbQ4XRRLD4EHrJfpSUIei6wvbcKp59Y23t3Xir2cBlZ/fh/95az8+vH8OgrJopRcawh8D6RZgSs7BmDKDq85dxTrkRvyOzXvtc/hB/fGMtxRU1WX8qXQH+/dE2LjqzJ99vK2T6hF6cOzq9SUGAuXwf1d/NJ+a82/Dt+R7CQQyDpxFWzFj9JRR/8H/ooQDR42YQLMvHu3c9ht2rSTvnh+z9rpTvtxfxm9vOaNI+NaoeQsn5nmDJIaJGXYxr+VtYB0wkmDIUUKj2hvhi9aF671uy5hCj+yYQDusoChiLtlE8/1kcI87DOWwKRfOfw953DCgKut9DxZf/IuGa3+Izxp36AxAiwrV7EJCUlMTDDz+M+cgOjH369CE/P7+9q9HqckvcvPPFrtrHC77dx+Ce8fTLaOLukoqCFvBQsugF9HAIa9Yg9LJDVFkScSoqu3LL+WZdLlAzHPvygi30SHXywnsb8AfCfLqxkiFnXUR0VQlhVxkVK+aiB3zY+4+neu0nePeuw5wxiF2HqtD1mlz/i5bu44qpfZn/9R6G901k/IAkEi5/kJJ3noD8XURPuZ6qr16ll89NGAhqYFU615qAoywGFUsjQ8ZCiK6toNzLq0cy8QB8/t1BhmUnMKRnQxd6CorBgGfnKnz71qP5vcSfewPBzYspX/cpSTc8RZUxgQXL9vPxigMADOmdQGqiHa8/RK/0aOZ9tQeAUFinyqsx/+v9taVv3F/FuN4GXIv/gWqxg2oASzQFEx4k9dBiAq4K1GQNf1jhtY+31ekvfYEwBwurSU2wM/frPTxywxgMCuiKimo0UvH5y0fKNNbcim9AzuHq2gDgeF+tzeW88T14+4tdjBmQdNJUorVUlUDBbkreeBTN7yH67Os4uteMz5xA3MU/pmzRC1St/qjmkzWaiRk/k0/2m9hfWFOH/QVVjOqTcMpT6YqKajDgXvcJ3q3foAX92AadXXs+g6pgMRnqrfNy2s1HXqPXTIONSceUko1r45c1LzAYsaT2pvK7D1FtTuKv+IUEAEI0UbsHAf369av9/4EDB/jkk0946623mvz+hARHW1SrxbYeqqh3zOULkZTkBCBYWYwpJolgZTFGZzyKWvcOlq47CJ9xBeUfvgBAVL+xlO9cy2qjk0tHWalw+U94Pbi8QfyBmsw4gZCGplpxbVmKY8hZVK5aAIBj9AUUvfMk6BrGkdNZ+GZRnXKOLkQrrvASZw9T/v0KQAFVxbPre+z9x1O1+iP0QdPICcQyKdHZ4s+qMUc/q0gSiW0GaXdH6az956nsPm6TxKOqvaFGP09Hr8G4MwfiO7gVU0I6YVc51d9/SNLMe4lK68H+fWW1AQDA1v2l9D8SUJyYVODExbqr97q49fzJsOEzNL8H+5mz+Tw/hv9+k8P5w8ZRuBp+NtCCIaRRWFb/Yr2wzENijI39+ZWYrUbio22Ak8DI86he8zGa30PsWT/AkdmHaGP9C3nv7pIG2+wPhjEc2fHYajU16XdNi+mLf8jZVK//HNViJ2bIREzxx6behGzDqU7pSbCw5rOyZY/Cu38DYx1lrMnIYFeeG4+/8Z/DiUKWUVTHpREqL8DaeziOXoMw2I+997oLBvDKoq21jxUFLp7Yi/j44zMcOVEnXUHRvP8DwJo1iEDhATSfi9izrsKRkU10I5updRcd3Y90lEhtd1vqsIXBu3fv5s477+Shhx6iV69eTX5faakLTet8t6KTY2yYjCrBI18gBlUhNcFGcXE11nAl5XOfImbK9VQue5v46f+D15Ze5/1WTx7lH/8VS8+haNVllC9/n5jxMzkrsAtPdV96ptb95XfYTKTE2emR4uRgYTWzz0ggZutrxJ51FSWL/oKt9wgCpbmUfvRX4s+/BdeWb/F++jw/vegn/H5hzciL9bjt5Af0iMNdUUWgNJ/4S+5FsUdTufgVVHs0ev8pKNsXM6jPEEpKEttkJCApyUlxcf0v+u4sEtsM0u7WKqs5Omv/eSpJ0VZsFmPtXWJFgYykqAY/z8QEOxWrP8V3cBuOMRfhWr+YkKsca9Zg/KX5hIrLqaz213vfURazAVWBox+TAkRZjbh9Nee+7qwUtK//hmqxYes7FvequYw65z7eNap8sbkSgPwiF8mxVsYOTGbNjro3Xgb0iOOb9bmMG5xK2B+iuLgai+aicsEcVEsUtr6jqVj6LsaUPvjjB9RvX0zDc/CT42xUuv1cdnYfLAZO+bumKArGvDVUr/+cqJHT8GxbStHHL+G44G4Cig1T2I1v1Ts1AYCigq7j2fUdjnEzCccN5NDKmmAkKcbWpN9rI0ECq94mVFmEY8zFuNZ+QsXaz2DQhbWf9bgBSThsI/lyzSGio8ycPy6L9DhrnfIt5bspWfA8isGEGhWD78BmHEPPxpI5kIql74EtFr3XmU1YA9I1Sf/ZeuWJDgoC1q5dy3333cejjz7KjBkzOqIKrS45xsLjPxzP4u8PEgxrXHBGD9LibKCD3xhL9OSrKV3wHI6R5xO01d8JMmSNJ/7iOzHEpRPyenBXlhOyR2GNTiSkKfRMiuKRm8by2Xc5xEdbOW9sFtFWI/deNYKlG/NZd6iSsefeg+o0E3/h7YRjs7CiYaguRE8dgDNjGHrJPtyWNMYNCmO3mRjeJ5FVWwp46PoxZKc4CKgKjgvuJmiIAkUhdvrd5Be72OwcwDkXD8WQ1Itw17t+EEJ0cbF2E4/dMo4v1hzC5QlywRk9yEpoeOGsohowZY8lISEDY1w6Jmc8poQsAhWFVK9aQFy/CWQkxhIfbaGsqiYYMBpUhvdJIBAIcbjExU+vG8O3G3IxGQxkpTr45TUD+XJLBRWuAP16JhE74BqwxRC2J0LqAD7apxA4cgMoKdZKrMOMCvzg/H4cLvWQW+wCYFifRAIhDUVRuOiMHrV19qsOos++FsXqJGxLxNJjCFp0WoPty0iwc/64rDrz52s2cRyI3WqgR7KjSesBdF1HScwmfsa96Bn/396dx0lVn/ke/5xau3pf6I1eAWn2HRVwAST2NTar0YhmMIbMJLnXV5wk9w4xhmtmMiZRbl5xifEmNyZmcV7jK8YtMmKcIRCNC8giKCog0DQNTdP0vtZ2fvePhoJm7dbqharv+y+quuqc8zTVT53n/H7n+U3BN/YqsG0Cju7vLSvURfB4NZbbS9ZN3ybQ2kjLy4/RVlfD68dH0ekPMWdSPiW5vRtdCuHGN2kBCaOvIJw9hqyCsVgZw/GfdqweV/c6ONNHZwHWWaMwAJY3GVdSBmlzlhFuqaN97xa8hWMI292j4q2bXiSjYAJ+d3qvjksknllmgJvw1tTUsGzZMh566CFmz57d5/cP9StZzhPz0k9PXt5QM00vPIC3YAydezaTdctquhKHn/Xe+jY/2/bUkZeZyIQRmTg5e/690+nAGNPjd2BOdM041tBBXmZid+eMyHxSQ/jEmbt1Yqj45Dag+8vjQj31T8Zj23a/3gsQj1c34jFmUNzR2tYnMdTz58U4HBaWde6Tw5NO/p6tE/cvJYQa8LsycJgw7mAzXZ7u+ev1bQHe23ecrkCYyaOGUZDlw7KsSE7szn0msj/L6t539+/PRPJhVyjMX7Ye5tXNVZTkpXD79WPITT91tT4Qsqmqa8cfDNPY2oXTYVFWlEFWcs8pK9Zp91udzNXn4w/ZVNa2sqeqieREN5MvyyYr2d2rk/8zXWhf3kADlr8Vf2oJFgZHw34q23xsPRSgrCid0ryUc96cfeH9nWhx/SnuL0sIt2DjJLDtRRInzqPLl4cr2ELXOy/gm34j/oSzL7TFCuXP6G1PBqEIuP/++3n22WcpLj51FWT58uXcdtttvXr/pfglZlk2Ce01BJNycbXXEkrIJHTGyr7tgTD3/b+3aTwx9/8fFk9g9vjcXmwc/vbeUX699gMAEhNcfPfOy8m/xPrkx2Nii8eYQXFHa1ufxKWYP/vqXL9np90JlpOw5cFjtxN0Jp12wv3pmx1YFnQFbdxOBxfqTnyhfVkWBG2D02Fh9fJ4Tm5voP+movE7i4asNDf1zcHIY5cVImRie/kj5c/obU8GYTrQ6tWrWb169UDvdlAZ46AzsQAMhBILzvp5IGRz+Hg7c2cUcriujXc+qOXI8faLXg2C7hGA3637KPK4oyvEX7ZUs6K8LOa/7EVEzsXpdERGC5x2J6HtL+FISMZXdiWNLz9G2twv4E/vbnccjZNZY+jVGirn21cwbLNt73FefH0/2ek+Pr9gNAWZiZ94e/1tKBQAAA5PAnCqCIj1AkAk2vQXM8g6A2F+9R8fsG13HQDjSjOZP72QOZPze7VcejhsEzpjaLy5PXBiOpDBckAoDC7HxQsKEZFLWXNHkG1vV+IPhMnPSqQwKwmnw4UzJYvmjb+nddMLWG4vuIfWSOmewy384oX3ATha38G+w8388KtzSPXpK1pE+o8aqw+yg7VtkQIA4MPKBqaNyen1dJ6MFC/XzTy1oIxlQfkVxYTDNo3tAZ7+y8fc98QmXt5cRdsZ/ZdFRGJFyDZ8cLCRjVur+eWL7/PYH3fy7v56QpYbz4ipWO4ETChA4rg5hJOGzpxxy4KPq5t6PNfRFeJ489ntRUVEokmXGQaZPxg+67lQ+OznztTSGaLqWCsel5NFV49g8qhsmtv9lOSmUJSdhG0M//bq7kiB8cz6vWDgxiuLNSIgIjGnpTPIurcqqT7WRrKvu53o48/u5Gd3zyCw/v9iudwkTy+nddOfyCoch5U/NarTWtzBZhwtRwhlleFqPwrGxp9cdNH3GQPFuT3nJ7ucDjJSvNE7OBGRc1ARMMhK8lJITfLQ0h4AICXRTUnehVcZbmgL8MPfvkPDiV7XM8fm8PcLx+M5bU5qmz/MgcPNFA1L4NDxLsYUJvP2rhquv7wQ13lWohQRuVQ1tfmpPtZGWrKbVfM8BC0Pf620cB/aRur8FYQtF6GkXIYNL8POKI5qAWBZFtTu5vjLj5N61S00bF2HO7uI5Bv+kYB17r7+pxtbnM6yuaN46W/7SUvy8pWlE8lIiu0Fr0Rk8KkIGGTpiW7+952Xs/tQEwaYMjqbZPeFZ2ntrW6KFAAAWz46xo1zSinNOdWv2ed18N3PeLA7mtncXsBV7g8wwyfhdlhD5qYuEZFoCNk2gZDN4mtGMj7PRcpH/w61e1laOJnOv24hcen/JJg7CWMgnD0x6vs3xmAKppAwYS4tbzyDIyEJz9yVBB0JvWrb6XM7WTSnhOtmFOJyWnidmqkrIv1PRcAAuVBLtawUL3NOtAM9sw3W6R2CTv7bPseGzuwE5DUh0v01tG95nrnDSrGPV5I2vAiDoXsNzMExVFrLiUhsMMB/bT3MH9bvBeBVj5MH7/g7rJf+BSq3kDhzIeHcCRfNO582N9ktxwjs2woOF7a/k/rdO3CVXU1a0sVHAk4GkuSJvVVue9PlTkQGh4qAftbUEeS9/fU0tvqZMCKTEXnJOHpxEt7eFWLXwUYO17UxtSyb9s4gew41UZqXyojhaST53LR3drdGG1eayfBhST3eH3Z4SJx8Pf4P/0boeCW+sstxlswgaAanAGjpDPH+gXrqmjqZNHIYpblJODQtSUQ+pca2AH/8y97I44lFPhwfv4Yd7MKdXYhduw9XUxXhtFK8/uMQ6CCUXoLn+EeQkErAm0VlbQs7DraTluRl0qgs0hPdfToGy4JgezMmOQt71p0MsxppPHIEn78WLy78SWe3ho51YWM4cLSN9/fXk5fpY8KILFISdMohMpToLzLKDBAM2SR4nHQEwjz6zLtU1nRf2X/xtX38w9JJTCzJIPkCydAAz762j43bDuPzunA6Hbzw132Rn980bxTf//srOVDTgsftZEReKgln9Kh2mgBd764j1FKHp6CMzj3vkDByOlbJLMwAFwJB2/DLP+1i14F6AF58bT/3fvFyLsvXYh0i0pPDYbBti+4ux2EM5786HrINbf4Qpw+EdgQMeJNh8b+wsQp2flzP1YfczEgMYG9bS8fut0m/ZjnHNj5F6rRyavOu4p9/vzsyClCcm8w9K2aelVMvxBiodpWwv2A5M1qO0vDaL3BdcQtdr/6UkC+V1MX/hN8aWm1J+9vHh1t48KmtkcfTx2bz35dMxKmLPyJDhoqAKKpr8fPMho/5uLqJz8wsYvLoYZECALq/KPZVN+HAcMWY87eoa2oP8tfthwGYdNkw3n6/psfP//T6fuZPL2T6ZcPOu42w5cFTPIGMrEIcxVMJfbgBZ1YxoUEYCTje3BUpAE56670jlBWM1YJmIhLh9dcT2PMGngkLoOkIdnsDdtFM7HMUAu3+EL97ZTcNrV3MnpTPW+9158mPDnfgXzyPZ/96kLfeP9r9XFUTQdtB+RXL8B/6iMb1v8GTfxkGw+bd9T2mAVXVtnG0oaPHPVa9kZ+ZxNu73PyxxsPnRl1FaPMzWO4E0pauoivOCgCn08GG7dU9ntv2UR318/3kpPVyepSI9DsVAVESCNs8+swODte1AfDHDR/j9bhI8rlo7zzVnz/B42JfdROzxuWe9wTY43KQ6HXR3hWiuc1PZmoCR+s7Ij9PT/bivtDa9CePKWssVpZNCAeO8eX4zcDfbBYMGyzL4pqpw3nrvaORhc2yMxI1T1REeupooOXtF/AcfJ9g7QE8xeNJGj4B23n2CfnHh1t458NaADJTElg27zIyUryMK83E6bQiBcBJ//XOIeaPKCDUchyAYH01SWOuJDN4dl70efr+1ehxObh57igcbTV0vLwbGzDBLkLH9uMozjpnIROrjDHkZfacoupxOfC64+d3IHIpUAuCKGluD0YKgJPe+bCWlQvHc3L0s/DElaVJo7IveAU8OcHJV5ZOwuW02H2wkcvH55J0YvqQx+XgK0sn9WgHeiHmxH+xPQgFwOGGDn74+y3c+/M32XOwkeXXl5Ge7CU/K4krxuXqBmER6SGYNZq0a24lcGQPxhjS5t9J8BwFAEDbiXuioDvXPr/xY4alJTC2NBOfy0FeVmKP15cVpcHxSnwjp5J354O4M4eDZTFt/HBKc7tfa1lw+/VjyE77ZD36XU6LhGALDoeTnC+uIXnGjfgPfYgj7L/4m2OIbRuunpxPdnr3VX+Hw+LLiyeQntS3ey1EpH9Z5hK7HFtf3zYkp5D4wzb3/XITdU2nVnm87foxXD9zOFV1HVTVttLQ3MXw7GQmj8zEc54WcJHuQBbUtwZobQ+Qk+EjbBuON3eRkewh/RLoH93SGeTeX7xFR9epURCnw+K7X7qc3HQfvjOuCJ3ZFSkexGPMoLijta1PYqjmz5O8zfup/+OPcKZlE2o8im/MLHxXfYGgM+ms1x5u6OT//NtWFlzevUK61+NkzsQ8LivOpK6ulUP1HTz89HYaW/2U5qfyP26aTG4yOMJdBF2p+AJ1hNypBC0v4UAn1c02Po+T7DQv1qfooGZZBm+olS5nKi67E8sOE3T1bWpRXw3Vv6nOQJjapk6SfW6GpXiIdme6oRp3f4rHmCH6cX/SHBprNB0oSrxOB99cPo1fr/2Ayppm5s8oYtaEXDAWxcOSKMlJ7r7y3duay0BWsoes5FMn/CkJ/ftFEk1Hjnf0KAAAwrahvrmL0uxLJw4RGUDeZJImziVh5hJMfRXhlmPYjnNflS/I8vH1W6aw5qmtBEPd0wxDIZuS4ekAFGUlcv9XZtHWGSI92YPbYREGwq7unNrpyY5sy+nxUZJ95h4+GWMsupzdCz6GHL64Hm/3eZx9vrdCRAaOioAoyktP4NtfmI4/GMbncUauefhDNofq2uj0hyjOSSGtj+3nLkXu80xX8ro0J1REzs2fkIN71m34jRMrdzxWzhjs83xNOSyLN9+riRQAcyblY4DXtlczPCuJo/VtJHhdFGUn9+oeKhGReKMiIMqcFiSetuBLyDY8vX5vpNtPSqKb1V+6guyUTzbn9FJRmJ3E6OJ09lY1RZ7LSfdRkqchOBE5v5Dpzp/GWJiLfEWdnNo0a2Ie9S1dvHmiQ1BWWgKzJ+Wz9m8HmD+9kNuuH41LrSlFRHqI44HKgVHb2BkpAABaO4K8v7/+Au+IDV6Xg7uWTeKOz45l4sgsbv3MaFb93QwtFiMiUWHbhutmFOFyWuRlJbH7YGPkZ/XNXbhO3He1YVs1tY1dg3WYMkQ57S4cdI8ieUx83bgtcpLOyPqZfY57AMLhoXtjXjSl+tzMnzqcz8woJBw2agkqIlFVnJ3Ev35lNofr2i/4OuUeOZ3L7iT07lrcOaV4ckfSsv7XpFz7BfyJwwf70EQGlIqAfpaX4WPm2By2fHQMgASPk0kjswb5qAaOMd0364mIRJsxhty0BFITPYwuTGNvdTMAqUmeSA+Gy8flkpveu8W6LMvgCPsJOxJw2n5shwcT5Y42MgQYgx0O0fDy4zh8SZHnROKNioB+5nY6uPPGscybXkinP0RJXgrZqV7lGxGRKPG5HXz95ikcONqCwSI73UfN8Tb+1+3TKc1Lwe3szYm8wV37AV0Hd5I4czFdO9bhzh9NKG+yCoEYE3ImkjS1nPZ3/xO7s420a28nmJQ/2IclMuBUBAyARI+L8cXpkccqAEREois5wcWk0sxIP/H8EwtV9ZaFwQS7aN/+Kl17NhFubyLjhq8BNsTRar/xwBtupfE/HsWRkEhCyWSaX/t3huWOwJ8xerAPTWRAqQgQEZF+4TAhnLafoDMJd7idsMOLbQ29rx3LAne4EwomkjxlAaHGIziS0nEWTyHUDwVAbXMX1cfaSE30UJybjLeXK8BLdARcyaTOuQUrKQPbl4m3dBJ2cu5gH5bIgBt62VhERIYEb8cRcHq6hy/tYJ9unLSwcVS9Q2flTlKuvpXWvz1NQukUTMmVmDMa09U2d/He/nqy03yMK0k/74rq/cGywF2/h9Y3/kD61TeDHcSVkQ8NR/C/uw7ntCWETfQKgUPHO/jXJzcROtEgomJOKTddO+JTrVIsfWOMRTB3YmRU3iqZo5vHJS6pCBARkbM4LAgd3Uvb1nUAJM/4LI5Rw7F7vei5A0diGp17N9O1byvGDpM4/lpCZxQArV0hfvjbd2jtCALwj5+fypSRmVGN5YLHacDyJBJqrKHu+Z9gQgEyPvMlvCOnYyWm449iAeBwWGzcXh0pAABefquSudMKGZbiucA7JdpOP+dXASDxSmOQIiJyFtuAp3AC4dYGwq0NeAon9LoAiMgqwT2sGBMK4B5WDFklZ72k0x+KFAAAxxo7P+WR952dlIOv7EpMKIDDm4iraBLBnIn4kwqivi/HGYuWWXSPRoiIDDQVASIichaHBf4D20iZ/TlSZn8O/4HtOPpwsuq0bAI71hE8XkXqVTcTPF5FYMc6nFbPlsEZyR7+25XdxUGSz82kUQPbQtmygCM7ad/5F1JmLQHLouU/f4nb7oj6vmzbMG96Ae7T7gFYfM1IsjQKICKDQNOBRETkLLYB1+hrMCdv5DUh+rLOYdg48I6bi7d0MsGsMobll0FiBn7T89qT2+ngpmtHMG9aAYkJrgFfVdwYsHJGM2zZPxHKGUvmyMvBhPFbvVtboK8KsxL5wVdnc+hYG6lJHoqyk0CzUURkEKgIEBGRcwo5TjsRttx9fr8/IRsSssEGf2bZeV/ndjrI7WNLz2gKuNMgOw0MhJML+3VfxsCwFC/DUrz9uh8RkYvRdCARERERkTijIkBEREREJM6oCBARERERiTMqAkRERERE4oyKABERERGROKMiQEREREQkzqgIEBERERGJMyoCRERERETizCW3WJijL+vWX6LiIcZzice44zFmUNzxuv+BEi9xni4eY4b4jDseY4b4jbs/WcYYLVguIiIiIhJHNB1IRERERCTOqAgQEREREYkzKgJEREREROKMigARERERkTijIkBEREREJM6oCBARERERiTMqAkRERERE4oyKABERERGROKMiQEREREQkzqgIGGRtbW0sXLiQ6upqAN58800WLVpEeXk5Dz300CAfXf947LHHqKiooKKigjVr1gDxEfcjjzzCjTfeSEVFBU8++SQQH3EDPPjgg9xzzz1AfMS8YsUKKioqWLJkCUuWLGHHjh1xEfdgiLccqvyp/BnrMSt/DiAjg+bdd981CxcuNBMmTDCHDh0ynZ2dZu7cuaaqqsoEg0GzcuVKs3HjxsE+zKh64403zK233mr8fr8JBALmjjvuMC+99FLMx71p0yazfPlyEwwGTWdnp5k/f7758MMPYz5uY4x58803zZVXXmm+/e1vx8Vn3LZtc/XVV5tgMBh5Lh7iHgzxlkOVP5U/Yz1m5c+BpZGAQfSHP/yB733ve+Tk5ACwc+dOSkpKKCoqwuVysWjRIl555ZVBPsroys7O5p577sHj8eB2uxk1ahSVlZUxH/cVV1zB7373O1wuF/X19YTDYVpaWmI+7qamJh566CG+9rWvAfHxGd+/fz8AK1euZPHixTz11FNxEfdgiLccqvyp/BnrMSt/DiwVAYPoBz/4ATNnzow8PnbsGNnZ2ZHHOTk51NbWDsah9ZvRo0czdepUACorK1m3bh2WZcV83ABut5tHH32UiooKZs+eHRf/3/fddx/f/OY3SU1NBeLjM97S0sLs2bP52c9+xm9+8xuefvppjhw5EvNxD4Z4y6HKn8qfsR6z8ufAUhEwhNi2jWVZkcfGmB6PY8nevXtZuXIlq1atoqioKG7ivvvuu3nrrbeoqamhsrIypuN+5plnyM/PZ/bs2ZHn4uEzPm3aNNasWUNKSgqZmZncfPPNPProozEf91AQD58vUP5U/uwWazGD8udAcw32AcgpeXl51NXVRR7X1dVFhrljydatW7n77ru59957qaioYPPmzTEf9759+wgEAowbNw6fz0d5eTmvvPIKTqcz8ppYi/vll1+mrq6OJUuW0NzcTEdHB4cPH47pmAG2bNlCMBiMfHkbYygoKIj5z/hQEA85VPlT+fOkWIsZlD8HmkYChpApU6Zw4MABDh48SDgcZu3atVx77bWDfVhRVVNTw1133cWPf/xjKioqgPiIu7q6mtWrVxMIBAgEAqxfv57ly5fHdNxPPvkka9eu5cUXX+Tuu+/muuuu44knnojpmAFaW1tZs2YNfr+ftrY2nn/+eb71rW/FfNxDQaznEuVP5c9YjhmUPweaRgKGEK/XywMPPMDXv/51/H4/c+fO5YYbbhjsw4qqX/3qV/j9fh544IHIc8uXL4/5uOfOncvOnTtZunQpTqeT8vJyKioqyMzMjOm4zxQPn/H58+ezY8cOli5dim3b3H777UybNi3m4x4KYv3zpfyp/Bnr/9fKnwPLMsaYwT4IEREREREZOJoOJCIiIiISZ1QEiIiIiIjEGRUBIiIiIiJxRkWAiIiIiEicUREgIiIiIhJnVASIiIiIiMQZFQES01auXElDQ8Onfs2mTZtYuHDhRfc3ZsyYc25r/fr13H///QCsWLGCV155herqaqZNm3bRbYqIDAblT5HYpsXCJKa98cYbUXnNp7VgwQIWLFjQ7/sREYkW5U+R2KaRAIlZ3/nOdwD44he/yObNm1mxYgWLFi1i8eLFvPDCC2e9pqamhg0bNrB8+XJuuukm5s2bx8MPP9zn/T788MMsW7aMJUuWsGHDBgCee+45vvrVr0YlLhGR/qb8KRL7NBIgMetHP/oRzz33HL/97W/5/Oc/z6pVqygvL6e2tpZbbrmFkpKSHq/JyMhg1apVPPDAA5SWllJbW8v8+fO54447+rTfwsJCvv/977Nnzx5WrFjBunXr+ilCEZH+ofwpEvtUBEjM27dvH36/n/LycgByc3MpLy/n9ddf7zGn1LIsfv7zn7Nx40bWrl3Lvn37MMbQ2dnZp/3ddtttAJSVlTFq1Ci2b98evWBERAaQ8qdI7NJ0IIl5lmVhWVaP54wxhEKhHs91dHSwbNkydu3axfjx41m1ahUulwtjTJ/253Cc+rOybRuXS7W2iFyalD9FYpeKAIlpTqeTgoICXC4Xr776KgC1tbX8+c9/Zs6cOZHXhEIhDh48SFtbG9/4xje47rrr2LRpE4FAANu2+7TP559/HoBdu3ZRVVXFlClTohuUiMgAUP4UiW0qsSWm3XDDDdx55508/vjj3H///fz0pz8lHA5z1113MWvWrMhrVqxYwSOPPMK8efP47Gc/i8fjoaysjMsuu4yDBw/i8Xh6vc9Dhw6xdOlSLMviJz/5Cenp6f0UnYhI/1H+FIltlunrWJ2IiIiIiFzSNBIg0gdPPPEEL7300jl/9uUvf5nFixcP8BGJiFwalD9FhhaNBIiIiIiIxBndGCwiIiIiEmdUBIiIiIiIxBkVASIiIiIicUZFgIiIiIhInFERICIiIiISZ/4/+yMhMUUXWKIAAAAASUVORK5CYII=",
"text/plain": [
"
"
],
"text/plain": [
" VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count \\\n",
"0 1.0 2021-01-01 00:30:10 2021-01-01 00:36:12 1.0 \n",
"1 1.0 2021-01-01 00:51:20 2021-01-01 00:52:19 1.0 \n",
"2 1.0 2021-01-01 00:43:30 2021-01-01 01:11:06 1.0 \n",
"3 1.0 2021-01-01 00:15:48 2021-01-01 00:31:01 0.0 \n",
"4 2.0 2021-01-01 00:31:49 2021-01-01 00:48:21 1.0 \n",
"... ... ... ... ... \n",
"1369760 NaN 2021-01-25 08:32:04 2021-01-25 08:49:32 NaN \n",
"1369761 NaN 2021-01-25 08:34:00 2021-01-25 09:04:00 NaN \n",
"1369762 NaN 2021-01-25 08:37:00 2021-01-25 08:53:00 NaN \n",
"1369763 NaN 2021-01-25 08:28:00 2021-01-25 08:50:00 NaN \n",
"1369764 NaN 2021-01-25 08:38:00 2021-01-25 08:50:00 NaN \n",
"\n",
" trip_distance RatecodeID store_and_fwd_flag PULocationID \\\n",
"0 2.10 1.0 N 142 \n",
"1 0.20 1.0 N 238 \n",
"2 14.70 1.0 N 132 \n",
"3 10.60 1.0 N 138 \n",
"4 4.94 1.0 N 68 \n",
"... ... ... ... ... \n",
"1369760 8.80 NaN NaN 135 \n",
"1369761 5.86 NaN NaN 42 \n",
"1369762 4.45 NaN NaN 14 \n",
"1369763 10.04 NaN NaN 175 \n",
"1369764 4.93 NaN NaN 248 \n",
"\n",
" DOLocationID payment_type fare_amount extra mta_tax tip_amount \\\n",
"0 43 2.0 8.00 3.00 0.5 0.00 \n",
"1 151 2.0 3.00 0.50 0.5 0.00 \n",
"2 165 1.0 42.00 0.50 0.5 8.65 \n",
"3 132 1.0 29.00 0.50 0.5 6.05 \n",
"4 33 1.0 16.50 0.50 0.5 4.06 \n",
"... ... ... ... ... ... ... \n",
"1369760 82 NaN 21.84 2.75 0.5 0.00 \n",
"1369761 161 NaN 26.67 2.75 0.5 0.00 \n",
"1369762 106 NaN 25.29 2.75 0.5 0.00 \n",
"1369763 216 NaN 28.24 2.75 0.5 0.00 \n",
"1369764 168 NaN 20.76 2.75 0.5 0.00 \n",
"\n",
" tolls_amount improvement_surcharge total_amount \\\n",
"0 0.0 0.3 11.80 \n",
"1 0.0 0.3 4.30 \n",
"2 0.0 0.3 51.95 \n",
"3 0.0 0.3 36.35 \n",
"4 0.0 0.3 24.36 \n",
"... ... ... ... \n",
"1369760 0.0 0.3 25.39 \n",
"1369761 0.0 0.3 30.22 \n",
"1369762 0.0 0.3 28.84 \n",
"1369763 0.0 0.3 31.79 \n",
"1369764 0.0 0.3 24.31 \n",
"\n",
" congestion_surcharge \n",
"0 2.5 \n",
"1 0.0 \n",
"2 0.0 \n",
"3 0.0 \n",
"4 2.5 \n",
"... ... \n",
"1369760 0.0 \n",
"1369761 0.0 \n",
"1369762 0.0 \n",
"1369763 0.0 \n",
"1369764 0.0 \n",
"\n",
"[1369765 rows x 18 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Faster Append with Modin's ``concat``"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our previous ``read_csv`` example operated on a relatively small dataframe. In the following example, we duplicate the same taxi dataset 100 times and then concatenate them together.\n",
"\n",
"Please note that this quickstart notebook is assumed to be run on a machine that has enough memory in order to be able to perform the operations both with pandas and Modin in a single pipeline (which at least doubles the amount of required memory). If your machine doesn't have enough resources to execute every cell of the notebook and you see an OOM issue, you most likely need to reduce ``N_copies`` in the cell below."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time to concat with pandas: 34.144 seconds\n"
]
}
],
"source": [
"N_copies= 100\n",
"start = time.time()\n",
"\n",
"big_pandas_df = pandas.concat([pandas_df for _ in range(N_copies)])\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to concat with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time to concat with Modin: 0.564 seconds\n"
]
},
{
"data": {
"text/markdown": [
"### Modin is 60.57x faster than pandas at `concat`!"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"start = time.time()\n",
"\n",
"big_modin_df = pd.concat([modin_df for _ in range(N_copies)])\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to concat with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `concat`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result dataset is around 19GB in size."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(apply_list_of_funcs pid=73415)\u001b[0m \n",
"\u001b[2m\u001b[36m(apply_list_of_funcs pid=73416)\u001b[0m \n",
"\n",
"Int64Index: 136976500 entries, 0 to 1369764\n",
"Data columns (total 18 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- --------------------- ------------------ ----- \n",
" 0 VendorID 127141300 non-null float64\n",
" 1 tpep_pickup_datetime 136976500 non-null datetime64[ns]\n",
" 2 tpep_dropoff_datetime 136976500 non-null datetime64[ns]\n",
" 3 passenger_count 127141300 non-null float64\n",
" 4 trip_distance 136976500 non-null float64\n",
" 5 RatecodeID 127141300 non-null float64\n",
" 6 store_and_fwd_flag 127141300 non-null object\n",
" 7 PULocationID 136976500 non-null int64\n",
" 8 DOLocationID 136976500 non-null int64\n",
" 9 payment_type 127141300 non-null float64\n",
" 10 fare_amount 136976500 non-null float64\n",
" 11 extra 136976500 non-null float64\n",
" 12 mta_tax 136976500 non-null float64\n",
" 13 tip_amount 136976500 non-null float64\n",
" 14 tolls_amount 136976500 non-null float64\n",
" 15 improvement_surcharge 136976500 non-null float64\n",
" 16 total_amount 136976500 non-null float64\n",
" 17 congestion_surcharge 136976500 non-null float64\n",
"dtypes: float64(13), datetime64[ns](2), int64(2), object(1)\n",
"memory usage: 19.4 GB\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"UserWarning: Distributing object. This may take some time.\n"
]
}
],
"source": [
"big_modin_df.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Faster ``apply`` over a single column\n",
"\n",
"The performance benefits of Modin becomes aparent when we operate on large gigabyte-scale datasets. For example, let's say that we want to round up the number across a single column via the ``apply`` operation. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time to apply with pandas: 43.969 seconds\n"
]
}
],
"source": [
"start = time.time()\n",
"rounded_trip_distance_pandas = big_pandas_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to apply with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time to apply with Modin: 1.225 seconds\n"
]
},
{
"data": {
"text/markdown": [
"### Modin is 35.88x faster than pandas at `apply` on one column!"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"start = time.time()\n",
"\n",
"rounded_trip_distance_modin = big_modin_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to apply with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `apply` on one column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Summary\n",
"\n",
"Hopefully, this tutorial demonstrated how Modin delivers significant speedup on pandas operations without the need for any extra effort. Throughout example, we moved from working with 100MBs of data to 20GBs of data all without having to change anything or manually optimize our code to achieve the level of scalable performance that Modin provides.\n",
"\n",
"Note that in this quickstart example, we've only shown ``read_csv``, ``concat``, ``apply``, but these are not the only pandas operations that Modin optimizes for. In fact, Modin covers [more than 90% of the pandas API](https://github.com/modin-project/modin/blob/main/README.md#pandas-api-coverage), yielding considerable speedups for many common operations."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/spreadsheet/requirements.txt
================================================
ray==1.1.0
git+https://github.com/modin-project/modin
git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5
================================================
FILE: examples/spreadsheet/tutorial.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## modin.spreadsheet\n",
"`modin.spreadsheet` is a Jupyter notebook widget that allows users to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, edit data and export reproducible code. \n",
"\n",
"This tutorial will showcase how to use `modin.spreadsheet`. Before starting, please install the required packages using `pip install -r requirements.txt` in the current directory. Then just run the cells; no editing required!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Please install the required packages using `pip install -r requirements.txt` in the current directory\n",
"# For all ways to install Modin see official documentation at:\n",
"# https://modin.readthedocs.io/en/latest/installation.html\n",
"import modin.pandas as pd\n",
"import modin.spreadsheet as mss"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create a Modin DataFrame\n",
"The following cells creates a DataFrame using a NYC taxi dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"columns_names = [\n",
" \"trip_id\", \"vendor_id\", \"pickup_datetime\", \"dropoff_datetime\", \"store_and_fwd_flag\",\n",
" \"rate_code_id\", \"pickup_longitude\", \"pickup_latitude\", \"dropoff_longitude\", \"dropoff_latitude\",\n",
" \"passenger_count\", \"trip_distance\", \"fare_amount\", \"extra\", \"mta_tax\", \"tip_amount\",\n",
" \"tolls_amount\", \"ehail_fee\", \"improvement_surcharge\", \"total_amount\", \"payment_type\",\n",
" \"trip_type\", \"pickup\", \"dropoff\", \"cab_type\", \"precipitation\", \"snow_depth\", \"snowfall\",\n",
" \"max_temperature\", \"min_temperature\", \"average_wind_speed\", \"pickup_nyct2010_gid\",\n",
" \"pickup_ctlabel\", \"pickup_borocode\", \"pickup_boroname\", \"pickup_ct2010\",\n",
" \"pickup_boroct2010\", \"pickup_cdeligibil\", \"pickup_ntacode\", \"pickup_ntaname\", \"pickup_puma\",\n",
" \"dropoff_nyct2010_gid\", \"dropoff_ctlabel\", \"dropoff_borocode\", \"dropoff_boroname\",\n",
" \"dropoff_ct2010\", \"dropoff_boroct2010\", \"dropoff_cdeligibil\", \"dropoff_ntacode\",\n",
" \"dropoff_ntaname\", \"dropoff_puma\",\n",
" ]\n",
"parse_dates=[\"pickup_datetime\", \"dropoff_datetime\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv('s3://modin-datasets/trips_data.csv', names=columns_names,\n",
" header=None, parse_dates=parse_dates)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generate a spreadsheet widget with the DataFrame\n",
"`mss.from_dataframe` takes in a DataFrame, optional configuration options, and returns a `SpreadsheetWidget`, which contains all the logic for displaying the spreadsheet view of the DataFrame. The object returned will not be rendered unless displayed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"spreadsheet = mss.from_dataframe(df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Displaying the Spreadsheet\n",
"The widget is displayed when the widget is returned by an input cell or passed to the `display` function e.g. `display(spreadsheet)`. When displayed, the SpreadsheetWidget will generate a transformation history cell that contains a record of the transformations applied to the DataFrame unless the cell already exists or the feature is disabled.\n",
"\n",
"### Basic Usage\n",
"`from_dataframe` creates a copy of the input DataFrame, so changes do not alter the original DataFrame.\n",
"\n",
"**Filter** - Each column can be filtered according to its datatype using the filter button to the right of the column header. Any number of columns can be filtered simultaneously.\\\n",
"**Sort** - Each column can be sorted by clicking on the column header. Assumptions on the order of the data should only be made according to the latest sort i.e. the 2nd last sort may not be in order even if grouped by the duplicates in the last sorted column.\\\n",
"**Cell Edit** - Double click on a cell to edit its value.\\\n",
"**Add Row**(toolbar) - Click on the `Add Row` button in the toolbar to duplicate the last row in the DataFrame.\\\n",
"**Remove Row**(toolbar) - Select row(s) on the spreadsheet and click the `Remove Row` button in the toolbar to remove them.\\\n",
"**Reset Filters**(toolbar) - Click on the `Reset Filters` button in the toolbar to remove all filters on the data.\\\n",
"**Reset Sort**(toolbar) - Click on the `Reset Sort` button in the toolbar to remove any sorting on the data.\n",
"\n",
"### Transformation History and Reproducible Code\n",
"The widget records the history of transformations, such as filtering, that occur on the spreadsheet. These transformations are updated in the `spreadsheet transformation history` cell as they happen and can be easily copied for reproducibility. The history can be cleared using the `Clear History` button in the toolbar.\n",
"\n",
"**Try making some changes to the spreadsheet!**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"spreadsheet"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exporting Changes\n",
"`to_dataframe` takes in a `SpreadsheetWidget` and returns a copy of the DataFrame reflecting the current state of the UI on the widget. Specifically, any filters, edits, or sorts will be applied on the returned Dataframe.\n",
"\n",
"**Export a DataFrame after making some changes on the spreadsheet UI**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"changed_df = mss.to_dataframe(spreadsheet)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"changed_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## SpreadsheetWidget API\n",
"The API on `SpreadsheetWidget` allows users to replicate some of the functionality on the GUI, but also provides other functionality such as applying the transformation history on another DataFrame or getting the DataFrame that matches the spreadsheet state like `to_dataframe`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Duplicates the `Reset Filters` button\n",
"spreadsheet.reset_filters()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Duplicates the `Reset Sort` button\n",
"spreadsheet.reset_sort()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"qgrid6f69f373-ae0e-423e-8e26-429f52e1669d": true
},
"outputs": [],
"source": [
"# Duplicates the `Clear History` button\n",
"spreadsheet.clear_history()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Gets the modified DataFrame that matches the changes to the spreadsheet\n",
"# This is the same functionality as `mss.to_dataframe`\n",
"spreadsheet.get_changed_df()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Retrieving and Applying Transformation History \n",
"The transformation history can be retrieved as a list of code snippets using the `get_history` API. The `apply_history` API will apply the transformations on the input DataFrame and return the resultant DataFrame."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"spreadsheet.get_history()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"another_df = df.copy()\n",
"spreadsheet.apply_history(another_df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional Example\n",
"Here is another example of how to use `from_dataframe` with configuration options."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mss.from_dataframe(df, show_toolbar=False, grid_options={'forceFitColumns': False, 'editable': False, 'highlightSelectedCell': True})"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: examples/tutorial/README.md
================================================
# Modin tutorial Jupyter Notebooks
Tutorial for how to use different features of Modin.
================================================
FILE: examples/tutorial/jupyter/README.md
================================================
# Jupyter notebook examples to run with Modin
Currently we provide tutorial notebooks for the following execution backends:
- [PandasOnRay](https://modin.readthedocs.io/en/latest/development/using_pandas_on_ray.html)
- [PandasOnDask](https://modin.readthedocs.io/en/latest/development/using_pandas_on_dask.html)
- [PandasOnMPI through unidist](https://modin.readthedocs.io/en/latest/development/using_pandas_on_mpi.html)
## Creating a development environment
To get required dependencies for `PandasOnRay`, `PandasOnDask` and `PandasOnUnidist` Jupyter Notebooks
you should create a development environment with `pip`
using `requirements.txt` file located in the respective directory:
```bash
pip install -r execution/pandas_on_ray/requirements.txt
```
to install dependencies needed to run notebooks with Modin on `PandasOnRay` execution or
```bash
pip install -r execution/pandas_on_dask/requirements.txt
```
to install dependencies needed to run notebooks with Modin on `PandasOnDask` execution or
```bash
pip install -r execution/pandas_on_unidist/requirements.txt
```
to install dependencies needed to run notebooks with Modin on `PandasOnUnidist` execution.
**Note:** Sometimes pip is installing every version of a package. If you encounter that issue,
please install every package listed in `requirements.txt` file individually with `pip install `.
## Run Jupyter Notebooks
A Jupyter Notebook server can be run from the current directory as follows:
```bash
jupyter notebook
```
Navigate to a concrete notebook (for example, to the `execution/pandas_on_ray/local/exercise_1.ipynb`).
**Note:** Since there are some specifics regarding the run of jupyter notebooks with the `Unidist` engine,
refer to [PandasOnUnidist](https://github.com/modin-project/modin/blob/main/examples/tutorial/jupyter/execution/pandas_on_unidist/README.md) document
to get more information on the matter.
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/Dockerfile
================================================
FROM continuumio/miniconda3
RUN conda install -c conda-forge psutil setproctitle
RUN pip install -r requirements-dev.txt
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/cluster/exercise_5.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 5: Setting up cluster environment\n",
"\n",
"**GOAL**: Learn how to set up a Dask cluster for Modin, connect Modin to a Dask cluster and run pandas queries on a cluster.\n",
"\n",
"**NOTE**: This exercise has extra requirements. Read instructions carefully before attempting. \n",
"\n",
"**This exercise instructs users on how to start a 500+ core Dask cluster, and it is not shut down until the end of exercise.**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Often in practice we have a need to exceed the capabilities of a single machine. Modin works and performs well \n",
"in both local mode and in a cluster environment. The key advantage of Modin is that your python code does not \n",
"change between local development and cluster execution. Users are not required to think about how many workers \n",
"exist or how to distribute and partition their data; Modin handles all of this seamlessly and transparently.\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Extra requirements for AWS authentication\n",
"\n",
"First of all, install the necessary dependencies in your environment:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install dask_cloudprovider[aws]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The next step is to setup your AWS credentials, namely, set ``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``\n",
"and ``AWS_SESSION_TOKEN`` (Optional) (refer to [AWS CLI environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) to get more insight on this):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"os.environ[\"AWS_ACCESS_KEY_ID\"] = \"\"\n",
"os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"\"\n",
"os.environ[\"AWS_SESSION_TOKEN\"] = \"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Starting and connecting to the cluster\n",
"\n",
"This example starts 1 scheduler node (m5.24xlarge) and 6 worker nodes (m5.24xlarge), 576 total CPUs. Keep in mind the scheduler node manages cluster operation but doesn't perform any execution.\n",
"\n",
"You can check the [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/on-demand/) page.\n",
"\n",
"Dask cluster can be deployed in different ways (refer to [Dask documentaion](https://docs.dask.org/en/latest/deploying.html) to get more information about it), but in this tutorial we will use the ``EC2Cluster`` from [dask_cloudprovider](https://cloudprovider.dask.org/en/latest/) to create and initialize a Dask cluster on Amazon Web Service (AWS).\n",
"\n",
"**Note**: EC2Cluster uses a docker container to run the scheduler and each of the workers. Probably you need to use another docker image depending on your python version and requirements. You can find more docker-images on [daskdev](https://hub.docker.com/u/daskdev) page.\n",
"\n",
"In the next cell you can see how the EC2Cluster is being created. Set your ``key_name`` and modify AWS settings as required before running it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from dask_cloudprovider.aws import EC2Cluster\n",
"\n",
"n_workers = 6\n",
"cluster = EC2Cluster(\n",
" # AWS parameters\n",
" key_name = \"\", # set your keyname\n",
" region = \"us-west-2\",\n",
" availability_zone = [\"us-west-2a\"],\n",
" ami = \"ami-0387d929287ab193e\",\n",
" instance_type = \"m5.24xlarge\",\n",
" vpc = \"vpc-002bd14c63f227832\",\n",
" subnet_id = \"subnet-09860dafd79720938\",\n",
" filesystem_size = 200, # in GB\n",
"\n",
" # DASK parameters\n",
" n_workers = n_workers,\n",
" docker_image = \"daskdev/dask:latest\",\n",
" debug = True,\n",
" security=False,\n",
")\n",
"\n",
"scheduler_adress = cluster.scheduler_address\n",
"print(f\"Scheduler IP address of Dask cluster: {scheduler_adress}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After creating the cluster you need to connect to it. To do this you should put the ``EC2Cluster`` instance or the scheduler IP address in ``distributed.Client``.\n",
"\n",
"When you connect to the cluster, the workers may not be initialized yet, so you need to wait for them using ``client.wait_for_workers``.\n",
"\n",
"Then you can call ``client.ncores()`` and check which workers are available and how many threads are used for each of them."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from distributed import Client\n",
"\n",
"client = Client(cluster)\n",
"# Or use an IP address connection if the cluster instance is unavailable:\n",
"# client = Client(f\"{scheduler_adress}:8687\")\n",
"\n",
"client.wait_for_workers(n_workers)\n",
"client.ncores()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After successful initialization of the cluster, you need to configure it.\n",
"\n",
"You can use plugins to install any requirements into workers:\n",
"* [InstallPlugin](https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.InstallPlugin)\n",
"* [PipInstall](https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.PipInstall)\n",
"* [CondaInstall](https://distributed.dask.org/en/stable/plugins.html#distributed.diagnostics.plugin.CondaInstall).\n",
"\n",
"You have to install Modin package on each worker using ``PipInstall`` plugin."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from dask.distributed import PipInstall\n",
"\n",
"client.register_plugin(PipInstall(packages=[\"modin\"]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you need an additional workers configuration, you can create your own [WorkerPlugin](https://distributed.dask.org/en/stable/plugins.html#worker-plugins) or function that will be executed on each worker upon calling ``client.run()``.\n",
"\n",
"**NOTE**: Dask cluster does not check if this plugin or function has been called before. Therefore, you need to take this into account when using them.\n",
"\n",
"In this tutorial a CSV file will be read, so you need to download it to each of the workers and local machine with the same global path."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from dask.distributed import Worker\n",
"import os\n",
"import urllib\n",
"\n",
"def dataset_upload(file_url, file_path):\n",
" try:\n",
" dir_name = os.path.dirname(file_path)\n",
" if not os.path.exists(dir_name):\n",
" os.makedirs(dir_name)\n",
" if os.path.exists(file_path):\n",
" return \"File has already existed.\"\n",
" else:\n",
" urllib.request.urlretrieve(file_url, file_path)\n",
" return \"OK\"\n",
" except Exception as ex:\n",
" return str(ex)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Set the directory where it should be downloaded (the local directory will be used by default):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"directory_path = \"./\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then you need to run `dataset_upload` function on all workers. As the result, you will get a dictionary, where the result of the function execution will be for each workers:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"file_path = os.path.join(os.path.abspath(directory_path), \"taxi.csv\")\n",
"client.run(dataset_upload, \"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\", file_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You have to also execute this function on the local machine:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dataset_upload(\"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\", file_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Congratulations! The cluster is now fully configured and we can start running Pandas queries."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Executing in a cluster environment\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Same as local mode Modin on cluster uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the [Modin Configuration Settings](https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list) section of the Modin documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Modin engine can be specified either by config\n",
"import modin.config as cfg\n",
"cfg.Engine.put(\"dask\")\n",
"\n",
"# or by setting the environment variable\n",
"# import os\n",
"# os.environ[\"MODIN_ENGINE\"] = \"dask\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can use Modin on the Dask cluster.\n",
"\n",
"Let's read the downloaded CSV file and execute such pandas operations as count, groupby and map:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import time\n",
"\n",
"t0 = time.perf_counter()\n",
"\n",
"df = pd.read_csv(file_path, quoting=3)\n",
"df_count = df.count()\n",
"df_groupby_count = df.groupby(\"passenger_count\").count()\n",
"df_map = df.map(str)\n",
"\n",
"t1 = time.perf_counter()\n",
"print(f\"Full script time is {(t1 - t0):.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shutting down the cluster\n",
"\n",
"Now that we have finished computation, we can shut down the cluster:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cluster.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### This ends the cluster exercise"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_1.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 1: How to use Modin\n",
"\n",
"**GOAL**: Learn how to import Modin to accelerate and scale pandas workflows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Modin is a drop-in replacement for pandas that distributes the computation \n",
"across all of the cores in your machine or in a cluster.\n",
"In practical terms, this means that you can continue using the same pandas scripts\n",
"as before and expect the behavior and results to be the same. The only thing that needs\n",
"to change is the import statement. Normally, you would change:\n",
"\n",
"```python\n",
"import pandas as pd\n",
"```\n",
"\n",
"to:\n",
"\n",
"```python\n",
"import modin.pandas as pd\n",
"```\n",
"\n",
"Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
pandas on a multicore laptop\n",
" \n",
" Modin on a multicore laptop\n",
" \n",
"\n",
"
\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for exercise: setting Modin engine\n",
"\n",
"Modin uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the [Modin Configuration Settings](https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list) section of the Modin documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Modin engine can be specified either by config\n",
"import modin.config as cfg\n",
"cfg.Engine.put(\"dask\")\n",
"\n",
"# or by setting the environment variable\n",
"# import os\n",
"# os.environ[\"MODIN_ENGINE\"] = \"dask\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for exercise: Dataframe constructor\n",
"\n",
"Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.\n",
"\n",
"```python\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\n",
"df = pd.DataFrame(frame_data)\n",
"```\n",
"\n",
"When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:\n",
"\n",
"```\n",
"UserWarning: Distributing object. This may take some time.\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Note: Do not change this code!\n",
"import numpy as np\n",
"import pandas\n",
"import sys\n",
"import modin"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas.__version__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin.__version__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Implement your answer here. You are also free to play with the size\n",
"# and shape of the DataFrame, but beware of exceeding your memory!\n",
"\n",
"import pandas as pd\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\n",
"df = pd.DataFrame(frame_data)\n",
"\n",
"# ***** Do not change the code below! It verifies that \n",
"# ***** the exercise has been done correctly. *****\n",
"\n",
"try:\n",
" assert df is not None\n",
" assert frame_data is not None\n",
" assert isinstance(frame_data, np.ndarray)\n",
"except:\n",
" raise AssertionError(\"Don't change too much of the original code!\")\n",
"assert \"modin.pandas\" in sys.modules, \"Not quite correct. Remember the single line of code change (See above)\"\n",
"\n",
"import modin.pandas\n",
"assert pd == modin.pandas, \"Remember the single line of code change (See above)\"\n",
"assert hasattr(df, \"_query_compiler\"), \"Make sure that `df` is a modin.pandas DataFrame.\"\n",
"\n",
"print(\"Success! You only need to change one line of code!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for Exercise: Data Interaction and Printing\n",
"\n",
"When interacting with data, it is very imporant to look at different parts of the data (e.g. `df.head()`). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the first 10 lines.\n",
"df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the DataFrame.\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Free cell for custom interaction (Play around here!)\n",
"df.add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.count()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please move on to [Exercise 2](./exercise_2.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_2.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 2: Speed improvements\n",
"\n",
"**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for Exercise: `read_csv` speedups\n",
"\n",
"The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation. We will import both Modin and pandas so that the speedups are evident.\n",
"\n",
"**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import time\n",
"from IPython.display import Markdown, display\n",
"\n",
"def printmd(string):\n",
" display(Markdown(string))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataset: 2015 NYC taxi trip data\n",
"\n",
"\n",
"We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\n",
"\n",
"**Size: ~1.8GB**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Modin execution engine setting:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.config as cfg\n",
"cfg.Engine.put(\"dask\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `pandas.read_csv`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_df = pandas.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to read with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Expect pandas to take >3 minutes on EC2, longer locally\n",
"\n",
"This is a good time to chat with your neighbor\n",
"Dicussion topics\n",
"- Do you work with a large amount of data daily?\n",
"- How big is your data?\n",
"- What’s the common use case of your data?\n",
"- Do you use any big data analytics tools?\n",
"- Do you use any interactive analytics tool?\n",
"- What’s are some drawbacks of your current interative analytic tools today?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `modin.pandas.read_csv`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_df = pd.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to read with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `read_csv`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Reduces\n",
"\n",
"In pandas, a reduce would be something along the lines of a `sum` or `count`. It computes some summary statistics about the rows or columns. We will be using `count`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_count = pandas_df.count()\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"\n",
"print(\"Time to count with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_count = modin_df.count()\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to count with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `count`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_count"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_count"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Map operations\n",
"\n",
"In pandas, map operations are operations that do a single pass over the data and do not change its shape. Operations like `isnull` and `applymap` are included in this. We will be using `isnull`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_isnull = pandas_df.isnull()\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"\n",
"print(\"Time to isnull with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_isnull = modin_df.isnull()\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to isnull with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `isnull`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_isnull"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_isnull"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Apply over a single column\n",
"\n",
"Sometimes we want to compute some summary statistics on a single column from our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"rounded_trip_distance_pandas = pandas_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"rounded_trip_distance_modin = modin_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to add a column with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `apply` on one column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rounded_trip_distance_pandas"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rounded_trip_distance_modin"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Add a column\n",
"\n",
"It is common to need to add a new column to an existing dataframe, here we show that this is significantly faster in Modin due to metadata management and an efficient zero copy implementation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"pandas_df[\"rounded_trip_distance\"] = rounded_trip_distance_pandas\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_df[\"rounded_trip_distance\"] = rounded_trip_distance_modin\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to add a column with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas add a column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please move on to [Exercise 3](./exercise_3.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_3.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 3: Not Implemented\n",
"\n",
"**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When functionality has not yet been implemented, we default to pandas\n",
"\n",
"\n",
"\n",
"We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\n",
"\n",
"When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\n",
"\n",
"```\n",
"UserWarning: `DataFrame.mask` defaulting to pandas implementation.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Default to pandas\n",
"\n",
"In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import numpy as np\n",
"import time\n",
"import modin.config as cfg\n",
"cfg.Engine.put(\"dask\")\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\n",
"df = pd.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df = pandas.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_start = time.time()\n",
"\n",
"print(df.mask(df < 50))\n",
"\n",
"modin_end = time.time()\n",
"print(\"Modin mask took {} seconds.\".format(round(modin_end - modin_start, 4)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_start = time.time()\n",
"\n",
"print(pandas_df.mask(pandas_df < 50))\n",
"\n",
"pandas_end = time.time()\n",
"print(\"pandas mask took {} seconds.\".format(round(pandas_end - pandas_start, 4)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Register custom functions\n",
"\n",
"Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Registering a custom function for all query compilers\n",
"\n",
"To register a custom function for a query compiler, we first need to import it:\n",
"\n",
"```python\n",
"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
"```\n",
"\n",
"The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\n",
"\n",
"If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\n",
"\n",
"```python\n",
"from modin.core.dataframe.algebra import TreeReduce\n",
"```\n",
"\n",
"Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\n",
"\n",
"```python\n",
"PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\n",
"```\n",
"\n",
"We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\n",
"\n",
"Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\n",
"\n",
"```python\n",
"def neg_kurtosis_func(self, **kwargs):\n",
" # The constructor allows you to pass in a query compiler as a keyword argument\n",
" return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\n",
"\n",
"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\n",
"```\n",
"\n",
"And then you can use it like you usually would:\n",
"\n",
"```python\n",
"df.neg_kurtosis_custom()\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
"from modin.core.dataframe.algebra import TreeReduce"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\n",
" pandas.DataFrame.kurtosis)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pandas._libs import lib\n",
"# The function signature came from the pandas documentation:\n",
"# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\n",
"def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\n",
" # We need to specify the axis for the query compiler\n",
" if axis in [None, lib.no_default]:\n",
" axis = 0\n",
" # The constructor allows you to pass in a query compiler as a keyword argument\n",
" # Reduce dimension is used for reduces\n",
" # We also pass all keyword arguments here to ensure correctness\n",
" return self._reduce_dimension(\n",
" self._query_compiler.neg_kurtosis_custom(\n",
" axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\n",
" )\n",
" )\n",
"\n",
"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Speed improvements\n",
"If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"pandas unary negation kurtosis took {} seconds.\".format(pandas_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"print(df.applymap(lambda x: ~x).kurtosis())\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Modin unary negation kurtosis took {} seconds.\".format(modin_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"custom_start = time.time()\n",
"\n",
"print(df.neg_kurtosis_custom())\n",
"\n",
"custom_end = time.time()\n",
"modin_custom_duration = custom_end - custom_start\n",
"print(\"Modin neg_kurtosis_custom took {} seconds.\".format(modin_custom_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Markdown, display\n",
"\n",
"display(Markdown(\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Congratulations! You have just implemented new DataFrame functionality!\n",
"\n",
"## Consider opening a pull request: https://github.com/modin-project/modin/pulls\n",
"\n",
"For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\n",
"\n",
"See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_mad_custom_start = time.time()\n",
"\n",
"# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\n",
"# Hint: Look at the kurtosis walkthrough above\n",
"\n",
"modin_mad_custom = ...\n",
"print(modin_mad_custom)\n",
"\n",
"modin_mad_custom_end = time.time()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Evaluation code, do not change!\n",
"modin_mad_start = time.time()\n",
"modin_mad = df.applymap(lambda x: x**2).mad()\n",
"print(modin_mad)\n",
"modin_mad_end = time.time()\n",
"\n",
"assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\n",
" \"Your implementation was too slow, or you used the chaining functions approach. Try again\"\n",
"assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \"Your result did not match the result of chaining the functions, try again\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now that you are able to create custom functions, you know enough to contribute to Modin!\n",
"\n",
"**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/local/exercise_4.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "99f41d2d",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "fdda1c9c",
"metadata": {},
"source": [
"# Exercise 4: Experimental Features\n",
"\n",
"**GOAL**: Explore some of the experimental features being added to Modin."
]
},
{
"cell_type": "markdown",
"id": "e7bf87a5",
"metadata": {},
"source": [
"### Concept for exercise: Spreadsheet\n",
"\n",
"For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.\n",
"\n",
"Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5d5c4a3e",
"metadata": {},
"outputs": [],
"source": [
"!jupyter nbextension enable --py --sys-prefix modin_spreadsheet"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dc8d5903",
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import modin.experimental.spreadsheet as mss\n",
"from modin.config import Engine\n",
"Engine.put(\"dask\")\n",
"\n",
"s3_path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"\n",
"modin_df = pd.read_csv(s3_path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3, nrows=1000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "145e7bbe",
"metadata": {},
"outputs": [],
"source": [
"spreadsheet = mss.from_dataframe(modin_df)\n",
"spreadsheet"
]
},
{
"cell_type": "markdown",
"id": "3c18b7f2",
"metadata": {},
"source": [
"### Thank you for participating!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/requirements.txt
================================================
fsspec>=2022.11.0
jupyterlab
ipywidgets
modin[dask]
modin[spreadsheet]
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_dask/test/test_notebooks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import os
import sys
import nbformat
MODIN_DIR = os.path.abspath(
os.path.join(os.path.dirname(__file__), *[".." for _ in range(6)])
)
sys.path.insert(0, MODIN_DIR)
from examples.tutorial.jupyter.execution.test.utils import ( # noqa: E402
_execute_notebook,
_replace_str,
download_taxi_dataset,
test_dataset_path,
)
local_notebooks_dir = "examples/tutorial/jupyter/execution/pandas_on_dask/local"
# in this notebook user should replace 'import pandas as pd' with
# 'import modin.pandas as pd' to make notebook work
def test_exercise_1():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_1_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_1.ipynb"),
as_version=nbformat.NO_CONVERT,
)
_replace_str(nb, "import pandas as pd", "import modin.pandas as pd")
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
# this notebook works "as is" but for testing purposes we can use smaller dataset
def test_exercise_2():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_2_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_2.ipynb"),
as_version=nbformat.NO_CONVERT,
)
new_cell = f'path = "{test_dataset_path}"\n' + download_taxi_dataset
_replace_str(
nb,
'path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
new_cell,
)
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
# in this notebook user should add custom mad implementation
# to make notebook work
def test_exercise_3():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_3_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_3.ipynb"),
as_version=nbformat.NO_CONVERT,
)
user_mad_implementation = """PandasQueryCompiler.sq_mad_custom = TreeReduce.register(lambda cell_value, **kwargs: cell_value ** 2,
pandas.DataFrame.mad)
def sq_mad_func(self, axis=None, skipna=True, level=None, **kwargs):
if axis is None:
axis = 0
return self._reduce_dimension(
self._query_compiler.sq_mad_custom(
axis=axis, skipna=skipna, level=level, **kwargs
)
)
pd.DataFrame.sq_mad_custom = sq_mad_func
modin_mad_custom = df.sq_mad_custom()
"""
_replace_str(nb, "modin_mad_custom = ...", user_mad_implementation)
nbformat.write(nb, modified_notebook_path)
# need to update example, `.mad` doesn't exist
# _execute_notebook(modified_notebook_path)
# this notebook works "as is" but for testing purposes we can use smaller dataset
def test_exercise_4():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_4_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_4.ipynb"),
as_version=nbformat.NO_CONVERT,
)
s3_path_cell = f's3_path = "{test_dataset_path}"\n' + download_taxi_dataset
_replace_str(
nb,
's3_path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
s3_path_cell,
)
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/Dockerfile
================================================
FROM continuumio/miniconda3
RUN conda install -c conda-forge psutil setproctitle
RUN pip install -r requirements-dev.txt
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/cluster/README.md
================================================
Scale your pandas workflows on a Ray cluster
**NOTE**: Before starting the exercise, please read the full instructions in the
[Modin documenation](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html).
The basic steps to run the script on a remote Ray cluster are:
Step 1. Install the necessary dependencies
```bash
pip install boto3
```
Step 2. Setup your AWS credentials.
```bash
aws configure
```
Step 3. Modify configuration file and start up the Ray cluster.
```bash
ray up modin-cluster.yaml
```
Step 4. Submit your script to the remote cluster.
```bash
ray submit modin-cluster.yaml exercise_5.py
```
Step 5. Shut down the Ray remote cluster.
```bash
ray down
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/cluster/exercise_5.py
================================================
import time
import ray
import modin.pandas as pd
ray.init(address="auto")
cpu_count = ray.cluster_resources()["CPU"]
assert cpu_count == 576, f"Expected 576 CPUs, but found {cpu_count}"
file_path = "big_yellow.csv"
t0 = time.perf_counter()
df = pd.read_csv(file_path, quoting=3)
df_count = df.count()
df_groupby_count = df.groupby("passenger_count").count()
df_map = df.map(str)
t1 = time.perf_counter()
print(f"Full script time is {(t1 - t0):.3f}") # noqa: T201
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/cluster/modin-cluster.yaml
================================================
# An unique identifier for the head node and workers of this cluster.
cluster_name: modin_init
# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 5
# The autoscaler will scale up the cluster faster with higher upscaling speed.
# E.g., if the task requires adding more nodes then autoscaler will gradually
# scale up the cluster in chunks of upscaling_speed*currently_running_nodes.
# This number should be > 0.
upscaling_speed: 1.0
# This executes all commands on all nodes in the docker container,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled.
docker:
# image: "rayproject/ray-ml:latest-gpu" # You can change this to latest-cpu if you don't need GPU support and want a faster startup
image: rayproject/ray:latest-cpu # use this one if you don't need ML dependencies, it's faster to pull
container_name: "ray_container"
# If true, pulls latest version of image. Otherwise, `docker run` will only pull the image
# if no cached version is present.
pull_before_run: True
run_options: # Extra options to pass into "docker run"
- --ulimit nofile=65536:65536
# Example of running a GPU head with CPU workers
# head_image: "rayproject/ray-ml:latest-gpu"
# Allow Ray to automatically detect GPUs
# worker_image: "rayproject/ray-ml:latest-cpu"
# worker_run_options: []
# If a node is idle for this many minutes, it will be removed.
idle_timeout_minutes: 5
# Cloud-provider specific configuration.
provider:
type: aws
region: us-west-2
# Availability zone(s), comma-separated, that nodes may be launched in.
# Nodes will be launched in the first listed availability zone and will
# be tried in the subsequent availability zones if launching fails.
availability_zone: us-west-2a,us-west-2b
# Whether to allow node reuse. If set to False, nodes will be terminated
# instead of stopped.
cache_stopped_nodes: False # If not present, the default is True.
# How Ray will authenticate with newly launched nodes.
auth:
ssh_user: ubuntu
# By default Ray creates a new private keypair, but you can also use your own.
# If you do so, make sure to also set "KeyName" in the head and worker node
# configurations below.
# ssh_private_key: /path/to/your/key.pem
# Tell the autoscaler the allowed node types and the resources they provide.
# The key is the name of the node type, which is just for debugging purposes.
# The node config specifies the launch config and physical instance type.
available_node_types:
ray.head.default:
# The node type's CPU and GPU resources are auto-detected based on AWS instance type.
# If desired, you can override the autodetected CPU and GPU resources advertised to the autoscaler.
# You can also set custom resources.
# For example, to mark a node type as having 1 CPU, 1 GPU, and 5 units of a resource called "custom", set
# resources: {"CPU": 1, "GPU": 1, "custom": 5}
resources: {}
# Provider-specific config for this node type, e.g. instance type. By default
# Ray will auto-configure unspecified fields such as SubnetId and KeyName.
# For more documentation on available fields, see:
# http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
node_config:
InstanceType: m5.24xlarge
# Default AMI for us-west-2.
# Check https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/aws/config.py
# for default images for other zones.
ImageId: ami-0387d929287ab193e
# You can provision additional disk space with a conf as follows
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 500
VolumeType: gp3
# Additional options in the boto docs.
ray.worker.default:
# The minimum number of worker nodes of this type to launch.
# This number should be >= 0.
min_workers: 5
# The maximum number of worker nodes of this type to launch.
# This takes precedence over min_workers.
max_workers: 5
# The node type's CPU and GPU resources are auto-detected based on AWS instance type.
# If desired, you can override the autodetected CPU and GPU resources advertised to the autoscaler.
# You can also set custom resources.
# For example, to mark a node type as having 1 CPU, 1 GPU, and 5 units of a resource called "custom", set
# resources: {"CPU": 1, "GPU": 1, "custom": 5}
resources: {}
# Provider-specific config for this node type, e.g. instance type. By default
# Ray will auto-configure unspecified fields such as SubnetId and KeyName.
# For more documentation on available fields, see:
# http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
node_config:
InstanceType: m5.24xlarge
# Default AMI for us-west-2.
# Check https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/aws/config.py
# for default images for other zones.
ImageId: ami-0387d929287ab193e
# You can provision additional disk space with a conf as follows
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 500
VolumeType: gp3
# Run workers on spot by default. Comment this out to use on-demand.
# NOTE: If relying on spot instances, it is best to specify multiple different instance
# types to avoid interruption when one instance type is experiencing heightened demand.
# Demand information can be found at https://aws.amazon.com/ec2/spot/instance-advisor/
# InstanceMarketOptions:
# MarketType: spot
# Additional options can be found in the boto docs, e.g.
# SpotOptions:
# MaxPrice: MAX_HOURLY_PRICE
# Additional options in the boto docs.
# Specify the node type of the head node (as configured above).
head_node_type: ray.head.default
# Files or directories to copy to the head and worker nodes. The format is a
# dictionary from REMOTE_PATH: LOCAL_PATH, e.g.
file_mounts: {
# "/path1/on/remote/machine": "/path1/on/local/machine",
# "/path2/on/remote/machine": "/path2/on/local/machine",
}
# Files or directories to copy from the head node to the worker nodes. The format is a
# list of paths. The same path on the head node will be copied to the worker node.
# This behavior is a subset of the file_mounts behavior. In the vast majority of cases
# you should just use file_mounts. Only use this if you know what you're doing!
cluster_synced_files: []
# Whether changes to directories in file_mounts or cluster_synced_files in the head node
# should sync to the worker node continuously
file_mounts_sync_continuously: False
# Patterns for files to exclude when running rsync up or rsync down
rsync_exclude:
- "**/.git"
- "**/.git/**"
# Pattern files to use for filtering out files when running rsync up or rsync down. The file is searched for
# in the source directory and recursively through all subdirectories. For example, if .gitignore is provided
# as a value, the behavior will match git's behavior for finding and using .gitignore files.
rsync_filter:
- ".gitignore"
# List of commands that will be run before `setup_commands`. If docker is
# enabled, these commands will run outside the container and before docker
# is setup.
initialization_commands: []
# List of shell commands to run to set up nodes.
setup_commands:
# Note: if you're developing Ray, you probably want to create a Docker image that
# has your Ray repo pre-cloned. Then, you can replace the pip installs
# below with a git checkout (and possibly a recompile).
# To run the nightly version of ray (as opposed to the latest), either use a rayproject docker image
# that has the "nightly" (e.g. "rayproject/ray-ml:nightly-gpu") or uncomment the following line:
# - pip install -U "ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl"
- conda create -n "modin" -c conda-forge modin "ray-default">=2.10.0,<3 -y
- conda activate modin && pip install -U fsspec>=2022.11.0 boto3
- echo "conda activate modin" >> ~/.bashrc
- wget https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv
- printf "VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,pickup_longitude,pickup_latitude,RateCodeID,store_and_fwd_flag,dropoff_longitude,dropoff_latitude,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge,airport_fee\n" > big_yellow.csv
- tail -n +2 yellow_tripdata_2015-01.csv{,}{,}{,}{,}{,}{,} >> big_yellow.csv
- echo 'export MODIN_RAY_CLUSTER=True' >> ~/.bashrc
# Custom commands that will be run on the head node after common setup.
head_setup_commands:
- echo 'export MODIN_REDIS_ADDRESS="localhost:6379"' >> ~/.bashrc
# Custom commands that will be run on worker nodes after common setup.
worker_setup_commands: []
# Command to start ray on the head node. You don't need to change this.
head_start_ray_commands:
- ray stop
- echo 'export MEMORY_STORE_SIZE=$(awk "/MemFree/ { printf \"%d \\n\", \$2*1024}" /proc/meminfo)' >> ~/.bashrc
- echo 'export TMPDIR="$(dirname $(mktemp tmp.XXXXXXXXXX -ut))"' >> ~/.bashrc
- ulimit -n 65536; ray start --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --object-store-memory=$MEMORY_STORE_SIZE --plasma-directory=$TMPDIR --dashboard-host=0.0.0.0
# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands:
- ray stop
- echo 'export MEMORY_STORE_SIZE=$(awk "/MemFree/ { printf \"%d \\n\", \$2*1024}" /proc/meminfo)' >> ~/.bashrc
- echo 'export TMPDIR="$(dirname $(mktemp tmp.XXXXXXXXXX -ut))"' >> ~/.bashrc
- ulimit -n 65536; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076 --object-store-memory=$MEMORY_STORE_SIZE --plasma-directory=$TMPDIR
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_1.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 1: How to use Modin\n",
"\n",
"**GOAL**: Learn how to import Modin to accelerate and scale pandas workflows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Modin is a drop-in replacement for pandas that distributes the computation \n",
"across all of the cores in your machine or in a cluster.\n",
"In practical terms, this means that you can continue using the same pandas scripts\n",
"as before and expect the behavior and results to be the same. The only thing that needs\n",
"to change is the import statement. Normally, you would change:\n",
"\n",
"```python\n",
"import pandas as pd\n",
"```\n",
"\n",
"to:\n",
"\n",
"```python\n",
"import modin.pandas as pd\n",
"```\n",
"\n",
"Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
pandas on a multicore laptop\n",
" \n",
" Modin on a multicore laptop\n",
" \n",
"\n",
"
\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for exercise: Dataframe constructor\n",
"\n",
"Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.\n",
"\n",
"```python\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\n",
"df = pd.DataFrame(frame_data)\n",
"```\n",
"\n",
"When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:\n",
"\n",
"```\n",
"UserWarning: Distributing object. This may take some time.\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Note: Do not change this code!\n",
"import numpy as np\n",
"import pandas\n",
"import sys\n",
"import modin"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas.__version__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin.__version__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Implement your answer here. You are also free to play with the size\n",
"# and shape of the DataFrame, but beware of exceeding your memory!\n",
"\n",
"import pandas as pd\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\n",
"df = pd.DataFrame(frame_data)\n",
"\n",
"# ***** Do not change the code below! It verifies that \n",
"# ***** the exercise has been done correctly. *****\n",
"\n",
"try:\n",
" assert df is not None\n",
" assert frame_data is not None\n",
" assert isinstance(frame_data, np.ndarray)\n",
"except:\n",
" raise AssertionError(\"Don't change too much of the original code!\")\n",
"assert \"modin.pandas\" in sys.modules, \"Not quite correct. Remember the single line of code change (See above)\"\n",
"\n",
"import modin.pandas\n",
"assert pd == modin.pandas, \"Remember the single line of code change (See above)\"\n",
"assert hasattr(df, \"_query_compiler\"), \"Make sure that `df` is a modin.pandas DataFrame.\"\n",
"\n",
"print(\"Success! You only need to change one line of code!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for Exercise: Data Interaction and Printing\n",
"\n",
"When interacting with data, it is very imporant to look at different parts of the data (e.g. `df.head()`). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the first 10 lines.\n",
"df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Print the DataFrame.\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Free cell for custom interaction (Play around here!)\n",
"df.add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df.count()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please move on to [Exercise 2](./exercise_2.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_2.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 2: Speed improvements\n",
"\n",
"**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for Exercise: `read_csv` speedups\n",
"\n",
"The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation.\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will import both Modin and pandas so that the speedups are evident.\n",
"\n",
"**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import time\n",
"from IPython.display import Markdown, display\n",
"\n",
"def printmd(string):\n",
" display(Markdown(string))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataset: 2015 NYC taxi trip data\n",
"\n",
"We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\n",
"\n",
"**Size: ~1.8GB**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Optional:** Note that the dataset takes a while to download. To speed things up a bit, if you prefer to download this file once locally, you can run the following code in the notebook:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# [Optional] Download data locally. This may take a few minutes to download.\n",
"# import urllib.request\n",
"# url_path = \"https://dask-data.s3.amazonaws.com/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"\n",
"# urllib.request.urlretrieve(url_path, \"taxi.csv\")\n",
"# path = \"taxi.csv\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `pandas.read_csv`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_df = pandas.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to read with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Expect pandas to take >3 minutes on EC2, longer locally\n",
"\n",
"This is a good time to chat with your neighbor\n",
"Dicussion topics\n",
"- Do you work with a large amount of data daily?\n",
"- How big is your data?\n",
"- What’s the common use case of your data?\n",
"- Do you use any big data analytics tools?\n",
"- Do you use any interactive analytics tool?\n",
"- What’s are some drawbacks of your current interative analytic tools today?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `modin.pandas.read_csv`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_df = pd.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to read with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `read_csv`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Reduces\n",
"\n",
"In pandas, a reduce would be something along the lines of a `sum` or `count`. It computes some summary statistics about the rows or columns. We will be using `count`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_count = pandas_df.count()\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"\n",
"print(\"Time to count with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_count = modin_df.count()\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to count with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `count`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_count"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_count"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Map operations\n",
"\n",
"In pandas, map operations are operations that do a single pass over the data and do not change its shape. Operations like `isnull` and `applymap` are included in this. We will be using `isnull`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_isnull = pandas_df.isnull()\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"\n",
"print(\"Time to isnull with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_isnull = modin_df.isnull()\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to isnull with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `isnull`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_isnull"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_isnull"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Apply over a single column\n",
"\n",
"Sometimes we want to compute some summary statistics on a single column from our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"rounded_trip_distance_pandas = pandas_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"rounded_trip_distance_modin = modin_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to add a column with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `apply` on one column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rounded_trip_distance_pandas"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rounded_trip_distance_modin"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Add a column\n",
"\n",
"It is common to need to add a new column to an existing dataframe, here we show that this is significantly faster in Modin due to metadata management and an efficient zero copy implementation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"pandas_df[\"rounded_trip_distance\"] = rounded_trip_distance_pandas\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_df[\"rounded_trip_distance\"] = rounded_trip_distance_modin\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to add a column with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas add a column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please move on to [Exercise 3](./exercise_3.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_3.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 3: Not Implemented\n",
"\n",
"**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When functionality has not yet been implemented, we default to pandas\n",
"\n",
"\n",
"\n",
"We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\n",
"\n",
"When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\n",
"\n",
"```\n",
"UserWarning: `DataFrame.mask` defaulting to pandas implementation.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Default to pandas\n",
"\n",
"In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import numpy as np\n",
"import time\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\n",
"df = pd.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df = pandas.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_start = time.time()\n",
"\n",
"print(df.mask(df < 50))\n",
"\n",
"modin_end = time.time()\n",
"print(\"Modin mask took {} seconds.\".format(round(modin_end - modin_start, 4)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_start = time.time()\n",
"\n",
"print(pandas_df.mask(pandas_df < 50))\n",
"\n",
"pandas_end = time.time()\n",
"print(\"pandas mask took {} seconds.\".format(round(pandas_end - pandas_start, 4)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Register custom functions\n",
"\n",
"Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Registering a custom function for all query compilers\n",
"\n",
"To register a custom function for a query compiler, we first need to import it:\n",
"\n",
"```python\n",
"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
"```\n",
"\n",
"The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\n",
"\n",
"If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\n",
"\n",
"```python\n",
"from modin.core.dataframe.algebra import TreeReduce\n",
"```\n",
"\n",
"Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\n",
"\n",
"```python\n",
"PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\n",
"```\n",
"\n",
"We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\n",
"\n",
"Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\n",
"\n",
"```python\n",
"def neg_kurtosis_func(self, **kwargs):\n",
" # The constructor allows you to pass in a query compiler as a keyword argument\n",
" return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\n",
"\n",
"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\n",
"```\n",
"\n",
"And then you can use it like you usually would:\n",
"\n",
"```python\n",
"df.neg_kurtosis_custom()\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
"from modin.core.dataframe.algebra import TreeReduce"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\n",
" pandas.DataFrame.kurtosis)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pandas._libs import lib\n",
"# The function signature came from the pandas documentation:\n",
"# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\n",
"def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\n",
" # We need to specify the axis for the query compiler\n",
" if axis in [None, lib.no_default]:\n",
" axis = 0\n",
" # The constructor allows you to pass in a query compiler as a keyword argument\n",
" # Reduce dimension is used for reduces\n",
" # We also pass all keyword arguments here to ensure correctness\n",
" return self._reduce_dimension(\n",
" self._query_compiler.neg_kurtosis_custom(\n",
" axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\n",
" )\n",
" )\n",
"\n",
"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Speed improvements\n",
"If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"pandas unary negation kurtosis took {} seconds.\".format(pandas_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"print(df.applymap(lambda x: ~x).kurtosis())\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Modin unary negation kurtosis took {} seconds.\".format(modin_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"custom_start = time.time()\n",
"\n",
"print(df.neg_kurtosis_custom())\n",
"\n",
"custom_end = time.time()\n",
"modin_custom_duration = custom_end - custom_start\n",
"print(\"Modin neg_kurtosis_custom took {} seconds.\".format(modin_custom_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Markdown, display\n",
"\n",
"display(Markdown(\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Congratulations! You have just implemented new DataFrame functionality!\n",
"\n",
"## Consider opening a pull request: https://github.com/modin-project/modin/pulls\n",
"\n",
"For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\n",
"\n",
"See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_mad_custom_start = time.time()\n",
"\n",
"# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\n",
"# Hint: Look at the kurtosis walkthrough above\n",
"\n",
"modin_mad_custom = ...\n",
"print(modin_mad_custom)\n",
"\n",
"modin_mad_custom_end = time.time()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Evaluation code, do not change!\n",
"modin_mad_start = time.time()\n",
"modin_mad = df.applymap(lambda x: x**2).mad()\n",
"print(modin_mad)\n",
"modin_mad_end = time.time()\n",
"\n",
"assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\n",
" \"Your implementation was too slow, or you used the chaining functions approach. Try again\"\n",
"assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \"Your result did not match the result of chaining the functions, try again\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now that you are able to create custom functions, you know enough to contribute to Modin!\n",
"\n",
"**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/local/exercise_4.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "99f41d2d",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "fdda1c9c",
"metadata": {},
"source": [
"# Exercise 4: Experimental Features\n",
"\n",
"**GOAL**: Explore some of the experimental features being added to Modin."
]
},
{
"cell_type": "markdown",
"id": "9b487c51",
"metadata": {},
"source": [
"### Concept for exercise: Progress Bar\n",
"\n",
"\n",
"Sometimes when running long functions on DataFrames, it can be hard to tell how much progress has been made, as well as how much longer the function will run. A progress bar allows users to see the estimated progress and completion time of each line they run, in environments such as a shell or Jupyter notebook.\n",
"\n",
"To enable Modin's Progress Bar, add the following lines of code after importing `modin.pandas`:\n",
"```python\n",
"from modin.config import ProgressBar\n",
"ProgressBar.enable()\n",
"```\n",
"\n",
"In this exercise, we'll see how the progress bar can improve our experience running dataframe queries!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f95d4874",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import numpy as np\n",
"from modin.config import ProgressBar\n",
"ProgressBar.enable()\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\n",
"df = pd.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "markdown",
"id": "6905bc6b",
"metadata": {},
"source": [
"On longer functions, its nice to be able to see an estimation of how much longer things will take!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "236ec8e2",
"metadata": {},
"outputs": [],
"source": [
"df = df.applymap(lambda x: ~x)\n",
"df"
]
},
{
"cell_type": "markdown",
"id": "e7bf87a5",
"metadata": {},
"source": [
"### Concept for exercise: Spreadsheet\n",
"\n",
"For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.\n",
"\n",
"Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5d5c4a3e",
"metadata": {},
"outputs": [],
"source": [
"!jupyter nbextension enable --py --sys-prefix modin_spreadsheet\n",
"ProgressBar.disable()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dc8d5903",
"metadata": {},
"outputs": [],
"source": [
"import modin.experimental.spreadsheet as mss\n",
"\n",
"s3_path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"\n",
"modin_df = pd.read_csv(s3_path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3, nrows=1000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "145e7bbe",
"metadata": {},
"outputs": [],
"source": [
"spreadsheet = mss.from_dataframe(modin_df)\n",
"spreadsheet"
]
},
{
"cell_type": "markdown",
"id": "3c18b7f2",
"metadata": {},
"source": [
"### Thank you for participating!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/requirements.txt
================================================
fsspec>=2022.11.0
jupyterlab
ipywidgets
tqdm>=4.60.0
modin[ray]
modin[spreadsheet]
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_ray/test/test_notebooks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import os
import sys
import nbformat
MODIN_DIR = os.path.abspath(
os.path.join(os.path.dirname(__file__), *[".." for _ in range(6)])
)
sys.path.insert(0, MODIN_DIR)
from examples.tutorial.jupyter.execution.test.utils import ( # noqa: E402
_execute_notebook,
_find_code_cell_idx,
_replace_str,
download_taxi_dataset,
test_dataset_path,
)
local_notebooks_dir = "examples/tutorial/jupyter/execution/pandas_on_ray/local"
# in this notebook user should replace 'import pandas as pd' with
# 'import modin.pandas as pd' to make notebook work
def test_exercise_1():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_1_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_1.ipynb"),
as_version=nbformat.NO_CONVERT,
)
_replace_str(nb, "import pandas as pd", "import modin.pandas as pd")
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
# this notebook works "as is" but for testing purposes we can use smaller dataset
def test_exercise_2():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_2_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_2.ipynb"),
as_version=nbformat.NO_CONVERT,
)
_replace_str(
nb,
'path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
'# path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
)
new_optional_cell = f'path = "{test_dataset_path}"\n' + download_taxi_dataset
optional_cell_idx = _find_code_cell_idx(nb, "[Optional] Download data locally.")
nb["cells"][optional_cell_idx]["source"] = new_optional_cell
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
# in this notebook user should add custom mad implementation
# to make notebook work
def test_exercise_3():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_3_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_3.ipynb"),
as_version=nbformat.NO_CONVERT,
)
user_mad_implementation = """PandasQueryCompiler.sq_mad_custom = TreeReduce.register(lambda cell_value, **kwargs: cell_value ** 2,
pandas.DataFrame.mad)
def sq_mad_func(self, axis=None, skipna=True, level=None, **kwargs):
if axis is None:
axis = 0
return self._reduce_dimension(
self._query_compiler.sq_mad_custom(
axis=axis, skipna=skipna, level=level, **kwargs
)
)
pd.DataFrame.sq_mad_custom = sq_mad_func
modin_mad_custom = df.sq_mad_custom()
"""
_replace_str(nb, "modin_mad_custom = ...", user_mad_implementation)
nbformat.write(nb, modified_notebook_path)
# need to update example, `.mad` doesn't exist
# _execute_notebook(modified_notebook_path)
# this notebook works "as is" but for testing purposes we can use smaller dataset
def test_exercise_4():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_4_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_4.ipynb"),
as_version=nbformat.NO_CONVERT,
)
s3_path_cell = f's3_path = "{test_dataset_path}"\n' + download_taxi_dataset
_replace_str(
nb,
's3_path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
s3_path_cell,
)
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/Dockerfile
================================================
FROM continuumio/miniconda3
RUN conda env create -f jupyter_unidist_env.yml
RUN conda install -c conda-forge psutil setproctitle
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/README.md
================================================
# Jupyter notebook examples to run with PandasOnUnidist
Currently, Modin supports `PandasOnUnidist` execution only with MPI backend of [unidist](https://github.com/modin-project/unidist).
There are some specifics on how to run a jupyter notebook with MPI, namely, you should use `mpiexec` command.
```bash
mpiexec -n 1 jupyter notebook
```
**Important**
MPI is not reliable yet to work in interactive environment such as jupyter notebooks. Thus, some things may not work.
For example, if you are experiencing the error `The kernel appears to have died. It will restart automatically.`,
you may want to modify `kernel.json` file or create a new one in order to fix the problem.
For simplicity, you can just run `setup_kernel.py` script located in this directory. This will install a new MPI enabled kernel,
which you can then select using the dropdown menu in your browser. Otherwise, you can follow the steps below:
1. First, what you should do is locate `kernel.json` file with `jupyter kernelspec list` command. It should generally be like this.
```bash
jupyter kernelspec list
Available kernels:
python3 $PREFIX/share/jupyter/kernels/python3
```
`kernel.json` file should be located in `python3` folder.
2. Second, you should make a copy of the `python3` folder, say to `python3mpi` folder.
```bash
cp -r $PREFIX/share/jupyter/kernels/python3 $PREFIX/share/jupyter/kernels/python3mpi
```
3. Third, modify `kernel.json` file in `python3mpi` folder to add `mpiexec -n 1` command
(like "mpiexec", "-n", "1") to the beginning of the launched command (`argv`).
4. Fourth, change `display_name` in `kernel.json` file to something like `Python 3 (ipykernel) with MPI`.
That way you can specifically select the Python kernel with MPI-enabled using the dropdown menu in your browser.
## Run Jupyter Notebooks with PandasOnUnidist
After the `setup_kernel.py` script is run or the steps above are done, you can run a jupyter notebook with `PandasOnUnidist` in a normal way.
```bash
jupyter notebook
```
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/jupyter_unidist_env.yml
================================================
name: jupyter_modin_on_unidist
channels:
- conda-forge
dependencies:
- pip
- fsspec>=2022.11.0
- jupyterlab
- ipywidgets
- modin-mpi
- pip:
- modin[spreadsheet]
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_1.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 1: How to use Modin\n",
"\n",
"**GOAL**: Learn how to import Modin to accelerate and scale pandas workflows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Modin is a drop-in replacement for pandas that distributes the computation \n",
"across all of the cores in your machine or in a cluster.\n",
"In practical terms, this means that you can continue using the same pandas scripts\n",
"as before and expect the behavior and results to be the same. The only thing that needs\n",
"to change is the import statement. Normally, you would change:\n",
"\n",
"```python\n",
"import pandas as pd\n",
"```\n",
"\n",
"to:\n",
"\n",
"```python\n",
"import modin.pandas as pd\n",
"```\n",
"\n",
"Changing this line of code will allow you to use all of the cores in your machine to do computation on your data. One of the major performance bottlenecks of pandas is that it only uses a single core for any given computation. Modin exposes an API that is identical to pandas, allowing you to continue interacting with your data as you would with pandas. There are no additional commands required to use Modin locally. Partitioning, scheduling, data transfer, and other related concerns are all handled by Modin under the hood."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
pandas on a multicore laptop\n",
" \n",
" Modin on a multicore laptop\n",
" \n",
"\n",
"
\n",
"\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for exercise: setting Modin engine\n",
"\n",
"Modin uses Ray as an execution engine by default so no additional action is required to start to use it. Alternatively, if you need to use another engine, it should be specified either by setting the Modin config or by setting Modin environment variable before the first operation with Modin as it is shown below. Also, note that the full list of Modin configs and corresponding environment variables can be found in the [Modin Configuration Settings](https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list) section of the Modin documentation.\n",
"\n",
"One of the execution engines that Modin uses is Unidist. Currently, Modin only supports MPI through unidist, so it should be specified either by setting the Unidist config or by setting Unidist environment variable. The full list of Unidist configs and corresponding environment variables can be found in the [Unidist Configuration Settings](https://unidist.readthedocs.io/en/latest/flow/unidist/config.html#unidist-configuration-settings-list) section of the Unidist documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Modin engine and Unidist backend can be specified either by config\n",
"import modin.config as modin_cfg\n",
"import unidist.config as unidist_cfg\n",
"modin_cfg.Engine.put(\"unidist\")\n",
"unidist_cfg.Backend.put(\"mpi\")\n",
"\n",
"# or by setting the environment variable\n",
"# import os\n",
"# os.environ[\"MODIN_ENGINE\"] = \"unidist\"\n",
"# os.environ[\"UNIDIST_BACKEND\"] = \"mpi\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for exercise: Dataframe constructor\n",
"\n",
"Often when playing around in pandas, it is useful to create a DataFrame with the constructor. That is where we will start.\n",
"\n",
"```python\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**10, 2**5))\n",
"df = pd.DataFrame(frame_data)\n",
"```\n",
"\n",
"When creating a dataframe from a non-distributed object, it will take extra time to partition the data. When this is happening, you will see this message:\n",
"\n",
"```\n",
"UserWarning: Distributing object. This may take some time.\n",
"```\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Note: Do not change this code!\n",
"import numpy as np\n",
"import pandas\n",
"import sys\n",
"import modin"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"pandas.__version__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"modin.__version__"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Implement your answer here. You are also free to play with the size\n",
"# and shape of the DataFrame, but beware of exceeding your memory!\n",
"\n",
"# import pandas as pd\n",
"import pandas as pd\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**5, 2**5))\n",
"df = pd.DataFrame(frame_data)\n",
"\n",
"# ***** Do not change the code below! It verifies that \n",
"# ***** the exercise has been done correctly. *****\n",
"\n",
"try:\n",
" assert df is not None\n",
" assert frame_data is not None\n",
" assert isinstance(frame_data, np.ndarray)\n",
"except:\n",
" raise AssertionError(\"Don't change too much of the original code!\")\n",
"assert \"modin.pandas\" in sys.modules, \"Not quite correct. Remember the single line of code change (See above)\"\n",
"\n",
"import modin.pandas\n",
"assert pd == modin.pandas, \"Remember the single line of code change (See above)\"\n",
"assert hasattr(df, \"_query_compiler\"), \"Make sure that `df` is a modin.pandas DataFrame.\"\n",
"\n",
"print(\"Success! You only need to change one line of code!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have created a toy example for playing around with the DataFrame, let's print it out in different ways."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept for Exercise: Data Interaction and Printing\n",
"\n",
"When interacting with data, it is very imporant to look at different parts of the data (e.g. `df.head()`). Here we will show that you can print the modin.pandas DataFrame in the same ways you would pandas."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Print the first 10 lines.\n",
"df.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Print the DataFrame.\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Free cell for custom interaction (Play around here!)\n",
"df.add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"df.count()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please move on to [Exercise 2](./exercise_2.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel) with MPI",
"language": "python",
"name": "python3mpi"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_2.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 2: Speed improvements\n",
"\n",
"**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for Exercise: `read_csv` speedups\n",
"\n",
"The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation. We will import both Modin and pandas so that the speedups are evident.\n",
"\n",
"**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import time\n",
"from IPython.display import Markdown, display\n",
"\n",
"def printmd(string):\n",
" display(Markdown(string))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataset: 2015 NYC taxi trip data\n",
"\n",
"\n",
"We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\n",
"\n",
"**Size: ~1.8GB**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Modin execution engine setting:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.config as modin_cfg\n",
"import unidist.config as unidist_cfg\n",
"modin_cfg.Engine.put(\"unidist\")\n",
"unidist_cfg.Backend.put(\"mpi\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `pandas.read_csv`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_df = pandas.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to read with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Expect pandas to take >3 minutes on EC2, longer locally\n",
"\n",
"This is a good time to chat with your neighbor\n",
"Dicussion topics\n",
"- Do you work with a large amount of data daily?\n",
"- How big is your data?\n",
"- What’s the common use case of your data?\n",
"- Do you use any big data analytics tools?\n",
"- Do you use any interactive analytics tool?\n",
"- What’s are some drawbacks of your current interative analytic tools today?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## `modin.pandas.read_csv`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_df = pd.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to read with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `read_csv`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Reduces\n",
"\n",
"In pandas, a reduce would be something along the lines of a `sum` or `count`. It computes some summary statistics about the rows or columns. We will be using `count`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_count = pandas_df.count()\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"\n",
"print(\"Time to count with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_count = modin_df.count()\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to count with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `count`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_count"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_count"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Map operations\n",
"\n",
"In pandas, map operations are operations that do a single pass over the data and do not change its shape. Operations like `isnull` and `applymap` are included in this. We will be using `isnull`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"pandas_isnull = pandas_df.isnull()\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"\n",
"print(\"Time to isnull with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_isnull = modin_df.isnull()\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to isnull with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `isnull`!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_isnull"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_isnull"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Apply over a single column\n",
"\n",
"Sometimes we want to compute some summary statistics on a single column from our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"rounded_trip_distance_pandas = pandas_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"rounded_trip_distance_modin = modin_df[\"trip_distance\"].apply(round)\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to add a column with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas at `apply` on one column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rounded_trip_distance_pandas"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rounded_trip_distance_modin"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Add a column\n",
"\n",
"It is common to need to add a new column to an existing dataframe, here we show that this is significantly faster in Modin due to metadata management and an efficient zero copy implementation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"pandas_df[\"rounded_trip_distance\"] = rounded_trip_distance_pandas\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"modin_df[\"rounded_trip_distance\"] = rounded_trip_distance_modin\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Time to add a column with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
"\n",
"printmd(\"### Modin is {}x faster than pandas add a column!\".format(round(pandas_duration / modin_duration, 2)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Are they equal?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please move on to [Exercise 3](./exercise_3.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel) with MPI",
"language": "python",
"name": "python3mpi"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_3.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 3: Not Implemented\n",
"\n",
"**GOAL**: Learn what happens when a function is not yet supported in Modin as well as how to extend Modin's functionality using the DataFrame Algebra."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When functionality has not yet been implemented, we default to pandas\n",
"\n",
"\n",
"\n",
"We convert a Modin dataframe to pandas to do the operation, then convert it back once it is finished. These operations will have a high overhead due to the communication involved and will take longer than pandas.\n",
"\n",
"When this is happening, a warning will be given to the user to inform them that this operation will take longer than usual. For example, `DataFrame.mask` is not yet implemented. In this case, when a user tries to use it, they will see this warning:\n",
"\n",
"```\n",
"UserWarning: `DataFrame.mask` defaulting to pandas implementation.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Default to pandas\n",
"\n",
"In this section of the exercise we will see first-hand how the runtime is affected by operations that are not implemented."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import pandas\n",
"import numpy as np\n",
"import time\n",
"import modin.config as modin_cfg\n",
"import unidist.config as unidist_cfg\n",
"modin_cfg.Engine.put(\"unidist\")\n",
"unidist_cfg.Backend.put(\"mpi\")\n",
"\n",
"frame_data = np.random.randint(0, 100, size=(2**18, 2**8))\n",
"df = pd.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_df = pandas.DataFrame(frame_data).add_prefix(\"col\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_start = time.time()\n",
"\n",
"print(df.mask(df < 50))\n",
"\n",
"modin_end = time.time()\n",
"print(\"Modin mask took {} seconds.\".format(round(modin_end - modin_start, 4)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pandas_start = time.time()\n",
"\n",
"print(pandas_df.mask(pandas_df < 50))\n",
"\n",
"pandas_end = time.time()\n",
"print(\"pandas mask took {} seconds.\".format(round(pandas_end - pandas_start, 4)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Concept for exercise: Register custom functions\n",
"\n",
"Modin's user-facing API is pandas, but it is possible that we do not yet support your favorite or most-needed functionalities. Your user-defined function may also be able to be executed more efficiently if you pre-define the type of function it is (e.g. map, reduce, etc.) using the DataFrame Algebra. To solve either case, it is possible to register a custom function to be applied to your data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Registering a custom function for all query compilers\n",
"\n",
"To register a custom function for a query compiler, we first need to import it:\n",
"\n",
"```python\n",
"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
"```\n",
"\n",
"The `PandasQueryCompiler` is responsible for defining and compiling the queries that can be operated on by Modin, and is specific to the pandas storage format. Any queries defined here must also both be compatible with and result in a `pandas.DataFrame`. Many functionalities are very simply implemented, as you can see in the current code: [Link](https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/core/storage_formats/pandas/query_compiler.py#L216).\n",
"\n",
"If we want to register a new function, we need to understand what kind of function it is. In our example, we will try to implement a `kurtosis` on the unary negation of the values in the dataframe, which is a map (unargy negation of each cell) followed by a reduce. So we next want to import the function type so we can use it in our definition:\n",
"\n",
"```python\n",
"from modin.core.dataframe.algebra import TreeReduce\n",
"```\n",
"\n",
"Then we can just use the `TreeReduce.register` `classmethod` and assign it to the `PandasQueryCompiler`:\n",
"\n",
"```python\n",
"PandasQueryCompiler.neg_kurtosis = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value, pandas.DataFrame.kurtosis)\n",
"```\n",
"\n",
"We include `**kwargs` to the `lambda` function since the query compiler will pass all keyword arguments to both the map and reduce functions.\n",
"\n",
"Finally, we want a handle to it from the `DataFrame`, so we need to create a way to do that:\n",
"\n",
"```python\n",
"def neg_kurtosis_func(self, **kwargs):\n",
" # The constructor allows you to pass in a query compiler as a keyword argument\n",
" return self.__constructor__(query_compiler=self._query_compiler.neg_kurtosis(**kwargs))\n",
"\n",
"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func\n",
"```\n",
"\n",
"And then you can use it like you usually would:\n",
"\n",
"```python\n",
"df.neg_kurtosis_custom()\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler\n",
"from modin.core.dataframe.algebra import TreeReduce"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"PandasQueryCompiler.neg_kurtosis_custom = TreeReduce.register(lambda cell_value, **kwargs: ~cell_value,\n",
" pandas.DataFrame.kurtosis)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pandas._libs import lib\n",
"# The function signature came from the pandas documentation:\n",
"# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.kurtosis.html\n",
"def neg_kurtosis_func(self, axis=lib.no_default, skipna=True, level=None, numeric_only=None, **kwargs):\n",
" # We need to specify the axis for the query compiler\n",
" if axis in [None, lib.no_default]:\n",
" axis = 0\n",
" # The constructor allows you to pass in a query compiler as a keyword argument\n",
" # Reduce dimension is used for reduces\n",
" # We also pass all keyword arguments here to ensure correctness\n",
" return self._reduce_dimension(\n",
" self._query_compiler.neg_kurtosis_custom(\n",
" axis=axis, skipna=skipna, level=level, numeric_only=numeric_only, **kwargs\n",
" )\n",
" )\n",
"\n",
"pd.DataFrame.neg_kurtosis_custom = neg_kurtosis_func"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Speed improvements\n",
"If we were to try and replicate this functionality using the pandas API, we would need to call `df.applymap` with our unary negation function, and subsequently `df.kurtosis` on the result of the first call. Let's see how this compares with our new, custom function!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"print(pandas_df.applymap(lambda cell_value: ~cell_value).kurtosis())\n",
"\n",
"end = time.time()\n",
"pandas_duration = end - start\n",
"print(\"pandas unary negation kurtosis took {} seconds.\".format(pandas_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start = time.time()\n",
"\n",
"print(df.applymap(lambda x: ~x).kurtosis())\n",
"\n",
"end = time.time()\n",
"modin_duration = end - start\n",
"print(\"Modin unary negation kurtosis took {} seconds.\".format(modin_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"custom_start = time.time()\n",
"\n",
"print(df.neg_kurtosis_custom())\n",
"\n",
"custom_end = time.time()\n",
"modin_custom_duration = custom_end - custom_start\n",
"print(\"Modin neg_kurtosis_custom took {} seconds.\".format(modin_custom_duration))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Markdown, display\n",
"\n",
"display(Markdown(\"### As expected, Modin is {}x faster than pandas when chaining the functions; however we see that our custom function is even faster than that - beating pandas by {}x, and Modin (when chaining the functions) by {}x!\".format(round(pandas_duration / modin_duration, 2), round(pandas_duration / modin_custom_duration, 2), round(modin_duration / modin_custom_duration, 2))))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Congratulations! You have just implemented new DataFrame functionality!\n",
"\n",
"## Consider opening a pull request: https://github.com/modin-project/modin/pulls\n",
"\n",
"For a complete list of what is implemented, see the [Supported APIs](https://modin.readthedocs.io/en/latest/supported_apis/index.html) section."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Test your knowledge: Add a custom function for another tree reduce: finding `DataFrame.mad` after squaring all of the values\n",
"\n",
"See the pandas documentation for the correct signature: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mad.html"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modin_mad_custom_start = time.time()\n",
"\n",
"# Implement your function here! Put the result of your custom squared `mad` in the variable `modin_mad_custom`\n",
"# Hint: Look at the kurtosis walkthrough above\n",
"\n",
"modin_mad_custom = ...\n",
"print(modin_mad_custom)\n",
"\n",
"modin_mad_custom_end = time.time()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Evaluation code, do not change!\n",
"modin_mad_start = time.time()\n",
"modin_mad = df.applymap(lambda x: x**2).mad()\n",
"print(modin_mad)\n",
"modin_mad_end = time.time()\n",
"\n",
"assert modin_mad_end - modin_mad_start > modin_mad_custom_end - modin_mad_custom_start, \\\n",
" \"Your implementation was too slow, or you used the chaining functions approach. Try again\"\n",
"assert modin_mad._to_pandas().equals(modin_mad_custom._to_pandas()), \"Your result did not match the result of chaining the functions, try again\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now that you are able to create custom functions, you know enough to contribute to Modin!\n",
"\n",
"**Please move on to [Exercise 4](./exercise_4.ipynb) when you are ready**"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel) with MPI",
"language": "python",
"name": "python3mpi"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/local/exercise_4.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "99f41d2d",
"metadata": {},
"source": [
"\n",
"\n",
"
Scale your pandas workflows by changing one line of code
\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "fdda1c9c",
"metadata": {},
"source": [
"# Exercise 4: Experimental Features\n",
"\n",
"**GOAL**: Explore some of the experimental features being added to Modin."
]
},
{
"cell_type": "markdown",
"id": "e7bf87a5",
"metadata": {},
"source": [
"### Concept for exercise: Spreadsheet\n",
"\n",
"For those who have worked with Excel, the Spreadsheet API will definitely feel familiar! The Spreadsheet API is a Jupyter notebook widget that allows us to interact with Modin DataFrames in a spreadsheet-like fashion while taking advantage of the underlying capabilities of Modin. The widget makes it quick and easy to explore, sort, filter, and edit data as well as export the changes as reproducible code.\n",
"\n",
"Let's look back at a subset of the 2015 NYC Taxi Data from Exercise 2, and see how the Spreadsheet API can make it easy to play with the data!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5d5c4a3e",
"metadata": {},
"outputs": [],
"source": [
"!jupyter nbextension enable --py --sys-prefix modin_spreadsheet"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dc8d5903",
"metadata": {},
"outputs": [],
"source": [
"import modin.pandas as pd\n",
"import modin.experimental.spreadsheet as mss\n",
"import modin.config as modin_cfg\n",
"import unidist.config as unidist_cfg\n",
"modin_cfg.Engine.put(\"unidist\")\n",
"unidist_cfg.Backend.put(\"mpi\")\n",
"\n",
"s3_path = \"s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv\"\n",
"modin_df = pd.read_csv(s3_path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"], quoting=3, nrows=1000)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "145e7bbe",
"metadata": {},
"outputs": [],
"source": [
"spreadsheet = mss.from_dataframe(modin_df)\n",
"spreadsheet"
]
},
{
"cell_type": "markdown",
"id": "3c18b7f2",
"metadata": {},
"source": [
"### Thank you for participating!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel) with MPI",
"language": "python",
"name": "python3mpi"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/setup_kernel.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import sys
from ipykernel import kernelspec
default_make_ipkernel_cmd = kernelspec.make_ipkernel_cmd
def custom_make_ipkernel_cmd(*args, **kwargs):
"""
Build modified Popen command list for launching an IPython kernel with MPI.
Parameters
----------
*args : iterable
Additional positional arguments to be passed in `default_make_ipkernel_cmd`.
**kwargs : dict
Additional keyword arguments to be passed in `default_make_ipkernel_cmd`.
Returns
-------
array
A Popen command list.
Notes
-----
The parameters of the function should be kept in sync with the ones of the original function.
"""
mpi_arguments = ["mpiexec", "-n", "1"]
arguments = default_make_ipkernel_cmd(*args, **kwargs)
return mpi_arguments + arguments
kernelspec.make_ipkernel_cmd = custom_make_ipkernel_cmd
if __name__ == "__main__":
kernel_name = "python3mpi"
display_name = "Python 3 (ipykernel) with MPI"
dest = kernelspec.install(
kernel_name=kernel_name, display_name=display_name, prefix=sys.prefix
)
print(f"Installed kernelspec {kernel_name} in {dest}") # noqa: T201
================================================
FILE: examples/tutorial/jupyter/execution/pandas_on_unidist/test/test_notebooks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import os
import sys
import nbformat
MODIN_DIR = os.path.abspath(
os.path.join(os.path.dirname(__file__), *[".." for _ in range(6)])
)
sys.path.insert(0, MODIN_DIR)
from examples.tutorial.jupyter.execution.test.utils import ( # noqa: E402
_execute_notebook,
_replace_str,
download_taxi_dataset,
set_kernel,
test_dataset_path,
)
# the kernel name "python3mpi" must match the one
# that is set up in `examples/tutorial/jupyter/execution/pandas_on_unidist/setup_kernel.py`
# for `Unidist` engine
set_kernel(kernel_name="python3mpi")
local_notebooks_dir = "examples/tutorial/jupyter/execution/pandas_on_unidist/local"
# in this notebook user should replace 'import pandas as pd' with
# 'import modin.pandas as pd' to make notebook work
def test_exercise_1():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_1_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_1.ipynb"),
as_version=nbformat.NO_CONVERT,
)
_replace_str(nb, "import pandas as pd", "import modin.pandas as pd")
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
# this notebook works "as is" but for testing purposes we can use smaller dataset
def test_exercise_2():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_2_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_2.ipynb"),
as_version=nbformat.NO_CONVERT,
)
new_cell = f'path = "{test_dataset_path}"\n' + download_taxi_dataset
_replace_str(
nb,
'path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
new_cell,
)
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
# in this notebook user should add custom mad implementation
# to make notebook work
def test_exercise_3():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_3_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_3.ipynb"),
as_version=nbformat.NO_CONVERT,
)
user_mad_implementation = """PandasQueryCompiler.sq_mad_custom = TreeReduce.register(lambda cell_value, **kwargs: cell_value ** 2,
pandas.DataFrame.mad)
def sq_mad_func(self, axis=None, skipna=True, level=None, **kwargs):
if axis is None:
axis = 0
return self._reduce_dimension(
self._query_compiler.sq_mad_custom(
axis=axis, skipna=skipna, level=level, **kwargs
)
)
pd.DataFrame.sq_mad_custom = sq_mad_func
modin_mad_custom = df.sq_mad_custom()
"""
_replace_str(nb, "modin_mad_custom = ...", user_mad_implementation)
nbformat.write(nb, modified_notebook_path)
# need to update example, `.mad` doesn't exist
# _execute_notebook(modified_notebook_path)
# this notebook works "as is" but for testing purposes we can use smaller dataset
def test_exercise_4():
modified_notebook_path = os.path.join(local_notebooks_dir, "exercise_4_test.ipynb")
nb = nbformat.read(
os.path.join(local_notebooks_dir, "exercise_4.ipynb"),
as_version=nbformat.NO_CONVERT,
)
s3_path_cell = f's3_path = "{test_dataset_path}"\n' + download_taxi_dataset
_replace_str(
nb,
's3_path = "s3://dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv"',
s3_path_cell,
)
nbformat.write(nb, modified_notebook_path)
_execute_notebook(modified_notebook_path)
================================================
FILE: examples/tutorial/jupyter/execution/test/utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
test_dataset_path = "taxi.csv"
download_taxi_dataset = f"""import os
import urllib.request
if not os.path.exists("{test_dataset_path}"):
url_path = "https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv"
urllib.request.urlretrieve(url_path, "{test_dataset_path}")
"""
# Default kernel name for ``ExecutePreprocessor`` to be created
_default_kernel_name = "python3"
def set_kernel(kernel_name):
"""
Set custom kernel for ``ExecutePreprocessor`` to be created.
Parameters
----------
kernel_name : str
Kernel name.
"""
global _default_kernel_name
_default_kernel_name = kernel_name
def make_execute_preprocessor():
"""
Make ``ExecutePreprocessor`` with the `_default_kernel_name`.
Returns
-------
nbconvert.preprocessors.ExecutePreprocessor
Execute processor entity.
Notes
-----
Note that `_default_kernel_name` can be changed for the concrete executions
(e.g., ``PandasOnUnidist`` with MPI backend).
"""
return ExecutePreprocessor(timeout=600, kernel_name=_default_kernel_name)
def _execute_notebook(notebook):
"""
Execute a jupyter notebook.
Parameters
----------
notebook : file-like or str
File-like object or path to the notebook to execute.
"""
nb = nbformat.read(notebook, as_version=nbformat.NO_CONVERT)
ep = make_execute_preprocessor()
ep.preprocess(nb)
def _find_code_cell_idx(nb, identifier):
"""
Find code cell index by provided ``identifier``.
Parameters
----------
nb : dict
Dictionary representation of the notebook to look for.
identifier : str
Unique string which target code cell should contain.
Returns
-------
int
Code cell index by provided ``identifier``.
Notes
-----
Assertion will be raised if ``identifier`` is found in
several code cells or isn't found at all.
"""
import_cell_idx = [
idx
for idx, cell in enumerate(nb["cells"])
if cell["cell_type"] == "code" and identifier in cell["source"]
]
assert len(import_cell_idx) == 1
return import_cell_idx[0]
def _replace_str(nb, original_str, str_to_replace):
"""
Replace ``original_str`` with ``str_to_replace`` in the provided notebook.
Parameters
----------
nb : dict
Dictionary representation of the notebook which requires replacement.
original_str : str
Original string which should be replaced.
str_to_replace : str
String to replace original string.
Notes
-----
Assertion will be raised if ``original_str`` is found in
several code cells or isn't found at all.
"""
import_cell_idx = _find_code_cell_idx(nb, original_str)
nb["cells"][import_cell_idx]["source"] = nb["cells"][import_cell_idx][
"source"
].replace(original_str, str_to_replace)
================================================
FILE: modin/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import warnings
from typing import Any, Optional, Tuple, Type, Union
from . import _version
def custom_formatwarning(
message: Union[Warning, str],
category: Type[Warning],
filename: str,
lineno: int,
line: Optional[str] = None,
) -> str:
# ignore everything except the message
return "{}: {}\n".format(category.__name__, message)
warnings.formatwarning = custom_formatwarning
# Filter numpy version warnings because they are not relevant
warnings.filterwarnings("ignore", message="numpy.dtype size changed")
warnings.filterwarnings("ignore", message="Large object of size")
def set_execution(engine: Any = None, storage_format: Any = None) -> Tuple[Any, Any]:
"""
Method to set the _pair_ of execution engine and storage format format simultaneously.
This is needed because there might be cases where switching one by one would be
impossible, as not all pairs of values are meaningful.
The method returns pair of old values, so it is easy to return back.
"""
from .config import Backend, Engine, Execution, StorageFormat
old_engine, old_storage_format = None, None
# defer callbacks until both entities are set
if engine is not None:
old_engine = Engine._put_nocallback(engine)
if storage_format is not None:
old_storage_format = StorageFormat._put_nocallback(storage_format)
# execute callbacks if something was changed
if old_engine is not None:
Engine._check_callbacks(old_engine)
if old_storage_format is not None:
StorageFormat._check_callbacks(old_storage_format)
old_backend = Backend.get()
Backend._put_nocallback(
Backend.get_backend_for_execution(
Execution(engine=Engine.get(), storage_format=StorageFormat.get())
)
)
Backend._check_callbacks(old_backend)
return old_engine, old_storage_format
__version__ = _version.get_versions()["version"]
================================================
FILE: modin/__main__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Command-line interface piece, called when user issues "python -m modin --foo"."""
import argparse
def main() -> None:
parser = argparse.ArgumentParser(
"python -m modin",
description="Drop-in pandas replacement; refer to https://modin.readthedocs.io/ for details.",
)
parser.add_argument(
"--versions",
action="store_true",
default=False,
help="Show versions of all known components",
)
args = parser.parse_args()
if args.versions:
from modin.utils import show_versions
show_versions()
if __name__ == "__main__":
main()
================================================
FILE: modin/_version.py
================================================
# This file helps to compute a version number in source trees obtained from
# git-archive tarball (such as those provided by githubs download-from-tag
# feature). Distribution tarballs (built by setup.py sdist) and build
# directories (produced by setup.py build) will contain a much shorter file
# that just contains the computed version number.
# This file is released into the public domain.
# Generated by versioneer-0.29
# https://github.com/python-versioneer/python-versioneer
"""Git implementation of _version.py."""
import errno
import functools
import os
import re
import subprocess
import sys
from typing import Any, Callable, Dict, List, Optional, Tuple
def get_keywords() -> Dict[str, str]:
"""Get the keywords needed to look up the version information."""
# these strings will be replaced by git during git-archive.
# setup.py/versioneer.py will grep for the variable names, so they must
# each be defined on a line of their own. _version.py will just call
# get_keywords().
git_refnames = "$Format:%d$"
git_full = "$Format:%H$"
git_date = "$Format:%ci$"
keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
return keywords
class VersioneerConfig:
"""Container for Versioneer configuration parameters."""
VCS: str
style: str
tag_prefix: str
parentdir_prefix: str
versionfile_source: str
verbose: bool
def get_config() -> VersioneerConfig:
"""Create, populate and return the VersioneerConfig() object."""
# these strings are filled in when 'setup.py versioneer' creates
# _version.py
cfg = VersioneerConfig()
cfg.VCS = "git"
cfg.style = "pep440"
cfg.tag_prefix = ""
cfg.parentdir_prefix = "modin-"
cfg.versionfile_source = "modin/_version.py"
cfg.verbose = False
return cfg
class NotThisMethod(Exception):
"""Exception raised if a method is not valid for the current scenario."""
LONG_VERSION_PY: Dict[str, str] = {}
HANDLERS: Dict[str, Dict[str, Callable]] = {}
def register_vcs_handler(vcs: str, method: str) -> Callable: # decorator
"""Create decorator to mark a method as the handler of a VCS."""
def decorate(f: Callable) -> Callable:
"""Store f in HANDLERS[vcs][method]."""
if vcs not in HANDLERS:
HANDLERS[vcs] = {}
HANDLERS[vcs][method] = f
return f
return decorate
def run_command(
commands: List[str],
args: List[str],
cwd: Optional[str] = None,
verbose: bool = False,
hide_stderr: bool = False,
env: Optional[Dict[str, str]] = None,
) -> Tuple[Optional[str], Optional[int]]:
"""Call the given command(s)."""
assert isinstance(commands, list)
process = None
popen_kwargs: Dict[str, Any] = {}
if sys.platform == "win32":
# This hides the console window if pythonw.exe is used
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
popen_kwargs["startupinfo"] = startupinfo
for command in commands:
try:
dispcmd = str([command] + args)
# remember shell=False, so use git.cmd on windows, not just git
process = subprocess.Popen(
[command] + args,
cwd=cwd,
env=env,
stdout=subprocess.PIPE,
stderr=(subprocess.PIPE if hide_stderr else None),
**popen_kwargs,
)
break
except OSError as e:
if e.errno == errno.ENOENT:
continue
if verbose:
print("unable to run %s" % dispcmd)
print(e)
return None, None
else:
if verbose:
print("unable to find command, tried %s" % (commands,))
return None, None
stdout = process.communicate()[0].strip().decode()
if process.returncode != 0:
if verbose:
print("unable to run %s (error)" % dispcmd)
print("stdout was %s" % stdout)
return None, process.returncode
return stdout, process.returncode
def versions_from_parentdir(
parentdir_prefix: str,
root: str,
verbose: bool,
) -> Dict[str, Any]:
"""Try to determine the version from the parent directory name.
Source tarballs conventionally unpack into a directory that includes both
the project name and a version string. We will also support searching up
two directory levels for an appropriately named parent directory
"""
rootdirs = []
for _ in range(3):
dirname = os.path.basename(root)
if dirname.startswith(parentdir_prefix):
return {
"version": dirname[len(parentdir_prefix) :],
"full-revisionid": None,
"dirty": False,
"error": None,
"date": None,
}
rootdirs.append(root)
root = os.path.dirname(root) # up a level
if verbose:
print(
"Tried directories %s but none started with prefix %s"
% (str(rootdirs), parentdir_prefix)
)
raise NotThisMethod("rootdir doesn't start with parentdir_prefix")
@register_vcs_handler("git", "get_keywords")
def git_get_keywords(versionfile_abs: str) -> Dict[str, str]:
"""Extract version information from the given file."""
# the code embedded in _version.py can just fetch the value of these
# keywords. When used from setup.py, we don't want to import _version.py,
# so we do it with a regexp instead. This function is not used from
# _version.py.
keywords: Dict[str, str] = {}
try:
with open(versionfile_abs, "r") as fobj:
for line in fobj:
if line.strip().startswith("git_refnames ="):
mo = re.search(r'=\s*"(.*)"', line)
if mo:
keywords["refnames"] = mo.group(1)
if line.strip().startswith("git_full ="):
mo = re.search(r'=\s*"(.*)"', line)
if mo:
keywords["full"] = mo.group(1)
if line.strip().startswith("git_date ="):
mo = re.search(r'=\s*"(.*)"', line)
if mo:
keywords["date"] = mo.group(1)
except OSError:
pass
return keywords
@register_vcs_handler("git", "keywords")
def git_versions_from_keywords(
keywords: Dict[str, str],
tag_prefix: str,
verbose: bool,
) -> Dict[str, Any]:
"""Get version information from git keywords."""
if "refnames" not in keywords:
raise NotThisMethod("Short version file found")
date = keywords.get("date")
if date is not None:
# Use only the last line. Previous lines may contain GPG signature
# information.
date = date.splitlines()[-1]
# git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant
# datestamp. However we prefer "%ci" (which expands to an "ISO-8601
# -like" string, which we must then edit to make compliant), because
# it's been around since git-1.5.3, and it's too difficult to
# discover which version we're using, or to work around using an
# older one.
date = date.strip().replace(" ", "T", 1).replace(" ", "", 1)
refnames = keywords["refnames"].strip()
if refnames.startswith("$Format"):
if verbose:
print("keywords are unexpanded, not using")
raise NotThisMethod("unexpanded keywords, not a git-archive tarball")
refs = {r.strip() for r in refnames.strip("()").split(",")}
# starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of
# just "foo-1.0". If we see a "tag: " prefix, prefer those.
TAG = "tag: "
tags = {r[len(TAG) :] for r in refs if r.startswith(TAG)}
if not tags:
# Either we're using git < 1.8.3, or there really are no tags. We use
# a heuristic: assume all version tags have a digit. The old git %d
# expansion behaves like git log --decorate=short and strips out the
# refs/heads/ and refs/tags/ prefixes that would let us distinguish
# between branches and tags. By ignoring refnames without digits, we
# filter out many common branch names like "release" and
# "stabilization", as well as "HEAD" and "master".
tags = {r for r in refs if re.search(r"\d", r)}
if verbose:
print("discarding '%s', no digits" % ",".join(refs - tags))
if verbose:
print("likely tags: %s" % ",".join(sorted(tags)))
for ref in sorted(tags):
# sorting will prefer e.g. "2.0" over "2.0rc1"
if ref.startswith(tag_prefix):
r = ref[len(tag_prefix) :]
# Filter out refs that exactly match prefix or that don't start
# with a number once the prefix is stripped (mostly a concern
# when prefix is '')
if not re.match(r"\d", r):
continue
if verbose:
print("picking %s" % r)
return {
"version": r,
"full-revisionid": keywords["full"].strip(),
"dirty": False,
"error": None,
"date": date,
}
# no suitable tags, so version is "0+unknown", but full hex is still there
if verbose:
print("no suitable tags, using unknown + full revision id")
return {
"version": "0+unknown",
"full-revisionid": keywords["full"].strip(),
"dirty": False,
"error": "no suitable tags",
"date": None,
}
@register_vcs_handler("git", "pieces_from_vcs")
def git_pieces_from_vcs(
tag_prefix: str, root: str, verbose: bool, runner: Callable = run_command
) -> Dict[str, Any]:
"""Get version from 'git describe' in the root of the source tree.
This only gets called if the git-archive 'subst' keywords were *not*
expanded, and _version.py hasn't already been rewritten with a short
version string, meaning we're inside a checked out source tree.
"""
GITS = ["git"]
if sys.platform == "win32":
GITS = ["git.cmd", "git.exe"]
# GIT_DIR can interfere with correct operation of Versioneer.
# It may be intended to be passed to the Versioneer-versioned project,
# but that should not change where we get our version from.
env = os.environ.copy()
env.pop("GIT_DIR", None)
runner = functools.partial(runner, env=env)
_, rc = runner(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=not verbose)
if rc != 0:
if verbose:
print("Directory %s not under git control" % root)
raise NotThisMethod("'git rev-parse --git-dir' returned error")
# if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty]
# if there isn't one, this yields HEX[-dirty] (no NUM)
describe_out, rc = runner(
GITS,
[
"describe",
"--tags",
"--dirty",
"--always",
"--long",
"--match",
f"{tag_prefix}[[:digit:]]*",
],
cwd=root,
)
# --long was added in git-1.5.5
if describe_out is None:
raise NotThisMethod("'git describe' failed")
describe_out = describe_out.strip()
full_out, rc = runner(GITS, ["rev-parse", "HEAD"], cwd=root)
if full_out is None:
raise NotThisMethod("'git rev-parse' failed")
full_out = full_out.strip()
pieces: Dict[str, Any] = {}
pieces["long"] = full_out
pieces["short"] = full_out[:7] # maybe improved later
pieces["error"] = None
branch_name, rc = runner(GITS, ["rev-parse", "--abbrev-ref", "HEAD"], cwd=root)
# --abbrev-ref was added in git-1.6.3
if rc != 0 or branch_name is None:
raise NotThisMethod("'git rev-parse --abbrev-ref' returned error")
branch_name = branch_name.strip()
if branch_name == "HEAD":
# If we aren't exactly on a branch, pick a branch which represents
# the current commit. If all else fails, we are on a branchless
# commit.
branches, rc = runner(GITS, ["branch", "--contains"], cwd=root)
# --contains was added in git-1.5.4
if rc != 0 or branches is None:
raise NotThisMethod("'git branch --contains' returned error")
branches = branches.split("\n")
# Remove the first line if we're running detached
if "(" in branches[0]:
branches.pop(0)
# Strip off the leading "* " from the list of branches.
branches = [branch[2:] for branch in branches]
if "master" in branches:
branch_name = "master"
elif not branches:
branch_name = None
else:
# Pick the first branch that is returned. Good or bad.
branch_name = branches[0]
pieces["branch"] = branch_name
# parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty]
# TAG might have hyphens.
git_describe = describe_out
# look for -dirty suffix
dirty = git_describe.endswith("-dirty")
pieces["dirty"] = dirty
if dirty:
git_describe = git_describe[: git_describe.rindex("-dirty")]
# now we have TAG-NUM-gHEX or HEX
if "-" in git_describe:
# TAG-NUM-gHEX
mo = re.search(r"^(.+)-(\d+)-g([0-9a-f]+)$", git_describe)
if not mo:
# unparsable. Maybe git-describe is misbehaving?
pieces["error"] = "unable to parse git-describe output: '%s'" % describe_out
return pieces
# tag
full_tag = mo.group(1)
if not full_tag.startswith(tag_prefix):
if verbose:
fmt = "tag '%s' doesn't start with prefix '%s'"
print(fmt % (full_tag, tag_prefix))
pieces["error"] = "tag '%s' doesn't start with prefix '%s'" % (
full_tag,
tag_prefix,
)
return pieces
pieces["closest-tag"] = full_tag[len(tag_prefix) :]
# distance: number of commits since tag
pieces["distance"] = int(mo.group(2))
# commit: short hex revision ID
pieces["short"] = mo.group(3)
else:
# HEX: no tags
pieces["closest-tag"] = None
out, rc = runner(GITS, ["rev-list", "HEAD", "--left-right"], cwd=root)
pieces["distance"] = len(out.split()) # total number of commits
# commit date: see ISO-8601 comment in git_versions_from_keywords()
date = runner(GITS, ["show", "-s", "--format=%ci", "HEAD"], cwd=root)[0].strip()
# Use only the last line. Previous lines may contain GPG signature
# information.
date = date.splitlines()[-1]
pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1)
return pieces
def plus_or_dot(pieces: Dict[str, Any]) -> str:
"""Return a + if we don't already have one, else return a ."""
if "+" in pieces.get("closest-tag", ""):
return "."
return "+"
def render_pep440(pieces: Dict[str, Any]) -> str:
"""Build up version string, with post-release "local version identifier".
Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you
get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty
Exceptions:
1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty]
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
if pieces["distance"] or pieces["dirty"]:
rendered += plus_or_dot(pieces)
rendered += "%d.g%s" % (pieces["distance"], pieces["short"])
if pieces["dirty"]:
rendered += ".dirty"
else:
# exception #1
rendered = "0+untagged.%d.g%s" % (pieces["distance"], pieces["short"])
if pieces["dirty"]:
rendered += ".dirty"
return rendered
def render_pep440_branch(pieces: Dict[str, Any]) -> str:
"""TAG[[.dev0]+DISTANCE.gHEX[.dirty]] .
The ".dev0" means not master branch. Note that .dev0 sorts backwards
(a feature branch will appear "older" than the master branch).
Exceptions:
1: no tags. 0[.dev0]+untagged.DISTANCE.gHEX[.dirty]
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
if pieces["distance"] or pieces["dirty"]:
if pieces["branch"] != "master":
rendered += ".dev0"
rendered += plus_or_dot(pieces)
rendered += "%d.g%s" % (pieces["distance"], pieces["short"])
if pieces["dirty"]:
rendered += ".dirty"
else:
# exception #1
rendered = "0"
if pieces["branch"] != "master":
rendered += ".dev0"
rendered += "+untagged.%d.g%s" % (pieces["distance"], pieces["short"])
if pieces["dirty"]:
rendered += ".dirty"
return rendered
def pep440_split_post(ver: str) -> Tuple[str, Optional[int]]:
"""Split pep440 version string at the post-release segment.
Returns the release segments before the post-release and the
post-release version number (or -1 if no post-release segment is present).
"""
vc = str.split(ver, ".post")
return vc[0], int(vc[1] or 0) if len(vc) == 2 else None
def render_pep440_pre(pieces: Dict[str, Any]) -> str:
"""TAG[.postN.devDISTANCE] -- No -dirty.
Exceptions:
1: no tags. 0.post0.devDISTANCE
"""
if pieces["closest-tag"]:
if pieces["distance"]:
# update the post release segment
tag_version, post_version = pep440_split_post(pieces["closest-tag"])
rendered = tag_version
if post_version is not None:
rendered += ".post%d.dev%d" % (post_version + 1, pieces["distance"])
else:
rendered += ".post0.dev%d" % (pieces["distance"])
else:
# no commits, use the tag as the version
rendered = pieces["closest-tag"]
else:
# exception #1
rendered = "0.post0.dev%d" % pieces["distance"]
return rendered
def render_pep440_post(pieces: Dict[str, Any]) -> str:
"""TAG[.postDISTANCE[.dev0]+gHEX] .
The ".dev0" means dirty. Note that .dev0 sorts backwards
(a dirty tree will appear "older" than the corresponding clean one),
but you shouldn't be releasing software with -dirty anyways.
Exceptions:
1: no tags. 0.postDISTANCE[.dev0]
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
if pieces["distance"] or pieces["dirty"]:
rendered += ".post%d" % pieces["distance"]
if pieces["dirty"]:
rendered += ".dev0"
rendered += plus_or_dot(pieces)
rendered += "g%s" % pieces["short"]
else:
# exception #1
rendered = "0.post%d" % pieces["distance"]
if pieces["dirty"]:
rendered += ".dev0"
rendered += "+g%s" % pieces["short"]
return rendered
def render_pep440_post_branch(pieces: Dict[str, Any]) -> str:
"""TAG[.postDISTANCE[.dev0]+gHEX[.dirty]] .
The ".dev0" means not master branch.
Exceptions:
1: no tags. 0.postDISTANCE[.dev0]+gHEX[.dirty]
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
if pieces["distance"] or pieces["dirty"]:
rendered += ".post%d" % pieces["distance"]
if pieces["branch"] != "master":
rendered += ".dev0"
rendered += plus_or_dot(pieces)
rendered += "g%s" % pieces["short"]
if pieces["dirty"]:
rendered += ".dirty"
else:
# exception #1
rendered = "0.post%d" % pieces["distance"]
if pieces["branch"] != "master":
rendered += ".dev0"
rendered += "+g%s" % pieces["short"]
if pieces["dirty"]:
rendered += ".dirty"
return rendered
def render_pep440_old(pieces: Dict[str, Any]) -> str:
"""TAG[.postDISTANCE[.dev0]] .
The ".dev0" means dirty.
Exceptions:
1: no tags. 0.postDISTANCE[.dev0]
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
if pieces["distance"] or pieces["dirty"]:
rendered += ".post%d" % pieces["distance"]
if pieces["dirty"]:
rendered += ".dev0"
else:
# exception #1
rendered = "0.post%d" % pieces["distance"]
if pieces["dirty"]:
rendered += ".dev0"
return rendered
def render_git_describe(pieces: Dict[str, Any]) -> str:
"""TAG[-DISTANCE-gHEX][-dirty].
Like 'git describe --tags --dirty --always'.
Exceptions:
1: no tags. HEX[-dirty] (note: no 'g' prefix)
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
if pieces["distance"]:
rendered += "-%d-g%s" % (pieces["distance"], pieces["short"])
else:
# exception #1
rendered = pieces["short"]
if pieces["dirty"]:
rendered += "-dirty"
return rendered
def render_git_describe_long(pieces: Dict[str, Any]) -> str:
"""TAG-DISTANCE-gHEX[-dirty].
Like 'git describe --tags --dirty --always -long'.
The distance/hash is unconditional.
Exceptions:
1: no tags. HEX[-dirty] (note: no 'g' prefix)
"""
if pieces["closest-tag"]:
rendered = pieces["closest-tag"]
rendered += "-%d-g%s" % (pieces["distance"], pieces["short"])
else:
# exception #1
rendered = pieces["short"]
if pieces["dirty"]:
rendered += "-dirty"
return rendered
def render(pieces: Dict[str, Any], style: str) -> Dict[str, Any]:
"""Render the given version pieces into the requested style."""
if pieces["error"]:
return {
"version": "unknown",
"full-revisionid": pieces.get("long"),
"dirty": None,
"error": pieces["error"],
"date": None,
}
if not style or style == "default":
style = "pep440" # the default
if style == "pep440":
rendered = render_pep440(pieces)
elif style == "pep440-branch":
rendered = render_pep440_branch(pieces)
elif style == "pep440-pre":
rendered = render_pep440_pre(pieces)
elif style == "pep440-post":
rendered = render_pep440_post(pieces)
elif style == "pep440-post-branch":
rendered = render_pep440_post_branch(pieces)
elif style == "pep440-old":
rendered = render_pep440_old(pieces)
elif style == "git-describe":
rendered = render_git_describe(pieces)
elif style == "git-describe-long":
rendered = render_git_describe_long(pieces)
else:
raise ValueError("unknown style '%s'" % style)
return {
"version": rendered,
"full-revisionid": pieces["long"],
"dirty": pieces["dirty"],
"error": None,
"date": pieces.get("date"),
}
def get_versions() -> Dict[str, Any]:
"""Get version information or return default if unable to do so."""
# I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have
# __file__, we can work backwards from there to the root. Some
# py2exe/bbfreeze/non-CPython implementations don't do __file__, in which
# case we can only use expanded keywords.
cfg = get_config()
verbose = cfg.verbose
try:
return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose)
except NotThisMethod:
pass
try:
root = os.path.realpath(__file__)
# versionfile_source is the relative path from the top of the source
# tree (where the .git directory might live) to this file. Invert
# this to find the root from __file__.
for _ in cfg.versionfile_source.split("/"):
root = os.path.dirname(root)
except NameError:
return {
"version": "0+unknown",
"full-revisionid": None,
"dirty": None,
"error": "unable to find root of source tree",
"date": None,
}
try:
pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose)
return render(pieces, cfg.style)
except NotThisMethod:
pass
try:
if cfg.parentdir_prefix:
return versions_from_parentdir(cfg.parentdir_prefix, root, verbose)
except NotThisMethod:
pass
return {
"version": "0+unknown",
"full-revisionid": None,
"dirty": None,
"error": "unable to compute version",
"date": None,
}
================================================
FILE: modin/config/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses config entities which can be used for Modin behavior tuning."""
from modin.config.envvars import (
AsvDataSizeConfig,
AsvImplementation,
AsyncReadMode,
AutoSwitchBackend,
Backend,
BackendJoinConsiderAllBackends,
BackendMergeCastInPlace,
BenchmarkMode,
CIAWSAccessKeyID,
CIAWSSecretAccessKey,
CpuCount,
DaskThreadsPerWorker,
DocModule,
DynamicPartitioning,
Engine,
EnvironmentVariable,
Execution,
GithubCI,
GpuCount,
IsDebug,
IsExperimental,
IsRayCluster,
LazyExecution,
LogFileSize,
LogMemoryInterval,
LogMode,
Memory,
MetricsMode,
MinColumnPartitionSize,
MinPartitionSize,
MinRowPartitionSize,
ModinNumpy,
NativePandasDeepCopy,
NativePandasMaxRows,
NativePandasTransferThreshold,
NPartitions,
PersistentPickle,
ProgressBar,
RangePartitioning,
RayInitCustomResources,
RayRedisAddress,
RayRedisPassword,
RayTaskCustomResources,
ReadSqlEngine,
ShowBackendSwitchProgress,
StorageFormat,
TestDatasetSize,
TestReadFromPostgres,
TestReadFromSqlServer,
TrackFileLeaks,
)
from modin.config.pubsub import Parameter, ValueSource, context
__all__ = [
"EnvironmentVariable",
"Parameter",
"ValueSource",
"context",
# General settings
"IsDebug",
"Engine",
"StorageFormat",
"CpuCount",
"GpuCount",
"Memory",
"Backend",
"BackendJoinConsiderAllBackends",
"BackendMergeCastInPlace",
"Execution",
"AutoSwitchBackend",
"ShowBackendSwitchProgress",
# Ray specific
"IsRayCluster",
"RayRedisAddress",
"RayRedisPassword",
"RayInitCustomResources",
"RayTaskCustomResources",
"LazyExecution",
# Dask specific
"DaskThreadsPerWorker",
# Native Pandas Specific
"NativePandasMaxRows",
"NativePandasTransferThreshold",
"NativePandasDeepCopy",
# Partitioning
"NPartitions",
"MinPartitionSize",
"MinRowPartitionSize",
"MinColumnPartitionSize",
# ASV specific
"TestDatasetSize",
"AsvImplementation",
"AsvDataSizeConfig",
# Specific features
"ProgressBar",
"BenchmarkMode",
"PersistentPickle",
"ModinNumpy",
"RangePartitioning",
"AsyncReadMode",
"ReadSqlEngine",
"IsExperimental",
"DynamicPartitioning",
# For tests
"TrackFileLeaks",
"TestReadFromSqlServer",
"TestReadFromPostgres",
"GithubCI",
"CIAWSSecretAccessKey",
"CIAWSAccessKeyID",
# Logging
"LogMode",
"LogMemoryInterval",
"LogFileSize",
"MetricsMode",
# Plugin settings
"DocModule",
]
================================================
FILE: modin/config/__main__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Content of this file should be executed if module `modin.config` is called.
If module is called (using `python -m modin.config`) configs help will be printed.
Using `-export_path` option configs description can be exported to the external CSV file
provided with this flag.
"""
import argparse
from textwrap import dedent
import pandas
import modin.config as cfg
def print_config_help() -> None:
"""Print configs help messages."""
for objname in sorted(cfg.__all__):
obj = getattr(cfg, objname)
if (
isinstance(obj, type)
and issubclass(obj, cfg.Parameter)
and not obj.is_abstract
):
print(f"{obj.get_help()}\n\tCurrent value: {obj.get()}") # noqa: T201
def export_config_help(filename: str) -> None:
"""
Export all configs help messages to the CSV file.
Parameters
----------
filename : str
Name of the file to export configs data.
"""
configs_data = []
default_values = dict(
RayRedisPassword="random string",
CpuCount="multiprocessing.cpu_count()",
NPartitions="equals to MODIN_CPUS env",
)
for objname in sorted(cfg.__all__):
obj = getattr(cfg, objname)
if (
isinstance(obj, type)
and issubclass(obj, cfg.Parameter)
and not obj.is_abstract
):
data = {
"Config Name": obj.__name__,
"Env. Variable Name": getattr(
obj, "varname", "not backed by environment"
),
"Default Value": default_values.get(obj.__name__, obj._get_default()),
# `Notes` `-` underlining can't be correctly parsed inside csv table by sphinx
"Description": dedent(obj.__doc__ or "").replace(
"Notes\n-----", "Notes:\n"
),
"Options": obj.choices,
}
configs_data.append(data)
pandas.DataFrame(
configs_data,
columns=[
"Config Name",
"Env. Variable Name",
"Default Value",
"Description",
"Options",
],
).to_csv(filename, index=False)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--export-path",
dest="export_path",
type=str,
required=False,
default=None,
help="File path to export configs data.",
)
export_path = parser.parse_args().export_path
if export_path:
export_config_help(export_path)
else:
print_config_help()
================================================
FILE: modin/config/envvars.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses Modin configs originated from environment variables."""
import os
import secrets
import sys
import warnings
from collections import namedtuple
from textwrap import dedent
from typing import Any, NoReturn, Optional
from packaging import version
from pandas.util._decorators import doc # type: ignore[attr-defined]
from modin import set_execution
from modin.config.pubsub import (
_TYPE_PARAMS,
_UNSET,
DeprecationDescriptor,
ExactStr,
Parameter,
ValueSource,
)
class EnvironmentVariable(Parameter, type=str, abstract=True):
"""Base class for environment variables-based configuration."""
varname: Optional[str] = None
@classmethod
def _get_value_from_config(cls) -> Any:
"""
Read the value from environment variable.
Returns
-------
Any
Config raw value if it's set, otherwise `_UNSET`.
"""
if cls.varname is None:
raise TypeError("varname should not be None")
if cls.varname not in os.environ:
return _UNSET
raw = os.environ[cls.varname]
if not _TYPE_PARAMS[cls.type].verify(raw):
# TODO: use and test a better error message, like "Invalid value
# for {cls.varname}: {raw}"
raise ValueError(f"Unsupported raw value: {raw}")
return _TYPE_PARAMS[cls.type].decode(raw)
@classmethod
def get_help(cls) -> str:
"""
Generate user-presentable help for the config.
Returns
-------
str
"""
help = f"{cls.varname}: {dedent(cls.__doc__ or 'Unknown').strip()}\n\tProvide {_TYPE_PARAMS[cls.type].help}"
if cls.choices:
help += f" (valid examples are: {', '.join(str(c) for c in cls.choices)})"
return help
class EnvWithSibilings(
EnvironmentVariable,
# 'type' is a mandatory parameter for '__init_subclasses__', so we have to pass something here,
# this doesn't force child classes to have 'str' type though, they actually can be any type
type=str,
):
"""Ensure values synchronization between sibling parameters."""
_update_sibling = True
@classmethod
def _sibling(cls) -> type["EnvWithSibilings"]:
"""Return a sibling parameter."""
raise NotImplementedError()
@classmethod
def get(cls) -> Any:
"""
Get parameter's value and ensure that it's equal to the sibling's value.
Returns
-------
Any
"""
sibling = cls._sibling()
if sibling._value is _UNSET and cls._value is _UNSET:
super().get()
with warnings.catch_warnings():
# filter warnings that can potentially come from the potentially deprecated sibling
warnings.filterwarnings("ignore", category=FutureWarning)
super(EnvWithSibilings, sibling).get()
if (
cls._value_source
== sibling._value_source
== ValueSource.GOT_FROM_CFG_SOURCE
):
raise ValueError(
f"Configuration is ambiguous. You cannot set '{cls.varname}' and '{sibling.varname}' at the same time."
)
# further we assume that there are only two valid sources for the variables: 'GOT_FROM_CFG' and 'DEFAULT',
# as otherwise we wouldn't ended-up in this branch at all, because all other ways of setting a value
# changes the '._value' attribute from '_UNSET' to something meaningful
from modin.error_message import ErrorMessage
if cls._value_source == ValueSource.GOT_FROM_CFG_SOURCE:
ErrorMessage.catch_bugs_and_request_email(
failure_condition=sibling._value_source != ValueSource.DEFAULT
)
sibling._value = cls._value
sibling._value_source = ValueSource.GOT_FROM_CFG_SOURCE
elif sibling._value_source == ValueSource.GOT_FROM_CFG_SOURCE:
ErrorMessage.catch_bugs_and_request_email(
failure_condition=cls._value_source != ValueSource.DEFAULT
)
cls._value = sibling._value
cls._value_source = ValueSource.GOT_FROM_CFG_SOURCE
else:
ErrorMessage.catch_bugs_and_request_email(
failure_condition=cls._value_source != ValueSource.DEFAULT
or sibling._value_source != ValueSource.DEFAULT
)
# propagating 'cls' default value to the sibling
sibling._value = cls._value
return super().get()
@classmethod
def put(cls, value: Any) -> None:
"""
Set a new value to this parameter as well as to its sibling.
Parameters
----------
value : Any
"""
super().put(value)
# avoid getting into an infinite recursion
if cls._update_sibling:
cls._update_sibling = False
try:
with warnings.catch_warnings():
# filter potential future warnings of the sibling
warnings.filterwarnings("ignore", category=FutureWarning)
cls._sibling().put(value)
finally:
cls._update_sibling = True
class EnvironmentVariableDisallowingExecutionAndBackendBothSet(
EnvironmentVariable,
type=EnvironmentVariable.type,
abstract=True,
):
"""Subclass to disallow getting this variable from the environment when both execution and backend are set in the environment."""
@classmethod
@doc(EnvironmentVariable._get_value_from_config.__doc__)
def _get_value_from_config(cls) -> str:
if Backend.varname in os.environ and (
Engine.varname in os.environ or StorageFormat.varname in os.environ
):
# Handling this case is tricky, in part because the combination of
# Backend and Engine/StorageFormat may be invalid. For now just
# disallow it.
raise ValueError("Can't specify both execution and backend in environment")
return super()._get_value_from_config()
class IsDebug(EnvironmentVariable, type=bool):
"""Force Modin engine to be "Python" unless specified by $MODIN_ENGINE."""
varname = "MODIN_DEBUG"
class Engine(
EnvironmentVariableDisallowingExecutionAndBackendBothSet,
type=str,
):
"""Distribution engine to run queries by."""
varname = "MODIN_ENGINE"
choices = ("Ray", "Dask", "Python", "Unidist", "Native")
NOINIT_ENGINES = {
"Python",
"Native",
} # engines that don't require initialization, useful for unit tests
has_custom_engine = False
@classmethod
def _get_default(cls) -> str:
"""
Get default value of the config.
Returns
-------
str
"""
from modin.utils import MIN_DASK_VERSION, MIN_RAY_VERSION, MIN_UNIDIST_VERSION
# If there's a custom engine, we don't need to check for any engine
# dependencies. Return the default "Python" engine.
if IsDebug.get() or cls.has_custom_engine:
return "Python"
try:
import ray
except ImportError:
pass
else:
if version.parse(ray.__version__) < MIN_RAY_VERSION:
raise ImportError(
'Please `pip install "modin[ray]"` to install compatible Ray '
+ "version "
+ f"(>={MIN_RAY_VERSION})."
)
return "Ray"
try:
import dask
import distributed
except ImportError:
pass
else:
if (
version.parse(dask.__version__) < MIN_DASK_VERSION
or version.parse(distributed.__version__) < MIN_DASK_VERSION
):
raise ImportError(
f'Please `pip install "modin[dask]"` to install compatible Dask version (>={MIN_DASK_VERSION}).'
)
return "Dask"
try:
import unidist
except ImportError:
pass
else:
if version.parse(unidist.__version__) < MIN_UNIDIST_VERSION:
raise ImportError(
'Please `pip install "unidist[mpi]"` to install compatible unidist on MPI '
+ "version "
+ f"(>={MIN_UNIDIST_VERSION})."
)
return "Unidist"
raise ImportError(
"Please refer to installation documentation page to install an engine"
)
@classmethod
@doc(Parameter.add_option.__doc__)
def add_option(cls, choice: Any) -> Any:
choice = super().add_option(choice)
cls.NOINIT_ENGINES.add(choice)
cls.has_custom_engine = True
return choice
@classmethod
def put(cls, value: str) -> None:
"""
Set the engine value.
Parameters
----------
value : str
Engine value to set.
"""
value = cls.normalize(value)
# Backend.put() will set Engine.
Backend.put(
Backend.get_backend_for_execution(
Execution(engine=value, storage_format=StorageFormat.get())
)
)
@classmethod
def get(cls) -> str:
"""
Get the engine value.
Returns
-------
str
Engine value.
"""
# We have to override get() because Engine may need to get its value
# from the OS's environment variables for Backend or Engine.
cls._warn_if_deprecated()
# First, check if we've already set the engine value.
if cls._value is not _UNSET:
return cls._value
engine_config_value = cls._get_value_from_config()
backend_config_value = Backend._get_value_from_config()
# If Engine is in the OS's configuration, use the configured Engine value.
# Otherwise, use the Backend config value if that exists. If it doesn't,
# fall back to the default Engine value.
cls._value = (
engine_config_value
if engine_config_value is not _UNSET
else (
Backend.get_execution_for_backend(backend_config_value).engine
if backend_config_value is not _UNSET
else cls._get_default()
)
)
return cls._value
class StorageFormat(EnvironmentVariableDisallowingExecutionAndBackendBothSet, type=str):
"""Engine to run on a single node of distribution."""
@classmethod
def put(cls, value: str) -> None:
"""
Set the storage format value.
Parameters
----------
value : str
Storage format value to set.
"""
value = cls.normalize(value)
# Backend.put() will set StorageFormat.
Backend.put(
Backend.get_backend_for_execution(
Execution(engine=Engine.get(), storage_format=value)
)
)
@classmethod
def get(cls) -> str:
"""
Get the storage format value.
Returns
-------
str
Storage format value.
"""
# We have to override get() because StorageFormat may need to get its
# value from the OS's environment variables for Backend or StorageFormat.
cls._warn_if_deprecated()
# First, check if we've already set the engine value.
if cls._value is not _UNSET:
return cls._value
storage_format_config_value = cls._get_value_from_config()
backend_config_value = Backend._get_value_from_config()
# If StorageFormat is in the OS's configuration, use the configured
# StorageFormat value. Otherwise, use the Backend config value if that
# exists. If it doesn't, fall back to the default StorageFormat value.
cls._value = (
storage_format_config_value
if storage_format_config_value is not _UNSET
else (
Backend.get_execution_for_backend(backend_config_value).storage_format
if backend_config_value is not _UNSET
else cls._get_default()
)
)
return cls._value
varname = "MODIN_STORAGE_FORMAT"
default = "Pandas"
choices = ("Pandas", "Native")
Execution = namedtuple("Execution", ["storage_format", "engine"])
class Backend(EnvironmentVariableDisallowingExecutionAndBackendBothSet, type=str):
"""
An alias for execution, i.e. the combination of StorageFormat and Engine.
Setting backend may change StorageFormat and/or Engine to the corresponding
respective values, and setting Engine or StorageFormat may change Backend.
Modin's built-in backends include:
- "Ray" <-> (StorageFormat="Pandas", Engine="Ray")
- "Dask" <-> (StorageFormat="Pandas", Engine="Dask")
- "Python_Test" <-> (StorageFormat="Pandas", Engine="Python")
- This execution mode is meant for testing only.
- "Unidist" <-> (StorageFormat="Pandas", Engine="Unidist")
- "Pandas" <-> (StorageFormat="Native", Engine="Native")
"""
_BACKEND_TO_EXECUTION: dict[str, Execution] = {}
_EXECUTION_TO_BACKEND: dict[Execution, str] = {}
varname: str = "MODIN_BACKEND"
choices: tuple[str, ...] = ("Ray", "Dask", "Python_Test", "Unidist", "Pandas")
@classmethod
def put(cls, value: str) -> None:
"""
Set the backend value.
Parameters
----------
value : str
Backend value to set.
"""
execution = cls.get_execution_for_backend(value)
set_execution(execution.engine, execution.storage_format)
@classmethod
def _get_default(cls) -> str:
"""
Get the default backend value.
Returns
-------
str
Default backend value.
"""
return cls._EXECUTION_TO_BACKEND[
Execution(StorageFormat._get_default(), Engine._get_default())
]
@classmethod
def register_backend(cls: type["Backend"], name: str, execution: Execution) -> None:
"""
Register a new backend.
Parameters
----------
name : str
Backend name.
execution : Execution
Execution that corresponds to the backend.
"""
name = cls.normalize(name)
super().add_option(name)
if name in cls._BACKEND_TO_EXECUTION:
raise ValueError(
f"Backend '{name}' is already registered with the execution {cls._BACKEND_TO_EXECUTION[name]}."
)
if execution in cls._EXECUTION_TO_BACKEND:
raise ValueError(
f"{execution} is already registered with the backend {cls._EXECUTION_TO_BACKEND[execution]}."
)
cls._BACKEND_TO_EXECUTION[name] = execution
cls._EXECUTION_TO_BACKEND[execution] = name
@classmethod
def add_option(cls, choice: str) -> NoReturn:
"""
Raise an exception for trying to add an option to Backend directly.
Parameters
----------
choice : str
Choice to add. Unused.
Raises
------
ValueError
Always.
"""
raise ValueError(
"Cannot add an option to Backend directly. Use Backend.register_backend instead."
)
@classmethod
def set_active_backends(cls, new_choices: tuple) -> None:
"""
Set the active backends available for manual and automatic switching.
Other backends may have been registered, and those backends remain registered, but the
set of engines that can be used is dynamically modified.
Parameters
----------
new_choices : tuple
Choices to add.
Raises
------
ValueError
Raises a ValueError when the set of new_choices are not already registered
"""
registered_backends = cls._BACKEND_TO_EXECUTION
for i in new_choices:
if i not in registered_backends:
raise ValueError(
f"Active backend choices {new_choices} are not all registered."
)
cls.choices = new_choices
@classmethod
def activate(cls, backend: str) -> None:
"""
Activate a backend that was previously registered.
This is a no-op if the backend is already active.
Parameters
----------
backend : str
Backend to activate.
Raises
------
ValueError
Raises a ValueError if backend was not previously registered.
"""
if backend not in cls._BACKEND_TO_EXECUTION:
raise ValueError(f"Unknown backend '{backend}' is not registered.")
cls.choices = (*cls.choices, backend)
@classmethod
def get_active_backends(cls) -> tuple[str, ...]:
"""
Get the active backends available for manual and automatic switching.
Returns
-------
tuple[str, ...]
returns the active set of backends for switching
"""
return cls.choices
@classmethod
def get_backend_for_execution(cls, execution: Execution) -> str:
"""
Get the backend for the execution.
Parameters
----------
execution : Execution
Execution to get the backend for.
Returns
-------
str
Backend for the execution.
"""
if execution not in cls._EXECUTION_TO_BACKEND:
raise ValueError(
f"{execution} has no known backend. Please register a "
+ "backend for it with Backend.register_backend()"
)
return cls._EXECUTION_TO_BACKEND[execution]
@classmethod
def get_execution_for_backend(cls, backend: str) -> Execution:
"""
Get the execution for the given backend.
Parameters
----------
backend : str
Backend to get the execution for.
Returns
-------
execution : Execution
The execution for the given backend
"""
if not isinstance(backend, str):
raise TypeError(
"Backend value should be a string, but instead it is "
+ f"{repr(backend)} of type {type(backend)}."
)
normalized_value = cls.normalize(backend)
if normalized_value not in cls.choices:
if normalized_value in cls._BACKEND_TO_EXECUTION:
raise ValueError(
f"Backend '{backend}' is not currently active. Activate it first with Backend.activate('{backend})'."
)
backend_choice_string = ", ".join(f"'{choice}'" for choice in cls.choices)
raise ValueError(
f"Unknown backend '{backend}'. Available backends are: "
+ backend_choice_string
)
if normalized_value not in cls._BACKEND_TO_EXECUTION:
raise ValueError(
f"Backend '{backend}' has no known execution. Please "
+ "register an execution for it with Backend.register_backend()."
)
return cls._BACKEND_TO_EXECUTION[normalized_value]
@classmethod
def get(cls) -> str:
"""
Get the backend.
Returns
-------
str
Backend.
"""
# We have to override get() because Backend may need to get its value
# from the OS's environment variables for Backend or Engine.
cls._warn_if_deprecated()
# First, check if we've already set the Backend value.
if cls._value is not _UNSET:
return cls._value
backend_config_value = Backend._get_value_from_config()
# If Backend is in the OS's configuration, use the configured Backend
# value. Otherwise, we need to figure out the Backend value based on
# the Engine and StorageFormat values.
cls._value = (
backend_config_value
if backend_config_value is not _UNSET
else cls.get_backend_for_execution(
Execution(storage_format=StorageFormat.get(), engine=Engine.get())
)
)
return cls._value
Backend.register_backend("Ray", Execution("Pandas", "Ray"))
Backend.register_backend("Dask", Execution("Pandas", "Dask"))
Backend.register_backend("Python_Test", Execution("Pandas", "Python"))
Backend.register_backend("Unidist", Execution("Pandas", "Unidist"))
Backend.register_backend("Pandas", Execution("Native", "Native"))
class AutoSwitchBackend(EnvironmentVariable, type=bool):
"""
Whether automatic backend switching is allowed.
When this flag is set, a Modin backend can attempt to automatically choose an appropriate backend
for different operations based on features of the input data. When disabled, backends should
avoid implicit backend switching outside of explicit operations like `to_pandas` and `to_ray`.
"""
varname = "MODIN_AUTO_SWITCH_BACKENDS"
default = False
@classmethod
def enable(cls) -> None:
"""Enable automatic backend switching."""
cls.put(True)
@classmethod
def disable(cls) -> None:
"""Disable automatic backend switching."""
cls.put(False)
class ShowBackendSwitchProgress(EnvironmentVariable, type=bool):
"""
Whether to show progress when switching between backends.
When enabled, progress messages are displayed during backend switches to inform users
about data transfer operations. When disabled, backend switches occur silently.
"""
varname = "MODIN_BACKEND_SWITCH_PROGRESS"
default = True
@classmethod
def enable(cls) -> None:
"""Enable backend switch progress display."""
cls.put(True)
@classmethod
def disable(cls) -> None:
"""Disable backend switch progress display."""
cls.put(False)
class IsExperimental(EnvironmentVariable, type=bool):
"""Whether to Turn on experimental features."""
varname = "MODIN_EXPERIMENTAL"
class IsRayCluster(EnvironmentVariable, type=bool):
"""Whether Modin is running on pre-initialized Ray cluster."""
varname = "MODIN_RAY_CLUSTER"
class RayRedisAddress(EnvironmentVariable, type=ExactStr):
"""Redis address to connect to when running in Ray cluster."""
varname = "MODIN_REDIS_ADDRESS"
class RayRedisPassword(EnvironmentVariable, type=ExactStr):
"""What password to use for connecting to Redis."""
varname = "MODIN_REDIS_PASSWORD"
default = secrets.token_hex(32)
class RayInitCustomResources(EnvironmentVariable, type=dict):
"""
Ray node's custom resources to initialize with.
Visit Ray documentation for more details:
https://docs.ray.io/en/latest/ray-core/scheduling/resources.html#custom-resources
Notes
-----
Relying on Modin to initialize Ray, you should set this config
for the proper initialization with custom resources.
"""
varname = "MODIN_RAY_INIT_CUSTOM_RESOURCES"
default = None
class RayTaskCustomResources(EnvironmentVariable, type=dict):
"""
Ray node's custom resources to request them in tasks or actors.
Visit Ray documentation for more details:
https://docs.ray.io/en/latest/ray-core/scheduling/resources.html#custom-resources
Notes
-----
You can use this config to limit the parallelism for the entire workflow
by setting the config at the very beginning.
>>> import modin.config as cfg
>>> cfg.RayTaskCustomResources.put({"special_hardware": 0.001})
This way each single remote task or actor will require 0.001 of "special_hardware" to run.
You can also use this config to limit the parallelism for a certain operation
by setting the config with context.
>>> with context(RayTaskCustomResources={"special_hardware": 0.001}):
... df.
This way each single remote task or actor will require 0.001 of "special_hardware" to run
within the context only.
"""
varname = "MODIN_RAY_TASK_CUSTOM_RESOURCES"
default = None
class CpuCount(EnvironmentVariable, type=int):
"""How many CPU cores to use during initialization of the Modin engine."""
varname = "MODIN_CPUS"
@classmethod
def _put(cls, value: int) -> None:
"""
Put specific value if CpuCount wasn't set by a user yet.
Parameters
----------
value : int
Config value to set.
Notes
-----
This method is used to set CpuCount from cluster resources internally
and should not be called by a user.
"""
if cls.get_value_source() == ValueSource.DEFAULT:
cls.put(value)
@classmethod
def _get_default(cls) -> int:
"""
Get default value of the config.
Returns
-------
int
"""
import multiprocessing
return multiprocessing.cpu_count()
@classmethod
def get(cls) -> int:
"""
Get ``CpuCount`` with extra checks.
Returns
-------
int
"""
cpu_count = super().get()
if cpu_count <= 0:
raise ValueError(f"`CpuCount` should be > 0; current value: {cpu_count}")
return cpu_count
class GpuCount(EnvironmentVariable, type=int):
"""How may GPU devices to utilize across the whole distribution."""
varname = "MODIN_GPUS"
class Memory(EnvironmentVariable, type=int):
"""
How much memory (in bytes) give to an execution engine.
Notes
-----
* In Ray case: the amount of memory to start the Plasma object store with.
* In Dask case: the amount of memory that is given to each worker depending on CPUs used.
"""
varname = "MODIN_MEMORY"
class NPartitions(EnvironmentVariable, type=int):
"""How many partitions to use for a Modin DataFrame (along each axis)."""
varname = "MODIN_NPARTITIONS"
@classmethod
def _put(cls, value: int) -> None:
"""
Put specific value if NPartitions wasn't set by a user yet.
Parameters
----------
value : int
Config value to set.
Notes
-----
This method is used to set NPartitions from cluster resources internally
and should not be called by a user.
"""
if cls.get_value_source() == ValueSource.DEFAULT:
cls.put(value)
@classmethod
def _get_default(cls) -> int:
"""
Get default value of the config.
Returns
-------
int
"""
return CpuCount.get()
@classmethod
def get(cls) -> int:
"""
Get ``NPartitions`` with extra checks.
Returns
-------
int
"""
nparts = super().get()
if nparts <= 0:
raise ValueError(f"`NPartitions` should be > 0; current value: {nparts}")
return nparts
class TestDatasetSize(EnvironmentVariable, type=str):
"""Dataset size for running some tests."""
varname = "MODIN_TEST_DATASET_SIZE"
choices = ("Small", "Normal", "Big")
class TrackFileLeaks(EnvironmentVariable, type=bool):
"""Whether to track for open file handles leakage during testing."""
varname = "MODIN_TEST_TRACK_FILE_LEAKS"
# Turn off tracking on Windows by default because
# psutil's open_files() can be extremely slow on Windows (up to adding a few hours).
# see https://github.com/giampaolo/psutil/pull/597
default = sys.platform != "win32"
class AsvImplementation(EnvironmentVariable, type=ExactStr):
"""Allows to select a library that we will use for testing performance."""
varname = "MODIN_ASV_USE_IMPL"
choices = ("modin", "pandas")
default = "modin"
class AsvDataSizeConfig(EnvironmentVariable, type=ExactStr):
"""Allows to override default size of data (shapes)."""
varname = "MODIN_ASV_DATASIZE_CONFIG"
default = None
class ProgressBar(EnvironmentVariable, type=bool):
"""Whether or not to show the progress bar."""
varname = "MODIN_PROGRESS_BAR"
default = False
@classmethod
def enable(cls) -> None:
"""Enable ``ProgressBar`` feature."""
cls.put(True)
@classmethod
def disable(cls) -> None:
"""Disable ``ProgressBar`` feature."""
cls.put(False)
@classmethod
def put(cls, value: bool) -> None:
"""
Set ``ProgressBar`` value only if synchronous benchmarking is disabled.
Parameters
----------
value : bool
Config value to set.
"""
if value and BenchmarkMode.get():
raise ValueError("ProgressBar isn't compatible with BenchmarkMode")
super().put(value)
class BenchmarkMode(EnvironmentVariable, type=bool):
"""Whether or not to perform computations synchronously."""
varname = "MODIN_BENCHMARK_MODE"
default = False
@classmethod
def put(cls, value: bool) -> None:
"""
Set ``BenchmarkMode`` value only if progress bar feature is disabled.
Parameters
----------
value : bool
Config value to set.
"""
if value and ProgressBar.get():
raise ValueError("BenchmarkMode isn't compatible with ProgressBar")
super().put(value)
class LogMode(EnvironmentVariable, type=ExactStr):
"""Set ``LogMode`` value if users want to opt-in."""
varname = "MODIN_LOG_MODE"
choices = ("enable", "disable")
default = "disable"
@classmethod
def enable(cls) -> None:
"""Enable all logging levels."""
cls.put("enable")
@classmethod
def disable(cls) -> None:
"""Disable logging feature."""
cls.put("disable")
class LogMemoryInterval(EnvironmentVariable, type=int):
"""Interval (in seconds) to profile memory utilization for logging."""
varname = "MODIN_LOG_MEMORY_INTERVAL"
default = 5
@classmethod
def put(cls, value: int) -> None:
"""
Set ``LogMemoryInterval`` with extra checks.
Parameters
----------
value : int
Config value to set.
"""
if value <= 0:
raise ValueError(f"Log memory Interval should be > 0, passed value {value}")
super().put(value)
@classmethod
def get(cls) -> int:
"""
Get ``LogMemoryInterval`` with extra checks.
Returns
-------
int
"""
log_memory_interval = super().get()
if log_memory_interval <= 0:
raise ValueError(
f"`LogMemoryInterval` should be > 0; current value: {log_memory_interval}"
)
return log_memory_interval
class LogFileSize(EnvironmentVariable, type=int):
"""Max size of logs (in MBs) to store per Modin job."""
varname = "MODIN_LOG_FILE_SIZE"
default = 10
@classmethod
def put(cls, value: int) -> None:
"""
Set ``LogFileSize`` with extra checks.
Parameters
----------
value : int
Config value to set.
"""
if value <= 0:
raise ValueError(f"Log file size should be > 0 MB, passed value {value}")
super().put(value)
@classmethod
def get(cls) -> int:
"""
Get ``LogFileSize`` with extra checks.
Returns
-------
int
"""
log_file_size = super().get()
if log_file_size <= 0:
raise ValueError(
f"`LogFileSize` should be > 0; current value: {log_file_size}"
)
return log_file_size
class MetricsMode(EnvironmentVariable, type=ExactStr):
"""
Set ``MetricsMode`` value to disable/enable metrics collection.
Metric handlers are registered through `add_metric_handler` and can
be used to record graphite-style timings or values. It is the
responsibility of the handler to define how those emitted metrics
are handled.
"""
varname = "MODIN_METRICS_MODE"
choices = ("enable", "disable")
default = "enable"
@classmethod
def enable(cls) -> None:
"""Enable all metric collection."""
cls.put("enable")
@classmethod
def disable(cls) -> None:
"""Disable all metric collection."""
cls.put("disable")
class PersistentPickle(EnvironmentVariable, type=bool):
"""Whether serialization should be persistent."""
varname = "MODIN_PERSISTENT_PICKLE"
# When set to off, it allows faster serialization which is only
# valid in current run (i.e. useless for saving to disk).
# When set to on, Modin objects could be saved to disk and loaded
# but serialization/deserialization could take more time.
default = False
class MinPartitionSize(EnvironmentVariable, type=int):
"""
Minimum number of rows/columns in a single pandas partition split.
Once a partition for a pandas dataframe has more than this many elements,
Modin adds another partition.
"""
varname = "MODIN_MIN_PARTITION_SIZE"
default = 32
@classmethod
def put(cls, value: int) -> None:
"""
Set ``MinPartitionSize`` with extra checks.
Parameters
----------
value : int
Config value to set.
"""
if value <= 0:
raise ValueError(f"Min partition size should be > 0, passed value {value}")
super().put(value)
@classmethod
def get(cls) -> int:
"""
Get ``MinPartitionSize`` with extra checks.
Returns
-------
int
"""
from modin.error_message import ErrorMessage
ErrorMessage.single_warning(
"`MinPartitionSize` is deprecated and will be removed in a future version. "
+ "This config has no longer effect, "
+ "use `MinRowPartitionSize` and `MinColumnPartitionSize` instead.",
FutureWarning,
)
min_partition_size = super().get()
if min_partition_size <= 0:
raise ValueError(
f"`MinPartitionSize` should be > 0; current value: {min_partition_size}"
)
return min_partition_size
class MinRowPartitionSize(EnvironmentVariable, type=int):
"""
Minimum number of rows in a single pandas partition split.
Once a partition for a pandas dataframe has more than this many elements,
Modin adds another partition.
"""
varname = "MODIN_MIN_ROW_PARTITION_SIZE"
default = 32
@classmethod
def put(cls, value: int) -> None:
"""
Set ``MinRowPartitionSize`` with extra checks.
Parameters
----------
value : int
Config value to set.
"""
if value <= 0:
raise ValueError(
f"Min row partition size should be > 0, passed value {value}"
)
super().put(value)
@classmethod
def get(cls) -> int:
"""
Get ``MinRowPartitionSize`` with extra checks.
Returns
-------
int
"""
min_row_partition_size = super().get()
if min_row_partition_size <= 0:
raise ValueError(
f"`MinRowPartitionSize` should be > 0; current value: {min_row_partition_size}"
)
return min_row_partition_size
class MinColumnPartitionSize(EnvironmentVariable, type=int):
"""
Minimum number of columns in a single pandas partition split.
Once a partition for a pandas dataframe has more than this many elements,
Modin adds another partition.
"""
varname = "MODIN_MIN_COLUMN_PARTITION_SIZE"
default = 32
@classmethod
def put(cls, value: int) -> None:
"""
Set ``MinColumnPartitionSize`` with extra checks.
Parameters
----------
value : int
Config value to set.
"""
if value <= 0:
raise ValueError(
f"Min column partition size should be > 0, passed value {value}"
)
super().put(value)
@classmethod
def get(cls) -> int:
"""
Get ``MinColumnPartitionSize`` with extra checks.
Returns
-------
int
"""
min_column_partition_size = super().get()
if min_column_partition_size <= 0:
raise ValueError(
f"`MinColumnPartitionSize` should be > 0; current value: {min_column_partition_size}"
)
return min_column_partition_size
class TestReadFromSqlServer(EnvironmentVariable, type=bool):
"""Set to true to test reading from SQL server."""
varname = "MODIN_TEST_READ_FROM_SQL_SERVER"
default = False
class TestReadFromPostgres(EnvironmentVariable, type=bool):
"""Set to true to test reading from Postgres."""
varname = "MODIN_TEST_READ_FROM_POSTGRES"
default = False
class GithubCI(EnvironmentVariable, type=bool):
"""Set to true when running Modin in GitHub CI."""
varname = "MODIN_GITHUB_CI"
default = False
class ModinNumpy(EnvironmentVariable, type=bool):
"""Set to true to use Modin's implementation of NumPy API."""
varname = "MODIN_NUMPY"
default = False
class RangePartitioning(EnvironmentVariable, type=bool):
"""
Set to true to use Modin's range-partitioning implementation where possible.
Please refer to documentation for cases where enabling this options would be beneficial:
https://modin.readthedocs.io/en/stable/flow/modin/experimental/range_partitioning_groupby.html
"""
varname = "MODIN_RANGE_PARTITIONING"
default = False
class CIAWSSecretAccessKey(EnvironmentVariable, type=str):
"""Set to AWS_SECRET_ACCESS_KEY when running mock S3 tests for Modin in GitHub CI."""
varname = "AWS_SECRET_ACCESS_KEY"
default = "foobar_secret"
class CIAWSAccessKeyID(EnvironmentVariable, type=str):
"""Set to AWS_ACCESS_KEY_ID when running mock S3 tests for Modin in GitHub CI."""
varname = "AWS_ACCESS_KEY_ID"
default = "foobar_key"
class AsyncReadMode(EnvironmentVariable, type=bool):
"""
It does not wait for the end of reading information from the source.
It basically means, that the reading function only launches tasks for the dataframe
to be read/created, but not ensures that the construction is finalized by the time
the reading function returns a dataframe.
This option was brought to improve performance of reading/construction
of Modin DataFrames, however it may also:
1. Increase the peak memory consumption. Since the garbage collection of the
temporary objects created during the reading is now also lazy and will only
be performed when the reading/construction is actually finished.
2. Can break situations when the source is manually deleted after the reading
function returns a result, for example, when reading inside of a context-block
that deletes the file on ``__exit__()``.
"""
varname = "MODIN_ASYNC_READ_MODE"
default = False
class ReadSqlEngine(EnvironmentVariable, type=str):
"""Engine to run `read_sql`."""
varname = "MODIN_READ_SQL_ENGINE"
default = "Pandas"
choices = ("Pandas", "Connectorx")
class LazyExecution(EnvironmentVariable, type=str):
"""
Lazy execution mode.
Supported values:
`Auto` - the execution mode is chosen by the engine for each operation (default value).
`On` - the lazy execution is performed wherever it's possible.
`Off` - the lazy execution is disabled.
"""
varname = "MODIN_LAZY_EXECUTION"
choices = ("Auto", "On", "Off")
default = "Auto"
class DocModule(EnvironmentVariable, type=ExactStr):
"""
The module to use that will be used for docstrings.
The value set here must be a valid, importable module. It should have
a `DataFrame`, `Series`, and/or several APIs directly (e.g. `read_csv`).
"""
varname = "MODIN_DOC_MODULE"
default = "pandas"
class DaskThreadsPerWorker(EnvironmentVariable, type=int):
"""Number of threads per Dask worker."""
varname = "MODIN_DASK_THREADS_PER_WORKER"
default = 1
class NativePandasMaxRows(EnvironmentVariable, type=int):
"""Maximum number of rows which can be processed using local, native, pandas."""
varname = "MODIN_NATIVE_MAX_ROWS"
default = 10_000_000
class NativePandasTransferThreshold(EnvironmentVariable, type=int):
"""
Targeted max number of dataframe rows which should be transferred between engines.
This is often the same value as MODIN_NATIVE_MAX_ROWS but it can be independently
set to change how transfer costs are considered.
"""
varname = "MODIN_NATIVE_MAX_XFER_ROWS"
default = 10_000_000
class NativePandasDeepCopy(EnvironmentVariable, type=bool):
"""
Whether to perform deep copies when transferring data with the native pandas backend.
Copies occur when constructing a Modin frame from a native pandas object with
`pd.DataFrame(pandas.DataFrame([]))`, or when creating a native pandas frame from a Modin one
via `df.modin.to_pandas()`.
Leaving this flag disabled produces significant performance improvements by reducing the number
of copy operations performed. However, it may create unexpected results if the user mutates
the Modin frame or native pandas frame in-place.
>>> import pandas # doctest: +SKIP
>>> import modin.pandas as pd # doctest: +SKIP
>>> from modin.config import Backend # doctest: + SKIP
>>> Backend.put("Pandas") # doctest: +SKIP
>>> pandas.set_option("mode.copy_on_write", False) # doctest: +SKIP
>>> native_df = pandas.DataFrame([0]) # doctest: +SKIP
>>> modin_df = pd.DataFrame(native_df) # doctest: +SKIP
>>> native_df.loc[0, 0] = -1 # doctest: +SKIP
>>> modin_df # doctest: +SKIP
0
0 -1
"""
varname = "MODIN_NATIVE_DEEP_COPY"
default = False
@classmethod
def enable(cls) -> None:
"""Enable deep copy on frames with the native pandas backend."""
cls.put(True)
@classmethod
def disable(cls) -> None:
"""Disable deep copy on frames with the native pandas backend."""
cls.put(False)
class BackendMergeCastInPlace(EnvironmentVariable, type=bool):
"""
Whether to cast a DataFrame in-place when performing a merge when using hybrid mode.
This flag modifies the behavior of a cast performed on operations involving more
than one type of query compiler. If enabled the actual cast will be performed in-place
and the input DataFrame will have a new backend. If disabled the original DataFrame
will remain on the same underlying engine.
"""
varname = "MODIN_BACKEND_MERGE_CAST_IN_PLACE"
default = True
@classmethod
def enable(cls) -> None:
"""Enable casting in place when performing a merge operation betwen two different compilers."""
cls.put(True)
@classmethod
def disable(cls) -> None:
"""Disable casting in place when performing a merge operation betwen two different compilers."""
cls.put(False)
class BackendJoinConsiderAllBackends(EnvironmentVariable, type=bool):
"""
Whether to consider all active backends when performing a pre-operation switch for join operations.
Only used when AutoSwitchBackend is active.
By default, only backends already present in the arguments of a join operation are considered when
switching backends. Enabling this flag will allow join operations that are registered
as pre-op switches to consider backends other than those directly present in the arguments.
"""
varname = "MODIN_BACKEND_JOIN_CONSIDER_ALL_BACKENDS"
default = True
@classmethod
def enable(cls) -> None:
"""Enable casting in place when performing a merge operation betwen two different compilers."""
cls.put(True)
@classmethod
def disable(cls) -> None:
"""Disable casting in place when performing a merge operation betwen two different compilers."""
cls.put(False)
class DynamicPartitioning(EnvironmentVariable, type=bool):
"""
Set to true to use Modin's dynamic-partitioning implementation where possible.
Please refer to documentation for cases where enabling this options would be beneficial:
https://modin.readthedocs.io/en/stable/usage_guide/optimization_notes/index.html#dynamic-partitioning-in-modin
"""
varname = "MODIN_DYNAMIC_PARTITIONING"
default = False
def _check_vars() -> None:
"""
Check validity of environment variables.
Look out for any environment variables that start with "MODIN_" prefix
that are unknown - they might be a typo, so warn a user.
"""
valid_names = {
obj.varname
for obj in globals().values()
if isinstance(obj, type)
and issubclass(obj, EnvironmentVariable)
and not obj.is_abstract
}
found_names = {name for name in os.environ if name.startswith("MODIN_")}
unknown = found_names - valid_names
deprecated: dict[str, DeprecationDescriptor] = {
obj.varname: obj._deprecation_descriptor
for obj in globals().values()
if isinstance(obj, type)
and issubclass(obj, EnvironmentVariable)
and not obj.is_abstract
and obj.varname is not None
and obj._deprecation_descriptor is not None
}
found_deprecated = found_names & deprecated.keys()
if unknown:
warnings.warn(
f"Found unknown environment variable{'s' if len(unknown) > 1 else ''},"
+ f" please check {'their' if len(unknown) > 1 else 'its'} spelling: "
+ ", ".join(sorted(unknown))
)
for depr_var in found_deprecated:
warnings.warn(
deprecated[depr_var].deprecation_message(use_envvar_names=True),
FutureWarning,
)
_check_vars()
================================================
FILE: modin/config/pubsub.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses ``Parameter`` class - base class for all configs."""
import contextlib
import warnings
from collections import defaultdict
from enum import IntEnum
from typing import (
TYPE_CHECKING,
Any,
Callable,
DefaultDict,
Iterator,
NamedTuple,
Optional,
Tuple,
cast,
)
if TYPE_CHECKING:
from modin.config.envvars import EnvironmentVariable
class DeprecationDescriptor:
"""
Describe deprecated parameter.
Parameters
----------
parameter : type[Parameter]
Deprecated parameter.
new_parameter : type[Parameter], optional
If there's a replacement parameter for the deprecated one, specify it here.
when_removed : str, optional
If known, the exact release when the deprecated parameter is planned to be removed.
"""
_parameter: type["Parameter"]
_new_parameter: Optional[type["Parameter"]]
_when_removed: str
def __init__(
self,
parameter: type["Parameter"],
new_parameter: Optional[type["Parameter"]] = None,
when_removed: Optional[str] = None,
):
self._parameter = parameter
self._new_parameter = new_parameter
self._when_removed = "a future" if when_removed is None else when_removed
def deprecation_message(self, use_envvar_names: bool = False) -> str:
"""
Generate a message to be used in a warning raised when using the deprecated parameter.
Parameters
----------
use_envvar_names : bool, default: False
Whether to use environment variable names in the warning. If ``True``, both
``self._parameter`` and ``self._new_parameter`` have to be a type of ``EnvironmentVariable``.
Returns
-------
str
"""
name = (
cast("EnvironmentVariable", self._parameter).varname
if use_envvar_names
else self._parameter.__name__
)
msg = f"'{name}' is deprecated and will be removed in {self._when_removed} version."
if self._new_parameter is not None:
new_name = (
cast("EnvironmentVariable", self._new_parameter).varname
if use_envvar_names
else self._new_parameter.__name__
)
msg += f" Use '{new_name}' instead."
return msg
class TypeDescriptor(NamedTuple):
"""
Class for config data manipulating of exact type.
Parameters
----------
decode : callable
Callable to decode config value from the raw data.
normalize : callable
Callable to bring different config value variations to
the single form.
verify : callable
Callable to check that config value satisfies given config
type requirements.
help : str
Class description string.
"""
decode: Callable[[str], object]
normalize: Callable[[object], object]
verify: Callable[[object], bool]
help: str
class ExactStr(str):
"""Class to be used in type params where no transformations are needed."""
_TYPE_PARAMS = {
str: TypeDescriptor(
decode=lambda value: value.strip().title(),
normalize=lambda value: str(value).strip().title(),
verify=lambda value: True,
help="a case-insensitive string",
),
ExactStr: TypeDescriptor(
decode=lambda value: value,
normalize=lambda value: value,
verify=lambda value: True,
help="a string",
),
bool: TypeDescriptor(
decode=lambda value: value.strip().lower() in {"true", "yes", "1"},
normalize=bool,
verify=lambda value: isinstance(value, bool)
or (
isinstance(value, str)
and value.strip().lower() in {"true", "yes", "1", "false", "no", "0"}
),
help="a boolean flag (any of 'true', 'yes' or '1' in case insensitive manner is considered positive)",
),
int: TypeDescriptor(
decode=lambda value: int(value.strip()),
normalize=int, # type: ignore
verify=lambda value: isinstance(value, int)
or (isinstance(value, str) and value.strip().isdigit()),
help="an integer value",
),
dict: TypeDescriptor(
decode=lambda value: {
key: int(val) if val.isdigit() else val
for key_value in value.split(",")
for key, val in [[v.strip() for v in key_value.split("=", maxsplit=1)]]
},
normalize=lambda value: (
value
if isinstance(value, dict)
else {
key: int(val) if val.isdigit() else val
for key_value in str(value).split(",")
for key, val in [[v.strip() for v in key_value.split("=", maxsplit=1)]]
}
),
verify=lambda value: isinstance(value, dict)
or (
isinstance(value, str)
and all(
key_value.find("=") not in (-1, len(key_value) - 1)
for key_value in value.split(",")
)
),
help="a sequence of KEY=VALUE values separated by comma (Example: 'KEY1=VALUE1,KEY2=VALUE2,KEY3=VALUE3')",
),
}
# special marker to distinguish unset value from None value
# as someone may want to use None as a real value for a parameter
_UNSET = object()
class ValueSource(IntEnum): # noqa: PR01
"""Class that describes the method of getting the value for a parameter."""
# got from default, i.e. neither user nor configuration source had the value
DEFAULT = 0
# set by user
SET_BY_USER = 1
# got from parameter configuration source, like environment variable
GOT_FROM_CFG_SOURCE = 2
class Parameter(object):
"""
Base class describing interface for configuration entities.
Attributes
----------
choices : Optional[Sequence[str]]
Array with possible options of ``Parameter`` values.
type : str
String that denotes ``Parameter`` type.
default : Optional[Any]
``Parameter`` default value.
is_abstract : bool, default: True
Whether or not ``Parameter`` is abstract.
_value_source : Optional[ValueSource]
Source of the ``Parameter`` value, should be set by
``ValueSource``.
_deprecation_descriptor : Optional[DeprecationDescriptor]
Indicate whether this parameter is deprecated.
"""
choices: Optional[Tuple[str, ...]] = None
type = str
default: Optional[Any] = None
is_abstract = True
_value_source: Optional[ValueSource] = None
_value: Any = _UNSET
_subs: list = []
_once: DefaultDict[Any, list] = defaultdict(list)
_deprecation_descriptor: Optional[DeprecationDescriptor] = None
@classmethod
def _warn_if_deprecated(cls) -> None:
"""Warn that the variable is deprecated if it has a deprecation descriptor."""
if cls._deprecation_descriptor is not None:
warnings.warn(
cls._deprecation_descriptor.deprecation_message(), FutureWarning
)
@classmethod
def _get_value_from_config(cls) -> Any:
"""
Read the value from config storage.
Returns
-------
Any
Config raw value if it's set, otherwise `_UNSET`.
Notes
-----
Config storage can be config file or environment variable or whatever.
Method should be implemented in the child class.
"""
raise NotImplementedError()
@classmethod
def get_help(cls) -> str:
"""
Generate user-presentable help for the option.
Returns
-------
str
Notes
-----
Method should be implemented in the child class.
"""
raise NotImplementedError()
def __init_subclass__(cls, type: Any, abstract: bool = False, **kw: dict):
"""
Initialize subclass.
Parameters
----------
type : Any
Type of the config.
abstract : bool, default: False
Whether config is abstract.
**kw : dict
Optional arguments for config initialization.
"""
assert type in _TYPE_PARAMS, f"Unsupported variable type: {type}"
cls.type = type
cls.is_abstract = abstract
cls._value = _UNSET
cls._subs = []
cls._once = defaultdict(list)
super().__init_subclass__(**kw)
@classmethod
def subscribe(cls, callback: Callable) -> None:
"""
Add `callback` to the `_subs` list and then execute it.
Parameters
----------
callback : callable
Callable to execute.
"""
cls._subs.append(callback)
callback(cls)
@classmethod
def _get_default(cls) -> Any:
"""
Get default value of the config.
Returns
-------
Any
"""
return cls.default
@classmethod
def get_value_source(cls) -> ValueSource:
"""
Get value source of the config.
Returns
-------
ValueSource
"""
if cls._value_source is None:
# dummy call to .get() to initialize the value
cls.get()
assert (
cls._value_source is not None
), "_value_source must be initialized by now in get()"
return cls._value_source
@classmethod
def get(cls) -> Any:
"""
Get config value.
Returns
-------
Any
Decoded and verified config value.
"""
cls._warn_if_deprecated()
if cls._value is _UNSET:
# get the value from env
config_value = cls._get_value_from_config()
if config_value is _UNSET:
cls._value = cls._get_default()
cls._value_source = ValueSource.DEFAULT
else:
cls._value = config_value
cls._value_source = ValueSource.GOT_FROM_CFG_SOURCE
return cls._value
@classmethod
def put(cls, value: Any) -> None:
"""
Set config value.
Parameters
----------
value : Any
Config value to set.
"""
cls._warn_if_deprecated()
cls._check_callbacks(cls._put_nocallback(value))
cls._value_source = ValueSource.SET_BY_USER
@classmethod
def normalize(cls, value: Any) -> Any:
"""
Normalize config value.
Parameters
----------
value : Any
Config value to normalize.
Returns
-------
Any
Normalized config value.
"""
return _TYPE_PARAMS[cls.type].normalize(value)
@classmethod
def once(cls, onvalue: Any, callback: Callable) -> None:
"""
Execute `callback` if config value matches `onvalue` value.
Otherwise accumulate callbacks associated with the given `onvalue`
in the `_once` container.
Parameters
----------
onvalue : Any
Config value to set.
callback : callable
Callable that should be executed if config value matches `onvalue`.
"""
onvalue = cls.normalize(onvalue)
if onvalue == cls.get():
callback(cls)
else:
cls._once[onvalue].append(callback)
@classmethod
def _put_nocallback(cls, value: Any) -> Any:
"""
Set config value without executing callbacks.
Parameters
----------
value : Any
Config value to set.
Returns
-------
Any
Replaced (old) config value.
"""
if not _TYPE_PARAMS[cls.type].verify(value):
raise ValueError(f"Unsupported value: {value}")
value = cls.normalize(value)
oldvalue, cls._value = cls.get(), value
return oldvalue
@classmethod
def _check_callbacks(cls, oldvalue: Any) -> None:
"""
Execute all needed callbacks if config value was changed.
Parameters
----------
oldvalue : Any
Previous (old) config value.
"""
if oldvalue == cls.get():
return
for callback in cls._subs:
callback(cls)
for callback in cls._once.pop(cls.get(), ()):
callback(cls)
@classmethod
def add_option(cls, choice: Any) -> Any:
"""
Add a new choice for the parameter.
Parameters
----------
choice : Any
New choice to add to the available choices.
Returns
-------
Any
Added choice normalized according to the parameter type.
"""
if cls.choices is not None:
if not _TYPE_PARAMS[cls.type].verify(choice):
raise ValueError(f"Unsupported choice value: {choice}")
choice = cls.normalize(choice)
if choice not in cls.choices:
cls.choices += (choice,)
return choice
raise TypeError("Cannot add a choice to a parameter where choices is None")
@contextlib.contextmanager
def context(**config: dict[str, Any]) -> Iterator[None]:
"""
Set a value(s) for the specified config(s) from ``modin.config`` in the scope of the context.
Parameters
----------
**config : dict[str, Any]
Keyword describing a name of a config variable from ``modin.config`` as a key
and a new value as a value.
Examples
--------
>>> RangePartitioning.get()
False
>>> with context(RangePartitioning=True):
... print(RangePartitioning.get()) # True
True
False
>>> RangePartitioning.get()
False
>>> with context(RangePartitioning=True, AsyncReadMode=True):
... print(RangePartitioning.get()) # True
... print(AsyncReadMode.get()) # True
True
True
>>> RangePartitioning.get()
False
>>> AsyncReadMode.get()
False
"""
import modin.config as cfg
old_values = {}
for name, val in config.items():
var = getattr(cfg, name)
old_values[var] = var.get()
var.put(val)
try:
yield
finally:
for var, val in old_values.items():
var.put(val)
__all__ = ["Parameter", "context"]
================================================
FILE: modin/conftest.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
# We turn off mypy type checks in this file because it's not imported anywhere
# type: ignore
import copy
import logging
import os
import platform
import shutil
import subprocess
import sys
import time
from collections import defaultdict
from contextlib import contextmanager
from typing import Iterable, Optional
import boto3
import numpy as np
import pandas
import pytest
import requests
import s3fs
from pandas.util._decorators import doc
from modin.config import Backend, Execution
assert (
"modin.utils" not in sys.modules
), "Do not import modin.utils before patching, or tests could fail"
# every import under this assert has to be postfixed with 'noqa: E402'
# as flake8 complains about that... but we _have_ to make sure we
# monkey-patch at the right spot, otherwise testing doc URLs might
# not catch all of them
import modin.utils # noqa: E402
_generated_doc_urls = set()
def _saving_make_api_url(token, _make_api_url=modin.utils._make_api_url):
url = _make_api_url(token)
_generated_doc_urls.add(url)
return url
modin.utils._make_api_url = _saving_make_api_url
import uuid # noqa: E402
import modin # noqa: E402
import modin.config # noqa: E402
import modin.pandas as pd # noqa: E402
import modin.tests.config # noqa: E402
from modin.config import ( # noqa: E402
AsyncReadMode,
BenchmarkMode,
GithubCI,
IsExperimental,
MinRowPartitionSize,
NPartitions,
)
from modin.core.execution.dispatching.factories import factories # noqa: E402
from modin.core.execution.python.implementations.pandas_on_python.io import ( # noqa: E402
PandasOnPythonIO,
)
from modin.core.storage_formats import ( # noqa: E402
BaseQueryCompiler,
PandasQueryCompiler,
)
from modin.core.storage_formats.pandas.query_compiler_caster import ( # noqa: E402
_CLASS_AND_BACKEND_TO_POST_OP_SWITCH_METHODS,
_CLASS_AND_BACKEND_TO_PRE_OP_SWITCH_METHODS,
_GENERAL_EXTENSIONS,
)
from modin.tests.pandas.utils import ( # noqa: E402
NROWS,
_make_csv_file,
get_unique_filename,
make_default_file,
)
def pytest_addoption(parser):
parser.addoption(
"--execution",
action="store",
default=None,
help="specifies execution to run tests on",
)
def set_experimental_env(mode):
IsExperimental.put(mode == "experimental")
@pytest.fixture(scope="session", autouse=True)
def enforce_config():
"""
A fixture that ensures that all checks for MODIN_* variables
are done using modin.config to prevent leakage
"""
orig_env = os.environ
modin_start = os.path.dirname(modin.__file__)
modin_exclude = [
os.path.dirname(modin.config.__file__),
os.path.dirname(modin.tests.config.__file__),
]
class PatchedEnv:
@staticmethod
def __check_var(name):
if name.upper().startswith("MODIN_"):
frame = sys._getframe()
try:
# get the path to module where caller of caller is defined;
# caller of this function is inside PatchedEnv, and we're
# interested in whomever called a method on PatchedEnv
caller_file = frame.f_back.f_back.f_code.co_filename
finally:
del frame
pkg_name = os.path.dirname(caller_file)
if pkg_name.startswith(modin_start):
assert any(
pkg_name.startswith(excl) for excl in modin_exclude
), "Do not access MODIN_ environment variable bypassing modin.config"
def __getitem__(self, name):
self.__check_var(name)
return orig_env[name]
def __setitem__(self, name, value):
self.__check_var(name)
orig_env[name] = value
def __delitem__(self, name):
self.__check_var(name)
del orig_env[name]
def pop(self, name, default=object()):
self.__check_var(name)
return orig_env.pop(name, default)
def get(self, name, default=None):
self.__check_var(name)
return orig_env.get(name, default)
def __contains__(self, name):
self.__check_var(name)
return name in orig_env
def __getattr__(self, name):
return getattr(orig_env, name)
def __iter__(self):
return iter(orig_env)
os.environ = PatchedEnv()
yield
os.environ = orig_env
BASE_EXECUTION_NAME = "BaseOnPython"
class TestQC(BaseQueryCompiler):
def __init__(self, modin_frame):
self._modin_frame = modin_frame
storage_format = property(
lambda self: "Base", doc=BaseQueryCompiler.storage_format.__doc__
)
engine = property(lambda self: "Python", doc=BaseQueryCompiler.engine.__doc__)
def finalize(self):
self._modin_frame.finalize()
def execute(self):
self.finalize()
self._modin_frame.wait_computations()
@classmethod
def from_pandas(cls, df, data_cls):
return cls(data_cls.from_pandas(df))
@classmethod
def from_arrow(cls, at, data_cls):
return cls(data_cls.from_arrow(at))
def free(self):
pass
def to_interchange_dataframe(
self, nan_as_null: bool = False, allow_copy: bool = True
):
raise NotImplementedError(
"The selected execution does not implement the DataFrame exchange protocol."
)
@classmethod
def from_interchange_dataframe(cls, df, data_cls):
raise NotImplementedError(
"The selected execution does not implement the DataFrame exchange protocol."
)
to_pandas = PandasQueryCompiler.to_pandas
default_to_pandas = PandasQueryCompiler.default_to_pandas
class BaseOnPythonIO(PandasOnPythonIO):
query_compiler_cls = TestQC
class BaseOnPythonFactory(factories.BaseFactory):
@classmethod
def prepare(cls):
cls.io_cls = BaseOnPythonIO
def set_base_execution(name=BASE_EXECUTION_NAME):
setattr(factories, f"{name}Factory", BaseOnPythonFactory)
Backend.register_backend(
"BaseOnPython",
Execution(
engine="Python",
storage_format="Base",
),
)
modin.set_execution(engine="python", storage_format=name.split("On")[0])
@pytest.fixture(scope="function")
def get_unique_base_execution():
"""Setup unique execution for a single function and yield its QueryCompiler that's suitable for inplace modifications."""
# It's better to use decimal IDs rather than hex ones due to factory names formatting
execution_id = int(uuid.uuid4().hex, 16)
format_name = f"Base{execution_id}"
engine_name = "Python"
execution_name = f"{format_name}On{engine_name}"
# Dynamically building all the required classes to form a new execution
base_qc = type(
format_name, (TestQC,), {"get_backend": (lambda self: execution_name)}
)
base_io = type(
f"{execution_name}IO", (BaseOnPythonIO,), {"query_compiler_cls": base_qc}
)
base_factory = type(
f"{execution_name}Factory",
(BaseOnPythonFactory,),
{"prepare": classmethod(lambda cls: setattr(cls, "io_cls", base_io))},
)
# Setting up the new execution
setattr(factories, f"{execution_name}Factory", base_factory)
Backend.register_backend(
execution_name, Execution(engine=engine_name, storage_format=format_name)
)
old_engine, old_format = modin.set_execution(
engine=engine_name, storage_format=format_name
)
yield base_qc
# Teardown the new execution
modin.set_execution(engine=old_engine, storage_format=old_format)
try:
delattr(factories, f"{execution_name}Factory")
except AttributeError:
pass
def pytest_configure(config):
execution = config.option.execution
if execution is None:
return
if execution == BASE_EXECUTION_NAME:
set_base_execution(BASE_EXECUTION_NAME)
config.addinivalue_line(
"filterwarnings", "default:.*defaulting to pandas.*:UserWarning"
)
else:
partition, engine = execution.split("On")
modin.set_execution(engine=engine, storage_format=partition)
def pytest_runtest_call(item):
custom_markers = ["xfail", "skip"]
# dynamicly adding custom markers to tests
for custom_marker in custom_markers:
for marker in item.iter_markers(name=f"{custom_marker}_executions"):
executions = marker.args[0]
if not isinstance(executions, list):
executions = [executions]
current_execution = modin.utils.get_current_execution()
reason = marker.kwargs.pop("reason", "")
item.add_marker(
getattr(pytest.mark, custom_marker)(
condition=current_execution in executions,
reason=f"Execution {current_execution} does not pass this test. {reason}",
**marker.kwargs,
)
)
_doc_pytest_fixture = """
Pytest fixture factory that makes temp {file_type} files for testing.
Yields:
Function that generates {file_type} files
"""
@pytest.fixture(scope="class")
def TestReadCSVFixture(tmp_path_factory):
tmp_path = tmp_path_factory.mktemp("TestReadCSVFixture")
creator = _make_csv_file(data_dir=tmp_path)
# each xdist worker spawned in separate process with separate namespace and dataset
pytest.csvs_names = {}
# test_read_csv_col_handling, test_read_csv_parsing
pytest.csvs_names["test_read_csv_regular"] = creator()
# test_read_csv_parsing
pytest.csvs_names["test_read_csv_yes_no"] = creator(
additional_col_values=["Yes", "true", "No", "false"],
)
# test_read_csv_col_handling
pytest.csvs_names["test_read_csv_blank_lines"] = creator(
add_blank_lines=True,
)
# test_read_csv_nans_handling
pytest.csvs_names["test_read_csv_nans"] = creator(
add_blank_lines=True,
additional_col_values=["", "N/A", "NA", "NULL", "custom_nan", "73"],
)
# test_read_csv_error_handling
pytest.csvs_names["test_read_csv_bad_lines"] = creator(
add_bad_lines=True,
)
yield
@pytest.fixture
@doc(_doc_pytest_fixture, file_type="csv")
def make_csv_file(tmp_path):
yield _make_csv_file(data_dir=tmp_path)
def create_fixture(file_type):
@doc(_doc_pytest_fixture, file_type=file_type)
def fixture(tmp_path):
yield make_default_file(file_type=file_type, data_dir=tmp_path)
return fixture
for file_type in ("json", "html", "excel", "feather", "stata", "hdf", "pickle", "fwf"):
fixture = create_fixture(file_type)
fixture.__name__ = f"make_{file_type}_file"
globals()[fixture.__name__] = pytest.fixture(fixture)
@pytest.fixture
def make_parquet_file():
"""Pytest fixture factory that makes a parquet file/dir for testing.
Yields:
Function that generates a parquet file/dir
"""
filenames = []
def _make_parquet_file(
filename,
nrows=NROWS,
ncols=2,
force=True,
range_index_start=0,
range_index_step=1,
range_index_name=None,
partitioned_columns=[],
row_group_size: Optional[int] = None,
):
"""Helper function to generate parquet files/directories.
Args:
filename: The name of test file, that should be created.
nrows: Number of rows for the dataframe.
ncols: Number of cols for the dataframe.
force: Create a new file/directory even if one already exists.
partitioned_columns: Create a partitioned directory using pandas.
row_group_size: Maximum size of each row group.
"""
if force or not os.path.exists(filename):
df = pandas.DataFrame(
{f"col{x + 1}": np.arange(nrows) for x in range(ncols)}
)
index = pandas.RangeIndex(
start=range_index_start,
stop=range_index_start + (nrows * range_index_step),
step=range_index_step,
name=range_index_name,
)
if (
range_index_start == 0
and range_index_step == 1
and range_index_name is None
):
assert df.index.equals(index)
else:
df.index = index
if len(partitioned_columns) > 0:
df.to_parquet(
filename,
partition_cols=partitioned_columns,
row_group_size=row_group_size,
)
else:
df.to_parquet(filename, row_group_size=row_group_size)
filenames.append(filename)
# Return function that generates parquet files
yield _make_parquet_file
# Delete parquet file that was created
for path in filenames:
if os.path.exists(path):
if os.path.isdir(path):
shutil.rmtree(path)
else:
os.remove(path)
@pytest.fixture
def make_sql_connection():
"""Sets up sql connections and takes them down after the caller is done.
Yields:
Factory that generates sql connection objects
"""
def _sql_connection(filename, table=""):
# Remove file if exists
if os.path.exists(filename):
os.remove(filename)
# Create connection and, if needed, table
conn = "sqlite:///{}".format(filename)
if table:
df = pandas.DataFrame(
{
"col1": [0, 1, 2, 3, 4, 5, 6],
"col2": [7, 8, 9, 10, 11, 12, 13],
"col3": [14, 15, 16, 17, 18, 19, 20],
"col4": [21, 22, 23, 24, 25, 26, 27],
"col5": [0, 0, 0, 0, 0, 0, 0],
}
)
df.to_sql(table, conn)
return conn
yield _sql_connection
@pytest.fixture(scope="class")
def TestReadGlobCSVFixture(tmp_path_factory):
tmp_path = tmp_path_factory.mktemp("TestReadGlobCSVFixture")
base_name = get_unique_filename(extension="")
pytest.glob_path = str(tmp_path / "{}_*.csv".format(base_name))
pytest.files = [str(tmp_path / "{}_{}.csv".format(base_name, i)) for i in range(11)]
for fname in pytest.files:
# Glob does not guarantee ordering so we have to remove the randomness in the generated csvs.
_make_csv_file(data_dir=tmp_path)(fname, row_size=11, remove_randomness=True)
yield
@pytest.fixture
def get_generated_doc_urls():
return lambda: _generated_doc_urls
@pytest.fixture
def set_num_partitions(request):
old_num_partitions = NPartitions.get()
NPartitions.put(request.param)
yield
NPartitions.put(old_num_partitions)
@pytest.fixture()
def set_benchmark_mode(request):
old_benchmark_mode = BenchmarkMode.get()
BenchmarkMode.put(request.param)
yield
BenchmarkMode.put(old_benchmark_mode)
@pytest.fixture
def set_async_read_mode(request):
old_async_read_mode = AsyncReadMode.get()
AsyncReadMode.put(request.param)
yield
AsyncReadMode.put(old_async_read_mode)
@pytest.fixture
def set_min_row_partition_size(request):
old_min_row_partition_size = MinRowPartitionSize.get()
MinRowPartitionSize.put(request.param)
yield
MinRowPartitionSize.put(old_min_row_partition_size)
ray_client_server = None
@pytest.fixture
def s3_storage_options(worker_id):
# # copied from pandas conftest.py:
# https://github.com/pandas-dev/pandas/blob/32f789fbc5d5a72d9d1ac14935635289eeac9009/pandas/tests/io/conftest.py#L45
# worker_id is a pytest fixture
if GithubCI.get():
url = "http://localhost:5000/"
else:
# If we hit this else-case, this test is being run locally. In that case, we want
# each worker to point to a different port for its mock S3 service. The easiest way
# to do that is to use the `worker_id`, which is unique, to determine what port to point
# to. We arbitrarily assign `5` as a worker id to the master worker, since we need a number
# for each worker, and we never run tests with more than `pytest -n 4`.
worker_id = "5" if worker_id == "master" else worker_id.lstrip("gw")
url = f"http://127.0.0.1:555{worker_id}/"
return {"client_kwargs": {"endpoint_url": url}}
@pytest.fixture(scope="session")
def monkeysession():
with pytest.MonkeyPatch.context() as mp:
yield mp
@pytest.fixture(scope="session")
def s3_base(worker_id, monkeysession):
"""
Fixture for mocking S3 interaction.
Sets up moto server in separate process locally.
Yields
------
str
URL for motoserver/moto CI service.
"""
# copied from pandas conftest.py
# still need access keys for https://github.com/getmoto/moto/issues/1924
monkeysession.setenv("AWS_ACCESS_KEY_ID", "foobar_key")
monkeysession.setenv("AWS_SECRET_ACCESS_KEY", "foobar_secret")
monkeysession.setenv("AWS_REGION", "us-west-2")
if GithubCI.get():
if sys.platform in ("darwin", "win32", "cygwin") or (
platform.machine() in ("arm64", "aarch64")
or platform.machine().startswith("armv")
):
# pandas comments say:
# DO NOT RUN on Windows/macOS/ARM, only Ubuntu
# - subprocess in CI can cause timeouts
# - GitHub Actions do not support
# container services for the above OSs
pytest.skip(
"S3 tests do not have a corresponding service in Windows, macOS "
+ "or ARM platforms"
)
else:
# assume CI has started moto in docker container:
# https://docs.getmoto.org/en/latest/docs/server_mode.html#run-using-docker
# It would be nice to start moto on another thread as in the
# instructions here:
# https://docs.getmoto.org/en/latest/docs/server_mode.html#start-within-python
# but that gives 403 forbidden error when we try to create the bucket
yield "http://localhost:5000"
else:
# Launching moto in server mode, i.e., as a separate process
# with an S3 endpoint on localhost
# If we hit this else-case, this test is being run locally. In that case, we want
# each worker to point to a different port for its mock S3 service. The easiest way
# to do that is to use the `worker_id`, which is unique, to determine what port to point
# to.
endpoint_port = (
5500 if worker_id == "master" else (5550 + int(worker_id.lstrip("gw")))
)
endpoint_uri = f"http://127.0.0.1:{endpoint_port}/"
# pipe to null to avoid logging in terminal
# TODO any way to throw the error from here? e.g. i had an annoying problem
# where I didn't have flask-cors and moto just failed .if there's an error
# in the popen command and we throw an error within the body of the context
# manager, the test just hangs forever.
with subprocess.Popen(
["moto_server", "s3", "-p", str(endpoint_port)],
stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
) as proc:
for _ in range(50):
try:
# OK to go once server is accepting connections
if requests.get(endpoint_uri).ok:
break
except Exception:
# try again while we still have retries
time.sleep(0.1)
else:
proc.terminate()
_, errs = proc.communicate()
raise RuntimeError(
"Could not connect to moto server after 50 tries. "
+ f"See stderr for extra info: {errs}"
)
yield endpoint_uri
proc.terminate()
@pytest.fixture
def s3_resource(s3_base):
"""
Set up S3 bucket with contents. The primary bucket name is "modin-test".
When running locally, this function should be safe even if there are multiple pytest
workers running in parallel because each worker gets its own endpoint. When running
in CI, we use a single endpoint for all workers, so we can't have multiple pytest
workers running in parallel.
"""
bucket = "modin-test"
conn = boto3.resource("s3", endpoint_url=s3_base)
cli = boto3.client("s3", endpoint_url=s3_base)
# https://github.com/getmoto/moto/issues/3292
# without location, I get
# botocore.exceptions.ClientError: An error occurred
# (IllegalLocationConstraintException) when calling the CreateBucket operation:
# The unspecified location constraint is incompatible for the region specific
# endpoint this request was sent to.
# even if I delete os.environ['AWS_REGION'] but somehow pandas can get away with
# this.
try:
cli.create_bucket(
Bucket=bucket, CreateBucketConfiguration={"LocationConstraint": "us-west-2"}
)
except Exception as e:
# OK if bucket already exists, but want to raise other exceptions.
# The exception raised by `create_bucket` is made using a factory,
# so we need to check using this method of reading the response rather
# than just checking the type of the exception.
response = getattr(e, "response", {})
error_code = response.get("Error", {}).get("Code", "")
if error_code not in ("BucketAlreadyOwnedByYou", "BucketAlreadyExists"):
raise
for _ in range(20):
# We want to wait until bucket creation is finished.
if cli.list_buckets()["Buckets"]:
break
time.sleep(0.1)
if not cli.list_buckets()["Buckets"]:
raise RuntimeError("Could not create bucket")
s3fs.S3FileSystem.clear_instance_cache()
s3 = s3fs.S3FileSystem(client_kwargs={"endpoint_url": s3_base})
test_s3_files = [
("modin-bugs/multiple_csv/", "modin/tests/pandas/data/multiple_csv/"),
(
"modin-bugs/test_data_dir.parquet/",
"modin/tests/pandas/data/test_data_dir.parquet/",
),
("modin-bugs/test_data.parquet", "modin/tests/pandas/data/test_data.parquet"),
("modin-bugs/test_data.json", "modin/tests/pandas/data/test_data.json"),
("modin-bugs/test_data.fwf", "modin/tests/pandas/data/test_data.fwf"),
("modin-bugs/test_data.feather", "modin/tests/pandas/data/test_data.feather"),
("modin-bugs/issue5159.parquet/", "modin/tests/pandas/data/issue5159.parquet/"),
]
for s3_key, file_name in test_s3_files:
s3.put(file_name, f"{bucket}/{s3_key}", recursive=s3_key.endswith("/"))
yield conn
s3.rm(bucket, recursive=True)
for _ in range(20):
# We want to wait until the deletion finishes.
if not cli.list_buckets()["Buckets"]:
break
time.sleep(0.1)
@pytest.fixture
def modify_config(request):
values = request.param
old_values = {}
for key, value in values.items():
old_values[key] = key.get()
key.put(value)
yield # waiting for the test to be completed
# restoring old parameters
for key, value in old_values.items():
try:
key.put(value)
except ValueError as e:
# sometimes bool env variables have 'None' as a default value, which
# causes a ValueError when we try to set this value back, as technically,
# only bool values are allowed (and 'None' is not a bool), in this case
# we try to set 'False' instead
if key.type == bool and value is None:
key.put(False)
else:
raise e
@contextmanager
def copy_and_restore(
dicts: Iterable[defaultdict],
) -> None:
"""
Make deep copies of defaultdicts and restore them upon exiting this context.
Ideally this function would be a fixture, but we want to pass it parameters
and use it in other fixtures, and it does not seem to be possible to pass
parameters from one fixture to another.
Parameters
----------
dicts : Iterable[defaultdict]
The dicts to copy and restore.
"""
try:
# Use a tuples of tuples instead of a dict mapping each original dict
# to its copy, because the original dict is not hashable.
original_dict_to_copy = tuple(
(original_dict, copy.deepcopy(original_dict)) for original_dict in dicts
)
yield
finally:
for original_dict, dict_copy in original_dict_to_copy:
original_dict.clear()
original_dict.update(dict_copy)
@pytest.fixture(autouse=True)
def clean_up_extensions():
with copy_and_restore(
(
pd.dataframe.DataFrame._extensions,
pd.Series._extensions,
pd.base.BasePandasDataset._extensions,
_GENERAL_EXTENSIONS,
pd.groupby.DataFrameGroupBy._extensions,
pd.groupby.SeriesGroupBy._extensions,
)
):
yield
from modin.pandas.api.extensions.extensions import _attrs_to_delete_on_test
for k, v in _attrs_to_delete_on_test.items():
for obj in v:
delattr(k, obj)
_attrs_to_delete_on_test.clear()
@pytest.fixture(autouse=True)
def clean_up_auto_backend_switching():
with copy_and_restore(
(
_CLASS_AND_BACKEND_TO_POST_OP_SWITCH_METHODS,
_CLASS_AND_BACKEND_TO_PRE_OP_SWITCH_METHODS,
)
):
yield
@pytest.fixture(autouse=True)
def assert_no_root_logging(caplog):
try:
import xgboost
except ImportError:
xgboost_path = None
else:
xgboost_path = os.path.dirname(xgboost.__file__)
root_logger = logging.getLogger()
# Capture logs at any level, i.e. at level >= logging.NOTSET.
with caplog.at_level(logging.NOTSET):
yield
# Note that because this code is in a fixture, we have to use
# caplog.get_records(when="call") instead of caplog.records:
# https://github.com/pytest-dev/pytest/issues/4033
assert not any(
record.name == root_logger.name
# Allow xgboost to log to root.
# TODO(https://github.com/modin-project/modin/issues/5194): Check
# whether we can remove this exception once we use a newer version of
# xgboost.
and not (xgboost_path is not None and record.pathname.startswith(xgboost_path))
for record in caplog.get_records(when="call")
)
================================================
FILE: modin/core/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's core functionality."""
================================================
FILE: modin/core/computation/__init__.py
================================================
================================================
FILE: modin/core/computation/align.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Core eval alignment algorithms. Forked from pandas.core.computation.align
"""
from __future__ import annotations
import warnings
from collections.abc import Sequence
from functools import (
partial,
wraps,
)
from typing import (
Callable,
)
import numpy as np
import pandas
import pandas.core.common as com
from pandas._typing import F
from pandas.core.base import PandasObject
from pandas.errors import PerformanceWarning
from modin.core.computation.common import result_type_many
from modin.pandas import DataFrame, Series
from modin.pandas.base import BasePandasDataset
def _align_core_single_unary_op(
term,
) -> tuple[partial | type[BasePandasDataset], dict[str, pandas.Index] | None]:
typ: partial | type[BasePandasDataset]
axes: dict[str, pandas.Index] | None = None
if isinstance(term.value, np.ndarray):
typ = partial(np.asanyarray, dtype=term.value.dtype)
else:
typ = type(term.value)
if hasattr(term.value, "axes"):
axes = _zip_axes_from_type(typ, term.value.axes)
return typ, axes
def _zip_axes_from_type(
typ: type[BasePandasDataset], new_axes: Sequence[pandas.Index]
) -> dict[str, pandas.Index]:
return {name: new_axes[i] for i, name in enumerate(typ._AXIS_ORDERS)}
def _any_pandas_objects(terms) -> bool:
"""
Check a sequence of terms for instances of PandasObject.
"""
return any(isinstance(term.value, PandasObject) for term in terms)
def _filter_special_cases(f) -> Callable[[F], F]:
@wraps(f)
def wrapper(terms):
# single unary operand
if len(terms) == 1:
return _align_core_single_unary_op(terms[0])
term_values = (term.value for term in terms)
# we don't have any pandas objects
if not _any_pandas_objects(terms):
return result_type_many(*term_values), None
return f(terms)
return wrapper
@_filter_special_cases
def _align_core(terms):
term_index = [i for i, term in enumerate(terms) if hasattr(term.value, "axes")]
term_dims = [terms[i].value.ndim for i in term_index]
ndims = pandas.Series(dict(zip(term_index, term_dims)))
# initial axes are the axes of the largest-axis'd term
biggest = terms[ndims.idxmax()].value
typ = biggest._constructor
axes = biggest.axes
naxes = len(axes)
gt_than_one_axis = naxes > 1
for value in (terms[i].value for i in term_index):
is_series = isinstance(value, Series)
is_series_and_gt_one_axis = is_series and gt_than_one_axis
for axis, items in enumerate(value.axes):
if is_series_and_gt_one_axis:
ax, itm = naxes - 1, value.index
else:
ax, itm = axis, items
if not axes[ax].is_(itm):
axes[ax] = axes[ax].union(itm)
for i, ndim in ndims.items():
for axis, items in zip(range(ndim), axes):
ti = terms[i].value
if hasattr(ti, "reindex"):
transpose = isinstance(ti, Series) and naxes > 1
reindexer = axes[naxes - 1] if transpose else items
term_axis_size = len(ti.axes[axis])
reindexer_size = len(reindexer)
ordm = np.log10(max(1, abs(reindexer_size - term_axis_size)))
if ordm >= 1 and reindexer_size >= 10000:
w = (
f"Alignment difference on axis {axis} is larger "
+ f"than an order of magnitude on term {repr(terms[i].name)}, "
+ f"by more than {ordm:.4g}; performance may suffer."
)
warnings.warn(w, category=PerformanceWarning)
obj = ti.reindex(reindexer, axis=axis, copy=False)
terms[i].update(obj)
terms[i].update(terms[i].value.values)
return typ, _zip_axes_from_type(typ, axes)
def align_terms(terms):
"""
Align a set of terms.
"""
try:
# flatten the parse tree (a nested list, really)
terms = list(com.flatten(terms))
except TypeError:
# can't iterate so it must just be a constant or single variable
if isinstance(terms.value, (Series, DataFrame)):
typ = type(terms.value)
return typ, _zip_axes_from_type(typ, terms.value.axes)
return np.result_type(terms.type), None
# if all resolved variables are numeric scalars
if all(term.is_scalar for term in terms):
return result_type_many(*(term.value for term in terms)).type, None
# perform the main alignment
typ, axes = _align_core(terms)
return typ, axes
def reconstruct_object(typ, obj, axes, dtype):
"""
Reconstruct an object given its type, raw value, and possibly empty
(None) axes.
Parameters
----------
typ : object
A type
obj : object
The value to use in the type constructor
axes : dict
The axes to use to construct the resulting pandas object
Returns
-------
ret : typ
An object of type ``typ`` with the value `obj` and possible axes
`axes`.
"""
try:
typ = typ.type
except AttributeError:
pass
res_t = np.result_type(obj.dtype, dtype)
if not isinstance(typ, partial) and issubclass(typ, PandasObject):
return typ(obj, dtype=res_t, **axes)
# special case for pathological things like ~True/~False
if hasattr(res_t, "type") and typ == np.bool_ and res_t != np.bool_:
ret_value = res_t.type(obj)
else:
ret_value = typ(obj).astype(res_t)
# The condition is to distinguish 0-dim array (returned in case of
# scalar) and 1 element array
# e.g. np.array(0) and np.array([0])
if (
len(obj.shape) == 1
and len(obj) == 1
and not isinstance(ret_value, np.ndarray)
):
ret_value = np.array([ret_value]).astype(res_t)
return ret_value
================================================
FILE: modin/core/computation/check.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Forked from pandas.core.computation.check
"""
from __future__ import annotations
from pandas.compat._optional import import_optional_dependency
ne = import_optional_dependency("numexpr", errors="warn")
NUMEXPR_INSTALLED = ne is not None
__all__ = ["NUMEXPR_INSTALLED"]
================================================
FILE: modin/core/computation/common.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Forked from pandas.core.computation.common
"""
from __future__ import annotations
from functools import reduce
import numpy as np
from pandas._config import get_option
from pandas.core.dtypes.cast import find_common_type
from pandas.core.dtypes.common import is_extension_array_dtype
def ensure_decoded(s) -> str:
"""
If we have bytes, decode them to unicode.
"""
if isinstance(s, (np.bytes_, bytes)):
s = s.decode(get_option("display.encoding"))
return s
def result_type_many(*arrays_and_dtypes):
"""
Wrapper around numpy.result_type which overcomes the NPY_MAXARGS (32)
argument limit.
"""
try:
return np.result_type(*arrays_and_dtypes)
except ValueError:
# we have > NPY_MAXARGS terms in our expression
return reduce(np.result_type, arrays_and_dtypes)
except TypeError:
arr_and_dtypes = list(arrays_and_dtypes)
ea_dtypes, non_ea_dtypes = [], []
for arr_or_dtype in arr_and_dtypes:
if is_extension_array_dtype(arr_or_dtype):
ea_dtypes.append(arr_or_dtype)
else:
non_ea_dtypes.append(arr_or_dtype)
if non_ea_dtypes:
try:
np_dtype = np.result_type(*non_ea_dtypes)
except ValueError:
np_dtype = reduce(np.result_type, arrays_and_dtypes)
return find_common_type(ea_dtypes + [np_dtype])
return find_common_type(ea_dtypes)
================================================
FILE: modin/core/computation/engines.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Engine classes for :func:`~pandas.eval`. Forked from pandas.core.computation.engines
"""
from __future__ import annotations
import abc
from typing import TYPE_CHECKING
from pandas.errors import NumExprClobberingError
from pandas.io.formats import printing
from modin.core.computation.align import (
align_terms,
reconstruct_object,
)
from modin.core.computation.ops import (
MATHOPS,
REDUCTIONS,
)
if TYPE_CHECKING:
from modin.core.computation.expr import Expr
_ne_builtins = frozenset(MATHOPS + REDUCTIONS)
def _check_ne_builtin_clash(expr: Expr) -> None:
"""
Attempt to prevent foot-shooting in a helpful way.
Parameters
----------
expr : Expr
Terms can contain
"""
names = expr.names
overlap = names & _ne_builtins
if overlap:
s = ", ".join([repr(x) for x in overlap])
raise NumExprClobberingError(
f'Variables in expression "{expr}" overlap with builtins: ({s})'
)
class AbstractEngine(metaclass=abc.ABCMeta):
"""Object serving as a base class for all engines."""
has_neg_frac = False
def __init__(self, expr) -> None:
self.expr = expr
self.aligned_axes = None
self.result_type = None
def convert(self) -> str:
"""
Convert an expression for evaluation.
Defaults to return the expression as a string.
"""
return printing.pprint_thing(self.expr)
def evaluate(self) -> object:
"""
Run the engine on the expression.
This method performs alignment which is necessary no matter what engine
is being used, thus its implementation is in the base class.
Returns
-------
object
The result of the passed expression.
"""
if not self._is_aligned:
self.result_type, self.aligned_axes = align_terms(self.expr.terms)
# make sure no names in resolvers and locals/globals clash
res = self._evaluate()
return reconstruct_object(
self.result_type, res, self.aligned_axes, self.expr.terms.return_type
)
@property
def _is_aligned(self) -> bool:
return self.aligned_axes is not None and self.result_type is not None
@abc.abstractmethod
def _evaluate(self):
"""
Return an evaluated expression.
Parameters
----------
env : Scope
The local and global environment in which to evaluate an
expression.
Notes
-----
Must be implemented by subclasses.
"""
class NumExprEngine(AbstractEngine):
"""NumExpr engine class"""
has_neg_frac = True
def _evaluate(self):
import numexpr as ne
# convert the expression to a valid numexpr expression
s = self.convert()
env = self.expr.env
scope = env.full_scope
_check_ne_builtin_clash(self.expr)
return ne.evaluate(s, local_dict=scope)
class PythonEngine(AbstractEngine):
"""
Evaluate an expression in Python space.
Mostly for testing purposes.
"""
has_neg_frac = False
def evaluate(self):
return self.expr()
def _evaluate(self) -> None:
pass
ENGINES: dict[str, type[AbstractEngine]] = {
"numexpr": NumExprEngine,
"python": PythonEngine,
}
================================================
FILE: modin/core/computation/eval.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Top level ``eval`` module. Forked from pandas.core.computation.eval
"""
from __future__ import annotations
import tokenize
import warnings
from pandas.core.dtypes.common import is_extension_array_dtype
from pandas.io.formats.printing import pprint_thing
from pandas.util._validators import validate_bool_kwarg
from modin.core.computation.check import NUMEXPR_INSTALLED
from modin.core.computation.engines import ENGINES
from modin.core.computation.expr import (
PARSERS,
Expr,
)
from modin.core.computation.ops import BinOp
from modin.core.computation.parsing import tokenize_string
from modin.core.computation.scope import ensure_scope
from modin.pandas.base import BasePandasDataset
def _check_engine(engine: str | None) -> str:
"""
Make sure a valid engine is passed.
Parameters
----------
engine : str
String to validate.
Raises
------
KeyError
* If an invalid engine is passed.
ImportError
* If numexpr was requested but doesn't exist.
Returns
-------
str
Engine name.
"""
if engine is None:
engine = "numexpr" if NUMEXPR_INSTALLED else "python"
if engine not in ENGINES:
valid_engines = list(ENGINES.keys())
raise KeyError(
f"Invalid engine '{engine}' passed, valid engines are {valid_engines}"
)
# TODO: validate this in a more general way (thinking of future engines
# that won't necessarily be import-able)
# Could potentially be done on engine instantiation
if engine == "numexpr" and not NUMEXPR_INSTALLED:
raise ImportError(
"'numexpr' is not installed or an unsupported version. Cannot use "
+ "engine='numexpr' for query/eval if 'numexpr' is not installed"
)
return engine
def _check_parser(parser: str):
"""
Make sure a valid parser is passed.
Parameters
----------
parser : str
Raises
------
KeyError
* If an invalid parser is passed
"""
if parser not in PARSERS:
raise KeyError(
f"Invalid parser '{parser}' passed, valid parsers are {PARSERS.keys()}"
)
def _check_resolvers(resolvers):
if resolvers is not None:
for resolver in resolvers:
if not hasattr(resolver, "__getitem__"):
name = type(resolver).__name__
raise TypeError(
f"Resolver of type '{name}' does not "
+ "implement the __getitem__ method"
)
def _check_expression(expr):
"""
Make sure an expression is not an empty string
Parameters
----------
expr : object
An object that can be converted to a string
Raises
------
ValueError
* If expr is an empty string
"""
if not expr:
raise ValueError("expr cannot be an empty string")
def _convert_expression(expr) -> str:
"""
Convert an object to an expression.
This function converts an object to an expression (a unicode string) and
checks to make sure it isn't empty after conversion. This is used to
convert operators to their string representation for recursive calls to
:func:`~pandas.eval`.
Parameters
----------
expr : object
The object to be converted to a string.
Returns
-------
str
The string representation of an object.
Raises
------
ValueError
* If the expression is empty.
"""
s = pprint_thing(expr)
_check_expression(s)
return s
def _check_for_locals(expr: str, stack_level: int, parser: str):
at_top_of_stack = stack_level == 0
not_pandas_parser = parser != "pandas"
if not_pandas_parser:
msg = "The '@' prefix is only supported by the pandas parser"
elif at_top_of_stack:
msg = (
"The '@' prefix is not allowed in top-level eval calls.\n"
+ "please refer to your variables by name without the '@' prefix."
)
if at_top_of_stack or not_pandas_parser:
for toknum, tokval in tokenize_string(expr):
if toknum == tokenize.OP and tokval == "@":
raise SyntaxError(msg)
def eval(
expr: str | BinOp, # we leave BinOp out of the docstr bc it isn't for users
parser: str = "pandas",
engine: str | None = None,
local_dict=None,
global_dict=None,
resolvers=(),
level: int = 0,
target=None,
inplace: bool = False,
):
"""
Evaluate a Python expression as a string using various backends.
The following arithmetic operations are supported: ``+``, ``-``, ``*``,
``/``, ``**``, ``%``, ``//`` (python engine only) along with the following
boolean operations: ``|`` (or), ``&`` (and), and ``~`` (not).
Additionally, the ``'pandas'`` parser allows the use of :keyword:`and`,
:keyword:`or`, and :keyword:`not` with the same semantics as the
corresponding bitwise operators. :class:`~pandas.Series` and
:class:`~pandas.DataFrame` objects are supported and behave as they would
with plain ol' Python evaluation.
Parameters
----------
expr : str
The expression to evaluate. This string cannot contain any Python
`statements
`__,
only Python `expressions
`__.
parser : {'pandas', 'python'}, default 'pandas'
The parser to use to construct the syntax tree from the expression. The
default of ``'pandas'`` parses code slightly different than standard
Python. Alternatively, you can parse an expression using the
``'python'`` parser to retain strict Python semantics. See the
:ref:`enhancing performance ` documentation for
more details.
engine : {'python', 'numexpr'}, default 'numexpr'
The engine used to evaluate the expression. Supported engines are
- None : tries to use ``numexpr``, falls back to ``python``
- ``'numexpr'`` : This default engine evaluates pandas objects using
numexpr for large speed ups in complex expressions with large frames.
- ``'python'`` : Performs operations as if you had ``eval``'d in top
level python. This engine is generally not that useful.
More backends may be available in the future.
local_dict : dict or None, optional
A dictionary of local variables, taken from locals() by default.
global_dict : dict or None, optional
A dictionary of global variables, taken from globals() by default.
resolvers : list of dict-like or None, optional
A list of objects implementing the ``__getitem__`` special method that
you can use to inject an additional collection of namespaces to use for
variable lookup. For example, this is used in the
:meth:`~DataFrame.query` method to inject the
``DataFrame.index`` and ``DataFrame.columns``
variables that refer to their respective :class:`~pandas.DataFrame`
instance attributes.
level : int, optional
The number of prior stack frames to traverse and add to the current
scope. Most users will **not** need to change this parameter.
target : object, optional, default None
This is the target object for assignment. It is used when there is
variable assignment in the expression. If so, then `target` must
support item assignment with string keys, and if a copy is being
returned, it must also support `.copy()`.
inplace : bool, default False
If `target` is provided, and the expression mutates `target`, whether
to modify `target` inplace. Otherwise, return a copy of `target` with
the mutation.
Returns
-------
ndarray, numeric scalar, DataFrame, Series, or None
The completion value of evaluating the given code or None if ``inplace=True``.
Raises
------
ValueError
There are many instances where such an error can be raised:
- `target=None`, but the expression is multiline.
- The expression is multiline, but not all them have item assignment.
An example of such an arrangement is this:
a = b + 1
a + 2
Here, there are expressions on different lines, making it multiline,
but the last line has no variable assigned to the output of `a + 2`.
- `inplace=True`, but the expression is missing item assignment.
- Item assignment is provided, but the `target` does not support
string item assignment.
- Item assignment is provided and `inplace=False`, but the `target`
does not support the `.copy()` method
See Also
--------
DataFrame.query : Evaluates a boolean expression to query the columns
of a frame.
DataFrame.eval : Evaluate a string describing operations on
DataFrame columns.
Notes
-----
The ``dtype`` of any objects involved in an arithmetic ``%`` operation are
recursively cast to ``float64``.
See the :ref:`enhancing performance ` documentation for
more details.
Examples
--------
>>> df = pd.DataFrame({"animal": ["dog", "pig"], "age": [10, 20]})
>>> df
animal age
0 dog 10
1 pig 20
We can add a new column using ``pd.eval``:
>>> pd.eval("double_age = df.age * 2", target=df)
animal age double_age
0 dog 10 20
1 pig 20 40
"""
inplace = validate_bool_kwarg(inplace, "inplace")
exprs: list[str | BinOp]
if isinstance(expr, str):
_check_expression(expr)
exprs = [e.strip() for e in expr.splitlines() if e.strip() != ""]
else:
# ops.BinOp; for internal compat, not intended to be passed by users
exprs = [expr]
multi_line = len(exprs) > 1
if multi_line and target is None:
raise ValueError(
"multi-line expressions are only valid in the "
+ "context of data, use DataFrame.eval"
)
engine = _check_engine(engine)
_check_parser(parser)
_check_resolvers(resolvers)
ret = None
first_expr = True
target_modified = False
for expr in exprs:
expr = _convert_expression(expr)
_check_for_locals(expr, level, parser)
# get our (possibly passed-in) scope
env = ensure_scope(
level + 1,
global_dict=global_dict,
local_dict=local_dict,
resolvers=resolvers,
target=target,
)
parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
if engine == "numexpr" and (
is_extension_array_dtype(parsed_expr.terms.return_type)
or getattr(parsed_expr.terms, "operand_types", None) is not None
and any(
is_extension_array_dtype(elem)
for elem in parsed_expr.terms.operand_types
)
):
warnings.warn(
"Engine has switched to 'python' because numexpr does not support "
+ "extension array dtypes. Please set your engine to python manually.",
RuntimeWarning,
)
engine = "python"
# construct the engine and evaluate the parsed expression
eng = ENGINES[engine]
eng_inst = eng(parsed_expr)
ret = eng_inst.evaluate()
if parsed_expr.assigner is None:
if multi_line:
raise ValueError(
"Multi-line expressions are only valid "
+ "if all expressions contain an assignment"
)
if inplace:
raise ValueError("Cannot operate inplace if there is no assignment")
# assign if needed
assigner = parsed_expr.assigner
if env.target is not None and assigner is not None:
target_modified = True
# if returning a copy, copy only on the first assignment
if not inplace and first_expr:
try:
target = env.target
if isinstance(target, BasePandasDataset):
target = target.copy(deep=True)
else:
target = target.copy()
except AttributeError as err:
raise ValueError("Cannot return a copy of the target") from err
else:
target = env.target
# TypeError is most commonly raised (e.g. int, list), but you
# get IndexError if you try to do this assignment on np.ndarray.
# we will ignore numpy warnings here; e.g. if trying
# to use a non-numeric indexer
try:
if inplace and isinstance(target, BasePandasDataset):
target.loc[:, assigner] = ret
else:
target[assigner] = ret # pyright: ignore[reportGeneralTypeIssues]
except (TypeError, IndexError) as err:
raise ValueError("Cannot assign expression output to target") from err
if not resolvers:
resolvers = ({assigner: ret},)
else:
# existing resolver needs updated to handle
# case of mutating existing column in copy
for resolver in resolvers:
if assigner in resolver:
resolver[assigner] = ret
break
else:
resolvers += ({assigner: ret},)
ret = None
first_expr = False
# We want to exclude `inplace=None` as being False.
return (target if target_modified else ret) if inplace is False else None
================================================
FILE: modin/core/computation/expr.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
:func:`~pandas.eval` parsers.
Forked from pandas.core.computation.expr
"""
from __future__ import annotations
import ast
import tokenize
from functools import (
partial,
reduce,
)
from keyword import iskeyword
from typing import (
Callable,
ClassVar,
TypeVar,
)
import numpy as np
import pandas.core.common as com
from pandas.errors import UndefinedVariableError
from pandas.io.formats import printing
from modin.core.computation.ops import (
ARITH_OPS_SYMS,
BOOL_OPS_SYMS,
CMP_OPS_SYMS,
LOCAL_TAG,
UNARY_OPS_SYMS,
BinOp,
Constant,
FuncNode,
Op,
Term,
UnaryOp,
is_term,
)
from modin.core.computation.parsing import (
clean_backtick_quoted_toks,
tokenize_string,
)
from modin.core.computation.scope import Scope
def _rewrite_assign(tok: tuple[int, str]) -> tuple[int, str]:
"""
Rewrite the assignment operator for PyTables expressions that use ``=``
as a substitute for ``==``.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tuple of int, str
Either the input or token or the replacement values
"""
toknum, tokval = tok
return toknum, "==" if tokval == "=" else tokval
def _replace_booleans(tok: tuple[int, str]) -> tuple[int, str]:
"""
Replace ``&`` with ``and`` and ``|`` with ``or`` so that bitwise
precedence is changed to boolean precedence.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tuple of int, str
Either the input or token or the replacement values
"""
toknum, tokval = tok
if toknum == tokenize.OP:
if tokval == "&":
return tokenize.NAME, "and"
elif tokval == "|":
return tokenize.NAME, "or"
return toknum, tokval
return toknum, tokval
def _replace_locals(tok: tuple[int, str]) -> tuple[int, str]:
"""
Replace local variables with a syntactically valid name.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tuple of int, str
Either the input or token or the replacement values
Notes
-----
This is somewhat of a hack in that we rewrite a string such as ``'@a'`` as
``'__pd_eval_local_a'`` by telling the tokenizer that ``__pd_eval_local_``
is a ``tokenize.OP`` and to replace the ``'@'`` symbol with it.
"""
toknum, tokval = tok
if toknum == tokenize.OP and tokval == "@":
return tokenize.OP, LOCAL_TAG
return toknum, tokval
def _compose2(f, g):
"""
Compose 2 callables.
"""
return lambda *args, **kwargs: f(g(*args, **kwargs))
def _compose(*funcs):
"""
Compose 2 or more callables.
"""
assert len(funcs) > 1, "At least 2 callables must be passed to compose"
return reduce(_compose2, funcs)
def _preparse(
source: str,
f=_compose(
_replace_locals, _replace_booleans, _rewrite_assign, clean_backtick_quoted_toks
),
) -> str:
"""
Compose a collection of tokenization functions.
Parameters
----------
source : str
A Python source code string
f : callable
This takes a tuple of (toknum, tokval) as its argument and returns a
tuple with the same structure but possibly different elements. Defaults
to the composition of ``_rewrite_assign``, ``_replace_booleans``, and
``_replace_locals``.
Returns
-------
str
Valid Python source code
Notes
-----
The `f` parameter can be any callable that takes *and* returns input of the
form ``(toknum, tokval)``, where ``toknum`` is one of the constants from
the ``tokenize`` module and ``tokval`` is a string.
"""
assert callable(f), "f must be callable"
return tokenize.untokenize(f(x) for x in tokenize_string(source))
def _is_type(t):
"""
Factory for a type checking function of type ``t`` or tuple of types.
"""
return lambda x: isinstance(x.value, t)
_is_list = _is_type(list)
_is_str = _is_type(str)
# partition all AST nodes
_all_nodes = frozenset(
node
for node in (getattr(ast, name) for name in dir(ast))
if isinstance(node, type) and issubclass(node, ast.AST)
)
def _filter_nodes(superclass, all_nodes=_all_nodes):
"""
Filter out AST nodes that are subclasses of ``superclass``.
"""
node_names = (node.__name__ for node in all_nodes if issubclass(node, superclass))
return frozenset(node_names)
_all_node_names = frozenset(x.__name__ for x in _all_nodes)
_mod_nodes = _filter_nodes(ast.mod)
_stmt_nodes = _filter_nodes(ast.stmt)
_expr_context_nodes = _filter_nodes(ast.expr_context)
_boolop_nodes = _filter_nodes(ast.boolop)
_handler_nodes = _filter_nodes(ast.excepthandler)
_arguments_nodes = _filter_nodes(ast.arguments)
_keyword_nodes = _filter_nodes(ast.keyword)
_alias_nodes = _filter_nodes(ast.alias)
# nodes that we don't support directly but are needed for parsing
_hacked_nodes = frozenset(["Assign", "Module", "Expr"])
_unsupported_expr_nodes = frozenset(
[
"Yield",
"GeneratorExp",
"IfExp",
"DictComp",
"SetComp",
"Repr",
"Lambda",
"Set",
"AST",
"Is",
"IsNot",
]
)
# these nodes are low priority or won't ever be supported (e.g., AST)
_unsupported_nodes = (
_stmt_nodes
| _mod_nodes
| _handler_nodes
| _arguments_nodes
| _keyword_nodes
| _alias_nodes
| _expr_context_nodes
| _unsupported_expr_nodes
) - _hacked_nodes
# we're adding a different assignment in some cases to be equality comparison
# and we don't want `stmt` and friends in their so get only the class whose
# names are capitalized
_base_supported_nodes = (_all_node_names - _unsupported_nodes) | _hacked_nodes
intersection = _unsupported_nodes & _base_supported_nodes
_msg = f"cannot both support and not support {intersection}"
assert not intersection, _msg
def _node_not_implemented(node_name: str) -> Callable[..., None]:
"""
Return a function that raises a NotImplementedError with a passed node name.
"""
def f(self, *args, **kwargs):
raise NotImplementedError(f"'{node_name}' nodes are not implemented")
return f
_T = TypeVar("_T")
def disallow(nodes: set[str]) -> Callable[[type[_T]], type[_T]]:
"""
Decorator to disallow certain nodes from parsing. Raises a
NotImplementedError instead.
Returns
-------
callable
"""
def disallowed(cls: type[_T]) -> type[_T]:
# error: "Type[_T]" has no attribute "unsupported_nodes"
cls.unsupported_nodes = () # type: ignore[attr-defined]
for node in nodes:
new_method = _node_not_implemented(node)
name = f"visit_{node}"
# error: "Type[_T]" has no attribute "unsupported_nodes"
cls.unsupported_nodes += (name,) # type: ignore[attr-defined]
setattr(cls, name, new_method)
return cls
return disallowed
def _op_maker(op_class, op_symbol):
"""
Return a function to create an op class with its symbol already passed.
Returns
-------
callable
"""
def f(self, node, *args, **kwargs):
"""
Return a partial function with an Op subclass with an operator already passed.
Returns
-------
callable
"""
return partial(op_class, op_symbol, *args, **kwargs)
return f
_op_classes = {"binary": BinOp, "unary": UnaryOp}
def add_ops(op_classes):
"""
Decorator to add default implementation of ops.
"""
def f(cls):
for op_attr_name, op_class in op_classes.items():
ops = getattr(cls, f"{op_attr_name}_ops")
ops_map = getattr(cls, f"{op_attr_name}_op_nodes_map")
for op in ops:
op_node = ops_map[op]
if op_node is not None:
made_op = _op_maker(op_class, op)
setattr(cls, f"visit_{op_node}", made_op)
return cls
return f
@disallow(_unsupported_nodes)
@add_ops(_op_classes)
class BaseExprVisitor(ast.NodeVisitor):
"""
Custom ast walker. Parsers of other engines should subclass this class
if necessary.
Parameters
----------
env : Scope
engine : str
parser : str
preparser : callable
"""
const_type: ClassVar[type[Term]] = Constant
term_type: ClassVar[type[Term]] = Term
binary_ops = CMP_OPS_SYMS + BOOL_OPS_SYMS + ARITH_OPS_SYMS
binary_op_nodes = (
"Gt",
"Lt",
"GtE",
"LtE",
"Eq",
"NotEq",
"In",
"NotIn",
"BitAnd",
"BitOr",
"And",
"Or",
"Add",
"Sub",
"Mult",
"Div",
"Pow",
"FloorDiv",
"Mod",
)
binary_op_nodes_map = dict(zip(binary_ops, binary_op_nodes))
unary_ops = UNARY_OPS_SYMS
unary_op_nodes = "UAdd", "USub", "Invert", "Not"
unary_op_nodes_map = dict(zip(unary_ops, unary_op_nodes))
rewrite_map = {
ast.Eq: ast.In,
ast.NotEq: ast.NotIn,
ast.In: ast.In,
ast.NotIn: ast.NotIn,
}
unsupported_nodes: tuple[str, ...]
def __init__(self, env, engine, parser, preparser=_preparse) -> None:
self.env = env
self.engine = engine
self.parser = parser
self.preparser = preparser
self.assigner = None
def visit(self, node, **kwargs):
if isinstance(node, str):
clean = self.preparser(node)
try:
node = ast.fix_missing_locations(ast.parse(clean))
except SyntaxError as e:
if any(iskeyword(x) for x in clean.split()):
e.msg = "Python keyword not valid identifier in numexpr query"
raise e
method = f"visit_{type(node).__name__}"
visitor = getattr(self, method)
return visitor(node, **kwargs)
def visit_Module(self, node, **kwargs):
if len(node.body) != 1:
raise SyntaxError("only a single expression is allowed")
expr = node.body[0]
return self.visit(expr, **kwargs)
def visit_Expr(self, node, **kwargs):
return self.visit(node.value, **kwargs)
def _rewrite_membership_op(self, node, left, right):
# the kind of the operator (is actually an instance)
op_instance = node.op
op_type = type(op_instance)
# must be two terms and the comparison operator must be ==/!=/in/not in
if is_term(left) and is_term(right) and op_type in self.rewrite_map:
left_list, right_list = map(_is_list, (left, right))
left_str, right_str = map(_is_str, (left, right))
# if there are any strings or lists in the expression
if left_list or right_list or left_str or right_str:
op_instance = self.rewrite_map[op_type]()
# pop the string variable out of locals and replace it with a list
# of one string, kind of a hack
if right_str:
name = self.env.add_tmp([right.value])
right = self.term_type(name, self.env)
if left_str:
name = self.env.add_tmp([left.value])
left = self.term_type(name, self.env)
op = self.visit(op_instance)
return op, op_instance, left, right
def _maybe_transform_eq_ne(self, node, left=None, right=None):
if left is None:
left = self.visit(node.left, side="left")
if right is None:
right = self.visit(node.right, side="right")
op, op_class, left, right = self._rewrite_membership_op(node, left, right)
return op, op_class, left, right
def _maybe_downcast_constants(self, left, right):
f32 = np.dtype(np.float32)
if (
left.is_scalar
and hasattr(left, "value")
and not right.is_scalar
and right.return_type == f32
):
# right is a float32 array, left is a scalar
name = self.env.add_tmp(np.float32(left.value))
left = self.term_type(name, self.env)
if (
right.is_scalar
and hasattr(right, "value")
and not left.is_scalar
and left.return_type == f32
):
# left is a float32 array, right is a scalar
name = self.env.add_tmp(np.float32(right.value))
right = self.term_type(name, self.env)
return left, right
def _maybe_eval(self, binop, eval_in_python):
# eval `in` and `not in` (for now) in "partial" python space
# things that can be evaluated in "eval" space will be turned into
# temporary variables. for example,
# [1,2] in a + 2 * b
# in that case a + 2 * b will be evaluated using numexpr, and the "in"
# call will be evaluated using isin (in python space)
return binop.evaluate(
self.env, self.engine, self.parser, self.term_type, eval_in_python
)
def _maybe_evaluate_binop(
self,
op,
op_class,
lhs,
rhs,
eval_in_python=("in", "not in"),
maybe_eval_in_python=("==", "!=", "<", ">", "<=", ">="),
):
res = op(lhs, rhs)
if res.has_invalid_return_type:
raise TypeError(
f"unsupported operand type(s) for {res.op}: "
+ f"'{lhs.type}' and '{rhs.type}'"
)
if self.engine != "pytables" and (
res.op in CMP_OPS_SYMS
and getattr(lhs, "is_datetime", False)
or getattr(rhs, "is_datetime", False)
):
# all date ops must be done in python bc numexpr doesn't work
# well with NaT
return self._maybe_eval(res, self.binary_ops)
if res.op in eval_in_python:
# "in"/"not in" ops are always evaluated in python
return self._maybe_eval(res, eval_in_python)
elif self.engine != "pytables":
if (
getattr(lhs, "return_type", None) == object
or getattr(rhs, "return_type", None) == object
):
# evaluate "==" and "!=" in python if either of our operands
# has an object return type
return self._maybe_eval(res, eval_in_python + maybe_eval_in_python)
return res
def visit_BinOp(self, node, **kwargs):
op, op_class, left, right = self._maybe_transform_eq_ne(node)
left, right = self._maybe_downcast_constants(left, right)
return self._maybe_evaluate_binop(op, op_class, left, right)
def visit_UnaryOp(self, node, **kwargs):
op = self.visit(node.op)
operand = self.visit(node.operand)
return op(operand)
def visit_Name(self, node, **kwargs) -> Term:
return self.term_type(node.id, self.env, **kwargs)
# TODO(py314): deprecated since Python 3.8. Remove after Python 3.14 is min
def visit_NameConstant(self, node, **kwargs) -> Term:
return self.const_type(node.value, self.env)
# TODO(py314): deprecated since Python 3.8. Remove after Python 3.14 is min
def visit_Num(self, node, **kwargs) -> Term:
return self.const_type(node.value, self.env)
def visit_Constant(self, node, **kwargs) -> Term:
return self.const_type(node.value, self.env)
# TODO(py314): deprecated since Python 3.8. Remove after Python 3.14 is min
def visit_Str(self, node, **kwargs) -> Term:
name = self.env.add_tmp(node.s)
return self.term_type(name, self.env)
def visit_List(self, node, **kwargs) -> Term:
name = self.env.add_tmp([self.visit(e)(self.env) for e in node.elts])
return self.term_type(name, self.env)
visit_Tuple = visit_List
def visit_Index(self, node, **kwargs):
"""df.index[4]"""
return self.visit(node.value)
def visit_Subscript(self, node, **kwargs) -> Term:
from modin.core.computation.eval import eval as pd_eval
value = self.visit(node.value)
slobj = self.visit(node.slice)
result = pd_eval(
slobj, local_dict=self.env, engine=self.engine, parser=self.parser
)
try:
# a Term instance
v = value.value[result]
except AttributeError:
# an Op instance
lhs = pd_eval(
value, local_dict=self.env, engine=self.engine, parser=self.parser
)
v = lhs[result]
name = self.env.add_tmp(v)
return self.term_type(name, env=self.env)
def visit_Slice(self, node, **kwargs) -> slice:
"""df.index[slice(4,6)]"""
lower = node.lower
if lower is not None:
lower = self.visit(lower).value
upper = node.upper
if upper is not None:
upper = self.visit(upper).value
step = node.step
if step is not None:
step = self.visit(step).value
return slice(lower, upper, step)
def visit_Assign(self, node, **kwargs):
"""
support a single assignment node, like
c = a + b
set the assigner at the top level, must be a Name node which
might or might not exist in the resolvers
"""
if len(node.targets) != 1:
raise SyntaxError("can only assign a single expression")
if not isinstance(node.targets[0], ast.Name):
raise SyntaxError("left hand side of an assignment must be a single name")
if self.env.target is None:
raise ValueError("cannot assign without a target object")
try:
assigner = self.visit(node.targets[0], **kwargs)
except UndefinedVariableError:
assigner = node.targets[0].id
self.assigner = getattr(assigner, "name", assigner)
if self.assigner is None:
raise SyntaxError(
"left hand side of an assignment must be a single resolvable name"
)
return self.visit(node.value, **kwargs)
def visit_Attribute(self, node, **kwargs):
attr = node.attr
value = node.value
ctx = node.ctx
if isinstance(ctx, ast.Load):
# resolve the value
resolved = self.visit(value).value
try:
v = getattr(resolved, attr)
name = self.env.add_tmp(v)
return self.term_type(name, self.env)
except AttributeError:
# something like datetime.datetime where scope is overridden
if isinstance(value, ast.Name) and value.id == attr:
return resolved
raise
raise ValueError(f"Invalid Attribute context {type(ctx).__name__}")
def visit_Call(self, node, side=None, **kwargs):
if isinstance(node.func, ast.Attribute) and node.func.attr != "__call__":
res = self.visit_Attribute(node.func)
elif not isinstance(node.func, ast.Name):
raise TypeError("Only named functions are supported")
else:
try:
res = self.visit(node.func)
except UndefinedVariableError:
# Check if this is a supported function name
try:
res = FuncNode(node.func.id)
except ValueError:
# Raise original error
raise
if res is None:
# error: "expr" has no attribute "id"
raise ValueError(
f"Invalid function call {node.func.id}" # type: ignore[attr-defined]
)
if hasattr(res, "value"):
res = res.value
if isinstance(res, FuncNode):
new_args = [self.visit(arg) for arg in node.args]
if node.keywords:
raise TypeError(
f'Function "{res.name}" does not support keyword arguments'
)
return res(*new_args)
else:
new_args = [self.visit(arg)(self.env) for arg in node.args]
for key in node.keywords:
if not isinstance(key, ast.keyword):
# error: "expr" has no attribute "id"
raise ValueError(
"keyword error in function call "
+ f"'{node.func.id}'" # type: ignore[attr-defined]
)
if key.arg:
kwargs[key.arg] = self.visit(key.value)(self.env)
name = self.env.add_tmp(res(*new_args, **kwargs))
return self.term_type(name=name, env=self.env)
def translate_In(self, op):
return op
def visit_Compare(self, node, **kwargs):
ops = node.ops
comps = node.comparators
# base case: we have something like a CMP b
if len(comps) == 1:
op = self.translate_In(ops[0])
binop = ast.BinOp(op=op, left=node.left, right=comps[0])
return self.visit(binop)
# recursive case: we have a chained comparison, a CMP b CMP c, etc.
left = node.left
values = []
for op, comp in zip(ops, comps):
new_node = self.visit(
ast.Compare(comparators=[comp], left=left, ops=[self.translate_In(op)])
)
left = comp
values.append(new_node)
return self.visit(ast.BoolOp(op=ast.And(), values=values))
def _try_visit_binop(self, bop):
if isinstance(bop, (Op, Term)):
return bop
return self.visit(bop)
def visit_BoolOp(self, node, **kwargs):
def visitor(x, y):
lhs = self._try_visit_binop(x)
rhs = self._try_visit_binop(y)
op, op_class, lhs, rhs = self._maybe_transform_eq_ne(node, lhs, rhs)
return self._maybe_evaluate_binop(op, node.op, lhs, rhs)
operands = node.values
return reduce(visitor, operands)
_python_not_supported = frozenset(["Dict", "BoolOp", "In", "NotIn"])
@disallow(
(_unsupported_nodes | _python_not_supported)
- (_boolop_nodes | frozenset(["BoolOp", "Attribute", "In", "NotIn", "Tuple"]))
)
class PandasExprVisitor(BaseExprVisitor):
def __init__(
self,
env,
engine,
parser,
preparser=partial(
_preparse,
f=_compose(_replace_locals, _replace_booleans, clean_backtick_quoted_toks),
),
) -> None:
super().__init__(env, engine, parser, preparser)
@disallow(_unsupported_nodes | _python_not_supported | frozenset(["Not"]))
class PythonExprVisitor(BaseExprVisitor):
def __init__(
self, env, engine, parser, preparser=lambda source, f=None: source
) -> None:
super().__init__(env, engine, parser, preparser=preparser)
class Expr:
"""
Object encapsulating an expression.
Parameters
----------
expr : str
engine : str, optional, default 'numexpr'
parser : str, optional, default 'pandas'
env : Scope, optional, default None
level : int, optional, default 2
"""
env: Scope
engine: str
parser: str
def __init__(
self,
expr,
engine: str = "numexpr",
parser: str = "pandas",
env: Scope | None = None,
level: int = 0,
) -> None:
self.expr = expr
self.env = env or Scope(level=level + 1)
self.engine = engine
self.parser = parser
self._visitor = PARSERS[parser](self.env, self.engine, self.parser)
self.terms = self.parse()
@property
def assigner(self):
return getattr(self._visitor, "assigner", None)
def __call__(self):
return self.terms(self.env)
def __repr__(self) -> str:
return printing.pprint_thing(self.terms)
def __len__(self) -> int:
return len(self.expr)
def parse(self):
"""
Parse an expression.
"""
return self._visitor.visit(self.expr)
@property
def names(self):
"""
Get the names in an expression.
"""
if is_term(self.terms):
return frozenset([self.terms.name])
return frozenset(term.name for term in com.flatten(self.terms))
PARSERS = {"python": PythonExprVisitor, "pandas": PandasExprVisitor}
================================================
FILE: modin/core/computation/ops.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Operator classes for eval. Forked from pandas.core.computation.ops
"""
from __future__ import annotations
import operator
from datetime import datetime
from functools import partial
from typing import (
TYPE_CHECKING,
Callable,
Literal,
)
import numpy as np
import pandas
import pandas.core.common as com
from pandas.core.dtypes.cast import find_common_type
from pandas.core.dtypes.common import (
is_list_like,
is_scalar,
)
from pandas.io.formats.printing import (
pprint_thing,
pprint_thing_encoded,
)
from modin.core.computation.common import (
ensure_decoded,
result_type_many,
)
from modin.core.computation.scope import DEFAULT_GLOBALS
if TYPE_CHECKING:
from collections.abc import (
Iterable,
Iterator,
)
REDUCTIONS = ("sum", "prod", "min", "max")
_unary_math_ops = (
"sin",
"cos",
"exp",
"log",
"expm1",
"log1p",
"sqrt",
"sinh",
"cosh",
"tanh",
"arcsin",
"arccos",
"arctan",
"arccosh",
"arcsinh",
"arctanh",
"abs",
"log10",
"floor",
"ceil",
)
_binary_math_ops = ("arctan2",)
MATHOPS = _unary_math_ops + _binary_math_ops
LOCAL_TAG = "__pd_eval_local_"
class Term:
def __new__(cls, name, env, side=None, encoding=None):
klass = Constant if not isinstance(name, str) else cls
# error: Argument 2 for "super" not an instance of argument 1
supr_new = super(Term, klass).__new__ # type: ignore[misc]
return supr_new(klass)
is_local: bool
def __init__(self, name, env, side=None, encoding=None) -> None:
# name is a str for Term, but may be something else for subclasses
self._name = name
self.env = env
self.side = side
tname = str(name)
self.is_local = tname.startswith(LOCAL_TAG) or tname in DEFAULT_GLOBALS
self._value = self._resolve_name()
self.encoding = encoding
@property
def local_name(self) -> str:
return self.name.replace(LOCAL_TAG, "")
def __repr__(self) -> str:
return pprint_thing(self.name)
def __call__(self, *args, **kwargs):
return self.value
def evaluate(self, *args, **kwargs) -> Term:
return self
def _resolve_name(self):
local_name = str(self.local_name)
is_local = self.is_local
if local_name in self.env.scope and isinstance(
self.env.scope[local_name], type
):
is_local = False
res = self.env.resolve(local_name, is_local=is_local)
self.update(res)
if hasattr(res, "ndim") and res.ndim > 2:
raise NotImplementedError(
"N-dimensional objects, where N > 2, are not supported with eval"
)
return res
def update(self, value) -> None:
"""
search order for local (i.e., @variable) variables:
scope, key_variable
[('locals', 'local_name'),
('globals', 'local_name'),
('locals', 'key'),
('globals', 'key')]
"""
key = self.name
# if it's a variable name (otherwise a constant)
if isinstance(key, str):
self.env.swapkey(self.local_name, key, new_value=value)
self.value = value
@property
def is_scalar(self) -> bool:
return is_scalar(self._value)
@property
def type(self):
try:
# potentially very slow for large, mixed dtype frames
return find_common_type(self._value.dtypes.values)
except AttributeError:
try:
# ndarray
return self._value.dtype
except AttributeError:
# scalar
return type(self._value)
return_type = type
@property
def raw(self) -> str:
return f"{type(self).__name__}(name={repr(self.name)}, type={self.type})"
@property
def is_datetime(self) -> bool:
try:
t = self.type.type
except AttributeError:
t = self.type
return issubclass(t, (datetime, np.datetime64))
@property
def value(self):
return self._value
@value.setter
def value(self, new_value) -> None:
self._value = new_value
@property
def name(self):
return self._name
@property
def ndim(self) -> int:
return self._value.ndim
class Constant(Term):
def _resolve_name(self):
return self._name
@property
def name(self):
return self.value
def __repr__(self) -> str:
# in python 2 str() of float
# can truncate shorter than repr()
return repr(self.name)
_bool_op_map = {"not": "~", "and": "&", "or": "|"}
class Op:
"""
Hold an operator of arbitrary arity.
"""
op: str
def __init__(self, op: str, operands: Iterable[Term | Op], encoding=None) -> None:
self.op = _bool_op_map.get(op, op)
self.operands = operands
self.encoding = encoding
def __iter__(self) -> Iterator:
return iter(self.operands)
def __repr__(self) -> str:
"""
Print a generic n-ary operator and its operands using infix notation.
"""
# recurse over the operands
parened = (f"({pprint_thing(opr)})" for opr in self.operands)
return pprint_thing(f" {self.op} ".join(parened))
@property
def return_type(self):
# clobber types to bool if the op is a boolean operator
if self.op in (CMP_OPS_SYMS + BOOL_OPS_SYMS):
return np.bool_
return result_type_many(*(term.type for term in com.flatten(self)))
@property
def has_invalid_return_type(self) -> bool:
types = self.operand_types
obj_dtype_set = frozenset([np.dtype("object")])
return self.return_type == object and types - obj_dtype_set
@property
def operand_types(self):
return frozenset(term.type for term in com.flatten(self))
@property
def is_scalar(self) -> bool:
return all(operand.is_scalar for operand in self.operands)
@property
def is_datetime(self) -> bool:
try:
t = self.return_type.type
except AttributeError:
t = self.return_type
return issubclass(t, (datetime, np.datetime64))
def _in(x, y):
"""
Compute the vectorized membership of ``x in y`` if possible, otherwise
use Python.
"""
try:
return x.isin(y)
except AttributeError:
if is_list_like(x):
try:
return y.isin(x)
except AttributeError:
pass
return x in y
def _not_in(x, y):
"""
Compute the vectorized membership of ``x not in y`` if possible,
otherwise use Python.
"""
try:
return ~x.isin(y)
except AttributeError:
if is_list_like(x):
try:
return ~y.isin(x)
except AttributeError:
pass
return x not in y
CMP_OPS_SYMS = (">", "<", ">=", "<=", "==", "!=", "in", "not in")
_cmp_ops_funcs = (
operator.gt,
operator.lt,
operator.ge,
operator.le,
operator.eq,
operator.ne,
_in,
_not_in,
)
_cmp_ops_dict = dict(zip(CMP_OPS_SYMS, _cmp_ops_funcs))
BOOL_OPS_SYMS = ("&", "|", "and", "or")
_bool_ops_funcs = (operator.and_, operator.or_, operator.and_, operator.or_)
_bool_ops_dict = dict(zip(BOOL_OPS_SYMS, _bool_ops_funcs))
ARITH_OPS_SYMS = ("+", "-", "*", "/", "**", "//", "%")
_arith_ops_funcs = (
operator.add,
operator.sub,
operator.mul,
operator.truediv,
operator.pow,
operator.floordiv,
operator.mod,
)
_arith_ops_dict = dict(zip(ARITH_OPS_SYMS, _arith_ops_funcs))
SPECIAL_CASE_ARITH_OPS_SYMS = ("**", "//", "%")
_binary_ops_dict = {}
for d in (_cmp_ops_dict, _bool_ops_dict, _arith_ops_dict):
_binary_ops_dict.update(d)
def is_term(obj) -> bool:
return isinstance(obj, Term)
class BinOp(Op):
"""
Hold a binary operator and its operands.
Parameters
----------
op : str
lhs : Term or Op
rhs : Term or Op
"""
def __init__(self, op: str, lhs, rhs) -> None:
super().__init__(op, (lhs, rhs))
self.lhs = lhs
self.rhs = rhs
self._disallow_scalar_only_bool_ops()
self.convert_values()
try:
self.func = _binary_ops_dict[op]
except KeyError as err:
# has to be made a list for python3
keys = list(_binary_ops_dict.keys())
raise ValueError(
f"Invalid binary operator {repr(op)}, valid operators are {keys}"
) from err
def __call__(self, env):
"""
Recursively evaluate an expression in Python space.
Parameters
----------
env : Scope
Returns
-------
object
The result of an evaluated expression.
"""
# recurse over the left/right nodes
left = self.lhs(env)
right = self.rhs(env)
return self.func(left, right)
def evaluate(self, env, engine: str, parser, term_type, eval_in_python):
"""
Evaluate a binary operation *before* being passed to the engine.
Parameters
----------
env : Scope
engine : str
parser : str
term_type : type
eval_in_python : list
Returns
-------
term_type
The "pre-evaluated" expression as an instance of ``term_type``
"""
if engine == "python":
res = self(env)
else:
# recurse over the left/right nodes
left = self.lhs.evaluate(
env,
engine=engine,
parser=parser,
term_type=term_type,
eval_in_python=eval_in_python,
)
right = self.rhs.evaluate(
env,
engine=engine,
parser=parser,
term_type=term_type,
eval_in_python=eval_in_python,
)
# base cases
if self.op in eval_in_python:
res = self.func(left.value, right.value)
else:
from modin.core.computation.eval import eval
res = eval(self, local_dict=env, engine=engine, parser=parser)
name = env.add_tmp(res)
return term_type(name, env=env)
def convert_values(self) -> None:
"""
Convert datetimes to a comparable value in an expression.
"""
def stringify(value):
encoder: Callable
if self.encoding is not None:
encoder = partial(pprint_thing_encoded, encoding=self.encoding)
else:
encoder = pprint_thing
return encoder(value)
lhs, rhs = self.lhs, self.rhs
if is_term(lhs) and lhs.is_datetime and is_term(rhs) and rhs.is_scalar:
v = rhs.value
if isinstance(v, (int, float)):
v = stringify(v)
v = pandas.Timestamp(ensure_decoded(v))
if v.tz is not None:
v = v.tz_convert("UTC")
self.rhs.update(v)
if is_term(rhs) and rhs.is_datetime and is_term(lhs) and lhs.is_scalar:
v = lhs.value
if isinstance(v, (int, float)):
v = stringify(v)
v = pandas.Timestamp(ensure_decoded(v))
if v.tz is not None:
v = v.tz_convert("UTC")
self.lhs.update(v)
def _disallow_scalar_only_bool_ops(self):
rhs = self.rhs
lhs = self.lhs
# GH#24883 unwrap dtype if necessary to ensure we have a type object
rhs_rt = rhs.return_type
rhs_rt = getattr(rhs_rt, "type", rhs_rt)
lhs_rt = lhs.return_type
lhs_rt = getattr(lhs_rt, "type", lhs_rt)
if (
(lhs.is_scalar or rhs.is_scalar)
and self.op in _bool_ops_dict
and (
not (
issubclass(rhs_rt, (bool, np.bool_))
and issubclass(lhs_rt, (bool, np.bool_))
)
)
):
raise NotImplementedError("cannot evaluate scalar only bool ops")
def isnumeric(dtype) -> bool:
return issubclass(np.dtype(dtype).type, np.number)
UNARY_OPS_SYMS = ("+", "-", "~", "not")
_unary_ops_funcs = (operator.pos, operator.neg, operator.invert, operator.invert)
_unary_ops_dict = dict(zip(UNARY_OPS_SYMS, _unary_ops_funcs))
class UnaryOp(Op):
"""
Hold a unary operator and its operands.
Parameters
----------
op : str
The token used to represent the operator.
operand : Term or Op
The Term or Op operand to the operator.
Raises
------
ValueError
* If no function associated with the passed operator token is found.
"""
def __init__(self, op: Literal["+", "-", "~", "not"], operand) -> None:
super().__init__(op, (operand,))
self.operand = operand
try:
self.func = _unary_ops_dict[op]
except KeyError as err:
raise ValueError(
f"Invalid unary operator {repr(op)}, valid operators are {UNARY_OPS_SYMS}"
) from err
def __call__(self, env) -> MathCall:
operand = self.operand(env)
# error: Cannot call function of unknown type
return self.func(operand) # type: ignore[operator]
def __repr__(self) -> str:
return pprint_thing(f"{self.op}({self.operand})")
@property
def return_type(self) -> np.dtype:
operand = self.operand
if operand.return_type == np.dtype("bool"):
return np.dtype("bool")
if isinstance(operand, Op) and (
operand.op in _cmp_ops_dict or operand.op in _bool_ops_dict
):
return np.dtype("bool")
return np.dtype("int")
class MathCall(Op):
def __init__(self, func, args) -> None:
super().__init__(func.name, args)
self.func = func
def __call__(self, env):
# error: "Op" not callable
operands = [op(env) for op in self.operands] # type: ignore[operator]
return self.func.func(*operands)
def __repr__(self) -> str:
operands = map(str, self.operands)
return pprint_thing(f"{self.op}({','.join(operands)})")
class FuncNode:
def __init__(self, name: str) -> None:
if name not in MATHOPS:
raise ValueError(f'"{name}" is not a supported function')
self.name = name
self.func = getattr(np, name)
def __call__(self, *args) -> MathCall:
return MathCall(self, args)
================================================
FILE: modin/core/computation/parsing.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
:func:`~pandas.eval` source string parsing functions. Forked from pandas.core.computation.parsing
"""
from __future__ import annotations
import token
import tokenize
from io import StringIO
from keyword import iskeyword
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import (
Hashable,
Iterator,
)
# A token value Python's tokenizer probably will never use.
BACKTICK_QUOTED_STRING = 100
def create_valid_python_identifier(name: str) -> str:
"""
Create valid Python identifiers from any string.
Check if name contains any special characters. If it contains any
special characters, the special characters will be replaced by
a special string and a prefix is added.
Raises
------
SyntaxError
If the returned name is not a Python valid identifier, raise an exception.
This can happen if there is a hashtag in the name, as the tokenizer will
than terminate and not find the backtick.
But also for characters that fall out of the range of (U+0001..U+007F).
"""
if name.isidentifier() and not iskeyword(name):
return name
# Create a dict with the special characters and their replacement string.
# EXACT_TOKEN_TYPES contains these special characters
# token.tok_name contains a readable description of the replacement string.
special_characters_replacements = {
char: f"_{token.tok_name[tokval]}_"
for char, tokval in (tokenize.EXACT_TOKEN_TYPES.items())
}
special_characters_replacements.update(
{
" ": "_",
"?": "_QUESTIONMARK_",
"!": "_EXCLAMATIONMARK_",
"$": "_DOLLARSIGN_",
"€": "_EUROSIGN_",
"°": "_DEGREESIGN_",
# Including quotes works, but there are exceptions.
"'": "_SINGLEQUOTE_",
'"': "_DOUBLEQUOTE_",
# Currently not possible. Terminates parser and won't find backtick.
# "#": "_HASH_",
}
)
name = "".join([special_characters_replacements.get(char, char) for char in name])
name = f"BACKTICK_QUOTED_STRING_{name}"
if not name.isidentifier():
raise SyntaxError(f"Could not convert '{name}' to a valid Python identifier.")
return name
def clean_backtick_quoted_toks(tok: tuple[int, str]) -> tuple[int, str]:
"""
Clean up a column name if surrounded by backticks.
Backtick quoted string are indicated by a certain tokval value. If a string
is a backtick quoted token it will processed by
:func:`_create_valid_python_identifier` so that the parser can find this
string when the query is executed.
In this case the tok will get the NAME tokval.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tok : Tuple[int, str]
Either the input or token or the replacement values
"""
toknum, tokval = tok
if toknum == BACKTICK_QUOTED_STRING:
return tokenize.NAME, create_valid_python_identifier(tokval)
return toknum, tokval
def clean_column_name(name: Hashable) -> Hashable:
"""
Function to emulate the cleaning of a backtick quoted name.
The purpose for this function is to see what happens to the name of
identifier if it goes to the process of being parsed a Python code
inside a backtick quoted string and than being cleaned
(removed of any special characters).
Parameters
----------
name : hashable
Name to be cleaned.
Returns
-------
name : hashable
Returns the name after tokenizing and cleaning.
Notes
-----
For some cases, a name cannot be converted to a valid Python identifier.
In that case :func:`tokenize_string` raises a SyntaxError.
In that case, we just return the name unmodified.
If this name was used in the query string (this makes the query call impossible)
an error will be raised by :func:`tokenize_backtick_quoted_string` instead,
which is not caught and propagates to the user level.
"""
try:
tokenized = tokenize_string(f"`{name}`")
tokval = next(tokenized)[1]
return create_valid_python_identifier(tokval)
except SyntaxError:
return name
def tokenize_backtick_quoted_string(
token_generator: Iterator[tokenize.TokenInfo], source: str, string_start: int
) -> tuple[int, str]:
"""
Creates a token from a backtick quoted string.
Moves the token_generator forwards till right after the next backtick.
Parameters
----------
token_generator : Iterator[tokenize.TokenInfo]
The generator that yields the tokens of the source string (Tuple[int, str]).
The generator is at the first token after the backtick (`)
source : str
The Python source code string.
string_start : int
This is the start of backtick quoted string inside the source string.
Returns
-------
tok: Tuple[int, str]
The token that represents the backtick quoted string.
The integer is equal to BACKTICK_QUOTED_STRING (100).
"""
string_end = None
for _, tokval, start, _, _ in token_generator:
if tokval == "`":
string_end = start[1]
break
assert string_end is not None
return BACKTICK_QUOTED_STRING, source[string_start:string_end]
def tokenize_string(source: str) -> Iterator[tuple[int, str]]:
"""
Tokenize a Python source code string.
Parameters
----------
source : str
The Python source code string.
Returns
-------
tok_generator : Iterator[Tuple[int, str]]
An iterator yielding all tokens with only toknum and tokval (Tuple[ing, str]).
"""
line_reader = StringIO(source).readline
token_generator = tokenize.generate_tokens(line_reader)
# Loop over all tokens till a backtick (`) is found.
# Then, take all tokens till the next backtick to form a backtick quoted string
for toknum, tokval, start, _, _ in token_generator:
if tokval == "`":
try:
yield tokenize_backtick_quoted_string(
token_generator, source, string_start=start[1] + 1
)
except Exception as err:
raise SyntaxError(f"Failed to parse backticks in '{source}'.") from err
else:
yield toknum, tokval
================================================
FILE: modin/core/computation/scope.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module for scope operations. Forked from pandas.core.computation.scope
"""
from __future__ import annotations
import datetime
import inspect
import itertools
import pprint
import struct
import sys
from collections import ChainMap
from io import StringIO
from typing import TypeVar
import numpy as np
import pandas
from pandas.errors import UndefinedVariableError
_KT = TypeVar("_KT")
_VT = TypeVar("_VT")
# https://docs.python.org/3/library/collections.html#chainmap-examples-and-recipes
class DeepChainMap(ChainMap[_KT, _VT]):
"""
Variant of ChainMap that allows direct updates to inner scopes.
Only works when all passed mapping are mutable.
"""
def __setitem__(self, key: _KT, value: _VT) -> None:
for mapping in self.maps:
if key in mapping:
mapping[key] = value
return
self.maps[0][key] = value
def __delitem__(self, key: _KT) -> None:
"""
Raises
------
KeyError
If `key` doesn't exist.
"""
for mapping in self.maps:
if key in mapping:
del mapping[key]
return
raise KeyError(key)
def ensure_scope(
level: int, global_dict=None, local_dict=None, resolvers=(), target=None
) -> Scope:
"""Ensure that we are grabbing the correct scope."""
return Scope(
level + 1,
global_dict=global_dict,
local_dict=local_dict,
resolvers=resolvers,
target=target,
)
def _replacer(x) -> str:
"""
Replace a number with its hexadecimal representation. Used to tag
temporary variables with their calling scope's id.
"""
# get the hex repr of the binary char and remove 0x and pad by pad_size
# zeros
try:
hexin = ord(x)
except TypeError:
# bytes literals masquerade as ints when iterating in py3
hexin = x
return hex(hexin)
def _raw_hex_id(obj) -> str:
"""Return the padded hexadecimal id of ``obj``."""
# interpret as a pointer since that's what really what id returns
packed = struct.pack("@P", id(obj))
return "".join([_replacer(x) for x in packed])
DEFAULT_GLOBALS = {
"Timestamp": pandas.Timestamp,
"datetime": datetime.datetime,
"True": True,
"False": False,
"list": list,
"tuple": tuple,
"inf": np.inf,
"Inf": np.inf,
}
def _get_pretty_string(obj) -> str:
"""
Return a prettier version of obj.
Parameters
----------
obj : object
Object to pretty print
Returns
-------
str
Pretty print object repr
"""
sio = StringIO()
pprint.pprint(obj, stream=sio) # noqa: T203
return sio.getvalue()
class Scope:
"""
Object to hold scope, with a few bells to deal with some custom syntax
and contexts added by pandas.
Parameters
----------
level : int
global_dict : dict or None, optional, default None
local_dict : dict or Scope or None, optional, default None
resolvers : list-like or None, optional, default None
target : object
Attributes
----------
level : int
scope : DeepChainMap
target : object
temps : dict
"""
__slots__ = ["level", "scope", "target", "resolvers", "temps"]
level: int
scope: DeepChainMap
resolvers: DeepChainMap
temps: dict
def __init__(
self, level: int, global_dict=None, local_dict=None, resolvers=(), target=None
) -> None:
self.level = level + 1
# shallow copy because we don't want to keep filling this up with what
# was there before if there are multiple calls to Scope/_ensure_scope
self.scope = DeepChainMap(DEFAULT_GLOBALS.copy())
self.target = target
if isinstance(local_dict, Scope):
self.scope.update(local_dict.scope)
if local_dict.target is not None:
self.target = local_dict.target
self._update(local_dict.level)
frame = sys._getframe(self.level)
try:
# shallow copy here because we don't want to replace what's in
# scope when we align terms (alignment accesses the underlying
# numpy array of pandas objects)
scope_global = self.scope.new_child(
(global_dict if global_dict is not None else frame.f_globals).copy()
)
self.scope = DeepChainMap(scope_global)
if not isinstance(local_dict, Scope):
scope_local = self.scope.new_child(
(local_dict if local_dict is not None else frame.f_locals).copy()
)
self.scope = DeepChainMap(scope_local)
finally:
del frame
# assumes that resolvers are going from outermost scope to inner
if isinstance(local_dict, Scope):
resolvers += tuple(local_dict.resolvers.maps)
self.resolvers = DeepChainMap(*resolvers)
self.temps = {}
def __repr__(self) -> str:
scope_keys = _get_pretty_string(list(self.scope.keys()))
res_keys = _get_pretty_string(list(self.resolvers.keys()))
return f"{type(self).__name__}(scope={scope_keys}, resolvers={res_keys})"
@property
def has_resolvers(self) -> bool:
"""
Return whether we have any extra scope.
For example, DataFrames pass Their columns as resolvers during calls to
``DataFrame.eval()`` and ``DataFrame.query()``.
Returns
-------
hr : bool
"""
return bool(len(self.resolvers))
def resolve(self, key: str, is_local: bool):
"""
Resolve a variable name in a possibly local context.
Parameters
----------
key : str
A variable name
is_local : bool
Flag indicating whether the variable is local or not (prefixed with
the '@' symbol)
Returns
-------
value : object
The value of a particular variable
"""
try:
# only look for locals in outer scope
if is_local:
return self.scope[key]
# not a local variable so check in resolvers if we have them
if self.has_resolvers:
return self.resolvers[key]
# if we're here that means that we have no locals and we also have
# no resolvers
assert not is_local and not self.has_resolvers
return self.scope[key]
except KeyError:
try:
# last ditch effort we look in temporaries
# these are created when parsing indexing expressions
# e.g., df[df > 0]
return self.temps[key]
except KeyError as err:
raise UndefinedVariableError(key, is_local) from err
def swapkey(self, old_key: str, new_key: str, new_value=None) -> None:
"""
Replace a variable name, with a potentially new value.
Parameters
----------
old_key : str
Current variable name to replace
new_key : str
New variable name to replace `old_key` with
new_value : object
Value to be replaced along with the possible renaming
"""
if self.has_resolvers:
maps = self.resolvers.maps + self.scope.maps
else:
maps = self.scope.maps
maps.append(self.temps)
for mapping in maps:
if old_key in mapping:
mapping[new_key] = new_value
return
def _get_vars(self, stack, scopes: list[str]) -> None:
"""
Get specifically scoped variables from a list of stack frames.
Parameters
----------
stack : list
A list of stack frames as returned by ``inspect.stack()``
scopes : sequence of strings
A sequence containing valid stack frame attribute names that
evaluate to a dictionary. For example, ('locals', 'globals')
"""
variables = itertools.product(scopes, stack)
for scope, (frame, _, _, _, _, _) in variables:
try:
d = getattr(frame, f"f_{scope}")
self.scope = DeepChainMap(self.scope.new_child(d))
finally:
# won't remove it, but DECREF it
# in Py3 this probably isn't necessary since frame won't be
# scope after the loop
del frame
def _update(self, level: int) -> None:
"""
Update the current scope by going back `level` levels.
Parameters
----------
level : int
"""
sl = level + 1
# add sl frames to the scope starting with the
# most distant and overwriting with more current
# makes sure that we can capture variable scope
stack = inspect.stack()
try:
self._get_vars(stack[:sl], scopes=["locals"])
finally:
# explcitly delete the stack according to the advice here:
# https://docs.python.org/3/library/inspect.html#inspect.Traceback
del stack[:], stack
def add_tmp(self, value) -> str:
"""
Add a temporary variable to the scope.
Parameters
----------
value : object
An arbitrary object to be assigned to a temporary variable.
Returns
-------
str
The name of the temporary variable created.
"""
name = f"{type(value).__name__}_{self.ntemps}_{_raw_hex_id(self)}"
# add to inner most scope
assert name not in self.temps
self.temps[name] = value
assert name in self.temps
# only increment if the variable gets put in the scope
return name
@property
def ntemps(self) -> int:
"""The number of temporary variables in this scope"""
return len(self.temps)
@property
def full_scope(self) -> DeepChainMap:
"""
Return the full scope for use with passing to engines transparently
as a mapping.
Returns
-------
vars : DeepChainMap
All variables in this scope.
"""
maps = [self.temps] + self.resolvers.maps + self.scope.maps
return DeepChainMap(*maps)
================================================
FILE: modin/core/dataframe/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe functionality."""
================================================
FILE: modin/core/dataframe/algebra/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin Dataframe algebra (core operators)."""
from .binary import Binary
from .fold import Fold
from .groupby import GroupByReduce
from .map import Map
from .operator import Operator
from .reduce import Reduce
from .tree_reduce import TreeReduce
__all__ = [
"Operator",
"Map",
"TreeReduce",
"Reduce",
"Fold",
"Binary",
"GroupByReduce",
]
================================================
FILE: modin/core/dataframe/algebra/binary.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses builder class for Binary operator."""
from __future__ import annotations
import warnings
from typing import TYPE_CHECKING, Any, Callable, Optional, Union
import numpy as np
import pandas
from pandas.api.types import is_bool_dtype, is_scalar
from modin.error_message import ErrorMessage
from .operator import Operator
if TYPE_CHECKING:
from pandas._typing import DtypeObj
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
def maybe_compute_dtypes_common_cast(
first: PandasQueryCompiler,
second: Union[PandasQueryCompiler, dict, list, tuple, np.ndarray, str, DtypeObj],
trigger_computations: bool = False,
axis: int = 0,
func: Optional[
Callable[[pandas.DataFrame, pandas.DataFrame], pandas.DataFrame]
] = None,
) -> Optional[pandas.Series]:
"""
Precompute data types for binary operations by finding common type between operands.
Parameters
----------
first : PandasQueryCompiler
First operand for which the binary operation would be performed later.
second : PandasQueryCompiler, dict, list, tuple, np.ndarray, str or DtypeObj
Second operand for which the binary operation would be performed later.
trigger_computations : bool, default: False
Whether to trigger computation of the lazy metadata for `first` and `second`.
If False is specified this method will return None if any of the operands doesn't
have materialized dtypes.
axis : int, default: 0
Axis to perform the binary operation along.
func : callable(pandas.DataFrame, pandas.DataFrame) -> pandas.DataFrame, optional
If specified, will use this function to perform the "try_sample" method
(see ``Binary.register()`` docs for more details).
Returns
-------
pandas.Series
The pandas series with precomputed dtypes or None if there's not enough metadata to compute it.
Notes
-----
The dtypes of the operands are supposed to be known.
"""
if not trigger_computations:
if not first.frame_has_materialized_dtypes:
return None
if isinstance(second, type(first)) and not second.frame_has_materialized_dtypes:
return None
dtypes_first = first.dtypes.to_dict()
if isinstance(second, type(first)):
dtypes_second = second.dtypes.to_dict()
columns_first = set(first.columns)
columns_second = set(second.columns)
common_columns = columns_first.intersection(columns_second)
# Here we want to XOR the sets in order to find the columns that do not
# belong to the intersection, these will be NaN columns in the result
mismatch_columns = columns_first ^ columns_second
elif isinstance(second, dict):
dtypes_second = {
key: pandas.api.types.pandas_dtype(type(value))
for key, value in second.items()
}
columns_first = set(first.columns)
columns_second = set(second.keys())
common_columns = columns_first.intersection(columns_second)
# Here we want to find the difference between the sets in order to find columns
# that are missing in the dictionary, this will be NaN columns in the result
mismatch_columns = columns_first.difference(columns_second)
else:
if isinstance(second, (list, tuple)):
second_dtypes_list = (
[pandas.api.types.pandas_dtype(type(value)) for value in second]
if axis == 1
# Here we've been given a column so it has only one dtype,
# Infering the dtype using `np.array`, TODO: maybe there's more efficient way?
else [np.array(second).dtype] * len(dtypes_first)
)
elif is_scalar(second) or isinstance(second, np.ndarray):
try:
dtype = getattr(second, "dtype", None) or pandas.api.types.pandas_dtype(
type(second)
)
except TypeError:
# For example, dtype '' not understood
dtype = pandas.Series(second).dtype
second_dtypes_list = [dtype] * len(dtypes_first)
else:
raise NotImplementedError(
f"Can't compute common type for {type(first)} and {type(second)}."
)
# We verify operands shapes at the front-end, invalid operands shouldn't be
# propagated to the query compiler level
ErrorMessage.catch_bugs_and_request_email(
failure_condition=len(second_dtypes_list) != len(dtypes_first),
extra_log="Shapes of the operands of a binary operation don't match",
)
dtypes_second = {
key: second_dtypes_list[idx] for idx, key in enumerate(dtypes_first.keys())
}
common_columns = first.columns
mismatch_columns = []
# If at least one column doesn't match, the result of the non matching column would be nan.
nan_dtype = pandas.api.types.pandas_dtype(type(np.nan))
dtypes = None
if func is not None:
try:
with warnings.catch_warnings():
warnings.filterwarnings("ignore")
df1 = pandas.DataFrame([[1] * len(common_columns)]).astype(
{i: dtypes_first[col] for i, col in enumerate(common_columns)}
)
df2 = pandas.DataFrame([[1] * len(common_columns)]).astype(
{i: dtypes_second[col] for i, col in enumerate(common_columns)}
)
dtypes = func(df1, df2).dtypes.set_axis(common_columns)
# it sometimes doesn't work correctly with strings, so falling back to
# the "common_cast" method in this case
except TypeError:
pass
if dtypes is None:
dtypes = pandas.Series(
[
pandas.core.dtypes.cast.find_common_type(
[
dtypes_first[x],
dtypes_second[x],
]
)
for x in common_columns
],
index=common_columns,
)
dtypes: pandas.Series = pandas.concat(
[
dtypes,
pandas.Series(
[nan_dtype] * (len(mismatch_columns)),
index=mismatch_columns,
),
]
)
return dtypes
def maybe_build_dtypes_series(
first: PandasQueryCompiler,
second: Union[PandasQueryCompiler, Any],
dtype: DtypeObj,
trigger_computations: bool = False,
) -> Optional[pandas.Series]:
"""
Build a ``pandas.Series`` describing dtypes of the result of a binary operation.
Parameters
----------
first : PandasQueryCompiler
First operand for which the binary operation would be performed later.
second : PandasQueryCompiler, list-like or scalar
Second operand for which the binary operation would be performed later.
dtype : DtypeObj
Dtype of the result.
trigger_computations : bool, default: False
Whether to trigger computation of the lazy metadata for `first` and `second`.
If False is specified this method will return None if any of the operands doesn't
have materialized columns.
Returns
-------
pandas.Series or None
The pandas series with precomputed dtypes or None if there's not enough metadata to compute it.
Notes
-----
Finds a union of columns and finds dtypes for all these columns.
"""
if not trigger_computations:
if not first.frame_has_columns_cache:
return None
if isinstance(second, type(first)) and not second.frame_has_columns_cache:
return None
columns_first = set(first.columns)
if isinstance(second, type(first)):
columns_second = set(second.columns)
columns_union = columns_first.union(columns_second)
else:
columns_union = columns_first
dtypes = pandas.Series([dtype] * len(columns_union), index=columns_union)
return dtypes
def try_compute_new_dtypes(
first: PandasQueryCompiler,
second: Union[PandasQueryCompiler, Any],
infer_dtypes: Optional[str] = None,
result_dtype: Optional[Union[DtypeObj, str]] = None,
axis: int = 0,
func: Optional[
Callable[[pandas.DataFrame, pandas.DataFrame], pandas.DataFrame]
] = None,
) -> Optional[pandas.Series]:
"""
Precompute resulting dtypes of the binary operation if possible.
The dtypes won't be precomputed if any of the operands doesn't have their dtypes materialized
or if the second operand type is not supported. Supported types: PandasQueryCompiler, list,
dict, tuple, np.ndarray.
Parameters
----------
first : PandasQueryCompiler
First operand of the binary operation.
second : PandasQueryCompiler, list-like or scalar
Second operand of the binary operation.
infer_dtypes : {"common_cast", "try_sample", "bool", None}, default: None
How dtypes should be infered (see ``Binary.register`` doc for more info).
result_dtype : np.dtype, optional
NumPy dtype of the result. If not specified it will be inferred from the `infer_dtypes` parameter.
axis : int, default: 0
Axis to perform the binary operation along.
func : callable(pandas.DataFrame, pandas.DataFrame) -> pandas.DataFrame, optional
A callable to be used for the "try_sample" method.
Returns
-------
pandas.Series or None
"""
if infer_dtypes is None and result_dtype is None:
return None
try:
if infer_dtypes == "bool" or is_bool_dtype(result_dtype):
dtypes = maybe_build_dtypes_series(
first, second, dtype=pandas.api.types.pandas_dtype(bool)
)
elif infer_dtypes == "common_cast":
dtypes = maybe_compute_dtypes_common_cast(
first, second, axis=axis, func=None
)
elif infer_dtypes == "try_sample":
if func is None:
raise ValueError(
"'func' must be specified if dtypes infering method is 'try_sample'"
)
dtypes = maybe_compute_dtypes_common_cast(
first, second, axis=axis, func=func
)
else:
# For now we only know how to handle `result_dtype == bool` as that's
# the only value that is being passed here right now, it's unclear
# how we should behave in case of an arbitrary dtype, so let's wait
# for at least one case to appear for this regard.
dtypes = None
except NotImplementedError:
dtypes = None
return dtypes
class Binary(Operator):
"""Builder class for Binary operator."""
@classmethod
def register(
cls,
func: Callable[..., pandas.DataFrame],
join_type: str = "outer",
sort: bool = None,
labels: str = "replace",
infer_dtypes: Optional[str] = None,
) -> Callable[..., PandasQueryCompiler]:
"""
Build template binary operator.
Parameters
----------
func : callable(pandas.DataFrame, [pandas.DataFrame, list-like, scalar]) -> pandas.DataFrame
Binary function to execute. Have to be able to accept at least two arguments.
join_type : {'left', 'right', 'outer', 'inner', None}, default: 'outer'
Type of join that will be used if indices of operands are not aligned.
sort : bool, default: None
Whether to sort index and columns or not.
labels : {"keep", "replace", "drop"}, default: "replace"
Whether keep labels from left Modin DataFrame, replace them with labels
from joined DataFrame or drop altogether to make them be computed lazily later.
infer_dtypes : {"common_cast", "try_sample", "bool", None}, default: None
How dtypes should be inferred.
* If "common_cast", casts to common dtype of operand columns.
* If "try_sample", creates small pandas DataFrames with dtypes of operands and
runs the `func` on them to determine output dtypes. If a ``TypeError`` is raised
during this process, fallback to "common_cast" method.
* If "bool", dtypes would be a boolean series with same size as that of operands.
* If ``None``, do not infer new dtypes (they will be computed manually once accessed).
Returns
-------
callable
Function that takes query compiler and executes binary operation.
"""
def caller(
query_compiler: PandasQueryCompiler,
other: Union[PandasQueryCompiler, Any],
broadcast: bool = False,
*args: tuple,
dtypes: Optional[Union[DtypeObj, str]] = None,
**kwargs: dict,
) -> PandasQueryCompiler:
"""
Apply binary `func` to passed operands.
Parameters
----------
query_compiler : PandasQueryCompiler
Left operand of `func`.
other : PandasQueryCompiler, list-like object or scalar
Right operand of `func`.
broadcast : bool, default: False
If `other` is a one-column query compiler, indicates whether it is a Series or not.
Frames and Series have to be processed differently, however we can't distinguish them
at the query compiler level, so this parameter is a hint that passed from a high level API.
*args : tuple,
Arguments that will be passed to `func`.
dtypes : "copy", scalar dtype or None, default: None
Dtypes of the result. "copy" to keep old dtypes and None to compute them on demand.
**kwargs : dict,
Arguments that will be passed to `func`.
Returns
-------
PandasQueryCompiler
Result of binary function.
"""
axis: int = kwargs.get("axis", 0)
if isinstance(other, type(query_compiler)) and broadcast:
assert (
len(other.columns) == 1
), "Invalid broadcast argument for `broadcast_apply`, too many columns: {}".format(
len(other.columns)
)
# Transpose on `axis=1` because we always represent an individual
# column or row as a single-column Modin DataFrame
if axis == 1:
other = other.transpose()
if dtypes != "copy":
dtypes = try_compute_new_dtypes(
query_compiler, other, infer_dtypes, dtypes, axis, func
)
shape_hint = None
if isinstance(other, type(query_compiler)):
if broadcast:
if (
query_compiler.frame_has_materialized_columns
and other.frame_has_materialized_columns
):
if (
len(query_compiler.columns) == 1
and len(other.columns) == 1
and query_compiler.columns.equals(other.columns)
):
shape_hint = "column"
return query_compiler.__constructor__(
query_compiler._modin_frame.broadcast_apply(
axis,
lambda left, right: func(
left, right.squeeze(), *args, **kwargs
),
other._modin_frame,
join_type=join_type,
labels=labels,
dtypes=dtypes,
),
shape_hint=shape_hint,
)
else:
if (
query_compiler.frame_has_materialized_columns
and other.frame_has_materialized_columns
):
if (
len(query_compiler.columns) == 1
and len(other.columns) == 1
and query_compiler.columns.equals(other.columns)
):
shape_hint = "column"
return query_compiler.__constructor__(
query_compiler._modin_frame.n_ary_op(
lambda x, y: func(x, y, *args, **kwargs),
[other._modin_frame],
join_type=join_type,
sort=sort,
labels=labels,
dtypes=dtypes,
),
shape_hint=shape_hint,
)
else:
# TODO: it's possible to chunk the `other` and broadcast them to partitions
# accordingly, in that way we will be able to use more efficient `._modin_frame.map()`
if isinstance(other, (dict, list, np.ndarray, pandas.Series)):
new_modin_frame = query_compiler._modin_frame.apply_full_axis(
axis,
lambda df: func(df, other, *args, **kwargs),
new_index=query_compiler.index,
new_columns=query_compiler.columns,
dtypes=dtypes,
)
else:
if (
query_compiler.frame_has_materialized_columns
and len(query_compiler._modin_frame.columns) == 1
and is_scalar(other)
):
shape_hint = "column"
new_modin_frame = query_compiler._modin_frame.map(
func,
func_args=(other, *args),
func_kwargs=kwargs,
dtypes=dtypes,
lazy=True,
)
return query_compiler.__constructor__(
new_modin_frame, shape_hint=shape_hint
)
return caller
================================================
FILE: modin/core/dataframe/algebra/default2pandas/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module default2pandas provides templates for a query compiler default-to-pandas methods."""
from .binary import BinaryDefault
from .cat import CatDefault
from .dataframe import DataFrameDefault
from .datetime import DateTimeDefault
from .default import DefaultMethod
from .groupby import GroupByDefault, SeriesGroupByDefault
from .list import ListDefault
from .resample import ResampleDefault
from .rolling import ExpandingDefault, RollingDefault
from .series import SeriesDefault
from .str import StrDefault
from .struct import StructDefault
__all__ = [
"DataFrameDefault",
"DateTimeDefault",
"SeriesDefault",
"StrDefault",
"BinaryDefault",
"ResampleDefault",
"RollingDefault",
"ExpandingDefault",
"DefaultMethod",
"CatDefault",
"GroupByDefault",
"SeriesGroupByDefault",
"ListDefault",
"StructDefault",
]
================================================
FILE: modin/core/dataframe/algebra/default2pandas/binary.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default binary functions builder class."""
import pandas
from pandas.core.dtypes.common import is_list_like
from .default import DefaultMethod
class BinaryDefault(DefaultMethod):
"""Build default-to-pandas methods which executes binary functions."""
@classmethod
def build_default_to_pandas(cls, fn, fn_name):
"""
Build function that do fallback to pandas for passed binary `fn`.
Parameters
----------
fn : callable
Binary function to apply to the casted to pandas frame and other operand.
fn_name : str
Function name which will be shown in default-to-pandas warning message.
Returns
-------
callable
Function that takes query compiler, does fallback to pandas and applies binary `fn`
to the casted to pandas frame.
"""
def bin_ops_wrapper(df, other, *args, **kwargs):
"""Apply specified binary function to the passed operands."""
squeeze_other = kwargs.pop("broadcast", False) or kwargs.pop(
"squeeze_other", False
)
squeeze_self = kwargs.pop("squeeze_self", False)
if squeeze_other:
other = other.squeeze(axis=1)
if squeeze_self:
df = df.squeeze(axis=1)
result = fn(df, other, *args, **kwargs)
if (
not isinstance(result, pandas.Series)
and not isinstance(result, pandas.DataFrame)
and is_list_like(result)
):
result = pandas.DataFrame(result)
return result
return super().build_default_to_pandas(bin_ops_wrapper, fn_name)
================================================
FILE: modin/core/dataframe/algebra/default2pandas/cat.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default applied-on-category functions builder class."""
from .series import SeriesDefault
class CatDefault(SeriesDefault):
"""Builder for default-to-pandas methods which is executed under category accessor."""
@classmethod
def frame_wrapper(cls, df):
"""
Get category accessor of the passed frame.
Parameters
----------
df : pandas.DataFrame
Returns
-------
pandas.core.arrays.categorical.CategoricalAccessor
"""
return df.squeeze(axis=1).cat
================================================
FILE: modin/core/dataframe/algebra/default2pandas/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default DataFrame functions builder class."""
import pandas
from modin.utils import _inherit_docstrings
from .default import DefaultMethod
@_inherit_docstrings(DefaultMethod)
class DataFrameDefault(DefaultMethod):
DEFAULT_OBJECT_TYPE = pandas.DataFrame
================================================
FILE: modin/core/dataframe/algebra/default2pandas/datetime.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default applied-on-datetime functions builder class."""
from .series import SeriesDefault
class DateTimeDefault(SeriesDefault):
"""Builder for default-to-pandas methods which is executed under datetime accessor."""
@classmethod
def frame_wrapper(cls, df):
"""
Get datetime accessor of the passed frame.
Parameters
----------
df : pandas.DataFrame
Returns
-------
pandas.core.indexes.accessors.DatetimeProperties
"""
return df.squeeze(axis=1).dt
================================================
FILE: modin/core/dataframe/algebra/default2pandas/default.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default functions builder class."""
import pandas
from pandas.core.dtypes.common import is_list_like
from modin.core.dataframe.algebra.operator import Operator
from modin.utils import MODIN_UNNAMED_SERIES_LABEL, try_cast_to_pandas
class ObjTypeDeterminer:
"""
Class that routes work to the frame.
Provides an instance which forwards all of the `__getattribute__` calls
to an object under which `key` function is applied.
"""
def __getattr__(self, key):
"""
Build function that executes `key` function over passed frame.
Parameters
----------
key : str
Returns
-------
callable
Function that takes DataFrame and executes `key` function on it.
"""
def func(df, *args, **kwargs):
"""Access specified attribute of the passed object and call it if it's callable."""
prop = getattr(df, key)
if callable(prop):
return prop(*args, **kwargs)
else:
return prop
return func
class DefaultMethod(Operator):
"""
Builder for default-to-pandas methods.
Attributes
----------
OBJECT_TYPE : str
Object type name that will be shown in default-to-pandas warning message.
DEFAULT_OBJECT_TYPE : object
Default place to search for a function.
"""
OBJECT_TYPE = "DataFrame"
DEFAULT_OBJECT_TYPE = ObjTypeDeterminer
@classmethod
def register(cls, func, obj_type=None, inplace=None, fn_name=None):
"""
Build function that do fallback to default pandas implementation for passed `func`.
Parameters
----------
func : callable or str,
Function to apply to the casted to pandas frame or its property accesed
by ``cls.frame_wrapper``.
obj_type : object, optional
If `func` is a string with a function name then `obj_type` provides an
object to search function in.
inplace : bool, optional
If True return an object to which `func` was applied, otherwise return
the result of `func`.
fn_name : str, optional
Function name which will be shown in default-to-pandas warning message.
If not specified, name will be deducted from `func`.
Returns
-------
callable
Function that takes query compiler, does fallback to pandas and applies `func`
to the casted to pandas frame or its property accesed by ``cls.frame_wrapper``.
"""
if isinstance(func, str):
if obj_type is None:
obj_type = cls.DEFAULT_OBJECT_TYPE
fn = getattr(obj_type, func)
else:
fn = func
if type(fn) is property:
if fn_name is None and hasattr(fn, "fget"):
# When `fn` is a property, `str(fn)` will be something like
# "". We instead check its `fget` method to get
# the name of the property.
# Note that this method is still imperfect because we cannot get the class name
# of the property. For example, we can only get "hour" from `Series.dt.hour`.
fn_name = f""
fn = cls.build_property_wrapper(fn)
else:
fn_name = getattr(fn, "__name__", str(fn)) if fn_name is None else fn_name
def applyier(df, *args, **kwargs):
"""
Apply target function to the casted to pandas frame.
This function is directly applied to the casted to pandas frame, executes target
function under it and processes result so it is possible to create a valid
query compiler from it.
"""
# pandas default implementation doesn't know how to handle `dtypes` keyword argument
kwargs.pop("dtypes", None)
df = cls.frame_wrapper(df)
result = fn(df, *args, **kwargs)
if (
not isinstance(result, pandas.Series)
and not isinstance(result, pandas.DataFrame)
and func not in ("to_numpy", pandas.DataFrame.to_numpy)
and func not in ("align", pandas.DataFrame.align)
and func not in ("divmod", pandas.Series.divmod)
and func not in ("rdivmod", pandas.Series.rdivmod)
and func not in ("to_list", pandas.Series.to_list)
and func not in ("corr", pandas.Series.corr)
and func not in ("to_dict", pandas.Series.to_dict)
and func not in ("mean", pandas.DataFrame.mean)
and func not in ("median", pandas.DataFrame.median)
and func not in ("skew", pandas.DataFrame.skew)
and func not in ("kurt", pandas.DataFrame.kurt)
):
# When applying a DatetimeProperties or TimedeltaProperties function,
# if we don't specify the dtype for the DataFrame, the frame might
# get the wrong dtype, e.g. for to_pydatetime in
# https://github.com/modin-project/modin/issues/4436
astype_kwargs = {}
dtype = getattr(result, "dtype", None)
if dtype and isinstance(
df,
(
pandas.core.indexes.accessors.DatetimeProperties,
pandas.core.indexes.accessors.TimedeltaProperties,
),
):
astype_kwargs["dtype"] = dtype
result = (
pandas.DataFrame(result, **astype_kwargs)
if is_list_like(result)
else pandas.DataFrame([result], **astype_kwargs)
)
if isinstance(result, pandas.Series):
if result.name is None:
result.name = MODIN_UNNAMED_SERIES_LABEL
result = result.to_frame()
inplace_method = kwargs.get("inplace", False)
if inplace is not None:
inplace_method = inplace
return result if not inplace_method else df
return cls.build_wrapper(applyier, fn_name)
@classmethod
# FIXME: this method is almost a duplicate of `cls.build_default_to_pandas`.
# Those two methods should be merged into a single one.
def build_wrapper(cls, fn, fn_name):
"""
Build function that do fallback to pandas for passed `fn`.
In comparison with ``cls.build_default_to_pandas`` this method also
casts function arguments to pandas before doing fallback.
Parameters
----------
fn : callable
Function to apply to the defaulted frame.
fn_name : str
Function name which will be shown in default-to-pandas warning message.
Returns
-------
callable
Method that does fallback to pandas and applies `fn` to the pandas frame.
"""
wrapper = cls.build_default_to_pandas(fn, fn_name)
def args_cast(self, *args, **kwargs):
"""
Preprocess `default_to_pandas` function arguments and apply default function.
Cast all Modin objects that function arguments contain to its pandas representation.
"""
args = try_cast_to_pandas(args)
kwargs = try_cast_to_pandas(kwargs)
return wrapper(self, *args, **kwargs)
return args_cast
@classmethod
def build_property_wrapper(cls, prop):
"""
Build function that accesses specified property of the frame.
Parameters
----------
prop : str
Property name to access.
Returns
-------
callable
Function that takes DataFrame and returns its value of `prop` property.
"""
def property_wrapper(df):
"""Get specified property of the passed object."""
return prop.fget(df)
return property_wrapper
@classmethod
def build_default_to_pandas(cls, fn, fn_name):
"""
Build function that do fallback to pandas for passed `fn`.
Parameters
----------
fn : callable
Function to apply to the defaulted frame.
fn_name : str
Function name which will be shown in default-to-pandas warning message.
Returns
-------
callable
Method that does fallback to pandas and applies `fn` to the pandas frame.
"""
fn.__name__ = f""
def wrapper(self, *args, **kwargs):
"""Do fallback to pandas for the specified function."""
return self.default_to_pandas(fn, *args, **kwargs)
return wrapper
@classmethod
def frame_wrapper(cls, df):
"""
Extract frame property to apply function on.
This method is executed under casted to pandas frame right before applying
a function passed to `register`, which gives an ability to transform frame somehow
or access its properties, by overriding this method in a child class.
Parameters
----------
df : pandas.DataFrame
Returns
-------
pandas.DataFrame
Notes
-----
Being a base implementation, this particular method does nothing with passed frame.
"""
return df
================================================
FILE: modin/core/dataframe/algebra/default2pandas/groupby.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default GroupBy functions builder class."""
import warnings
from typing import Any
import pandas
from pandas.core.dtypes.common import is_list_like
# Defines a set of string names of functions that are executed in a transform-way in groupby
from pandas.core.groupby.base import transformation_kernels
from modin.utils import MODIN_UNNAMED_SERIES_LABEL, hashable
from .default import DefaultMethod
# FIXME: there is no sence of keeping `GroupBy` and `GroupByDefault` logic in a different
# classes. They should be combined.
class GroupBy:
"""Builder for GroupBy aggregation functions."""
agg_aliases = [
"agg",
"dict_agg",
pandas.core.groupby.DataFrameGroupBy.agg,
pandas.core.groupby.DataFrameGroupBy.aggregate,
]
@staticmethod
def is_transformation_kernel(agg_func: Any) -> bool:
"""
Check whether a passed aggregation function is a transformation.
Transformation means that the result of the function will be broadcasted
to the frame's original shape.
Parameters
----------
agg_func : Any
Returns
-------
bool
"""
return hashable(agg_func) and agg_func in transformation_kernels.union(
# these methods are also producing transpose-like result in a sense we understand it
# (they're non-aggregative functions), however are missing in the pandas dictionary
{"nth", "head", "tail"}
)
@classmethod
def _call_groupby(cls, df, *args, **kwargs): # noqa: PR01
"""Call .groupby() on passed `df`."""
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=FutureWarning)
return df.groupby(*args, **kwargs)
@classmethod
def validate_by(cls, by):
"""
Build valid `by` parameter for `pandas.DataFrame.groupby`.
Cast all DataFrames in `by` parameter to Series or list of Series in case
of multi-column frame.
Parameters
----------
by : DateFrame, Series, index label or list of such
Object which indicates groups for GroupBy.
Returns
-------
Series, index label or list of such
By parameter with all DataFrames casted to Series.
"""
def try_cast_series(df):
"""Cast one-column frame to Series."""
if isinstance(df, pandas.DataFrame):
df = df.squeeze(axis=1)
if not isinstance(df, pandas.Series):
return df
if df.name == MODIN_UNNAMED_SERIES_LABEL:
df.name = None
return df
if isinstance(by, pandas.DataFrame):
by = [try_cast_series(column) for _, column in by.items()]
elif isinstance(by, pandas.Series):
by = [try_cast_series(by)]
elif isinstance(by, list):
by = [try_cast_series(o) for o in by]
return by
@classmethod
def inplace_applyier_builder(cls, key, func=None):
"""
Bind actual aggregation function to the GroupBy aggregation method.
Parameters
----------
key : callable
Function that takes GroupBy object and evaluates passed aggregation function.
func : callable or str, optional
Function that takes DataFrame and aggregate its data. Will be applied
to each group at the grouped frame.
Returns
-------
callable,
Function that executes aggregation under GroupBy object.
"""
inplace_args = [] if func is None else [func]
def inplace_applyier(grp, *func_args, **func_kwargs):
return key(grp, *inplace_args, *func_args, **func_kwargs)
return inplace_applyier
@classmethod
def get_func(cls, key, **kwargs):
"""
Extract aggregation function from groupby arguments.
Parameters
----------
key : callable or str
Default aggregation function. If aggregation function is not specified
via groupby arguments, then `key` function is used.
**kwargs : dict
GroupBy arguments that may contain aggregation function.
Returns
-------
callable
Aggregation function.
Notes
-----
There are two ways of how groupby aggregation can be invoked:
1. Explicitly with query compiler method: `qc.groupby_sum()`.
2. By passing aggregation function as an argument: `qc.groupby_agg("sum")`.
Both are going to produce the same result, however in the first case actual aggregation
function can be extracted from the method name, while for the second only from the method arguments.
"""
if "agg_func" in kwargs:
return cls.inplace_applyier_builder(key, kwargs["agg_func"])
elif "func_dict" in kwargs:
return cls.inplace_applyier_builder(key, kwargs["func_dict"])
else:
return cls.inplace_applyier_builder(key)
@classmethod
def build_aggregate_method(cls, key):
"""
Build function for `QueryCompiler.groupby_agg` that can be executed as default-to-pandas.
Parameters
----------
key : callable or str
Default aggregation function. If aggregation function is not specified
via groupby arguments, then `key` function is used.
Returns
-------
callable
Function that executes groupby aggregation.
"""
def fn(
df,
by,
axis,
groupby_kwargs,
agg_args,
agg_kwargs,
drop=False,
**kwargs,
):
"""Group DataFrame and apply aggregation function to each group."""
by = cls.validate_by(by)
grp = cls._call_groupby(df, by, axis=axis, **groupby_kwargs)
agg_func = cls.get_func(key, **kwargs)
result = agg_func(grp, *agg_args, **agg_kwargs)
return result
return fn
@classmethod
def build_groupby_reduce_method(cls, agg_func):
"""
Build function for `QueryCompiler.groupby_*` that can be executed as default-to-pandas.
Parameters
----------
agg_func : callable or str
Default aggregation function. If aggregation function is not specified
via groupby arguments, then `agg_func` function is used.
Returns
-------
callable
Function that executes groupby aggregation.
"""
def fn(
df, by, axis, groupby_kwargs, agg_args, agg_kwargs, drop=False, **kwargs
):
"""Group DataFrame and apply aggregation function to each group."""
if not isinstance(by, (pandas.Series, pandas.DataFrame)):
by = cls.validate_by(by)
grp = cls._call_groupby(df, by, axis=axis, **groupby_kwargs)
grp_agg_func = cls.get_func(agg_func, **kwargs)
return grp_agg_func(
grp,
*agg_args,
**agg_kwargs,
)
if isinstance(by, pandas.DataFrame):
by = by.squeeze(axis=1)
if (
drop
and isinstance(by, pandas.Series)
and by.name in df
and df[by.name].equals(by)
):
by = [by.name]
if isinstance(by, pandas.DataFrame):
df = pandas.concat([df] + [by[[o for o in by if o not in df]]], axis=1)
by = list(by.columns)
groupby_kwargs = groupby_kwargs.copy()
as_index = groupby_kwargs.pop("as_index", True)
groupby_kwargs["as_index"] = True
grp = cls._call_groupby(df, by, axis=axis, **groupby_kwargs)
func = cls.get_func(agg_func, **kwargs)
result = func(grp, *agg_args, **agg_kwargs)
method = kwargs.get("method")
if isinstance(result, pandas.Series):
result = result.to_frame(
MODIN_UNNAMED_SERIES_LABEL if result.name is None else result.name
)
if not as_index:
if isinstance(by, pandas.Series):
# 1. If `drop` is True then 'by' Series represents a column from the
# source frame and so the 'by' is internal.
# 2. If method is 'size' then any 'by' is considered to be internal.
# This is a hacky legacy from the ``groupby_size`` implementation:
# https://github.com/modin-project/modin/issues/3739
internal_by = (by.name,) if drop or method == "size" else tuple()
else:
internal_by = by
cls.handle_as_index_for_dataframe(
result,
internal_by,
by_cols_dtypes=(
df.index.dtypes.values
if isinstance(df.index, pandas.MultiIndex)
else (df.index.dtype,)
),
by_length=len(by),
drop=drop,
method=method,
inplace=True,
)
if result.index.name == MODIN_UNNAMED_SERIES_LABEL:
result.index.name = None
return result
return fn
@classmethod
def is_aggregate(cls, key): # noqa: PR01
"""Check whether `key` is an alias for pandas.GroupBy.aggregation method."""
return key in cls.agg_aliases
@classmethod
def build_groupby(cls, func):
"""
Build function that groups DataFrame and applies aggregation function to the every group.
Parameters
----------
func : callable or str
Default aggregation function. If aggregation function is not specified
via groupby arguments, then `func` function is used.
Returns
-------
callable
Function that takes pandas DataFrame and does GroupBy aggregation.
"""
if cls.is_aggregate(func):
return cls.build_aggregate_method(func)
return cls.build_groupby_reduce_method(func)
@classmethod
def handle_as_index_for_dataframe(
cls,
result,
internal_by_cols,
by_cols_dtypes=None,
by_length=None,
selection=None,
partition_idx=0,
drop=True,
method=None,
inplace=False,
):
"""
Handle `as_index=False` parameter for the passed GroupBy aggregation result.
Parameters
----------
result : DataFrame
Frame containing GroupBy aggregation result computed with `as_index=True`
parameter (group names are located at the frame's index).
internal_by_cols : list-like
Internal 'by' columns.
by_cols_dtypes : list-like, optional
Data types of the internal 'by' columns. Required to do special casing
in case of categorical 'by'. If not specified, assume that there is no
categorical data in 'by'.
by_length : int, optional
Amount of keys to group on (including frame columns and external objects like list, Series, etc.)
If not specified, consider `by_length` to be equal ``len(internal_by_cols)``.
selection : label or list of labels, optional
Set of columns that were explicitly selected for aggregation (for example
via dict-aggregation). If not specified assuming that aggregation was
applied to all of the available columns.
partition_idx : int, default: 0
Positional index of the current partition.
drop : bool, default: True
Indicates whether or not any of the `by` data came from the same frame.
method : str, optional
Name of the groupby function. This is a hint to be able to do special casing.
Note: this parameter is a legacy from the ``groupby_size`` implementation,
it's a hacky one and probably will be removed in the future: https://github.com/modin-project/modin/issues/3739.
inplace : bool, default: False
Modify the DataFrame in place (do not create a new object).
Returns
-------
DataFrame
GroupBy aggregation result with the considered `as_index=False` parameter.
"""
if not inplace:
result = result.copy()
reset_index, drop, lvls_to_drop, cols_to_drop = cls.handle_as_index(
result_cols=result.columns,
result_index_names=result.index.names,
internal_by_cols=internal_by_cols,
by_cols_dtypes=by_cols_dtypes,
by_length=by_length,
selection=selection,
partition_idx=partition_idx,
drop=drop,
method=method,
)
if len(lvls_to_drop) > 0:
result.index = result.index.droplevel(lvls_to_drop)
if len(cols_to_drop) > 0:
result.drop(columns=cols_to_drop, inplace=True)
if reset_index:
result.reset_index(drop=drop, inplace=True)
return result
@staticmethod
def handle_as_index(
result_cols,
result_index_names,
internal_by_cols,
by_cols_dtypes=None,
by_length=None,
selection=None,
partition_idx=0,
drop=True,
method=None,
):
"""
Compute hints to process ``as_index=False`` parameter for the GroupBy result.
This function resolves naming conflicts of the index levels to insert and the column labels
for the GroupBy result. The logic of this function assumes that the initial GroupBy result
was computed as ``as_index=True``.
Parameters
----------
result_cols : pandas.Index
Columns of the GroupBy result.
result_index_names : list-like
Index names of the GroupBy result.
internal_by_cols : list-like
Internal 'by' columns.
by_cols_dtypes : list-like, optional
Data types of the internal 'by' columns. Required to do special casing
in case of categorical 'by'. If not specified, assume that there is no
categorical data in 'by'.
by_length : int, optional
Amount of keys to group on (including frame columns and external objects like list, Series, etc.)
If not specified, consider `by_length` to be equal ``len(internal_by_cols)``.
selection : label or list of labels, optional
Set of columns that were explicitly selected for aggregation (for example
via dict-aggregation). If not specified assuming that aggregation was
applied to all of the available columns.
partition_idx : int, default: 0
Positional index of the current partition.
drop : bool, default: True
Indicates whether or not any of the `by` data came from the same frame.
method : str, optional
Name of the groupby function. This is a hint to be able to do special casing.
Note: this parameter is a legacy from the ``groupby_size`` implementation,
it's a hacky one and probably will be removed in the future: https://github.com/modin-project/modin/issues/3739.
Returns
-------
reset_index : bool
Indicates whether to reset index to the default one (0, 1, 2 ... n) at this partition.
drop_index : bool
If `reset_index` is True, indicates whether to drop all index levels (True) or insert them into the
resulting columns (False).
lvls_to_drop : list of ints
Contains numeric indices of the levels of the result index to drop as intersected.
cols_to_drop : list of labels
Contains labels of the columns to drop from the result as intersected.
Examples
--------
>>> groupby_result = compute_groupby_without_processing_as_index_parameter()
>>> if not as_index:
>>> reset_index, drop, lvls_to_drop, cols_to_drop = handle_as_index(**extract_required_params(groupby_result))
>>> if len(lvls_to_drop) > 0:
>>> groupby_result.index = groupby_result.index.droplevel(lvls_to_drop)
>>> if len(cols_to_drop) > 0:
>>> groupby_result = groupby_result.drop(columns=cols_to_drop)
>>> if reset_index:
>>> groupby_result_with_processed_as_index_parameter = groupby_result.reset_index(drop=drop)
>>> else:
>>> groupby_result_with_processed_as_index_parameter = groupby_result
"""
if by_length is None:
by_length = len(internal_by_cols)
reset_index = method != "transform" and (by_length > 0 or selection is not None)
# If the method is "size" then the result contains only one unique named column
# and we don't have to worry about any naming conflicts, so inserting all of
# the "by" into the result (just a fast-path)
if method == "size":
return reset_index, False, [], []
# Pandas logic of resolving naming conflicts is the following:
# 1. If any categorical is in 'by' and 'by' is multi-column, then the categorical
# index is prioritized: drop intersected columns and insert all of the 'by' index
# levels to the frame as columns.
# 2. Otherwise, aggregation result is prioritized: drop intersected index levels and
# insert the filtered ones to the frame as columns.
if by_cols_dtypes is not None:
keep_index_levels = (
by_length > 1
and selection is None
and any(isinstance(x, pandas.CategoricalDtype) for x in by_cols_dtypes)
)
else:
keep_index_levels = False
# 1. We insert 'by'-columns to the result at the beginning of the frame and so only to the
# first partition, if partition_idx != 0 we just drop the index. If there are no columns
# that are required to drop (keep_index_levels is True) then we can exit here.
# 2. We don't insert 'by'-columns to the result if 'by'-data came from a different
# frame (drop is False), there's only one exception for this rule: if the `method` is "size",
# so if (drop is False) and method is not "size" we just drop the index and so can exit here.
if (not keep_index_levels and partition_idx != 0) or (
not drop and method != "size"
):
return reset_index, True, [], []
if not isinstance(internal_by_cols, pandas.Index):
if not is_list_like(internal_by_cols):
internal_by_cols = [internal_by_cols]
internal_by_cols = pandas.Index(internal_by_cols)
internal_by_cols = (
internal_by_cols[
~internal_by_cols.str.startswith(MODIN_UNNAMED_SERIES_LABEL, na=False)
]
if hasattr(internal_by_cols, "str")
else internal_by_cols
)
if selection is not None and not isinstance(selection, pandas.Index):
selection = pandas.Index(selection)
lvls_to_drop = []
cols_to_drop = []
if not keep_index_levels:
# We want to insert only these internal-by-cols that are not presented
# in the result in order to not create naming conflicts
if selection is None:
cols_to_insert = frozenset(internal_by_cols) - frozenset(result_cols)
else:
cols_to_insert = frozenset(
# We have to use explicit 'not in' check and not just difference
# of sets because of specific '__contains__' operator in case of
# scalar 'col' and MultiIndex 'selection'.
col
for col in internal_by_cols
if col not in selection
)
else:
cols_to_insert = internal_by_cols
# We want to drop such internal-by-cols that are presented
# in the result in order to not create naming conflicts
cols_to_drop = frozenset(internal_by_cols) & frozenset(result_cols)
if partition_idx == 0:
lvls_to_drop = [
i
for i, name in enumerate(result_index_names)
if name not in cols_to_insert
]
else:
lvls_to_drop = result_index_names
drop = False
if len(lvls_to_drop) == len(result_index_names):
drop = True
lvls_to_drop = []
return reset_index, drop, lvls_to_drop, cols_to_drop
class SeriesGroupBy(GroupBy):
"""Builder for GroupBy aggregation functions for Series."""
@classmethod
def _call_groupby(cls, df, *args, **kwargs): # noqa: PR01
"""Call .groupby() on passed `df` squeezed to Series."""
# We can end up here by two means - either by "true" call
# like Series().groupby() or by df.groupby()[item].
if len(df.columns) == 1:
# Series().groupby() case
return df.squeeze(axis=1).groupby(*args, **kwargs)
# In second case surrounding logic will supplement grouping columns,
# so we need to drop them after grouping is over; our originally
# selected column is always the first, so use it
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=FutureWarning)
return df.groupby(*args, **kwargs)[df.columns[0]]
class GroupByDefault(DefaultMethod):
"""Builder for default-to-pandas GroupBy aggregation functions."""
_groupby_cls = GroupBy
OBJECT_TYPE = "GroupBy"
@classmethod
def register(cls, func, **kwargs):
"""
Build default-to-pandas GroupBy aggregation function.
Parameters
----------
func : callable or str
Default aggregation function. If aggregation function is not specified
via groupby arguments, then `func` function is used.
**kwargs : kwargs
Additional arguments that will be passed to function builder.
Returns
-------
callable
Functiom that takes query compiler and defaults to pandas to do GroupBy
aggregation.
"""
return super().register(
cls._groupby_cls.build_groupby(func), fn_name=func.__name__, **kwargs
)
# This specifies a `pandas.DataFrameGroupBy` method to pass the `agg_func` to,
# it's based on `how` to apply it. Going by pandas documentation:
# 1. `.aggregate(func)` applies func row/column wise.
# 2. `.apply(func)` applies func to a DataFrames, holding a whole group (group-wise).
# 3. `.transform(func)` is the same as `.apply()` but also broadcast the `func`
# result to the group's original shape.
# 4. 'direct' mode means that the passed `func` has to be applied directly
# to the `pandas.DataFrameGroupBy` object.
_aggregation_methods_dict = {
"axis_wise": pandas.core.groupby.DataFrameGroupBy.aggregate,
"group_wise": pandas.core.groupby.DataFrameGroupBy.apply,
"transform": pandas.core.groupby.DataFrameGroupBy.transform,
"direct": lambda grp, func, *args, **kwargs: func(grp, *args, **kwargs),
}
@classmethod
def get_aggregation_method(cls, how):
"""
Return `pandas.DataFrameGroupBy` method that implements the passed `how` UDF applying strategy.
Parameters
----------
how : {"axis_wise", "group_wise", "transform"}
`how` parameter of the ``BaseQueryCompiler.groupby_agg``.
Returns
-------
callable(pandas.DataFrameGroupBy, callable, *args, **kwargs) -> [pandas.DataFrame | pandas.Series]
Notes
-----
Visit ``BaseQueryCompiler.groupby_agg`` doc-string for more information about `how` parameter.
"""
return cls._aggregation_methods_dict[how]
class SeriesGroupByDefault(GroupByDefault):
"""Builder for default-to-pandas GroupBy aggregation functions for Series."""
_groupby_cls = SeriesGroupBy
_aggregation_methods_dict = {
"axis_wise": pandas.core.groupby.SeriesGroupBy.aggregate,
"group_wise": pandas.core.groupby.SeriesGroupBy.apply,
"transform": pandas.core.groupby.SeriesGroupBy.transform,
"direct": lambda grp, func, *args, **kwargs: func(grp, *args, **kwargs),
}
================================================
FILE: modin/core/dataframe/algebra/default2pandas/list.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default applied-on-list accessor functions builder class."""
from .series import SeriesDefault
class ListDefault(SeriesDefault):
"""Builder for default-to-pandas methods which is executed under list accessor."""
@classmethod
def frame_wrapper(cls, df):
"""
Get list accessor of the passed frame.
Parameters
----------
df : pandas.DataFrame
Returns
-------
pandas.core.arrays.arrow.ListAccessor
"""
return df.squeeze(axis=1).list
================================================
FILE: modin/core/dataframe/algebra/default2pandas/resample.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default Resamle functions builder class."""
from .default import DefaultMethod
# FIXME: there is no sence of keeping `Resampler` and `ResampleDefault` logic in a different
# classes. They should be combined.
class Resampler:
"""Builder class for resampled aggregation functions."""
@classmethod
def build_resample(cls, func, squeeze_self):
"""
Build function that resamples time-series data and does aggregation.
Parameters
----------
func : callable
Aggregation function to execute under resampled frame.
squeeze_self : bool
Whether or not to squeeze frame before resampling.
Returns
-------
callable
Function that takes pandas DataFrame and applies aggregation
to resampled time-series data.
"""
def fn(df, resample_kwargs, *args, **kwargs):
"""Resample time-series data of the passed frame and apply specified aggregation."""
if squeeze_self:
df = df.squeeze(axis=1)
resampler = df.resample(**resample_kwargs)
if type(func) is property:
return func.fget(resampler)
return func(resampler, *args, **kwargs)
return fn
class ResampleDefault(DefaultMethod):
"""Builder for default-to-pandas resampled aggregation functions."""
OBJECT_TYPE = "Resampler"
@classmethod
def register(cls, func, squeeze_self=False, **kwargs):
"""
Build function that do fallback to pandas and aggregate resampled data.
Parameters
----------
func : callable
Aggregation function to execute under resampled frame.
squeeze_self : bool, default: False
Whether or not to squeeze frame before resampling.
**kwargs : kwargs
Additional arguments that will be passed to function builder.
Returns
-------
callable
Function that takes query compiler and does fallback to pandas to resample
time-series data and apply aggregation on it.
"""
return super().register(
Resampler.build_resample(func, squeeze_self),
fn_name=func.__name__,
**kwargs
)
================================================
FILE: modin/core/dataframe/algebra/default2pandas/rolling.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default Rolling functions builder class."""
from .default import DefaultMethod
class RollingDefault(DefaultMethod):
"""Builder for default-to-pandas aggregation on a rolling window functions."""
OBJECT_TYPE = "Rolling"
@classmethod
def _build_rolling(cls, func):
"""
Build function that creates a rolling window and executes `func` on it.
Parameters
----------
func : callable
Function to execute on a rolling window.
Returns
-------
callable
Function that takes pandas DataFrame and applies `func` on a rolling window.
"""
def fn(df, rolling_kwargs, *args, **kwargs):
"""Create rolling window for the passed frame and execute specified `func` on it."""
roller = df.rolling(**rolling_kwargs)
if type(func) is property:
return func.fget(roller)
return func(roller, *args, **kwargs)
return fn
@classmethod
def register(cls, func, **kwargs):
"""
Build function that do fallback to pandas to apply `func` on a rolling window.
Parameters
----------
func : callable
Function to execute on a rolling window.
**kwargs : kwargs
Additional arguments that will be passed to function builder.
Returns
-------
callable
Function that takes query compiler and defaults to pandas to apply aggregation
`func` on a rolling window.
"""
return super().register(
cls._build_rolling(func), fn_name=func.__name__, **kwargs
)
class ExpandingDefault(DefaultMethod):
"""Builder for default-to-pandas aggregation on an expanding window functions."""
OBJECT_TYPE = "Expanding"
@classmethod
def _build_expanding(cls, func, squeeze_self):
"""
Build function that creates an expanding window and executes `func` on it.
Parameters
----------
func : callable
Function to execute on a expanding window.
squeeze_self : bool
Whether or not to squeeze frame before executing the window function.
Returns
-------
callable
Function that takes pandas DataFrame and applies `func` on a expanding window.
"""
def fn(df, rolling_args, *args, **kwargs):
"""Create rolling window for the passed frame and execute specified `func` on it."""
if squeeze_self:
df = df.squeeze(axis=1)
roller = df.expanding(*rolling_args)
if type(func) is property:
return func.fget(roller)
return func(roller, *args, **kwargs)
return fn
@classmethod
def register(cls, func, squeeze_self=False, **kwargs):
"""
Build function that do fallback to pandas to apply `func` on a expanding window.
Parameters
----------
func : callable
Function to execute on an expanding window.
squeeze_self : bool, default: False
Whether or not to squeeze frame before executing the window function.
**kwargs : kwargs
Additional arguments that will be passed to function builder.
Returns
-------
callable
Function that takes query compiler and defaults to pandas to apply aggregation
`func` on an expanding window.
"""
return super().register(
cls._build_expanding(func, squeeze_self=squeeze_self),
fn_name=func.__name__,
**kwargs
)
================================================
FILE: modin/core/dataframe/algebra/default2pandas/series.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default Series functions builder class."""
from .default import DefaultMethod
class SeriesDefault(DefaultMethod):
"""Builder for default-to-pandas methods which is executed under Series."""
OBJECT_TYPE = "Series"
@classmethod
def frame_wrapper(cls, df):
"""
Squeeze passed DataFrame to be able to process Series-specific functions on it.
Parameters
----------
df : pandas.DataFrame
One-column DataFrame to squeeze.
Returns
-------
pandas.Series
"""
return df.squeeze(axis=1)
================================================
FILE: modin/core/dataframe/algebra/default2pandas/str.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default applied-on-str functions builder class."""
from .series import SeriesDefault
class StrDefault(SeriesDefault):
"""Builder for default-to-pandas methods which is executed under `str` accessor."""
@classmethod
def frame_wrapper(cls, df):
"""
Get `str` accessor of the passed frame.
Parameters
----------
df : pandas.DataFrame
Returns
-------
pandas.core.strings.accessor.StringMethods
"""
return df.squeeze(axis=1).str
================================================
FILE: modin/core/dataframe/algebra/default2pandas/struct.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses default applied-on-struct accessor functions builder class."""
from .series import SeriesDefault
class StructDefault(SeriesDefault):
"""Builder for default-to-pandas methods which is executed under struct accessor."""
@classmethod
def frame_wrapper(cls, df):
"""
Get struct accessor of the passed frame.
Parameters
----------
df : pandas.DataFrame
Returns
-------
pandas.core.arrays.arrow.StructAccessor
"""
return df.squeeze(axis=1).struct
================================================
FILE: modin/core/dataframe/algebra/fold.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses builder class for Fold operator."""
from __future__ import annotations
from typing import TYPE_CHECKING, Callable, Optional
from .operator import Operator
if TYPE_CHECKING:
import pandas
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
class Fold(Operator):
"""Builder class for Fold functions."""
@classmethod
def register(
cls, fold_function: Callable[..., pandas.DataFrame], shape_preserved=False
) -> Callable[..., PandasQueryCompiler]:
"""
Build Fold operator that will be performed across rows/columns.
Parameters
----------
fold_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.DataFrame
Function to apply across rows/columns.
shape_preserved : bool, default: False
Whether the shape of the dataframe is preserved or not
after applying a function.
Returns
-------
callable
Function that takes query compiler and executes Fold function.
"""
def caller(
query_compiler: PandasQueryCompiler,
fold_axis: Optional[int] = None,
*args: tuple,
new_index=None,
new_columns=None,
**kwargs: dict,
) -> PandasQueryCompiler:
"""
Execute Fold function against passed query compiler.
Parameters
----------
query_compiler : PandasQueryCompiler
The query compiler to execute the function on.
fold_axis : int, optional
0 or None means apply across full column partitions. 1 means
apply across full row partitions.
*args : tuple
Additional arguments passed to `fold_function`.
new_index : list-like, optional
The index of the result.
new_columns : list-like, optional
The columns of the result.
**kwargs: dict
Additional keyword arguments passed to `fold_function`.
Returns
-------
PandasQueryCompiler
A new query compiler representing the result of executing the
function.
"""
return query_compiler.__constructor__(
query_compiler._modin_frame.fold(
cls.validate_axis(fold_axis),
lambda x: fold_function(x, *args, **kwargs),
new_index=new_index,
new_columns=new_columns,
shape_preserved=shape_preserved,
)
)
return caller
================================================
FILE: modin/core/dataframe/algebra/groupby.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses builder class for GroupByReduce operator."""
from __future__ import annotations
from typing import TYPE_CHECKING, Callable, Optional, Union
import pandas
from modin.core.dataframe.pandas.metadata import ModinIndex
from modin.error_message import ErrorMessage
from modin.utils import MODIN_UNNAMED_SERIES_LABEL, hashable
from .default2pandas.groupby import GroupBy, GroupByDefault
from .tree_reduce import TreeReduce
if TYPE_CHECKING:
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
class GroupByReduce(TreeReduce):
"""
Builder class for GroupBy aggregation functions.
Attributes
----------
ID_LEVEL_NAME : str
It's supposed that implementations may produce multiple temporary
columns per one source column in an intermediate phase. In order
for these columns to be processed accordingly at the Reduce phase,
an implementation must store unique names for such temporary
columns in the ``ID_LEVEL_NAME`` level. Duplicated names are not allowed.
_GROUPBY_REDUCE_IMPL_FLAG : str
Attribute indicating that a callable should be treated as an
implementation for one of the TreeReduce phases rather than an
arbitrary aggregation. Note: this attribute should be considered private.
"""
ID_LEVEL_NAME: str = "__ID_LEVEL_NAME__"
_GROUPBY_REDUCE_IMPL_FLAG: str = "__groupby_reduce_impl_func__"
@classmethod
def register(
cls,
map_func: Union[str, dict, Callable[..., pandas.DataFrame]],
reduce_func: Optional[Union[str, dict, Callable[..., pandas.DataFrame]]] = None,
**call_kwds: dict,
) -> Callable[..., PandasQueryCompiler]:
"""
Build template GroupBy aggregation function.
Resulted function is applied in parallel via TreeReduce algorithm.
Parameters
----------
map_func : str, dict or callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject` at the map phase. If ``str`` was passed it will
be treated as a DataFrameGroupBy's method name.
reduce_func : str, dict or callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame, optional
Function to apply to the ``DataFrameGroupBy`` at the reduce phase. If not specified
will be set the same as 'map_func'.
**call_kwds : dict
Kwargs that will be passed to the returned function.
Returns
-------
callable
Function that takes query compiler and executes GroupBy aggregation
with TreeReduce algorithm.
"""
if reduce_func is None:
reduce_func = map_func
def build_fn(name):
return lambda df, *args, **kwargs: getattr(df, name)(*args, **kwargs)
if isinstance(map_func, str):
map_func = build_fn(map_func)
if isinstance(reduce_func, str):
reduce_func = build_fn(reduce_func)
assert not (
isinstance(map_func, dict) ^ isinstance(reduce_func, dict)
) and not (
callable(map_func) ^ callable(reduce_func)
), "Map and reduce functions must be either both dict or both callable."
return lambda *args, **kwargs: cls.caller(
*args, map_func=map_func, reduce_func=reduce_func, **kwargs, **call_kwds
)
@classmethod
def register_implementation(
cls,
map_func: Callable[..., pandas.DataFrame],
reduce_func: Callable[..., pandas.DataFrame],
) -> None:
"""
Register callables to be recognized as an implementations of tree-reduce phases.
Parameters
----------
map_func : callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame
Callable to register.
reduce_func : callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame
Callable to register.
"""
setattr(map_func, cls._GROUPBY_REDUCE_IMPL_FLAG, True)
setattr(reduce_func, cls._GROUPBY_REDUCE_IMPL_FLAG, True)
@classmethod
def map(
cls,
df: pandas.DataFrame,
map_func: Callable[..., pandas.DataFrame],
axis: int,
groupby_kwargs: dict,
agg_args: list,
agg_kwargs: dict,
other: Optional[pandas.DataFrame] = None,
by=None,
drop: bool = False,
) -> pandas.DataFrame:
"""
Execute Map phase of GroupByReduce.
Groups DataFrame and applies map function. Groups will be
preserved in the results index for the following reduce phase.
Parameters
----------
df : pandas.DataFrame
Serialized frame to group.
map_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject`.
axis : {0, 1}
Axis to group and apply aggregation function along. 0 means index axis
when 1 means column axis.
groupby_kwargs : dict
Dictionary which carries arguments for `pandas.DataFrame.groupby`.
agg_args : list-like
Positional arguments to pass to the aggregation functions.
agg_kwargs : dict
Keyword arguments to pass to the aggregation functions.
other : pandas.DataFrame, optional
Serialized frame, whose columns are used to determine the groups.
If not specified, `by` parameter is used.
by : level index name or list of such labels, optional
Index levels, that is used to determine groups.
If not specified, `other` parameter is used.
drop : bool, default: False
Indicates whether or not by-data came from the `self` frame.
Returns
-------
pandas.DataFrame
GroupBy aggregation result for one particular partition.
"""
# Set `as_index` to True to track the metadata of the grouping object
# It is used to make sure that between phases we are constructing the
# right index and placing columns in the correct order.
groupby_kwargs["as_index"] = True
groupby_kwargs["observed"] = True
# We have to filter func-dict BEFORE inserting broadcasted 'by' columns
# to avoid multiple aggregation results for 'by' cols in case they're
# present in the func-dict:
apply_func = cls.get_callable(
map_func,
df,
# We won't be able to preserve the order as the Map phase would likely
# produce some temporary columns that won't fit into the original
# aggregation order. It doesn't matter much as we restore the original
# order at the Reduce phase.
preserve_aggregation_order=False,
)
if other is not None:
# Other is a broadcasted partition that represents 'by' data to group on.
# If 'drop' then the 'by' data came from the 'self' frame, thus
# inserting missed columns to the partition to group on them.
if drop or isinstance(
other := other.squeeze(axis=axis ^ 1), pandas.DataFrame
):
df = pandas.concat(
[df] + [other[[o for o in other if o not in df]]],
axis=1,
)
other = list(other.columns)
by_part = other
else:
by_part = by
result = apply_func(
df.groupby(by=by_part, axis=axis, **groupby_kwargs), *agg_args, **agg_kwargs
)
# Result could not always be a frame, so wrapping it into DataFrame
return pandas.DataFrame(result)
@classmethod
def reduce(
cls,
df: pandas.DataFrame,
reduce_func: Union[dict, Callable[..., pandas.DataFrame]],
axis: int,
groupby_kwargs: dict,
agg_args: list,
agg_kwargs: dict,
partition_idx: int = 0,
drop: bool = False,
method: Optional[str] = None,
finalizer_fn: Optional[Callable[[pandas.DataFrame], pandas.DataFrame]] = None,
) -> pandas.DataFrame:
"""
Execute Reduce phase of GroupByReduce.
Combines groups from the Map phase and applies reduce function.
Parameters
----------
df : pandas.DataFrame
Serialized frame which contain groups to combine.
reduce_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject`.
axis : {0, 1}
Axis to group and apply aggregation function along. 0 means index axis
when 1 means column axis.
groupby_kwargs : dict
Dictionary which carries arguments for `pandas.DataFrame.groupby`.
agg_args : list-like
Positional arguments to pass to the aggregation functions.
agg_kwargs : dict
Keyword arguments to pass to the aggregation functions.
partition_idx : int, default: 0
Internal index of column partition to which this function is applied.
drop : bool, default: False
Indicates whether or not by-data came from the `self` frame.
method : str, optional
Name of the groupby function. This is a hint to be able to do special casing.
finalizer_fn : callable(pandas.DataFrame) -> pandas.DataFrame, optional
A callable to execute at the end a groupby kernel against groupby result.
Returns
-------
pandas.DataFrame
GroupBy aggregation result.
"""
# Wrapping names into an Index should be unnecessary, however
# there is a bug in pandas with intersection that forces us to do so:
# https://github.com/pandas-dev/pandas/issues/39699
by_part = pandas.Index(df.index.names)
groupby_kwargs = groupby_kwargs.copy()
as_index = groupby_kwargs.get("as_index", True)
# Set `as_index` to True to track the metadata of the grouping object
groupby_kwargs["as_index"] = True
# since now index levels contain out 'by', in the reduce phace
# we want to group on these levels
groupby_kwargs["level"] = list(range(len(df.index.names)))
apply_func = cls.get_callable(reduce_func, df)
result = apply_func(
df.groupby(axis=axis, **groupby_kwargs), *agg_args, **agg_kwargs
)
if not as_index:
idx = df.index
GroupBy.handle_as_index_for_dataframe(
result,
by_part,
by_cols_dtypes=(
idx.dtypes.values
if isinstance(idx, pandas.MultiIndex) and hasattr(idx, "dtypes")
else (idx.dtype,)
),
by_length=len(by_part),
selection=reduce_func.keys() if isinstance(reduce_func, dict) else None,
partition_idx=partition_idx,
drop=drop,
method=method,
inplace=True,
)
# Result could not always be a frame, so wrapping it into DataFrame
result = pandas.DataFrame(result)
if result.index.name == MODIN_UNNAMED_SERIES_LABEL:
result.index.name = None
return result if finalizer_fn is None else finalizer_fn(result)
@classmethod
def caller(
cls,
query_compiler: PandasQueryCompiler,
by,
map_func: Union[dict, Callable[..., pandas.DataFrame]],
reduce_func: Union[dict, Callable[..., pandas.DataFrame]],
axis: int,
groupby_kwargs: dict,
agg_args: list,
agg_kwargs: dict,
drop: bool = False,
method: Optional[str] = None,
default_to_pandas_func: Optional[Callable[..., pandas.DataFrame]] = None,
finalizer_fn: Optional[Callable[[pandas.DataFrame], pandas.DataFrame]] = None,
) -> PandasQueryCompiler:
"""
Execute GroupBy aggregation with TreeReduce approach.
Parameters
----------
query_compiler : PandasQueryCompiler
Frame to group.
by : PandasQueryCompiler, column or index label, Grouper or list of such
Object that determine groups.
map_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject` at the Map phase.
reduce_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject` at the Reduce phase.
axis : {0, 1}
Axis to group and apply aggregation function along. 0 means index axis
when 1 means column axis.
groupby_kwargs : dict
Dictionary which carries arguments for pandas.DataFrame.groupby.
agg_args : list-like
Positional arguments to pass to the aggregation functions.
agg_kwargs : dict
Keyword arguments to pass to the aggregation functions.
drop : bool, default: False
Indicates whether or not by-data came from the `self` frame.
method : str, optional
Name of the GroupBy aggregation function. This is a hint to be able to do special casing.
default_to_pandas_func : callable(pandas.DataFrameGroupBy) -> pandas.DataFrame, optional
The pandas aggregation function equivalent to the `map_func + reduce_func`.
Used in case of defaulting to pandas. If not specified `map_func` is used.
finalizer_fn : callable(pandas.DataFrame) -> pandas.DataFrame, optional
A callable to execute at the end a groupby kernel against groupby result.
Returns
-------
PandasQueryCompiler
QueryCompiler which carries the result of GroupBy aggregation.
"""
is_unsupported_axis = axis != 0
# Defaulting to pandas in case of an empty frame as we can't process it properly.
# Higher API level won't pass empty data here unless the frame has delayed
# computations. So we apparently lose some laziness here (due to index access)
# because of the inability to process empty groupby natively.
is_empty_data = (
len(query_compiler.columns) == 0 or len(query_compiler.index) == 0
)
is_grouping_using_by_arg = (
groupby_kwargs.get("level", None) is None and by is not None
)
is_unsupported_by_arg = isinstance(by, pandas.Grouper) or (
not hashable(by) and not isinstance(by, type(query_compiler))
)
if (
is_unsupported_axis
or is_empty_data
or (is_grouping_using_by_arg and is_unsupported_by_arg)
):
if default_to_pandas_func is None:
default_to_pandas_func = (
(lambda grp: grp.agg(map_func))
if isinstance(map_func, dict)
else map_func
)
default_to_pandas_func = GroupByDefault.register(default_to_pandas_func)
return default_to_pandas_func(
query_compiler,
by=by,
axis=axis,
groupby_kwargs=groupby_kwargs,
agg_args=agg_args,
agg_kwargs=agg_kwargs,
drop=drop,
)
# The bug only occurs in the case of Categorical 'by', so we might want to check whether any of
# the 'by' dtypes is Categorical before going into this branch, however triggering 'dtypes'
# computation if they're not computed may take time, so we don't do it
if not groupby_kwargs.get("sort", True) and isinstance(
by, type(query_compiler)
):
ErrorMessage.mismatch_with_pandas(
operation="df.groupby(categorical_by, sort=False)",
message=(
"the groupby keys will be sorted anyway, although the 'sort=False' was passed. "
+ "See the following issue for more details: "
+ "https://github.com/modin-project/modin/issues/3571"
),
)
groupby_kwargs = groupby_kwargs.copy()
groupby_kwargs["sort"] = True
map_fn, reduce_fn = cls.build_map_reduce_functions(
by=by,
axis=axis,
groupby_kwargs=groupby_kwargs,
map_func=map_func,
reduce_func=reduce_func,
agg_args=agg_args,
agg_kwargs=agg_kwargs,
drop=drop,
method=method,
finalizer_fn=finalizer_fn,
)
# If `by` is a ModinFrame, then its partitions will be broadcasted to every
# `self` partition in a way determined by engine (modin_frame.groupby_reduce)
# Otherwise `by` was already bound to the Map function in `build_map_reduce_functions`.
broadcastable_by = getattr(by, "_modin_frame", None)
apply_indices = list(map_func.keys()) if isinstance(map_func, dict) else None
if (
broadcastable_by is not None
and groupby_kwargs.get("as_index", True)
and broadcastable_by.has_materialized_dtypes
):
new_index = ModinIndex(
# actual value will be assigned on a parent update
value=None,
axis=0,
dtypes=broadcastable_by.dtypes,
)
else:
new_index = None
new_modin_frame = query_compiler._modin_frame.groupby_reduce(
axis,
broadcastable_by,
map_fn,
reduce_fn,
apply_indices=apply_indices,
new_index=new_index,
)
result = query_compiler.__constructor__(new_modin_frame)
return result
@classmethod
def get_callable(
cls,
agg_func: Union[dict, Callable[..., pandas.DataFrame]],
df: pandas.DataFrame,
preserve_aggregation_order: bool = True,
) -> Callable[..., pandas.DataFrame]:
"""
Build aggregation function to apply to each group at this particular partition.
If it's dictionary aggregation — filters aggregation dictionary for keys which
this particular partition contains, otherwise do nothing with passed function.
Parameters
----------
agg_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Aggregation function.
df : pandas.DataFrame
Serialized partition which contains available columns.
preserve_aggregation_order : bool, default: True
Whether to manually restore the order of columns for the result specified
by the `agg_func` keys (only makes sense when `agg_func` is a dictionary).
Returns
-------
callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame
Aggregation function that can be safely applied to this particular partition.
"""
if not isinstance(agg_func, dict):
return agg_func
grp_has_id_level = df.columns.names[0] == cls.ID_LEVEL_NAME
# The 'id' level prevents us from a lookup for the original
# partition's columns. So dropping the level.
partition_columns = frozenset(
df.columns.droplevel(0) if grp_has_id_level else df.columns
)
partition_dict = {k: v for k, v in agg_func.items() if k in partition_columns}
return cls._build_callable_for_dict(
partition_dict, preserve_aggregation_order, grp_has_id_level
)
@classmethod
def _build_callable_for_dict(
cls,
agg_dict: dict,
preserve_aggregation_order: bool = True,
grp_has_id_level: bool = False,
) -> Callable[..., pandas.DataFrame]:
"""
Build callable for an aggregation dictionary.
Parameters
----------
agg_dict : dict
Aggregation dictionary.
preserve_aggregation_order : bool, default: True
Whether to manually restore the order of columns for the result specified
by the `agg_func` keys (only makes sense when `agg_func` is a dictionary).
grp_has_id_level : bool, default: False
Whether the frame we're grouping on has intermediate columns
(see ``cls.ID_LEVEL_NAME``).
Returns
-------
callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame
"""
# We have to keep this import away from the module level to avoid circular import
from modin.pandas.utils import walk_aggregation_dict
# We now filter aggregation functions into those that could be applied natively
# using pandas (pandas_grp_obj.agg(**native_aggs)) and those that require
# special treatment (custom_aggs).
custom_aggs = {}
native_aggs = {}
result_columns = []
for col, func, func_name, col_renaming_required in walk_aggregation_dict(
agg_dict
):
# Filter dictionary
dict_to_add = (
custom_aggs if cls.is_registered_implementation(func) else native_aggs
)
new_value = func if func_name is None else (func_name, func)
old_value = dict_to_add.get(col, None)
if old_value is not None:
ErrorMessage.catch_bugs_and_request_email(
failure_condition=not isinstance(old_value, list),
extra_log="Expected for all aggregation values to be a list when at least "
+ f"one column has multiple aggregations. Got: {old_value} {type(old_value)}",
)
old_value.append(new_value)
else:
# Pandas knows that it has to modify the resulting columns if it meets
# a function wrapped into a list. Renaming is required if either a new
# column name was explicitly specified, or multiple functions were
# specified per one column, or if any other column in the aggregation
# is going to be renamed.
dict_to_add[col] = [new_value] if col_renaming_required else new_value
# Construct resulting columns
if col_renaming_required:
func_name = str(func) if func_name is None else func_name
result_columns.append(
(*(col if isinstance(col, tuple) else (col,)), func_name)
)
else:
result_columns.append(col)
result_columns = pandas.Index(result_columns)
def aggregate_on_dict(grp_obj, *args, **kwargs):
"""Aggregate the passed groupby object."""
if len(native_aggs) == 0:
native_agg_res = None
elif grp_has_id_level:
# Adding the 'id' level to the aggregation keys so they match `grp_obj` columns
native_aggs_modified = {
(
cls.ID_LEVEL_NAME,
*(key if isinstance(key, tuple) else (key,)),
): value
for key, value in native_aggs.items()
}
native_agg_res = grp_obj.agg(native_aggs_modified)
# Dropping the 'id' level from the resulted frame
native_agg_res.columns = native_agg_res.columns.droplevel(0)
else:
native_agg_res = grp_obj.agg(native_aggs)
custom_results = []
insert_id_levels = False
for col, func, func_name, col_renaming_required in walk_aggregation_dict(
custom_aggs
):
if grp_has_id_level:
cols_without_ids = grp_obj.obj.columns.droplevel(0)
if isinstance(cols_without_ids, pandas.MultiIndex):
# We may have multiple columns matching the `col` in
# a MultiIndex case, that's why use `.get_locs` here
col_pos = cols_without_ids.get_locs(col)
else:
# `pandas.Index` doesn't have `.get_locs` method
col_pos = cols_without_ids.get_loc(col)
agg_key = grp_obj.obj.columns[col_pos]
else:
agg_key = [col]
result = func(grp_obj[agg_key])
# The `func` may have discarded an ID-level if there were any.
# So checking for this again.
result_has_id_level = result.columns.names[0] == cls.ID_LEVEL_NAME
insert_id_levels |= result_has_id_level
if col_renaming_required:
func_name = str(func) if func_name is None else func_name
if result_has_id_level:
result.columns = pandas.MultiIndex.from_tuples(
[
# `old_col[0]` stores values from the 'id'
# level, the ones we want to preserve here
(old_col[0], col, func_name)
for old_col in result.columns
],
names=[
result.columns.names[0],
result.columns.names[1],
None,
],
)
else:
result.columns = pandas.MultiIndex.from_tuples(
[(col, func_name)] * len(result.columns),
names=[result.columns.names[0], None],
)
custom_results.append(result)
if insert_id_levels:
# As long as any `result` has an id-level we have to insert the level
# into every `result` so the number of levels matches
for idx, ext_result in enumerate(custom_results):
if ext_result.columns.names[0] != cls.ID_LEVEL_NAME:
custom_results[idx] = pandas.concat(
[ext_result],
keys=[cls.ID_LEVEL_NAME],
names=[cls.ID_LEVEL_NAME],
axis=1,
copy=False,
)
if native_agg_res is not None:
native_agg_res = pandas.concat(
[native_agg_res],
keys=[cls.ID_LEVEL_NAME],
names=[cls.ID_LEVEL_NAME],
axis=1,
copy=False,
)
native_res_part = [] if native_agg_res is None else [native_agg_res]
parts = [*native_res_part, *custom_results]
if parts:
result = pandas.concat(parts, axis=1, copy=False)
else:
result = pandas.DataFrame(columns=result_columns)
# The order is naturally preserved if there's no custom aggregations
if preserve_aggregation_order and len(custom_aggs):
result = result.reindex(result_columns, axis=1)
return result
return aggregate_on_dict
@classmethod
def is_registered_implementation(cls, func: Callable) -> bool:
"""
Check whether the passed `func` was registered as a TreeReduce implementation.
Parameters
----------
func : callable
Returns
-------
bool
"""
return callable(func) and hasattr(func, cls._GROUPBY_REDUCE_IMPL_FLAG)
@classmethod
def build_map_reduce_functions(
cls,
by,
axis: int,
groupby_kwargs: dict,
map_func: Union[dict, Callable[..., pandas.DataFrame]],
reduce_func: Union[dict, Callable[..., pandas.DataFrame]],
agg_args: list,
agg_kwargs: dict,
drop: bool = False,
method: Optional[str] = None,
finalizer_fn: Callable[[pandas.DataFrame], pandas.DataFrame] = None,
) -> tuple[Callable, Callable]:
"""
Bind appropriate arguments to map and reduce functions.
Parameters
----------
by : BaseQueryCompiler, column or index label, Grouper or list of such
Object that determine groups.
axis : {0, 1}
Axis to group and apply aggregation function along. 0 means index axis
when 1 means column axis.
groupby_kwargs : dict
Dictionary which carries arguments for pandas.DataFrame.groupby.
map_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject` at the Map phase.
reduce_func : dict or callable(pandas.DataFrameGroupBy) -> pandas.DataFrame
Function to apply to the `GroupByObject` at the Reduce phase.
agg_args : list-like
Positional arguments to pass to the aggregation functions.
agg_kwargs : dict
Keyword arguments to pass to the aggregation functions.
drop : bool, default: False
Indicates whether or not by-data came from the `self` frame.
method : str, optional
Name of the GroupBy aggregation function. This is a hint to be able to do special casing.
finalizer_fn : callable(pandas.DataFrame) -> pandas.DataFrame, optional
A callable to execute at the end a groupby kernel against groupby result.
Returns
-------
Tuple of callable
Tuple of map and reduce functions with bound arguments.
"""
# if by is a query compiler, then it will be broadcasted explicit via
# groupby_reduce method of the modin frame and so we don't want secondary
# implicit broadcastion via passing it as an function argument.
if hasattr(by, "_modin_frame"):
by = None
def _map(
df: pandas.DataFrame,
other: Optional[pandas.DataFrame] = None,
**kwargs: dict,
) -> pandas.DataFrame:
def wrapper(
df: pandas.DataFrame, other: Optional[pandas.DataFrame] = None
) -> pandas.DataFrame:
return cls.map(
df,
other=other,
axis=axis,
by=by,
groupby_kwargs=groupby_kwargs.copy(),
map_func=map_func,
agg_args=agg_args,
agg_kwargs=agg_kwargs,
drop=drop,
**kwargs,
)
try:
result = wrapper(df, other)
# This will happen with Arrow buffer read-only errors. We don't want to copy
# all the time, so this will try to fast-path the code first.
except ValueError:
result = wrapper(df.copy(), other if other is None else other.copy())
return result
def _reduce(df: pandas.DataFrame, **call_kwargs: dict) -> pandas.DataFrame:
def wrapper(df: pandas.DataFrame):
return cls.reduce(
df,
axis=axis,
groupby_kwargs=groupby_kwargs,
reduce_func=reduce_func,
agg_args=agg_args,
agg_kwargs=agg_kwargs,
drop=drop,
method=method,
finalizer_fn=finalizer_fn,
**call_kwargs,
)
try:
result = wrapper(df)
# This will happen with Arrow buffer read-only errors. We don't want to copy
# all the time, so this will try to fast-path the code first.
except ValueError:
result = wrapper(df.copy())
return result
return _map, _reduce
================================================
FILE: modin/core/dataframe/algebra/map.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses builder class for Map operator."""
from __future__ import annotations
from typing import TYPE_CHECKING, Callable
from .operator import Operator
if TYPE_CHECKING:
import pandas
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
class Map(Operator):
"""Builder class for Map operator."""
@classmethod
def register(
cls,
function: Callable[..., pandas.DataFrame],
*call_args: tuple,
**call_kwds: dict,
) -> Callable[..., PandasQueryCompiler]:
"""
Build Map operator that will be performed across each partition.
Parameters
----------
function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.DataFrame
Function that will be applied to the each partition.
Function takes `pandas.DataFrame` and returns `pandas.DataFrame`
of the same shape.
*call_args : tuple
Args that will be passed to the returned function.
**call_kwds : dict
Kwargs that will be passed to the returned function.
Returns
-------
callable
Function that takes query compiler and executes map function.
"""
def caller(
query_compiler: PandasQueryCompiler, *args: tuple, **kwargs: dict
) -> PandasQueryCompiler:
"""Execute Map function against passed query compiler."""
shape_hint = call_kwds.pop("shape_hint", None) or query_compiler._shape_hint
return query_compiler.__constructor__(
query_compiler._modin_frame.map(
lambda x: function(x, *args, **kwargs), *call_args, **call_kwds
),
shape_hint=shape_hint,
)
return caller
================================================
FILE: modin/core/dataframe/algebra/operator.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module contains an interface for operator builder classes."""
from __future__ import annotations
from typing import Callable, Optional
class Operator(object):
"""Interface for building operators that can execute in parallel across partitions."""
def __init__(self) -> None:
raise ValueError(
"Please use {}.register instead of the constructor".format(
type(self).__name__
)
)
@classmethod
def register(cls, func: Callable, **kwargs: dict):
"""
Build operator that applies source function across the entire dataset.
Parameters
----------
func : callable
Source function.
**kwargs : dict
Kwargs that will be passed to the builder function.
Returns
-------
callable
"""
raise NotImplementedError("Please implement in child class")
@classmethod
def validate_axis(cls, axis: Optional[int]) -> int:
"""
Ensure that axis to apply function on has valid value.
Parameters
----------
axis : int, optional
0 or None means apply on index, 1 means apply on columns.
Returns
-------
int
Integer representation of given axis.
"""
return 0 if axis is None else axis
================================================
FILE: modin/core/dataframe/algebra/reduce.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses builder class for Reduce operator."""
from __future__ import annotations
from typing import TYPE_CHECKING, Callable, Optional
from .operator import Operator
if TYPE_CHECKING:
import pandas
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
class Reduce(Operator):
"""Builder class for Reduce operator."""
@classmethod
def register(
cls,
reduce_function: Callable[..., pandas.Series],
axis: Optional[int] = None,
shape_hint: Optional[str] = None,
) -> Callable[..., PandasQueryCompiler]:
"""
Build Reduce operator that will be performed across rows/columns.
It's used if `func` reduces the dimension of partitions in contrast to `Fold`.
Parameters
----------
reduce_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.Series
Source function.
axis : int, optional
Axis to apply function along.
shape_hint : {"row", "column", None}, default: None
Shape hint for the results known to be a column or a row, otherwise None.
Returns
-------
callable
Function that takes query compiler and executes Reduce function.
"""
def caller(
query_compiler: PandasQueryCompiler, *args: tuple, **kwargs: dict
) -> PandasQueryCompiler:
"""Execute Reduce function against passed query compiler."""
_axis = kwargs.get("axis") if axis is None else axis
return query_compiler.__constructor__(
query_compiler._modin_frame.reduce(
cls.validate_axis(_axis),
lambda x: reduce_function(x, *args, **kwargs),
),
shape_hint=shape_hint,
)
return caller
================================================
FILE: modin/core/dataframe/algebra/tree_reduce.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses builder class for TreeReduce operator."""
from __future__ import annotations
from typing import TYPE_CHECKING, Callable, Optional
from .operator import Operator
if TYPE_CHECKING:
import pandas
from pandas._typing import DtypeObj
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
class TreeReduce(Operator):
"""Builder class for TreeReduce operator."""
@classmethod
def register(
cls,
map_function: Optional[Callable[..., pandas.DataFrame]],
reduce_function: Optional[Callable[..., pandas.Series]] = None,
axis: Optional[int] = None,
compute_dtypes: Optional[Callable[..., DtypeObj]] = None,
) -> Callable[..., PandasQueryCompiler]:
"""
Build TreeReduce operator.
Parameters
----------
map_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.DataFrame
Source map function.
reduce_function : callable(pandas.DataFrame, *args, **kwargs) -> pandas.Series, optional
Source reduce function.
axis : int, optional
Specifies axis to apply function along.
compute_dtypes : callable(pandas.Series, *func_args, **func_kwargs) -> DtypeObj, optional
Callable for computing dtypes.
Returns
-------
callable
Function that takes query compiler and executes passed functions
with TreeReduce algorithm.
"""
if reduce_function is None:
reduce_function = map_function
def caller(
query_compiler: PandasQueryCompiler, *args: tuple, **kwargs: dict
) -> PandasQueryCompiler:
"""Execute TreeReduce function against passed query compiler."""
_axis = kwargs.get("axis") if axis is None else axis
new_dtypes = None
if compute_dtypes and query_compiler.frame_has_materialized_dtypes:
new_dtypes = str(compute_dtypes(query_compiler.dtypes, *args, **kwargs))
return query_compiler.__constructor__(
query_compiler._modin_frame.tree_reduce(
cls.validate_axis(_axis),
lambda x: map_function(x, *args, **kwargs),
lambda y: reduce_function(y, *args, **kwargs),
dtypes=new_dtypes,
)
)
return caller
================================================
FILE: modin/core/dataframe/base/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe classes."""
================================================
FILE: modin/core/dataframe/base/dataframe/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe class and Axis and JoinType Enums."""
================================================
FILE: modin/core/dataframe/base/dataframe/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module contains class ModinDataframe.
ModinDataframe is a parent abstract class for any dataframe class.
"""
from abc import ABC, abstractmethod
from typing import Callable, Dict, Hashable, List, Optional, Union
from modin.core.dataframe.base.dataframe.utils import Axis, JoinType
class ModinDataframe(ABC):
"""
An abstract class that represents the Parent class for any Dataframe class.
This class is intended to specify the behaviors that a Dataframe must implement.
For more details about how these methods were chosen, please refer to this
(https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) paper, which specifies
a Dataframe algebra that this class exposes.
"""
@abstractmethod
def take_2d_labels_or_positional(
self,
row_labels: Optional[List[Hashable]] = None,
row_positions: Optional[List[int]] = None,
col_labels: Optional[List[Hashable]] = None,
col_positions: Optional[List[int]] = None,
) -> "ModinDataframe":
"""
Mask rows and columns in the dataframe.
Allow users to perform selection and projection on the row and column labels (named notation),
in addition to the row and column number (positional notation).
Parameters
----------
row_labels : list of hashable, optional
The row labels to extract.
row_positions : list of int, optional
The row positions to extract.
col_labels : list of hashable, optional
The column labels to extract.
col_positions : list of int, optional
The column positions to extract.
Returns
-------
ModinDataframe
A new ModinDataframe from the mask provided.
Notes
-----
If both `row_labels` and `row_positions` are provided, a ValueError is raised.
The same rule applies for `col_labels` and `col_positions`.
"""
pass
@abstractmethod
def filter_by_types(self, types: List[Hashable]) -> "ModinDataframe":
"""
Allow the user to specify a type or set of types by which to filter the columns.
Parameters
----------
types : list of hashables
The types to filter columns by.
Returns
-------
ModinDataframe
A new ModinDataframe with only the columns whose dtypes appear in `types`.
"""
pass
@abstractmethod
def map(
self,
function: Callable,
axis: Optional[Union[int, Axis]] = None,
dtypes: Optional[str] = None,
new_columns: Optional[List[Hashable]] = None,
) -> "ModinDataframe":
"""
Apply a user-defined function row-wise if `axis`=0, column-wise if `axis`=1, and cell-wise if `axis` is None.
Parameters
----------
function : callable(row|col|cell) -> row|col|cell
The function to map across the dataframe.
axis : int or modin.core.dataframe.base.utils.Axis, optional
The axis to map over.
dtypes : str, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
new_columns : List[Hashable], optional
New column labels of the result, its length has to be identical
to the older columns. If not specified, old column labels are preserved.
Returns
-------
ModinDataframe
A new ModinDataframe with the map applied.
Notes
-----
This does not change the shape of the dataframe.
"""
pass
@abstractmethod
def filter(self, axis: Union[int, Axis], condition: Callable) -> "ModinDataframe":
"""
Filter data based on the function provided along the specified axis.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to filter over.
condition : callable(row|col) -> bool
The function to use for the filter. This function should filter the
data itself. It accepts either a row or column (depending on the axis argument) and
returns True to keep the row/col, and False to drop it.
Returns
-------
ModinDataframe
A new ModinDataframe filtered by content according to the filter provided by condition.
"""
pass
@abstractmethod
def explode(
self,
axis: Union[int, Axis],
function: Callable,
result_schema: Optional[Dict[Hashable, type]] = None,
) -> "ModinDataframe":
"""
Explode data based on the function provided along the specified axis.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to expand over.
function : callable
The function to use to expand the data. This function should accept one
row/column, and return multiple.
result_schema : dictionary, optional
Mapping from column labels to data types that represents the types of the output dataframe.
Returns
-------
ModinDataframe
A new ModinDataframe with the specified axis expanded.
Notes
-----
Only one axis can be expanded at a time.
The user-defined function may increase the number of rows (columns if axis=1),
but it should not remove or drop rows.
"""
pass
@abstractmethod
def window(
self,
axis: Union[int, Axis],
reduce_fn: Callable,
window_size: int,
result_schema: Optional[Dict[Hashable, type]] = None,
) -> "ModinDataframe":
"""
Apply a sliding window operator that acts as a GROUPBY on each window, reducing each window to a single row (column).
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to slide over.
reduce_fn : callable(rowgroup|colgroup) -> row|col
The reduce function to apply over the data.
window_size : int
The number of row/columns to pass to the function.
(The size of the sliding window).
result_schema : dictionary, optional
Mapping from column labels to data types that represents the types of the output dataframe.
Returns
-------
ModinDataframe
A new ModinDataframe with the reduce function applied over windows of the specified
axis.
Notes
-----
The user-defined reduce function must reduce each window's column
(row if axis=1) down to a single value.
"""
pass
@abstractmethod
def groupby(
self,
axis: Union[int, Axis],
by: Union[str, List[str]],
operator: Callable,
result_schema: Optional[Dict[Hashable, type]] = None,
) -> "ModinDataframe":
"""
Generate groups based on values in the input column(s) and perform the specified operation on each.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to apply the grouping over.
by : string or list of strings
One or more column labels to use for grouping.
operator : callable
The operation to carry out on each of the groups. The operator is another
algebraic operator with its own user-defined function parameter, depending
on the output desired by the user.
result_schema : dictionary, optional
Mapping from column labels to data types that represents the types of the output dataframe.
Returns
-------
ModinDataframe
A new ModinDataframe containing the groupings specified, with the operator
applied to each group.
Notes
-----
No communication between groups is allowed in this algebra implementation.
The number of rows (columns if axis=1) returned by the user-defined function
passed to the groupby may be at most the number of rows in the group, and
may be as small as a single row.
Unlike the pandas API, an intermediate "GROUP BY" object is not present in this
algebra implementation.
"""
pass
@abstractmethod
def reduce(
self,
axis: Union[int, Axis],
function: Callable,
dtypes: Optional[str] = None,
) -> "ModinDataframe":
"""
Perform a user-defined aggregation on the specified axis, where the axis reduces down to a singleton.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the reduce over.
function : callable(row|col) -> single value
The reduce function to apply to each column.
dtypes : str, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
Returns
-------
ModinDataframe
A new ModinDataframe with the same columns as the previous, with only a single row.
Notes
-----
The user-defined function must reduce to a single value.
"""
pass
@abstractmethod
def tree_reduce(
self,
axis: Union[int, Axis],
map_func: Callable,
reduce_func: Optional[Callable] = None,
dtypes: Optional[str] = None,
) -> "ModinDataframe":
"""
Perform a user-defined aggregation on the specified axis, where the axis reduces down to a singleton using a tree-reduce computation pattern.
The map function is applied first over multiple partitions of a column, and then the reduce
function (if specified, otherwise the map function is applied again) is applied to the
results to produce a single value.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the tree reduce over.
map_func : callable(row|col) -> row|col|single value
The map function to apply to each column.
reduce_func : callable(row|col) -> single value, optional
The reduce function to apply to the results of the map function.
dtypes : str, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
Returns
-------
ModinDataframe
A new ModinDataframe with the same columns as the previous, with only a single row.
Notes
-----
The user-defined function must reduce to a single value.
If the user-defined function requires access to the entire column, please use reduce instead.
"""
pass
@abstractmethod
def infer_types(self, columns_list: List[str]) -> "ModinDataframe":
"""
Determine the compatible type shared by all values in the specified columns, and coerce them to that type.
Parameters
----------
columns_list : list of strings
List of column labels to infer and induce types over.
Returns
-------
ModinDataframe
A new ModinDataframe with the inferred schema.
"""
pass
@abstractmethod
def join(
self,
axis: Union[int, Axis],
condition: Callable,
other: "ModinDataframe",
join_type: Union[str, JoinType],
) -> "ModinDataframe":
"""
Join this dataframe with the other.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the join on.
condition : callable
Function that determines which rows should be joined. The condition can be a
simple equality, e.g. "left.col1 == right.col1" or can be arbitrarily complex.
other : ModinDataframe
The other data to join with, i.e. the right dataframe.
join_type : string {"inner", "left", "right", "outer"} or modin.core.dataframe.base.utils.JoinType
The type of join to perform.
Returns
-------
ModinDataframe
A new ModinDataframe that is the result of applying the specified join over the two
dataframes.
Notes
-----
During the join, this dataframe is considered the left, while the other is
treated as the right.
Only inner joins, left outer, right outer, and full outer joins are currently supported.
Support for other join types (e.g. natural join) may be implemented in the future.
"""
pass
@abstractmethod
def concat(
self,
axis: Union[int, Axis],
others: Union["ModinDataframe", List["ModinDataframe"]],
) -> "ModinDataframe":
"""
Append rows/columns along the specified axis from multiple dataframes.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis on which to perform the concatenation.
others : ModinDataframe or list of ModinDataframes
The other ModinDataframe(s) to concatenate.
Returns
-------
ModinDataframe
A new ModinDataframe that is the result of concatenating the dataframes over the
specified axis.
Notes
-----
The concat operator incurs fixed overheads, and so this algebra places no
limit to the number of dataframes that may be concatenated in this way.
"""
pass
@abstractmethod
def transpose(self) -> "ModinDataframe":
"""
Swap the row and column axes.
Returns
-------
ModinDataframe
A new ModinDataframe with the row and column axes swapped.
Notes
-----
Transposing a dataframe is expensive, and so it is performed lazily. The axes are swapped
logically immediately, but the physical swap does not occur until absolutely necessary,
which helps motivate the axis argument to the other operators in this algebra.
"""
pass
@abstractmethod
def to_labels(self, column_labels: Union[str, List[str]]) -> "ModinDataframe":
"""
Replace the row labels with one or more columns of data.
Parameters
----------
column_labels : string or list of strings
Column label(s) to use as the new row labels.
Returns
-------
ModinDataframe
A new ModinDataframe with the row labels replaced by the specified columns.
Notes
-----
When multiple column labels are specified, a hierarchical set of labels is created, ordered by the ordering
of labels in the input.
"""
pass
@abstractmethod
def from_labels(self) -> "ModinDataframe":
"""
Move the row labels into the data at position 0, and sets the row labels to the positional notation.
Returns
-------
ModinDataframe
A new ModinDataframe with the row labels moved into the data.
Notes
-----
In the case that the dataframe has hierarchical labels, all label "levels" are inserted into the dataframe
in the order they occur in the labels, with the outermost being in position 0.
"""
pass
@abstractmethod
def rename(
self,
new_row_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,
new_col_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,
) -> "ModinDataframe":
"""
Replace the row and column labels with the specified new labels.
Parameters
----------
new_row_labels : dictionary or callable, optional
Mapping or callable that relates old row labels to new labels.
new_col_labels : dictionary or callable, optional
Mapping or callable that relates old col labels to new labels.
Returns
-------
ModinDataframe
A new ModinDataframe with the new row and column labels.
"""
pass
@abstractmethod
def sort_by(
self,
axis: Union[int, Axis],
labels: Union[str, List[str]],
ascending: bool = True,
) -> "ModinDataframe":
"""
Logically reorder rows (columns if axis=1) lexicographically by the data in a column or set of columns.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the sort over.
labels : string or list of strings
Column (row if axis=1) label(s) to use to determine lexicographical ordering. If multiple
columns (rows if axis=1) are provided, the sort is performed on the first column (row if axis=1),
with ties broken by the other columns (rows if axis=1) provided.
ascending : boolean, default: True
Whether to sort in ascending or descending order.
Returns
-------
ModinDataframe
A new ModinDataframe sorted into lexicographical order by the specified column(s).
"""
pass
================================================
FILE: modin/core/dataframe/base/dataframe/utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module contains useful enums for Modin.
Axis is an enum that represents the `axis` argument for dataframe operations.
JoinType is an enum that represents the `join_type` or `how` argument for the join algebra operator.
"""
from enum import Enum
from typing import Dict, List, Sequence, Tuple, cast
import pandas
from pandas._typing import IndexLabel
from pandas.api.types import is_scalar
from pandas.core.dtypes.common import is_integer_dtype
class Axis(Enum): # noqa: PR01
"""
An enum that represents the `axis` argument provided to the algebra operators.
The enum has 3 values - ROW_WISE to represent the row axis, COL_WISE to represent the
column axis, and CELL_WISE to represent no axis. ROW_WISE operations iterate over the rows
COL_WISE operations over the columns, and CELL_WISE operations over any of the partitioning
schemes that are supported in Modin (row-wise, column-wise, or block-wise).
"""
ROW_WISE = 0
COL_WISE = 1
CELL_WISE = None
class JoinType(Enum): # noqa: PR01
"""
An enum that represents the `join_type` argument provided to the algebra operators.
The enum has 4 values - INNER to represent inner joins, LEFT to represent left joins, RIGHT to
represent right joins, and OUTER to represent outer joins.
"""
INNER = "inner"
LEFT = "left"
RIGHT = "right"
OUTER = "outer"
def join_columns(
left: pandas.Index,
right: pandas.Index,
left_on: IndexLabel,
right_on: IndexLabel,
suffixes: Tuple[str, str],
) -> Tuple[pandas.Index, Dict[IndexLabel, IndexLabel], Dict[IndexLabel, IndexLabel]]:
"""
Compute resulting columns for the two dataframes being merged.
Parameters
----------
left : pandas.Index
Columns of the left frame to join.
right : pandas.Index
Columns of the right frame to join.
left_on : list-like or scalar
Column names on which the frames are joined in the left DataFrame.
right_on : list-like or scalar
Column names on which the frames are joined in the right DataFrame.
suffixes : tuple[str, str]
A 2-length sequence containing suffixes to append to the intersected columns.
Returns
-------
pandas.Index, dict[IndexLabel -> IndexLabel], dict[IndexLabel -> IndexLabel]
Returns columns for the resulting frame and mappings of old to new column
names for `left` and `right` accordingly.
Raises
------
NotImplementedError
Raised when one of the keys to join is an index level, pandas behaviour is really
complicated in this case, so we're not supporting this case for now.
"""
# using `cast` to make `mypy` acknowledged that the variable now ensured to be `Sequence[IndexLabel]`
left_on = cast(Sequence[IndexLabel], [left_on] if is_scalar(left_on) else left_on)
right_on = cast(
Sequence[IndexLabel], [right_on] if is_scalar(right_on) else right_on
)
# handling a simple case of merging on one column and when the column is located in an index
if len(left_on) == 1 and len(right_on) == 1 and left_on[0] == right_on[0]:
if left_on[0] not in left and right_on[0] not in right:
# in this case the 'on' column will stay in the index, so we can simply
# drop the 'left/right_on' values and proceed as normal
left_on = []
right_on = []
# in other cases, we can simply add the index name to columns and proceed as normal
# on python 3.9 with pandas-stubs 2.2, these lines will warn about insert being an untyped call,
# but this error is no longer present on higher versions
elif left_on[0] not in left:
left = left.insert(loc=0, item=left_on[0]) # type: ignore[no-untyped-call, unused-ignore]
elif right_on[0] not in right:
right = right.insert(loc=0, item=right_on[0]) # type: ignore[no-untyped-call, unused-ignore]
if any(col not in left for col in left_on) or any(
col not in right for col in right_on
):
raise NotImplementedError(
"Cases, where one of the keys to join is an index level, are not yet supported."
)
left_conflicts = set(left) & (set(right) - set(right_on))
right_conflicts = set(right) & (set(left) - set(left_on))
conflicting_cols = left_conflicts | right_conflicts
def _get_new_name(col: IndexLabel, suffix: str) -> IndexLabel:
if col in conflicting_cols:
return (
(f"{col[0]}{suffix}", *col[1:])
if isinstance(col, tuple)
else f"{col}{suffix}"
)
else:
return col
left_renamer: Dict[IndexLabel, IndexLabel] = {}
right_renamer: Dict[IndexLabel, IndexLabel] = {}
new_left: List = []
new_right: List = []
for col in left:
new_name = _get_new_name(col, suffixes[0])
new_left.append(new_name)
left_renamer[col] = new_name
for col in right:
# If we're joining on the column that exists in both frames then it was already
# taken from the 'left', don't want to take it again from the 'right'.
if not (col in left_on and col in right_on):
new_name = _get_new_name(col, suffixes[1])
new_right.append(new_name)
right_renamer[col] = new_name
new_columns = pandas.Index(new_left + new_right)
return new_columns, left_renamer, right_renamer
def is_trivial_index(index: pandas.Index) -> bool:
"""
Check if the index is a trivial index, i.e. a sequence [0..n].
Parameters
----------
index : pandas.Index
An index to check.
Returns
-------
bool
"""
if len(index) == 0:
return True
if isinstance(index, pandas.RangeIndex):
return index.start == 0 and index.step == 1
if not (isinstance(index, pandas.Index) and is_integer_dtype(index)):
return False
return (
index.is_monotonic_increasing
and index.is_unique
and index.min() == 0
and index.max() == len(index) - 1
)
================================================
FILE: modin/core/dataframe/base/interchange/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe functionality related to data exchange protocols."""
================================================
FILE: modin/core/dataframe/base/interchange/dataframe_protocol/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Base Modin Dataframe functionality related to the dataframe exchange protocol.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
"""
================================================
FILE: modin/core/dataframe/base/interchange/dataframe_protocol/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Dataframe exchange protocol implementation.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
"""
from abc import ABC, abstractmethod
from typing import Any, Dict, Iterable, Optional, Sequence, Tuple, TypedDict
from .utils import ColumnNullType, DlpackDeviceType, DTypeKind
class ColumnBuffers(TypedDict): # noqa: GL08
# first element is a buffer containing the column data;
# second element is the data buffer's associated dtype
data: Tuple["ProtocolBuffer", Any]
# first element is a buffer containing mask values indicating missing data;
# second element is the mask value buffer's associated dtype.
# None if the null representation is not a bit or byte mask
validity: Optional[Tuple["ProtocolBuffer", Any]]
# first element is a buffer containing the offset values for
# variable-size binary data (e.g., variable-length strings);
# second element is the offsets buffer's associated dtype.
# None if the data buffer does not have an associated offsets buffer
offsets: Optional[Tuple["ProtocolBuffer", Any]]
class CategoricalDescription(TypedDict): # noqa: GL08
# whether the ordering of dictionary indices is semantically meaningful
is_ordered: bool
# whether a column-style mapping of categorical values to other objects exists
is_dictionary: bool
# None if not a column-style categorical.
categories: Optional["ProtocolColumn"]
class ProtocolBuffer(ABC):
"""
Data in the buffer is guaranteed to be contiguous in memory.
Note that there is no dtype attribute present, a buffer can be thought of
as simply a block of memory. However, if the column that the buffer is
attached to has a dtype that's supported by DLPack and ``__dlpack__`` is
implemented, then that dtype information will be contained in the return
value from ``__dlpack__``.
This distinction is useful to support both (a) data exchange via DLPack on a
buffer and (b) dtypes like variable-length strings which do not have a
fixed number of bytes per element.
"""
@property
@abstractmethod
def bufsize(self) -> int:
"""
Buffer size in bytes.
Returns
-------
int
"""
pass
@property
@abstractmethod
def ptr(self) -> int:
"""
Pointer to start of the buffer as an integer.
Returns
-------
int
"""
pass
@abstractmethod
def __dlpack__(self) -> Any:
"""
Produce DLPack capsule (see array API standard).
DLPack not implemented in NumPy yet, so leave it out here.
Raises
------
``TypeError`` if the buffer contains unsupported dtypes.
``NotImplementedError`` if DLPack support is not implemented.
Notes
-----
Useful to have to connect to array libraries. Support optional because
it's not completely trivial to implement for a Python-only library.
"""
pass
@abstractmethod
def __dlpack_device__(self) -> Tuple[DlpackDeviceType, Optional[int]]:
"""
Device type and device ID for where the data in the buffer resides.
Uses device type codes matching DLPack. Enum members are:
- CPU = 1
- CUDA = 2
- CPU_PINNED = 3
- OPENCL = 4
- VULKAN = 7
- METAL = 8
- VPI = 9
- ROCM = 10
Returns
-------
tuple
Device type and device ID.
Notes
-----
Must be implemented even if ``__dlpack__`` is not.
"""
pass
class ProtocolColumn(ABC):
"""
A column object, with only the methods and properties required by the interchange protocol defined.
A column can contain one or more chunks. Each chunk can contain up to three
buffers - a data buffer, a mask buffer (depending on null representation),
and an offsets buffer (if variable-size binary; e.g., variable-length strings).
TBD: Arrow has a separate "null" dtype, and has no separate mask concept.
Instead, it seems to use "children" for both columns with a bit mask,
and for nested dtypes. Unclear whether this is elegant or confusing.
This design requires checking the null representation explicitly.
The Arrow design requires checking:
1. the ARROW_FLAG_NULLABLE (for sentinel values)
2. if a column has two children, combined with one of those children
having a null dtype.
Making the mask concept explicit seems useful. One null dtype would
not be enough to cover both bit and byte masks, so that would mean
even more checking if we did it the Arrow way.
TBD: there's also the "chunk" concept here, which is implicit in Arrow as
multiple buffers per array (= column here). Semantically it may make
sense to have both: chunks were meant for example for lazy evaluation
of data which doesn't fit in memory, while multiple buffers per column
could also come from doing a selection operation on a single
contiguous buffer.
Given these concepts, one would expect chunks to be all of the same
size (say a 10,000 row dataframe could have 10 chunks of 1,000 rows),
while multiple buffers could have data-dependent lengths. Not an issue
in pandas if one column is backed by a single NumPy array, but in
Arrow it seems possible.
Are multiple chunks *and* multiple buffers per column necessary for
the purposes of this interchange protocol, or must producers either
reuse the chunk concept for this or copy the data?
Notes
-----
This ProtocolColumn object can only be produced by ``__dataframe__``,
so doesn't need its own version or ``__column__`` protocol.
"""
@abstractmethod
def size(self) -> int:
"""
Size of the column, in elements.
Corresponds to `DataFrame.num_rows()` if column is a single chunk;
equal to size of this current chunk otherwise.
Is a method rather than a property because it may cause a (potentially
expensive) computation for some dataframe implementations.
Returns
-------
int
Size of the column, in elements.
"""
pass
@property
@abstractmethod
def offset(self) -> int:
"""
Get the offset of first element.
May be > 0 if using chunks; for example for a column
with N chunks of equal size M (only the last chunk may be shorter),
``offset = n * M``, ``n = 0 .. N-1``.
Returns
-------
int
The offset of first element.
"""
pass
@property
@abstractmethod
def dtype(self) -> Tuple[DTypeKind, int, str, str]:
"""
Dtype description as a tuple ``(kind, bit-width, format string, endianness)``.
* Kind : DTypeKind
* Bit-width : the number of bits as an integer
* Format string : data type description format string in Apache Arrow C
Data Interface format.
* Endianness : current only native endianness (``=``) is supported
Returns
-------
tuple
``(kind, bit-width, format string, endianness)``.
Notes
-----
- Kind specifiers are aligned with DLPack where possible
(hence the jump to 20, leave enough room for future extension).
- Masks must be specified as boolean with either bit width 1 (for bit masks)
or 8 (for byte masks).
- Dtype width in bits was preferred over bytes
- Endianness isn't too useful, but included now in case in the future
we need to support non-native endianness
- Went with Apache Arrow format strings over NumPy format strings
because they're more complete from a dataframe perspective
- Format strings are mostly useful for datetime specification, and for categoricals.
- For categoricals, the format string describes the type of the categorical
in the data buffer. In case of a separate encoding of the categorical
(e.g. an integer to string mapping), this can be derived from ``self.describe_categorical``.
- Data types not included: complex, Arrow-style null, binary, decimal,
and nested (list, struct, map, union) dtypes.
"""
pass
@property
@abstractmethod
def describe_categorical(self) -> CategoricalDescription:
"""
If the dtype is categorical, there are two options.
- There are only values in the data buffer.
- There is a separate non-categorical Column encoding categorical values.
TBD: are there any other in-memory representations that are needed?
Returns
-------
dict
Content of returned dict:
- "is_ordered" : bool, whether the ordering of dictionary indices is
semantically meaningful.
- "is_dictionary" : bool, whether a mapping of
categorical values to other objects exists
- "categories" : Column representing the (implicit) mapping of indices to
category values (e.g. an array of cat1, cat2, ...).
None if not a dictionary-style categorical.
Raises
------
``TypeError`` if the dtype is not categorical.
"""
pass
@property
@abstractmethod
def describe_null(self) -> Tuple[ColumnNullType, Any]:
"""
Return the missing value (or "null") representation the column dtype uses.
Return as a tuple ``(kind, value)``.
* Kind: ColumnNullType
* Value : if kind is "sentinel value", the actual value. If kind is a bit
mask or a byte mask, the value (0 or 1) indicating a missing value. None
otherwise.
Returns
-------
tuple
``(kind, value)``.
"""
pass
@property
@abstractmethod
def null_count(self) -> int:
"""
Get number of null elements, if known.
Returns
-------
int
Notes
-----
Arrow uses -1 to indicate "unknown", but None seems cleaner.
"""
pass
@property
@abstractmethod
def metadata(self) -> Dict[str, Any]:
"""
Get the metadata for the column.
See `DataFrame.metadata` for more details.
Returns
-------
dict
"""
pass
@abstractmethod
def num_chunks(self) -> int:
"""
Return the number of chunks the column consists of.
Returns
-------
int
The number of chunks the column consists of.
"""
pass
@abstractmethod
def get_chunks(self, n_chunks: Optional[int] = None) -> Iterable["ProtocolColumn"]:
"""
Return an iterator yielding the chunks.
By default ``n_chunks=None``, yields the chunks that the data is stored as by the producer.
If given, ``n_chunks`` must be a multiple of ``self.num_chunks()``,
meaning the producer must subdivide each chunk before yielding it.
Parameters
----------
n_chunks : int, optional
Number of chunks to yield.
Yields
------
DataFrame
A ``DataFrame`` object(s).
Raises
------
``RuntimeError`` if ``n_chunks`` is not a multiple of ``self.num_chunks()``.
"""
pass
@abstractmethod
def get_buffers(self) -> ColumnBuffers:
"""
Return a dictionary containing the underlying buffers.
Returns
-------
dict
- "data": a two-element tuple whose first element is a buffer
containing the data and whose second element is the data buffer's associated dtype.
- "validity": a two-element tuple whose first element is a buffer
containing mask values indicating missing data and
whose second element is the mask value buffer's
associated dtype. None if the null representation is not a bit or byte mask.
- "offsets": a two-element tuple whose first element is a buffer
containing the offset values for variable-size binary data
(e.g., variable-length strings) and whose second element is the offsets
buffer's associated dtype. None if the data buffer does not have
an associated offsets buffer.
"""
pass
class ProtocolDataframe(ABC):
"""
A data frame class, with only the methods required by the interchange protocol defined.
Instances of this (private) class are returned from
``modin.core.dataframe.base.dataframe.dataframe.ModinDataframe.__dataframe__``
as objects with the methods and attributes defined on this class.
A "data frame" represents an ordered collection of named columns.
A column's "name" must be a unique string. Columns may be accessed by name or by position.
This could be a public data frame class, or an object with the methods and
attributes defined on this ProtocolDataframe class could be returned from the
``__dataframe__`` method of a public data frame class in a library adhering
to the dataframe interchange protocol specification.
"""
version = 0 # version of the protocol
@abstractmethod
def __dataframe__(
self, nan_as_null: bool = False, allow_copy: bool = True
) -> "ProtocolDataframe":
"""
Construct a new dataframe interchange object, potentially changing the parameters.
See more about the protocol in https://data-apis.org/dataframe-protocol/latest/index.html.
Parameters
----------
nan_as_null : bool, default: False
A keyword intended for the consumer to tell the producer
to overwrite null values in the data with ``NaN``.
This currently has no effect; once support for nullable extension
dtypes is added, this value should be propagated to columns.
allow_copy : bool, default: True
A keyword that defines whether or not the library is allowed
to make a copy of the data. For example, copying data would be necessary
if a library supports strided buffers, given that this protocol
specifies contiguous buffers. Currently, if the flag is set to ``False``
and a copy is needed, a ``RuntimeError`` will be raised.
Returns
-------
ProtocolDataframe
"""
pass
@property
@abstractmethod
def metadata(self) -> Dict[str, Any]:
"""
Get the metadata for the data frame, as a dictionary with string keys.
The contents of `metadata` may be anything, they are meant for a library
to store information that it needs to, e.g., roundtrip losslessly or
for two implementations to share data that is not (yet) part of the
interchange protocol specification. For avoiding collisions with other
entries, please add name the keys with the name of the library
followed by a period and the desired name, e.g, ``pandas.indexcol``.
Returns
-------
dict
"""
pass
@abstractmethod
def num_columns(self) -> int:
"""
Return the number of columns in the ProtocolDataframe.
Returns
-------
int
The number of columns in the ProtocolDataframe.
"""
pass
@abstractmethod
def num_rows(self) -> Optional[int]:
"""
Return the number of rows in the ProtocolDataframe, if available.
Returns
-------
int
The number of rows in the ProtocolDataframe.
"""
pass
@abstractmethod
def num_chunks(self) -> int:
"""
Return the number of chunks the ProtocolDataframe consists of.
Returns
-------
int
The number of chunks the ProtocolDataframe consists of.
"""
pass
@abstractmethod
def column_names(self) -> Iterable[str]:
"""
Return an iterator yielding the column names.
Yields
------
str
The name of the column(s).
"""
pass
@abstractmethod
def get_column(self, i: int) -> ProtocolColumn:
"""
Return the column at the indicated position.
Parameters
----------
i : int
Positional index of the column to be returned.
Returns
-------
Column
The column at the indicated position.
"""
pass
@abstractmethod
def get_column_by_name(self, name: str) -> ProtocolColumn:
"""
Return the column whose name is the indicated name.
Parameters
----------
name : str
String label of the column to be returned.
Returns
-------
Column
The column whose name is the indicated name.
"""
pass
@abstractmethod
def get_columns(self) -> Iterable[ProtocolColumn]:
"""
Return an iterator yielding the columns.
Yields
------
Column
The ``Column`` object(s).
"""
pass
@abstractmethod
def select_columns(self, indices: Sequence[int]) -> "ProtocolDataframe":
"""
Create a new ProtocolDataframe by selecting a subset of columns by index.
Parameters
----------
indices : Sequence[int]
Column indices to be selected out of the ProtocolDataframe.
Returns
-------
ProtocolDataframe
A new ProtocolDataframe with selected a subset of columns by index.
"""
pass
@abstractmethod
def select_columns_by_name(self, names: Sequence[str]) -> "ProtocolDataframe":
"""
Create a new ProtocolDataframe by selecting a subset of columns by name.
Parameters
----------
names : Sequence[str]
Column names to be selected out of the ProtocolDataframe.
Returns
-------
ProtocolDataframe
A new ProtocolDataframe with selected a subset of columns by name.
"""
pass
@abstractmethod
def get_chunks(
self, n_chunks: Optional[int] = None
) -> Iterable["ProtocolDataframe"]:
"""
Return an iterator yielding the chunks.
By default `n_chunks=None`, yields the chunks that the data is stored as by the producer.
If given, `n_chunks` must be a multiple of `self.num_chunks()`,
meaning the producer must subdivide each chunk before yielding it.
Parameters
----------
n_chunks : int, optional
Number of chunks to yield.
Yields
------
ProtocolDataframe
A ``ProtocolDataframe`` object(s).
Raises
------
``RuntimeError`` if ``n_chunks`` is not a multiple of ``self.num_chunks()``.
"""
pass
================================================
FILE: modin/core/dataframe/base/interchange/dataframe_protocol/utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Dataframe exchange protocol implementation.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
"""
import enum
import re
from typing import Optional, Union
import numpy as np
import pandas
from pandas.api.types import is_datetime64_dtype
class DTypeKind(enum.IntEnum): # noqa PR01
"""
Integer enum for data types.
Attributes
----------
INT : int
Matches to signed integer data type.
UINT : int
Matches to unsigned integer data type.
FLOAT : int
Matches to floating point data type.
BOOL : int
Matches to boolean data type.
STRING : int
Matches to string data type (UTF-8 encoded).
DATETIME : int
Matches to datetime data type.
CATEGORICAL : int
Matches to categorical data type.
"""
INT = 0
UINT = 1
FLOAT = 2
BOOL = 20
STRING = 21 # UTF-8
DATETIME = 22
CATEGORICAL = 23
class ColumnNullType(enum.IntEnum): # noqa PR01
"""
Integer enum for null type representation.
Attributes
----------
NON_NULLABLE : int
Non-nullable column.
USE_NAN : int
Use explicit float NaN value.
USE_SENTINEL : int
Sentinel value besides NaN.
USE_BITMASK : int
The bit is set/unset representing a null on a certain position.
USE_BYTEMASK : int
The byte is set/unset representing a null on a certain position.
"""
NON_NULLABLE = 0
USE_NAN = 1
USE_SENTINEL = 2
USE_BITMASK = 3
USE_BYTEMASK = 4
class DlpackDeviceType(enum.IntEnum): # noqa PR01
"""Integer enum for device type codes matching DLPack."""
CPU = 1
CUDA = 2
CPU_PINNED = 3
OPENCL = 4
VULKAN = 7
METAL = 8
VPI = 9
ROCM = 10
class ArrowCTypes:
"""
Enum for Apache Arrow C type format strings.
The Arrow C data interface:
https://arrow.apache.org/docs/format/CDataInterface.html#data-type-description-format-strings
"""
NULL = "n"
BOOL = "b"
INT8 = "c"
UINT8 = "C"
INT16 = "s"
UINT16 = "S"
INT32 = "i"
UINT32 = "I"
INT64 = "l"
UINT64 = "L"
FLOAT16 = "e"
FLOAT32 = "f"
FLOAT64 = "g"
STRING = "u" # utf-8
DATE32 = "tdD"
DATE64 = "tdm"
# Resoulution:
# - seconds -> 's'
# - miliseconds -> 'm'
# - microseconds -> 'u'
# - nanoseconds -> 'n'
TIMESTAMP = "ts{resolution}:{tz}"
TIME = "tt{resolution}"
class Endianness:
"""Enum indicating the byte-order of a data-type."""
LITTLE = "<"
BIG = ">"
NATIVE = "="
NA = "|"
def pandas_dtype_to_arrow_c(dtype: Union[np.dtype, pandas.CategoricalDtype]) -> str:
"""
Represent pandas `dtype` as a format string in Apache Arrow C notation.
Parameters
----------
dtype : np.dtype
Datatype of pandas DataFrame to represent.
Returns
-------
str
Format string in Apache Arrow C notation of the given `dtype`.
"""
if isinstance(dtype, pandas.CategoricalDtype):
return ArrowCTypes.INT64
elif dtype == pandas.api.types.pandas_dtype("O"):
return ArrowCTypes.STRING
format_str = getattr(ArrowCTypes, dtype.name.upper(), None)
if format_str is not None:
return format_str
if is_datetime64_dtype(dtype):
# Selecting the first char of resolution string:
# dtype.str -> ' None:
"""
Raise a ``RuntimeError`` mentioning that there's a copy required.
Parameters
----------
copy_reason : str, optional
The reason of making a copy. Should fit to the following format:
'The copy occurred due to {copy_reason}.'.
"""
msg = "Copy required but 'allow_copy=False' is set."
if copy_reason:
msg += f" The copy occurred due to {copy_reason}."
raise RuntimeError(msg)
================================================
FILE: modin/core/dataframe/base/partitioning/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe classes related to its partitioning."""
================================================
FILE: modin/core/dataframe/base/partitioning/axis_partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base class of an axis partition for a Modin Dataframe."""
from abc import ABC, abstractmethod
from typing import Any, Callable, Iterable, Optional, Tuple, Type, Union
from modin.logging import ClassLogger
from modin.logging.config import LogLevel
class BaseDataframeAxisPartition(
ABC, ClassLogger, modin_layer="VIRTUAL-PARTITION", log_level=LogLevel.DEBUG
): # pragma: no cover
"""
An abstract class that represents the parent class for any axis partition class.
This class is intended to simplify the way that operations are performed.
Attributes
----------
_PARTITIONS_METADATA_LEN : int
The number of metadata values that the object of `partition_type` consumes.
"""
@property
@abstractmethod
def list_of_blocks(self) -> list:
"""Get the list of physical partition objects that compose this partition."""
pass
def apply(
self,
func: Callable,
*args: Iterable,
num_splits: Optional[int] = None,
other_axis_partition: Optional["BaseDataframeAxisPartition"] = None,
maintain_partitioning: bool = True,
lengths: Optional[Iterable] = None,
manual_partition: bool = False,
**kwargs: dict,
) -> Any:
"""
Apply a function to this axis partition along full axis.
Parameters
----------
func : callable
The function to apply. This will be preprocessed according to
the corresponding `BaseDataframePartition` objects.
*args : iterable
Positional arguments to pass to `func`.
num_splits : int, default: None
The number of times to split the result object.
other_axis_partition : BaseDataframeAxisPartition, default: None
Another `BaseDataframeAxisPartition` object to be applied
to func. This is for operations that are between two data sets.
maintain_partitioning : bool, default: True
Whether to keep the partitioning in the same
orientation as it was previously or not. This is important because we may be
operating on an individual axis partition and not touching the rest.
In this case, we have to return the partitioning to its previous
orientation (the lengths will remain the same). This is ignored between
two axis partitions.
lengths : iterable, default: None
The list of lengths to shuffle the partition into.
manual_partition : bool, default: False
If True, partition the result with `lengths`.
**kwargs : dict
Additional keywords arguments to be passed in `func`.
Returns
-------
list
A list of `BaseDataframePartition` objects.
Notes
-----
The procedures that invoke this method assume full axis
knowledge. Implement this method accordingly.
You must return a list of `BaseDataframePartition` objects from this method.
"""
pass
# Child classes must have these in order to correctly subclass.
partition_type: Type
_PARTITIONS_METADATA_LEN = 0
def _wrap_partitions(
self, partitions: list, extract_metadata: Optional[bool] = None
) -> list:
"""
Wrap remote partition objects with `BaseDataframePartition` class.
Parameters
----------
partitions : list
List of remotes partition objects to be wrapped with `BaseDataframePartition` class.
extract_metadata : bool, optional
Whether the partitions list contains information about partition's metadata.
If `None` was passed will take the argument's value from the value of `cls._PARTITIONS_METADATA_LEN`.
Returns
-------
list
List of wrapped remote partition objects.
"""
assert self.partition_type is not None
if extract_metadata is None:
# If `_PARTITIONS_METADATA_LEN == 0` then the execution doesn't support metadata
# and thus we should never try extracting it, otherwise assuming that the common
# approach of always passing the metadata is used.
extract_metadata = bool(self._PARTITIONS_METADATA_LEN)
if extract_metadata:
# Here we recieve a 1D array of futures describing partitions and their metadata as:
# [object_id{partition_idx}, metadata{partition_idx}_{metadata_idx}, ...]
# Here's an example of such array:
# [
# object_id1, metadata1_1, metadata1_2, ..., metadata1_PARTITIONS_METADATA_LEN,
# object_id2, metadata2_1, ..., metadata2_PARTITIONS_METADATA_LEN,
# ...
# object_idN, metadataN_1, ..., metadataN_PARTITIONS_METADATA_LEN,
# ]
return [
self.partition_type(*init_args)
for init_args in zip(
# `partition_type` consumes `(object_id, *metadata)`, thus adding `+1`
*[iter(partitions)]
* (1 + self._PARTITIONS_METADATA_LEN)
)
]
else:
return [self.partition_type(object_id) for object_id in partitions]
def force_materialization(
self, get_ip: bool = False
) -> "BaseDataframeAxisPartition":
"""
Materialize axis partitions into a single partition.
Parameters
----------
get_ip : bool, default: False
Whether to get node ip address to a single partition or not.
Returns
-------
BaseDataframeAxisPartition
An axis partition containing only a single materialized partition.
"""
materialized = self.apply(
lambda x: x, num_splits=1, maintain_partitioning=False
)
return type(self)(materialized, get_ip=get_ip) # type: ignore[call-arg]
def unwrap(
self, squeeze: bool = False, get_ip: bool = False
) -> Union[list, Tuple[list, list]]:
"""
Unwrap partitions from this axis partition.
Parameters
----------
squeeze : bool, default: False
Flag used to unwrap only one partition.
get_ip : bool, default: False
Whether to get node ip address to each partition or not.
Returns
-------
list
List of partitions from this axis partition.
Notes
-----
If `get_ip=True`, a tuple of lists of Ray.ObjectRef/Dask.Future to node ip addresses and
unwrapped partitions, respectively, is returned if Ray/Dask is used as an engine
(i.e. [(Ray.ObjectRef/Dask.Future, Ray.ObjectRef/Dask.Future), ...]).
"""
if squeeze and len(self.list_of_blocks) == 1:
if get_ip:
# TODO(https://github.com/modin-project/modin/issues/5176): Stop ignoring the list_of_ips
# check once we know that we're not calling list_of_ips on python axis partitions
return self.list_of_ips[0], self.list_of_blocks[0] # type: ignore[attr-defined]
else:
return self.list_of_blocks[0]
else:
if get_ip:
return list(zip(self.list_of_ips, self.list_of_blocks)) # type: ignore[attr-defined]
else:
return self.list_of_blocks
================================================
FILE: modin/core/dataframe/pandas/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe classes optimized for pandas storage format."""
================================================
FILE: modin/core/dataframe/pandas/dataframe/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe class optimized for pandas storage format."""
================================================
FILE: modin/core/dataframe/pandas/dataframe/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module contains class PandasDataframe.
PandasDataframe is a parent abstract class for any dataframe class
for pandas storage format.
"""
from __future__ import annotations
import datetime
import re
from abc import ABC, abstractmethod
from functools import cached_property
from typing import TYPE_CHECKING, Callable, Dict, Hashable, List, Optional, Union
import numpy as np
import pandas
from pandas._libs.lib import no_default
from pandas.api.types import is_object_dtype
from pandas.core.dtypes.common import is_dtype_equal, is_list_like, is_numeric_dtype
from pandas.core.indexes.api import Index, RangeIndex
from modin.config import (
IsRayCluster,
MinColumnPartitionSize,
MinRowPartitionSize,
NPartitions,
)
from modin.core.dataframe.base.dataframe.dataframe import ModinDataframe
from modin.core.dataframe.base.dataframe.utils import Axis, JoinType, is_trivial_index
from modin.core.dataframe.pandas.dataframe.utils import (
ShuffleSortFunctions,
add_missing_categories_to_groupby,
lazy_metadata_decorator,
)
from modin.core.dataframe.pandas.metadata import (
DtypesDescriptor,
LazyProxyCategoricalDtype,
ModinDtypes,
ModinIndex,
)
from modin.core.storage_formats.pandas.parsers import (
find_common_type_cat as find_common_type,
)
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
from modin.core.storage_formats.pandas.utils import get_length_list
from modin.error_message import ErrorMessage
from modin.logging import ClassLogger
from modin.logging.config import LogLevel
from modin.pandas.indexing import is_range_like
from modin.pandas.utils import (
check_both_not_none,
get_pandas_backend,
is_full_grab_slice,
)
from modin.utils import MODIN_UNNAMED_SERIES_LABEL
if TYPE_CHECKING:
from pandas._typing import npt
from modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (
ProtocolDataframe,
)
from modin.core.dataframe.pandas.partitioning.partition_manager import (
PandasDataframePartitionManager,
)
class PandasDataframe(
ABC, ClassLogger, modin_layer="CORE-DATAFRAME", log_level=LogLevel.DEBUG
):
"""
An abstract class that represents the parent class for any pandas storage format dataframe class.
This class provides interfaces to run operations on dataframe partitions.
Parameters
----------
partitions : np.ndarray
A 2D NumPy array of partitions.
index : sequence or callable, optional
The index for the dataframe. Converted to a ``pandas.Index``.
Is computed from partitions on demand if not specified.
If ``callable() -> (pandas.Index, list of row lengths or None)`` type,
then the calculation will be delayed until `self.index` is called.
columns : sequence, optional
The columns object for the dataframe. Converted to a ``pandas.Index``.
Is computed from partitions on demand if not specified.
row_lengths : list, optional
The length of each partition in the rows. The "height" of
each of the block partitions. Is computed if not provided.
column_widths : list, optional
The width of each partition in the columns. The "width" of
each of the block partitions. Is computed if not provided.
dtypes : pandas.Series or callable, optional
The data types for the dataframe columns.
pandas_backend : {"pyarrow", None}, optional
Backend used by pandas.
"""
_partition_mgr_cls: PandasDataframePartitionManager
_query_compiler_cls = PandasQueryCompiler
# These properties flag whether or not we are deferring the metadata synchronization
_deferred_index: bool = False
_deferred_column: bool = False
_index_cache: ModinIndex = None
_columns_cache: ModinIndex = None
_dtypes: Optional[ModinDtypes] = None
_pandas_backend: Optional[str] = None
@property
def storage_format(self) -> str:
"""
The storage format for this frame's data.
Returns
-------
str
The storage format.
"""
return "Pandas"
@property
@abstractmethod
def engine(self) -> str:
"""
The engine for this frame.
Returns
-------
str
The engine.
"""
pass
@cached_property
def __constructor__(self) -> type[PandasDataframe]:
"""
Create a new instance of this object.
Returns
-------
callable
"""
return type(self)
def __init__(
self,
partitions,
index=None,
columns=None,
row_lengths=None,
column_widths=None,
dtypes: Optional[Union[pandas.Series, ModinDtypes, Callable]] = None,
pandas_backend: Optional[str] = None,
):
self._partitions = partitions
self.set_index_cache(index)
self.set_columns_cache(columns)
self._row_lengths_cache = row_lengths
self._column_widths_cache = column_widths
self._pandas_backend = pandas_backend
if pandas_backend != "pyarrow" or len(partitions) == 0:
# If the backend is pyarrow and there are no partitions, the computed dtype otherwise becomes NaN,
# which means we lost the dtype, so actually set it in that case
self.set_dtypes_cache(dtypes)
else:
# In this case, the type precomputation may be incorrect; we need
# to know the type algebra precisely. Considering the number of operations
# and different combinations of backends, the best solution would be to
# introduce optimizations gradually, with a large number of tests.
self.set_dtypes_cache(None)
self._validate_axes_lengths()
self._filter_empties(compute_metadata=False)
def _validate_axes_lengths(self):
"""Validate that labels are split correctly if split is known."""
if (
self._row_lengths_cache is not None
and self.has_materialized_index
and len(self.index) > 0
):
# An empty frame can have 0 rows but a nonempty index. If the frame
# does have rows, the number of rows must equal the size of the
# index.
num_rows = sum(self._row_lengths_cache)
if num_rows > 0:
ErrorMessage.catch_bugs_and_request_email(
num_rows != len(self.index),
f"Row lengths: {num_rows} != {len(self.index)}",
)
ErrorMessage.catch_bugs_and_request_email(
any(val < 0 for val in self._row_lengths_cache),
f"Row lengths cannot be negative: {self._row_lengths_cache}",
)
if (
self._column_widths_cache is not None
and self.has_materialized_columns
and len(self.columns) > 0
):
# An empty frame can have 0 column but a nonempty column index. If
# the frame does have columns, the number of columns must equal the
# size of the columns.
num_columns = sum(self._column_widths_cache)
if num_columns > 0:
ErrorMessage.catch_bugs_and_request_email(
num_columns != len(self.columns),
f"Column widths: {num_columns} != {len(self.columns)}",
)
ErrorMessage.catch_bugs_and_request_email(
any(val < 0 for val in self._column_widths_cache),
f"Column widths cannot be negative: {self._column_widths_cache}",
)
@property
def num_parts(self) -> int:
"""
Get the total number of partitions for this frame.
Returns
-------
int
"""
return np.prod(self._partitions.shape)
@property
def row_lengths(self):
"""
Compute the row partitions lengths if they are not cached.
Returns
-------
list
A list of row partitions lengths.
"""
if self._row_lengths_cache is None:
if len(self._partitions.T) > 0:
row_parts = self._partitions.T[0]
self._row_lengths_cache = self._get_lengths(row_parts, Axis.ROW_WISE)
else:
self._row_lengths_cache = []
return self._row_lengths_cache
@classmethod
def _get_lengths(cls, parts, axis):
"""
Get list of dimensions for all the provided parts.
Parameters
----------
parts : list
List of parttions.
axis : {0, 1}
The axis along which to get the lengths (0 - length across rows or, 1 - width across columns).
Returns
-------
list
"""
if axis == Axis.ROW_WISE:
return [part.length() for part in parts]
else:
return [part.width() for part in parts]
def __len__(self) -> int:
"""
Return length of index axis.
Returns
-------
int
"""
if self.has_materialized_index:
_len = len(self.index)
else:
_len = sum(self.row_lengths)
return _len
@property
def column_widths(self):
"""
Compute the column partitions widths if they are not cached.
Returns
-------
list
A list of column partitions widths.
"""
if self._column_widths_cache is None:
if len(self._partitions) > 0:
col_parts = self._partitions[0]
self._column_widths_cache = self._get_lengths(col_parts, Axis.COL_WISE)
else:
self._column_widths_cache = []
return self._column_widths_cache
def _set_axis_lengths_cache(self, value, axis=0):
"""
Set the row/column lengths cache for the specified axis.
Parameters
----------
value : list of ints
axis : int, default: 0
0 for row lengths and 1 for column widths.
"""
if axis == 0:
self._row_lengths_cache = value
else:
self._column_widths_cache = value
def _get_axis_lengths_cache(self, axis=0):
"""
Get partition's shape caches along the specified axis if avaliable.
Parameters
----------
axis : int, default: 0
0 - get row lengths cache, 1 - get column widths cache.
Returns
-------
list of ints or None
If the cache is computed return a list of ints, ``None`` otherwise.
"""
return self._row_lengths_cache if axis == 0 else self._column_widths_cache
def _get_axis_lengths(self, axis: int = 0) -> List[int]:
"""
Get row lengths/column widths.
Parameters
----------
axis : int, default: 0
Returns
-------
list of ints
"""
return self.row_lengths if axis == 0 else self.column_widths
@property
def has_dtypes_cache(self) -> bool:
"""
Check if the dtypes cache exists.
Returns
-------
bool
"""
return self._dtypes is not None
@property
def has_materialized_dtypes(self) -> bool:
"""
Check if dataframe has materialized index cache.
Returns
-------
bool
"""
return self.has_dtypes_cache and self._dtypes.is_materialized
def copy_dtypes_cache(self):
"""
Copy the dtypes cache.
Returns
-------
pandas.Series, callable or None
If there is an pandas.Series in the cache, then copying occurs.
"""
dtypes_cache = None
if self.has_dtypes_cache:
dtypes_cache = self._dtypes.copy()
return dtypes_cache
def _maybe_update_proxies(self, dtypes, new_parent=None):
"""
Update lazy proxies stored inside of `dtypes` with a new parent inplace.
Parameters
----------
dtypes : pandas.Series, ModinDtypes or callable
new_parent : object, optional
A new parent to link the proxies to. If not specified
will consider the `self` to be a new parent.
Returns
-------
pandas.Series, ModinDtypes or callable
"""
new_parent = new_parent or self
if isinstance(dtypes, ModinDtypes):
dtypes = dtypes.maybe_specify_new_frame_ref(new_parent)
if isinstance(dtypes, pandas.Series):
LazyProxyCategoricalDtype.update_dtypes(dtypes, new_parent)
return dtypes
def set_dtypes_cache(self, dtypes):
"""
Set dtypes cache.
Parameters
----------
dtypes : pandas.Series, ModinDtypes, callable or None
"""
dtypes = self._maybe_update_proxies(dtypes)
if dtypes is None and self.has_materialized_columns:
# try to set a descriptor instead of 'None' to be more flexible in
# dtypes computing
try:
self._dtypes = ModinDtypes(
DtypesDescriptor(
cols_with_unknown_dtypes=self.columns.tolist(), parent_df=self
)
)
except NotImplementedError:
self._dtypes = None
elif isinstance(dtypes, ModinDtypes) or dtypes is None:
self._dtypes = dtypes
else:
self._dtypes = ModinDtypes(dtypes)
@property
def dtypes(self):
"""
Compute the data types if they are not cached.
Returns
-------
pandas.Series
A pandas Series containing the data types for this dataframe.
"""
if self.has_dtypes_cache:
dtypes = self._dtypes.get()
else:
dtypes = self._compute_dtypes()
self.set_dtypes_cache(dtypes)
# During materialization, we can find out the backend and, if it
# is suitable, use the ability to pre-calculate types.
self._pandas_backend = get_pandas_backend(dtypes)
return dtypes
def get_dtypes_set(self):
"""
Get a set of dtypes that are in this dataframe.
Returns
-------
set
"""
if isinstance(self._dtypes, ModinDtypes):
return self._dtypes.get_dtypes_set()
return set(self.dtypes.values)
def _compute_dtypes(self, columns=None) -> pandas.Series:
"""
Compute the data types via TreeReduce pattern for the specified columns.
Parameters
----------
columns : list-like, optional
Columns to compute dtypes for. If not specified compute dtypes
for all the columns in the dataframe.
Returns
-------
pandas.Series
A pandas Series containing the data types for this dataframe.
"""
def dtype_builder(df):
return df.apply(lambda col: find_common_type(col.values), axis=0)
if columns is not None:
# Sorting positions to request columns in the order they're stored (it's more efficient)
numeric_indices = sorted(self.columns.get_indexer_for(columns))
if any(pos < 0 for pos in numeric_indices):
raise KeyError(
f"Some of the columns are not in index: subset={columns}; columns={self.columns}"
)
obj = self.take_2d_labels_or_positional(
col_labels=self.columns[numeric_indices].tolist()
)
else:
obj = self
# For now we will use a pandas Series for the dtypes.
if len(obj.columns) > 0:
dtypes = (
obj.tree_reduce(0, lambda df: df.dtypes, dtype_builder)
.to_pandas()
.iloc[0]
)
else:
dtypes = pandas.Series([])
# reset name to None because we use MODIN_UNNAMED_SERIES_LABEL internally
dtypes.name = None
return dtypes
def set_index_cache(self, index):
"""
Set index cache.
Parameters
----------
index : sequence, callable or None
"""
if index is None:
self._index_cache = ModinIndex(self, axis=0)
elif isinstance(index, ModinIndex):
# update reference with the new frame to not pollute memory
self._index_cache = index.maybe_specify_new_frame_ref(self, axis=0)
else:
self._index_cache = ModinIndex(index)
def set_columns_cache(self, columns):
"""
Set columns cache.
Parameters
----------
columns : sequence, callable or None
"""
if columns is None:
self._columns_cache = ModinIndex(self, axis=1)
elif isinstance(columns, ModinIndex):
# update reference with the new frame to not pollute memory
self._columns_cache = columns.maybe_specify_new_frame_ref(self, axis=1)
else:
self._columns_cache = ModinIndex(columns)
def set_axis_cache(self, value, axis=0):
"""
Set cache for the specified axis (index or columns).
Parameters
----------
value : sequence, callable or None
axis : int, default: 0
"""
if axis == 0:
self.set_index_cache(value)
else:
self.set_columns_cache(value)
def has_axis_cache(self, axis=0) -> bool:
"""
Check if the cache for the specified axis exists.
Parameters
----------
axis : int, default: 0
Returns
-------
bool
"""
return self.has_index_cache if axis == 0 else self.has_columns_cache
@property
def has_index_cache(self):
"""
Check if the index cache exists.
Returns
-------
bool
"""
return self._index_cache is not None
def copy_index_cache(self, copy_lengths=False):
"""
Copy the index cache.
Parameters
----------
copy_lengths : bool, default: False
Whether to copy the stored partition lengths to the
new index object.
Returns
-------
pandas.Index, callable or ModinIndex
If there is an pandas.Index in the cache, then copying occurs.
"""
idx_cache = self._index_cache
if self.has_index_cache:
idx_cache = self._index_cache.copy(copy_lengths)
return idx_cache
def _get_axis_cache(self, axis=0) -> ModinIndex:
"""
Get axis cache for the specified axis if available.
Parameters
----------
axis : int, default: 0
Returns
-------
ModinIndex
"""
return self._index_cache if axis == 0 else self._columns_cache
@property
def has_columns_cache(self):
"""
Check if the columns cache exists.
Returns
-------
bool
"""
return self._columns_cache is not None
def copy_columns_cache(self, copy_lengths=False):
"""
Copy the columns cache.
Parameters
----------
copy_lengths : bool, default: False
Whether to copy the stored partition lengths to the
new index object.
Returns
-------
pandas.Index or None
If there is an pandas.Index in the cache, then copying occurs.
"""
columns_cache = self._columns_cache
if columns_cache is not None:
columns_cache = columns_cache.copy(copy_lengths)
return columns_cache
def copy_axis_cache(self, axis=0, copy_lengths=False):
"""
Copy the axis cache (index or columns).
Parameters
----------
axis : int, default: 0
copy_lengths : bool, default: False
Whether to copy the stored partition lengths to the
new index object.
Returns
-------
pandas.Index, callable or None
If there is an pandas.Index in the cache, then copying occurs.
"""
if axis == 0:
return self.copy_index_cache(copy_lengths)
else:
return self.copy_columns_cache(copy_lengths)
@property
def has_materialized_index(self):
"""
Check if dataframe has materialized index cache.
Returns
-------
bool
"""
return self.has_index_cache and self._index_cache.is_materialized
@property
def has_materialized_columns(self):
"""
Check if dataframe has materialized columns cache.
Returns
-------
bool
"""
return self.has_columns_cache and self._columns_cache.is_materialized
def _validate_set_axis(self, new_labels, old_labels):
"""
Validate the possibility of replacement of old labels with the new labels.
Parameters
----------
new_labels : list-like
The labels to replace with.
old_labels : list-like
The labels to replace.
Returns
-------
list-like
The validated labels.
"""
new_labels = (
ModinIndex(new_labels)
if not isinstance(new_labels, ModinIndex)
else new_labels
)
old_len = len(old_labels)
new_len = len(new_labels)
if old_len != new_len:
raise ValueError(
f"Length mismatch: Expected axis has {old_len} elements, "
+ f"new values have {new_len} elements"
)
return new_labels
def _get_index(self):
"""
Get the index from the cache object.
Returns
-------
pandas.Index
An index object containing the row labels.
"""
if self.has_index_cache:
index, row_lengths = self._index_cache.get(return_lengths=True)
else:
index, row_lengths = self._compute_axis_labels_and_lengths(0)
self.set_index_cache(index)
if self._row_lengths_cache is None:
self._row_lengths_cache = row_lengths
return index
def _get_columns(self):
"""
Get the columns from the cache object.
Returns
-------
pandas.Index
An index object containing the column labels.
"""
if self.has_columns_cache:
columns, column_widths = self._columns_cache.get(return_lengths=True)
else:
columns, column_widths = self._compute_axis_labels_and_lengths(1)
self.set_columns_cache(columns)
if self._column_widths_cache is None:
self._column_widths_cache = column_widths
return columns
def _set_index(self, new_index):
"""
Replace the current row labels with new labels.
Parameters
----------
new_index : list-like
The new row labels.
"""
if self.has_materialized_index:
new_index = self._validate_set_axis(new_index, self._index_cache)
self.set_index_cache(new_index)
self.synchronize_labels(axis=0)
def _set_columns(self, new_columns):
"""
Replace the current column labels with new labels.
Parameters
----------
new_columns : list-like
The new column labels.
"""
if self.has_materialized_columns:
# do not set new columns if they're identical to the previous ones
if (
isinstance(new_columns, pandas.Index)
and self.columns.identical(new_columns)
) or (
not isinstance(new_columns, pandas.Index)
and np.array_equal(self.columns.values, new_columns)
):
return
new_columns = self._validate_set_axis(new_columns, self._columns_cache)
if isinstance(self._dtypes, ModinDtypes):
try:
new_dtypes = self._dtypes.set_index(new_columns)
except NotImplementedError:
# can raise on duplicated labels
new_dtypes = None
elif isinstance(self._dtypes, pandas.Series):
new_dtypes = self.dtypes.set_axis(new_columns)
else:
new_dtypes = None
self.set_columns_cache(new_columns)
# we have to set new dtypes cache after columns,
# so the 'self.columns' and 'new_dtypes.index' indices would match
self.set_dtypes_cache(new_dtypes)
self.synchronize_labels(axis=1)
columns = property(_get_columns, _set_columns)
index = property(_get_index, _set_index)
@property
def axes(self):
"""
Get index and columns that can be accessed with an `axis` integer.
Returns
-------
list
List with two values: index and columns.
"""
return [self.index, self.columns]
def get_axis(self, axis: int = 0) -> pandas.Index:
"""
Get index object for the requested axis.
Parameters
----------
axis : {0, 1}, default: 0
Returns
-------
pandas.Index
"""
return self.index if axis == 0 else self.columns
def _compute_axis_labels_and_lengths(self, axis: int, partitions=None):
"""
Compute the labels for specific `axis`.
Parameters
----------
axis : int
Axis to compute labels along.
partitions : np.ndarray, optional
A 2D NumPy array of partitions from which labels will be grabbed.
If not specified, partitions will be taken from `self._partitions`.
Returns
-------
pandas.Index
Labels for the specified `axis`.
List of int
Size of partitions alongside specified `axis`.
"""
if partitions is None:
partitions = self._partitions
new_index, internal_idx = self._partition_mgr_cls.get_indices(axis, partitions)
return new_index, list(map(len, internal_idx))
def _filter_empties(self, compute_metadata=True):
"""
Remove empty partitions from `self._partitions` to avoid triggering excess computation.
Parameters
----------
compute_metadata : bool, default: True
Trigger the computations for partition sizes and labels if they're not done already.
"""
if not compute_metadata and (
self._row_lengths_cache is None or self._column_widths_cache is None
):
# do not trigger the computations
return
if (
self.has_materialized_index
and len(self.index) == 0
or self.has_materialized_columns
and len(self.columns) == 0
or sum(self.row_lengths) == 0
or sum(self.column_widths) == 0
):
# This is the case for an empty frame. We don't want to completely remove
# all metadata and partitions so for the moment, we won't prune if the frame
# is empty.
# TODO: Handle empty dataframes better
return
self._partitions = np.array(
[
[
self._partitions[i][j]
for j in range(len(self._partitions[i]))
if j < len(self.column_widths) and self.column_widths[j] != 0
]
for i in range(len(self._partitions))
if i < len(self.row_lengths) and self.row_lengths[i] != 0
]
)
new_col_widths = [w for w in self.column_widths if w != 0]
new_row_lengths = [r for r in self.row_lengths if r != 0]
# check whether an axis partitioning was modified and if we should reset the lengths id for 'ModinIndex'
if new_col_widths != self.column_widths:
self.set_columns_cache(self.copy_columns_cache(copy_lengths=False))
if new_row_lengths != self.row_lengths:
self.set_index_cache(self.copy_index_cache(copy_lengths=False))
self._column_widths_cache = new_col_widths
self._row_lengths_cache = new_row_lengths
def synchronize_labels(self, axis=None):
"""
Set the deferred axes variables for the ``PandasDataframe``.
Parameters
----------
axis : int, optional
The deferred axis.
0 for the index, 1 for the columns.
"""
if axis is None:
self._deferred_index = True
self._deferred_column = True
elif axis == 0:
self._deferred_index = True
else:
self._deferred_column = True
def _propagate_index_objs(self, axis=None) -> None:
"""
Synchronize labels by applying the index object for specific `axis` to the `self._partitions` lazily.
Adds `set_axis` function to call-queue of each partition from `self._partitions`
to apply new axis.
Parameters
----------
axis : int, optional
The axis to apply to. If it's None applies to both axes.
"""
self._filter_empties(compute_metadata=False)
if axis is None or axis == 0:
cum_row_lengths = np.cumsum([0] + self.row_lengths)
if axis is None or axis == 1:
cum_col_widths = np.cumsum([0] + self.column_widths)
if axis is None:
def apply_idx_objs(df, idx, cols):
# We should make at least one copy to avoid the data modification problem
# that may arise when sharing buffers from distributed storage
# (zero-copy pickling).
return df.set_axis(idx, axis="index").set_axis(
cols, axis="columns", copy=False
)
self._partitions = np.array(
[
[
self._partitions[i][j].add_to_apply_calls(
apply_idx_objs,
idx=self.index[
slice(cum_row_lengths[i], cum_row_lengths[i + 1])
],
cols=self.columns[
slice(cum_col_widths[j], cum_col_widths[j + 1])
],
length=self.row_lengths[i],
width=self.column_widths[j],
)
for j in range(len(self._partitions[i]))
]
for i in range(len(self._partitions))
]
)
self._deferred_index = False
self._deferred_column = False
elif axis == 0:
def apply_idx_objs(df, idx):
return df.set_axis(idx, axis="index")
self._partitions = np.array(
[
[
self._partitions[i][j].add_to_apply_calls(
apply_idx_objs,
idx=self.index[
slice(cum_row_lengths[i], cum_row_lengths[i + 1])
],
length=self.row_lengths[i],
width=(
self.column_widths[j]
if self._column_widths_cache is not None
else None
),
)
for j in range(len(self._partitions[i]))
]
for i in range(len(self._partitions))
]
)
self._deferred_index = False
elif axis == 1:
def apply_idx_objs(df, cols):
return df.set_axis(cols, axis="columns")
self._partitions = np.array(
[
[
self._partitions[i][j].add_to_apply_calls(
apply_idx_objs,
cols=self.columns[
slice(cum_col_widths[j], cum_col_widths[j + 1])
],
length=(
self.row_lengths[i]
if self._row_lengths_cache is not None
else None
),
width=self.column_widths[j],
)
for j in range(len(self._partitions[i]))
]
for i in range(len(self._partitions))
]
)
self._deferred_column = False
else:
ErrorMessage.catch_bugs_and_request_email(
axis is not None and axis not in [0, 1]
)
@lazy_metadata_decorator(apply_axis=None)
def take_2d_labels_or_positional(
self,
row_labels: Optional[List[Hashable]] = None,
row_positions: Optional[List[int]] = None,
col_labels: Optional[List[Hashable]] = None,
col_positions: Optional[List[int]] = None,
) -> PandasDataframe:
"""
Lazily select columns or rows from given indices.
Parameters
----------
row_labels : list of hashable, optional
The row labels to extract.
row_positions : list-like of ints, optional
The row positions to extract.
col_labels : list of hashable, optional
The column labels to extract.
col_positions : list-like of ints, optional
The column positions to extract.
Returns
-------
PandasDataframe
A new PandasDataframe from the mask provided.
Notes
-----
If both `row_labels` and `row_positions` are provided, a ValueError is raised.
The same rule applies for `col_labels` and `col_positions`.
"""
if check_both_not_none(row_labels, row_positions):
raise ValueError(
"Both row_labels and row_positions were provided - "
+ "please provide only one of row_labels and row_positions."
)
if check_both_not_none(col_labels, col_positions):
raise ValueError(
"Both col_labels and col_positions were provided - "
+ "please provide only one of col_labels and col_positions."
)
if row_labels is not None:
# Get numpy array of positions of values from `row_labels`
if isinstance(self.index, pandas.MultiIndex):
row_positions = np.zeros(len(row_labels), dtype="int64")
# we can't use .get_locs(row_labels) because the function
# requires a different format for row_labels
for idx, label in enumerate(row_labels):
if isinstance(label, str):
label = [label]
# get_loc can return slice that _take_2d_positional can't handle
row_positions[idx] = self.index.get_locs(label)[0]
else:
row_positions = self.index.get_indexer_for(row_labels)
if col_labels is not None:
# Get numpy array of positions of values from `col_labels`
if isinstance(self.columns, pandas.MultiIndex):
col_positions = np.zeros(len(col_labels), dtype="int64")
# we can't use .get_locs(col_labels) because the function
# requires a different format for row_labels
for idx, label in enumerate(col_labels):
if isinstance(label, str):
label = [label]
# get_loc can return slice that _take_2d_positional can't handle
col_positions[idx] = self.columns.get_locs(label)[0]
else:
col_positions = self.columns.get_indexer_for(col_labels)
return self._take_2d_positional(row_positions, col_positions)
def _get_sorted_positions(self, positions):
"""
Sort positions if necessary.
Parameters
----------
positions : Sequence[int]
Returns
-------
Sequence[int]
"""
# Helper for take_2d_positional
if is_range_like(positions) and positions.step > 0:
sorted_positions = positions
else:
sorted_positions = np.sort(positions)
return sorted_positions
def _get_new_lengths(self, partitions_dict, *, axis: int) -> List[int]:
"""
Find lengths of new partitions.
Parameters
----------
partitions_dict : dict
axis : int
Returns
-------
list[int]
"""
# Helper for take_2d_positional
if axis == 0:
axis_lengths = self.row_lengths
else:
axis_lengths = self.column_widths
new_lengths = [
len(
# Row lengths for slice are calculated as the length of the slice
# on the partition. Often this will be the same length as the current
# length, but sometimes it is different, thus the extra calculation.
range(*part_indexer.indices(axis_lengths[part_idx]))
if isinstance(part_indexer, slice)
else part_indexer
)
for part_idx, part_indexer in partitions_dict.items()
]
return new_lengths
def _get_new_index_obj(
self, positions, sorted_positions, axis: int
) -> tuple[pandas.Index, slice | npt.NDArray[np.intp]]:
"""
Find the new Index object for take_2d_positional result.
Parameters
----------
positions : Sequence[int]
sorted_positions : Sequence[int]
axis : int
Returns
-------
pandas.Index
slice or Sequence[int]
"""
# Helper for take_2d_positional
# Use the slice to calculate the new columns
if axis == 0:
idx = self.index
else:
idx = self.columns
# TODO: Support fast processing of negative-step ranges
if is_range_like(positions) and positions.step > 0:
# pandas Index is more likely to preserve its metadata if the indexer
# is slice
monotonic_idx = slice(positions.start, positions.stop, positions.step)
else:
monotonic_idx = np.asarray(sorted_positions, dtype=np.intp)
new_idx = idx[monotonic_idx]
return new_idx, monotonic_idx
def _take_2d_positional(
self,
row_positions: Optional[List[int]] = None,
col_positions: Optional[List[int]] = None,
) -> PandasDataframe:
"""
Lazily select columns or rows from given indices.
Parameters
----------
row_positions : list-like of ints, optional
The row positions to extract.
col_positions : list-like of ints, optional
The column positions to extract.
Returns
-------
PandasDataframe
A new PandasDataframe from the mask provided.
"""
indexers = []
for axis, indexer in enumerate((row_positions, col_positions)):
if is_range_like(indexer):
if indexer.step == 1 and len(indexer) == len(self.get_axis(axis)):
# By this function semantics, `None` indexer is a full-axis access
indexer = None
elif indexer is not None and not isinstance(indexer, pandas.RangeIndex):
# Pure python's range is not fully compatible with a list of ints,
# converting it to ``pandas.RangeIndex``` that is compatible.
indexer = pandas.RangeIndex(
indexer.start, indexer.stop, indexer.step
)
else:
ErrorMessage.catch_bugs_and_request_email(
failure_condition=not (indexer is None or is_list_like(indexer)),
extra_log="Mask takes only list-like numeric indexers, "
+ f"received: {type(indexer)}",
)
if isinstance(indexer, list):
indexer = np.array(indexer, dtype=np.int64)
indexers.append(indexer)
row_positions, col_positions = indexers
if col_positions is None and row_positions is None:
return self.copy()
# quite fast check that allows skip sorting
must_sort_row_pos = row_positions is not None and not np.all(
row_positions[1:] >= row_positions[:-1]
)
must_sort_col_pos = col_positions is not None and not np.all(
col_positions[1:] >= col_positions[:-1]
)
if col_positions is None and row_positions is not None:
# Check if the optimization that first takes part of the data using the mask
# operation so that later less data is concatenated into a whole column is useful.
# In the case when only a small portion of the data is discarded, the overhead of the
# engine (for putting data in and out of storage) can exceed the resulting speedup.
all_rows = None
if self.has_materialized_index:
all_rows = len(self.index)
elif self._row_lengths_cache or must_sort_row_pos:
all_rows = sum(self.row_lengths)
# 'base_num_cols' specifies the number of columns that the dataframe should have
# in order to jump to 'reordered_labels' in case of len(row_positions) / len(self) >= base_ratio;
# these variables may be a subject to change in order to tune performance more accurately
base_num_cols = 10
base_ratio = 0.2
# Example:
# len(self.columns): 10 == base_num_cols -> min ratio to jump to reorder_labels: 0.2 == base_ratio
# len(self.columns): 15 -> min ratio to jump to reorder_labels: 0.3
# len(self.columns): 20 -> min ratio to jump to reorder_labels: 0.4
# ...
# len(self.columns): 49 -> min ratio to jump to reorder_labels: 0.98
# len(self.columns): 50 -> min ratio to jump to reorder_labels: 1.0
# len(self.columns): 55 -> min ratio to jump to reorder_labels: 1.0
# ...
if (all_rows and len(row_positions) > 0.9 * all_rows) or (
must_sort_row_pos
and len(row_positions) * base_num_cols
>= min(
all_rows * len(self.columns) * base_ratio,
len(row_positions) * base_num_cols,
)
):
return self._reorder_labels(
row_positions=row_positions, col_positions=col_positions
)
sorted_row_positions = sorted_col_positions = None
if row_positions is not None:
if must_sort_row_pos:
sorted_row_positions = self._get_sorted_positions(row_positions)
else:
sorted_row_positions = row_positions
# Get dict of row_parts as {row_index: row_internal_indices}
row_partitions_dict = self._get_dict_of_block_index(
0, sorted_row_positions, are_indices_sorted=True
)
new_row_lengths = self._get_new_lengths(row_partitions_dict, axis=0)
new_index, _ = self._get_new_index_obj(
row_positions, sorted_row_positions, axis=0
)
else:
row_partitions_dict = {i: slice(None) for i in range(len(self._partitions))}
new_row_lengths = self._row_lengths_cache
new_index = self.copy_index_cache(copy_lengths=True)
if col_positions is not None:
if must_sort_col_pos:
sorted_col_positions = self._get_sorted_positions(col_positions)
else:
sorted_col_positions = col_positions
# Get dict of col_parts as {col_index: col_internal_indices}
col_partitions_dict = self._get_dict_of_block_index(
1, sorted_col_positions, are_indices_sorted=True
)
new_col_widths = self._get_new_lengths(col_partitions_dict, axis=1)
new_columns, monotonic_col_idx = self._get_new_index_obj(
col_positions, sorted_col_positions, axis=1
)
ErrorMessage.catch_bugs_and_request_email(
failure_condition=sum(new_col_widths) != len(new_columns),
extra_log=f"{sum(new_col_widths)} != {len(new_columns)}.\n"
+ f"{col_positions}\n{self.column_widths}\n{col_partitions_dict}",
)
if self.has_materialized_dtypes:
new_dtypes = self.dtypes.iloc[monotonic_col_idx]
elif isinstance(self._dtypes, ModinDtypes):
try:
supported_monotonic_col_idx = monotonic_col_idx
if isinstance(monotonic_col_idx, slice):
supported_monotonic_col_idx = pandas.RangeIndex(
monotonic_col_idx.start,
monotonic_col_idx.stop,
monotonic_col_idx.step,
).to_list()
new_dtypes = self._dtypes.lazy_get(
supported_monotonic_col_idx, numeric_index=True
)
# can raise either on missing cache or on duplicated labels
except (ValueError, NotImplementedError):
new_dtypes = None
else:
new_dtypes = None
else:
col_partitions_dict = {
i: slice(None) for i in range(len(self._partitions.T))
}
new_col_widths = self._column_widths_cache
new_columns = self.copy_columns_cache(copy_lengths=True)
new_dtypes = self.copy_dtypes_cache()
new_partitions = np.array(
[
[
self._partitions[row_idx][col_idx].mask(
row_internal_indices, col_internal_indices
)
for col_idx, col_internal_indices in col_partitions_dict.items()
]
for row_idx, row_internal_indices in row_partitions_dict.items()
]
)
intermediate = self.__constructor__(
new_partitions,
new_index,
new_columns,
new_row_lengths,
new_col_widths,
new_dtypes,
pandas_backend=self._pandas_backend,
)
return self._maybe_reorder_labels(
intermediate,
row_positions,
col_positions,
)
def _maybe_reorder_labels(
self,
intermediate: PandasDataframe,
row_positions,
col_positions,
) -> PandasDataframe:
"""
Call re-order labels on take_2d_labels_or_positional result if necessary.
Parameters
----------
intermediate : PandasDataFrame
row_positions : list-like of ints, optional
The row positions to extract.
col_positions : list-like of ints, optional
The column positions to extract.
Returns
-------
PandasDataframe
"""
# Check if monotonically increasing, return if it is. Fast track code path for
# common case to keep it fast.
if (
row_positions is None
# Fast range processing of non-positive-step ranges is not yet supported
or (is_range_like(row_positions) and row_positions.step > 0)
or len(row_positions) == 1
or np.all(row_positions[1:] >= row_positions[:-1])
) and (
col_positions is None
# Fast range processing of non-positive-step ranges is not yet supported
or (is_range_like(col_positions) and col_positions.step > 0)
or len(col_positions) == 1
or np.all(col_positions[1:] >= col_positions[:-1])
):
return intermediate
# The new labels are often smaller than the old labels, so we can't reuse the
# original order values because those were mapped to the original data. We have
# to reorder here based on the expected order from within the data.
# To do so, we "unsort" the indices by using np.argsort() twice, as inspired by
# https://stackoverflow.com/questions/2483696/undo-or-reverse-argsort-python,
# meaning that `new_row_order` must be so `np.sort(row_positions)[new_row_order] == row_positions`
# This is achieved by first calculating the indices which would sort `row_positions`,
# and then by calculating new indices that would sort "sorting indices" themselves.
# First argsort brings us to the proper "index space" (according to smaller labels count),
# and the second re-orders them to match the original data.
new_row_order, new_col_order = None, None
if is_range_like(row_positions):
if row_positions.step < 0:
# do not need to re-order positive-step-ranges
new_row_order = pandas.RangeIndex(len(row_positions) - 1, -1, -1)
elif row_positions is not None:
new_row_order = np.argsort(
np.argsort(np.asarray(row_positions, dtype=np.intp))
)
if is_range_like(col_positions):
if col_positions.step < 0:
new_col_order = pandas.RangeIndex(len(col_positions) - 1, -1, -1)
elif col_positions is not None:
new_col_order = np.argsort(
np.argsort(np.asarray(col_positions, dtype=np.intp))
)
return intermediate._reorder_labels(
row_positions=new_row_order, col_positions=new_col_order
)
@lazy_metadata_decorator(apply_axis="rows")
def from_labels(self) -> PandasDataframe:
"""
Convert the row labels to a column of data, inserted at the first position.
Gives result by similar way as `pandas.DataFrame.reset_index`. Each level
of `self.index` will be added as separate column of data.
Returns
-------
PandasDataframe
A PandasDataframe with new columns from index labels.
"""
new_row_labels = pandas.RangeIndex(len(self.index))
if self.index.nlevels > 1:
level_names = [
(
self.index.names[i]
if self.index.names[i] is not None
else "level_{}".format(i)
)
for i in range(self.index.nlevels)
]
else:
level_names = [
(
self.index.names[0]
if self.index.names[0] is not None
else (
"index" if "index" not in self.columns else "level_{}".format(0)
)
)
]
names = tuple(level_names) if len(level_names) > 1 else level_names[0]
new_dtypes = self.index.to_frame(name=names).dtypes
try:
new_dtypes = ModinDtypes.concat([new_dtypes, self._dtypes])
except NotImplementedError:
# can raise on duplicated labels
new_dtypes = None
# We will also use the `new_column_names` in the calculation of the internal metadata, so this is a
# lightweight way of ensuring the metadata matches.
if self.columns.nlevels > 1:
# Column labels are different for multilevel index.
new_column_names = pandas.MultiIndex.from_tuples(
# Set level names on the 1st columns level and fill up empty level names with empty string.
# Expand tuples in level names. This is how reset_index works when col_level col_fill are not specified.
[
tuple(
list(level) + [""] * (self.columns.nlevels - len(level))
if isinstance(level, tuple)
else [level] + [""] * (self.columns.nlevels - 1)
)
for level in level_names
],
names=self.columns.names,
)
else:
new_column_names = pandas.Index(level_names, tupleize_cols=False)
new_columns = new_column_names.append(self.columns)
def from_labels_executor(
df: pandas.DataFrame, **kwargs
) -> pandas.DataFrame: # pragma: no cover
# Setting the names here ensures that external and internal metadata always match.
df.index.names = new_column_names
# Handling of a case when columns have the same name as one of index levels names.
# In this case `df.reset_index` provides errors related to columns duplication.
# This case is possible because columns metadata updating is deferred. To workaround
# `df.reset_index` error we allow columns duplication in "if" branch via `concat`.
if any(name_level in df.columns for name_level in df.index.names):
columns_to_add = df.index.to_frame()
columns_to_add.reset_index(drop=True, inplace=True)
df = df.reset_index(drop=True)
result = pandas.concat([columns_to_add, df], axis=1, copy=False)
else:
result = df.reset_index()
# Put the index back to the original due to GH#4394
result.index = df.index
return result
new_parts = self._partition_mgr_cls.apply_func_to_select_indices(
0,
self._partitions,
from_labels_executor,
[0],
keep_remaining=True,
)
new_column_widths = [
self.index.nlevels + self.column_widths[0]
] + self.column_widths[1:]
result = self.__constructor__(
new_parts,
new_row_labels,
new_columns,
row_lengths=self._row_lengths_cache,
column_widths=new_column_widths,
dtypes=new_dtypes,
pandas_backend=self._pandas_backend,
)
# Set flag for propagating deferred row labels across dataframe partitions
result.synchronize_labels(axis=0)
return result
def to_labels(self, column_list: List[Hashable]) -> PandasDataframe:
"""
Move one or more columns into the row labels. Previous labels are dropped.
Parameters
----------
column_list : list of hashable
The list of column names to place as the new row labels.
Returns
-------
PandasDataframe
A new PandasDataframe that has the updated labels.
"""
extracted_columns = self.take_2d_labels_or_positional(
col_labels=column_list
).to_pandas()
if len(column_list) == 1:
new_labels = pandas.Index(
extracted_columns.squeeze(axis=1), name=column_list[0]
)
else:
new_labels = pandas.MultiIndex.from_frame(
extracted_columns, names=column_list
)
result = self.take_2d_labels_or_positional(
col_labels=[i for i in self.columns if i not in extracted_columns.columns]
)
result.index = new_labels
return result
@lazy_metadata_decorator(apply_axis="both")
def _reorder_labels(self, row_positions=None, col_positions=None):
"""
Reorder the column and or rows in this DataFrame.
Parameters
----------
row_positions : list of int, optional
The ordered list of new row orders such that each position within the list
indicates the new position.
col_positions : list of int, optional
The ordered list of new column orders such that each position within the
list indicates the new position.
Returns
-------
PandasDataframe
A new PandasDataframe with reordered columns and/or rows.
"""
new_dtypes = self.copy_dtypes_cache()
if row_positions is not None:
# We want to preserve the frame's partitioning so passing in ``keep_partitioning=True``
# in order to use the cached `row_lengths` values for the new frame.
# If the frame's is re-partitioned using the "standard" partitioning,
# then knowing that, we can compute new row lengths.
ordered_rows = self._partition_mgr_cls.map_axis_partitions(
0,
self._partitions,
lambda df: df.iloc[row_positions],
keep_partitioning=True,
)
row_idx = self.index[row_positions]
if len(row_idx) != len(self.index):
# The frame was re-partitioned along the 0 axis during reordering using
# the "standard" partitioning. Knowing the standard partitioning scheme
# we are able to compute new row lengths.
new_lengths = get_length_list(
axis_len=len(row_idx),
num_splits=ordered_rows.shape[0],
min_block_size=MinRowPartitionSize.get(),
)
else:
# If the frame's partitioning was preserved then
# we can use previous row lengths cache
new_lengths = self._row_lengths_cache
else:
ordered_rows = self._partitions
row_idx = self.copy_index_cache(copy_lengths=True)
new_lengths = self._row_lengths_cache
if col_positions is not None:
# We want to preserve the frame's partitioning so passing in ``keep_partitioning=True``
# in order to use the cached `column_widths` values for the new frame.
# If the frame's is re-partitioned using the "standard" partitioning,
# then knowing that, we can compute new column widths.
ordered_cols = self._partition_mgr_cls.map_axis_partitions(
1,
ordered_rows,
lambda df: df.iloc[:, col_positions],
keep_partitioning=True,
)
col_idx = self.columns[col_positions]
if self.has_materialized_dtypes:
new_dtypes = self.dtypes.iloc[col_positions]
elif isinstance(self._dtypes, ModinDtypes):
try:
new_dtypes = self._dtypes.lazy_get(col_idx)
# can raise on duplicated labels
except NotImplementedError:
new_dtypes = None
if len(col_idx) != len(self.columns):
# The frame was re-partitioned along the 1 axis during reordering using
# the "standard" partitioning. Knowing the standard partitioning scheme
# we are able to compute new column widths.
new_widths = get_length_list(
axis_len=len(col_idx),
num_splits=ordered_cols.shape[1],
min_block_size=MinColumnPartitionSize.get(),
)
else:
# If the frame's partitioning was preserved then
# we can use previous column widths cache
new_widths = self._column_widths_cache
else:
ordered_cols = ordered_rows
col_idx = self.copy_columns_cache(copy_lengths=True)
new_widths = self._column_widths_cache
return self.__constructor__(
ordered_cols,
row_idx,
col_idx,
new_lengths,
new_widths,
new_dtypes,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis=None)
def copy(self):
"""
Copy this object.
Returns
-------
PandasDataframe
A copied version of this object.
"""
return self.__constructor__(
self._partitions,
self.copy_index_cache(copy_lengths=True),
self.copy_columns_cache(copy_lengths=True),
self._row_lengths_cache,
self._column_widths_cache,
self.copy_dtypes_cache(),
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def astype(self, col_dtypes, errors: str = "raise"):
"""
Convert the columns dtypes to given dtypes.
Parameters
----------
col_dtypes : dictionary of {col: dtype,...} or str
Where col is the column name and dtype is a NumPy dtype.
errors : {'raise', 'ignore'}, default: 'raise'
Control raising of exceptions on invalid data for provided dtype.
Returns
-------
BaseDataFrame
Dataframe with updated dtypes.
"""
new_dtypes = None
self_dtypes = self.dtypes
# When casting to "category" we have to make up the whole axis partition
# to get the properly encoded table of categories. Every block partition
# will store the encoded table. That can lead to higher memory footprint.
# TODO: Revisit if this hurts users.
use_full_axis_cast = False
if isinstance(col_dtypes, dict):
for column, dtype in col_dtypes.items():
if not is_dtype_equal(dtype, self_dtypes[column]):
if new_dtypes is None:
new_dtypes = self_dtypes.copy()
# Update the new dtype series to the proper pandas dtype
new_dtype = pandas.api.types.pandas_dtype(dtype)
if self.engine == "Dask" and hasattr(dtype, "_is_materialized"):
# FIXME: https://github.com/dask/distributed/issues/8585
_ = dtype._materialize_categories()
# We cannot infer without computing the dtype if new dtype is categorical
if isinstance(new_dtype, pandas.CategoricalDtype):
new_dtypes[column] = LazyProxyCategoricalDtype._build_proxy(
# Actual parent will substitute `None` at `.set_dtypes_cache`
parent=None,
column_name=column,
materializer=lambda parent, column: parent._compute_dtypes(
columns=[column]
)[column],
)
use_full_axis_cast = True
else:
new_dtypes[column] = new_dtype
def astype_builder(df):
"""Compute new partition frame with dtypes updated."""
return df.astype(
{k: v for k, v in col_dtypes.items() if k in df}, errors=errors
)
else:
# Assume that the dtype is a scalar.
if not (self_dtypes == col_dtypes).all():
new_dtypes = self_dtypes.copy()
new_dtype = pandas.api.types.pandas_dtype(col_dtypes)
if self.engine == "Dask" and hasattr(new_dtype, "_is_materialized"):
# FIXME: https://github.com/dask/distributed/issues/8585
_ = new_dtype._materialize_categories()
if isinstance(new_dtype, pandas.CategoricalDtype):
new_dtypes[:] = new_dtypes.to_frame().apply(
lambda column: LazyProxyCategoricalDtype._build_proxy(
# Actual parent will substitute `None` at `.set_dtypes_cache`
parent=None,
column_name=column.index[0],
materializer=lambda parent, column: parent._compute_dtypes(
columns=[column]
)[column],
)
)[0]
use_full_axis_cast = True
else:
new_dtypes[:] = new_dtype
def astype_builder(df):
"""Compute new partition frame with dtypes updated."""
return df.astype(col_dtypes, errors=errors)
if new_dtypes is None:
return self.copy()
if use_full_axis_cast:
new_frame = self._partition_mgr_cls.map_axis_partitions(
0, self._partitions, astype_builder, keep_partitioning=True
)
else:
new_frame = self._partition_mgr_cls.lazy_map_partitions(
self._partitions, astype_builder
)
return self.__constructor__(
new_frame,
self.copy_index_cache(copy_lengths=True),
self.copy_columns_cache(copy_lengths=True),
self._row_lengths_cache,
self._column_widths_cache,
new_dtypes,
pandas_backend=get_pandas_backend(new_dtypes),
)
def numeric_columns(self, include_bool=True):
"""
Return the names of numeric columns in the frame.
Parameters
----------
include_bool : bool, default: True
Whether to consider boolean columns as numeric.
Returns
-------
list
List of column names.
"""
columns = []
for col, dtype in zip(self.columns, self.dtypes):
if is_numeric_dtype(dtype) and (
include_bool or (not include_bool and dtype != np.bool_)
):
columns.append(col)
return columns
def _get_dict_of_block_index(self, axis, indices, are_indices_sorted=False):
"""
Convert indices to an ordered dict mapping partition (or block) index to internal indices in said partition.
Parameters
----------
axis : {0, 1}
The axis along which to get the indices (0 - rows, 1 - columns).
indices : list of int, slice
A list of global indices to convert.
are_indices_sorted : bool, default: False
Flag indicating whether the `indices` sequence is sorted by ascending or not.
Note: the internal algorithm requires for the `indices` to be sorted, this
flag is used for optimization in order to not sort already sorted data.
Be careful when passing ``True`` for this flag, if the data appears to be unsorted
with the flag set to ``True`` this would lead to undefined behavior.
Returns
-------
dict
A mapping from partition index to list of internal indices which correspond to `indices` in each
partition.
"""
# TODO: Support handling of slices with specified 'step'. For now, converting them into a range
if isinstance(indices, slice) and (
indices.step is not None and indices.step != 1
):
indices = range(*indices.indices(len(self.get_axis(axis))))
# Fasttrack slices
if isinstance(indices, slice) or (is_range_like(indices) and indices.step == 1):
# Converting range-like indexer to slice
indices = slice(indices.start, indices.stop, indices.step)
if is_full_grab_slice(indices, sequence_len=len(self.get_axis(axis))):
return dict(
zip(
range(self._partitions.shape[axis]),
[slice(None)] * self._partitions.shape[axis],
)
)
# Empty selection case
if indices.start == indices.stop and indices.start is not None:
return dict()
if indices.start is None or indices.start == 0:
last_part, last_idx = list(
self._get_dict_of_block_index(axis, [indices.stop]).items()
)[0]
dict_of_slices = dict(zip(range(last_part), [slice(None)] * last_part))
dict_of_slices.update({last_part: slice(last_idx[0])})
return dict_of_slices
elif indices.stop is None or indices.stop >= len(self.get_axis(axis)):
first_part, first_idx = list(
self._get_dict_of_block_index(axis, [indices.start]).items()
)[0]
dict_of_slices = dict({first_part: slice(first_idx[0], None)})
num_partitions = np.size(self._partitions, axis=axis)
part_list = range(first_part + 1, num_partitions)
dict_of_slices.update(
dict(zip(part_list, [slice(None)] * len(part_list)))
)
return dict_of_slices
else:
first_part, first_idx = list(
self._get_dict_of_block_index(axis, [indices.start]).items()
)[0]
last_part, last_idx = list(
self._get_dict_of_block_index(axis, [indices.stop]).items()
)[0]
if first_part == last_part:
return dict({first_part: slice(first_idx[0], last_idx[0])})
else:
if last_part - first_part == 1:
return dict(
# FIXME: this dictionary creation feels wrong - it might not maintain the order
{
first_part: slice(first_idx[0], None),
last_part: slice(None, last_idx[0]),
}
)
else:
dict_of_slices = dict({first_part: slice(first_idx[0], None)})
part_list = range(first_part + 1, last_part)
dict_of_slices.update(
dict(zip(part_list, [slice(None)] * len(part_list)))
)
dict_of_slices.update({last_part: slice(None, last_idx[0])})
return dict_of_slices
if isinstance(indices, list):
# Converting python list to numpy for faster processing
indices = np.array(indices, dtype=np.int64)
# Fasttrack empty numpy array
if isinstance(indices, np.ndarray) and indices.size == 0:
# This will help preserve metadata stored in empty dataframes (indexes and dtypes)
# Otherwise, we will get an empty `new_partitions` array, from which it will
# no longer be possible to obtain metadata
return dict([(0, np.array([], dtype=np.int64))])
negative_mask = np.less(indices, 0)
has_negative = np.any(negative_mask)
if has_negative:
# We're going to modify 'indices' inplace in a numpy way, so doing a copy/converting indices to numpy.
indices = (
indices.copy()
if isinstance(indices, np.ndarray)
else np.array(indices, dtype=np.int64)
)
indices[negative_mask] = indices[negative_mask] % len(self.get_axis(axis))
# If the `indices` array was modified because of the negative indices conversion
# then the original order was broken and so we have to sort anyway:
if has_negative or not are_indices_sorted:
indices = np.sort(indices)
if axis == 0:
bins = np.array(self.row_lengths)
else:
bins = np.array(self.column_widths)
# INT_MAX to make sure we don't try to compute on partitions that don't exist.
cumulative = np.append(bins[:-1].cumsum(), np.iinfo(bins.dtype).max)
def internal(block_idx: int, global_index):
"""Transform global index to internal one for given block (identified by its index)."""
return (
global_index
if not block_idx
else np.subtract(
global_index, cumulative[min(block_idx, len(cumulative) - 1) - 1]
)
)
partition_ids = np.digitize(indices, cumulative)
count_for_each_partition = np.array(
[(partition_ids == i).sum() for i in range(len(cumulative))]
).cumsum()
# Compute the internal indices and pair those with the partition index.
# If the first partition has any values we need to return, compute those
# first to make the list comprehension easier. Otherwise, just append the
# rest of the values to an empty list.
if count_for_each_partition[0] > 0:
first_partition_indices = [
(0, internal(0, indices[slice(count_for_each_partition[0])]))
]
else:
first_partition_indices = []
partition_ids_with_indices = first_partition_indices + [
(
i,
internal(
i,
indices[
slice(
count_for_each_partition[i - 1],
count_for_each_partition[i],
)
],
),
)
for i in range(1, len(count_for_each_partition))
if count_for_each_partition[i] > count_for_each_partition[i - 1]
]
return dict(partition_ids_with_indices)
@staticmethod
def _join_index_objects(axis, indexes, how, sort, fill_value=None):
"""
Join the pair of index objects (columns or rows) by a given strategy.
Unlike Index.join() in pandas, if `axis` is 1, `sort` is False,
and `how` is "outer", the result will _not_ be sorted.
Parameters
----------
axis : {0, 1}
The axis index object to join (0 - rows, 1 - columns).
indexes : list(Index)
The indexes to join on.
how : {'left', 'right', 'inner', 'outer', None}
The type of join to join to make. If `None` then joined index
considered to be the first index in the `indexes` list.
sort : boolean
Whether or not to sort the joined index.
fill_value : any, optional
Value to use for missing values.
Returns
-------
(Index, func)
Joined index with make_reindexer func.
"""
assert isinstance(indexes, list)
# define helper functions
def merge(left_index, right_index):
"""Combine a pair of indices depending on `axis`, `how` and `sort` from outside."""
if axis == 1 and how == "outer" and not sort:
return left_index.union(right_index, sort=False)
else:
return left_index.join(right_index, how=how, sort=sort)
# define condition for joining indexes
all_indices_equal = all(indexes[0].equals(index) for index in indexes[1:])
do_join_index = how is not None and not all_indices_equal
# define condition for joining indexes with getting indexers
need_indexers = (
axis == 0
and not all_indices_equal
and any(not index.is_unique for index in indexes)
)
indexers = None
# perform joining indexes
if do_join_index:
if len(indexes) == 2 and need_indexers:
# in case of count of indexes > 2 we should perform joining all indexes
# after that get indexers
# in the fast path we can obtain joined_index and indexers in one call
indexers = [None, None]
joined_index, indexers[0], indexers[1] = indexes[0].join(
indexes[1], how=how, sort=sort, return_indexers=True
)
else:
joined_index = indexes[0]
# TODO: revisit for performance
for index in indexes[1:]:
joined_index = merge(joined_index, index)
else:
joined_index = indexes[0].copy()
if need_indexers and indexers is None:
indexers = [index.get_indexer_for(joined_index) for index in indexes]
def make_reindexer(do_reindex: bool, frame_idx: int):
"""Create callback that reindexes the dataframe using newly computed index."""
# the order of the frames must match the order of the indexes
if not do_reindex:
return lambda df: df
if need_indexers:
assert indexers is not None
return lambda df: df._reindex_with_indexers(
{0: [joined_index, indexers[frame_idx]]},
copy=True,
allow_dups=True,
fill_value=fill_value,
)
return lambda df: df.reindex(joined_index, axis=axis, fill_value=fill_value)
return joined_index, make_reindexer
# Internal methods
# These methods are for building the correct answer in a modular way.
# Please be careful when changing these!
def _build_treereduce_func(self, axis, func):
"""
Properly formats a TreeReduce result so that the partitioning is correct.
Parameters
----------
axis : int
The axis along which to apply the function.
func : callable
The function to apply.
Returns
-------
callable
A function to be shipped to the partitions to be executed.
Notes
-----
This should be used for any TreeReduce style operation that results in a
reduced data dimensionality (dataframe -> series).
"""
def _tree_reduce_func(df, *args, **kwargs):
"""Tree-reducer function itself executing `func`, presenting the resulting pandas.Series as pandas.DataFrame."""
series_result = func(df, *args, **kwargs)
if axis == 0 and isinstance(series_result, pandas.Series):
# In the case of axis=0, we need to keep the shape of the data
# consistent with what we have done. In the case of a reduce, the
# data for axis=0 should be a single value for each column. By
# transposing the data after we convert to a DataFrame, we ensure that
# the columns of the result line up with the columns from the data.
# axis=1 does not have this requirement because the index already will
# line up with the index of the data based on how pandas creates a
# DataFrame from a Series.
result = pandas.DataFrame(series_result).T
result.index = [MODIN_UNNAMED_SERIES_LABEL]
else:
result = pandas.DataFrame(series_result)
if isinstance(series_result, pandas.Series):
result.columns = [MODIN_UNNAMED_SERIES_LABEL]
return result
return _tree_reduce_func
def _compute_tree_reduce_metadata(self, axis, new_parts, dtypes=None):
"""
Compute the metadata for the result of reduce function.
Parameters
----------
axis : int
The axis on which reduce function was applied.
new_parts : NumPy 2D array
Partitions with the result of applied function.
dtypes : str, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
Returns
-------
PandasDataframe
Modin series (1xN frame) containing the reduced data.
"""
new_axes, new_axes_lengths = [0, 0], [0, 0]
new_axes[axis] = [MODIN_UNNAMED_SERIES_LABEL]
new_axes[axis ^ 1] = self.get_axis(axis ^ 1)
new_axes_lengths[axis] = [1]
new_axes_lengths[axis ^ 1] = self._get_axis_lengths(axis ^ 1)
if dtypes == "copy":
dtypes = self.copy_dtypes_cache()
elif dtypes is not None:
dtypes = pandas.Series(
[pandas.api.types.pandas_dtype(dtypes)] * len(new_axes[1]),
index=new_axes[1],
)
result = self.__constructor__(
new_parts,
*new_axes,
*new_axes_lengths,
dtypes,
pandas_backend=self._pandas_backend,
)
return result
@lazy_metadata_decorator(apply_axis="both")
def reduce(
self,
axis: Union[int, Axis],
function: Callable,
dtypes: Optional[str] = None,
) -> PandasDataframe:
"""
Perform a user-defined aggregation on the specified axis, where the axis reduces down to a singleton. Requires knowledge of the full axis for the reduction.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the reduce over.
function : callable(row|col) -> single value
The reduce function to apply to each column.
dtypes : str, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
Returns
-------
PandasDataframe
Modin series (1xN frame) containing the reduced data.
Notes
-----
The user-defined function must reduce to a single value.
"""
axis = Axis(axis)
function = self._build_treereduce_func(axis.value, function)
new_parts = self._partition_mgr_cls.map_axis_partitions(
axis.value, self._partitions, function
)
return self._compute_tree_reduce_metadata(axis.value, new_parts, dtypes=dtypes)
@lazy_metadata_decorator(apply_axis="opposite", axis_arg=0)
def tree_reduce(
self,
axis: Union[int, Axis],
map_func: Callable,
reduce_func: Optional[Callable] = None,
dtypes: Optional[str] = None,
) -> PandasDataframe:
"""
Apply function that will reduce the data to a pandas Series.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the tree reduce over.
map_func : callable(row|col) -> row|col
Callable function to map the dataframe.
reduce_func : callable(row|col) -> single value, optional
Callable function to reduce the dataframe.
If none, then apply map_func twice.
dtypes : str, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
Returns
-------
PandasDataframe
A new dataframe.
"""
axis = Axis(axis)
map_func = self._build_treereduce_func(axis.value, map_func)
if reduce_func is None:
reduce_func = map_func
else:
reduce_func = self._build_treereduce_func(axis.value, reduce_func)
map_parts = self._partition_mgr_cls.map_partitions(self._partitions, map_func)
reduce_parts = self._partition_mgr_cls.map_axis_partitions(
axis.value, map_parts, reduce_func
)
return self._compute_tree_reduce_metadata(
axis.value, reduce_parts, dtypes=dtypes
)
@lazy_metadata_decorator(apply_axis=None)
def map(
self,
func: Callable,
dtypes: Optional[str] = None,
new_columns: Optional[pandas.Index] = None,
func_args=None,
func_kwargs=None,
lazy=False,
) -> PandasDataframe:
"""
Perform a function that maps across the entire dataset.
Parameters
----------
func : callable(row|col|cell) -> row|col|cell
The function to apply.
dtypes : dtypes of the result, optional
The data types for the result. This is an optimization
because there are functions that always result in a particular data
type, and this allows us to avoid (re)computing it.
new_columns : pandas.Index, optional
New column labels of the result, its length has to be identical
to the older columns. If not specified, old column labels are preserved.
func_args : iterable, optional
Positional arguments for the 'func' callable.
func_kwargs : dict, optional
Keyword arguments for the 'func' callable.
lazy : bool, default: False
Whether to prefer lazy execution or not.
Returns
-------
PandasDataframe
A new dataframe.
"""
map_fn = (
self._partition_mgr_cls.lazy_map_partitions
if lazy
else self._partition_mgr_cls.map_partitions
)
new_partitions = map_fn(self._partitions, func, func_args, func_kwargs)
if new_columns is not None and self.has_materialized_columns:
assert len(new_columns) == len(
self.columns
), "New column's length must be identical to the previous columns"
elif new_columns is None:
new_columns = self.copy_columns_cache(copy_lengths=True)
if isinstance(dtypes, str) and dtypes == "copy":
dtypes = self.copy_dtypes_cache()
elif dtypes is not None and not isinstance(dtypes, pandas.Series):
if isinstance(new_columns, ModinIndex):
# Materializing lazy columns in order to build dtype's index
new_columns = new_columns.get(return_lengths=False)
dtypes = pandas.Series(
[pandas.api.types.pandas_dtype(dtypes)] * len(new_columns),
index=new_columns,
)
return self.__constructor__(
new_partitions,
self.copy_index_cache(copy_lengths=True),
new_columns,
self._row_lengths_cache,
self._column_widths_cache,
dtypes=dtypes,
pandas_backend=self._pandas_backend,
)
def window(
self,
axis: Union[int, Axis],
reduce_fn: Callable,
window_size: int,
result_schema: Optional[Dict[Hashable, type]] = None,
) -> PandasDataframe:
"""
Apply a sliding window operator that acts as a GROUPBY on each window, and reduces down to a single row (column) per window.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to slide over.
reduce_fn : callable(rowgroup|colgroup) -> row|col
The reduce function to apply over the data.
window_size : int
The number of row/columns to pass to the function.
(The size of the sliding window).
result_schema : dict, optional
Mapping from column labels to data types that represents the types of the output dataframe.
Returns
-------
PandasDataframe
A new PandasDataframe with the reduce function applied over windows of the specified
axis.
Notes
-----
The user-defined reduce function must reduce each window’s column
(row if axis=1) down to a single value.
"""
pass
@lazy_metadata_decorator(apply_axis="both")
def fold(self, axis, func, new_index=None, new_columns=None, shape_preserved=False):
"""
Perform a function across an entire axis.
Parameters
----------
axis : int
The axis to apply over.
func : callable
The function to apply.
new_index : list-like, optional
The index of the result.
new_columns : list-like, optional
The columns of the result.
shape_preserved : bool, default: False
Whether the shape of the dataframe is preserved or not
after applying a function.
Returns
-------
PandasDataframe
A new dataframe.
"""
new_row_lengths = None
new_column_widths = None
if shape_preserved:
if new_index is None:
new_index = self.copy_index_cache(copy_lengths=True)
if new_columns is None:
new_columns = self.copy_columns_cache(copy_lengths=True)
new_row_lengths = self._row_lengths_cache
new_column_widths = self._column_widths_cache
new_partitions = self._partition_mgr_cls.map_axis_partitions(
axis, self._partitions, func, keep_partitioning=True
)
return self.__constructor__(
new_partitions,
new_index,
new_columns,
row_lengths=new_row_lengths,
column_widths=new_column_widths,
pandas_backend=self._pandas_backend,
)
def infer_objects(self) -> PandasDataframe:
"""
Attempt to infer better dtypes for object columns.
Attempts soft conversion of object-dtyped columns, leaving non-object and unconvertible
columns unchanged. The inference rules are the same as during normal Series/DataFrame
construction.
Returns
-------
PandasDataframe
A new PandasDataframe with the inferred schema.
"""
obj_cols = [
col for col, dtype in enumerate(self.dtypes) if is_object_dtype(dtype)
]
return self.infer_types(obj_cols)
def infer_types(self, col_labels: List[str]) -> PandasDataframe:
"""
Determine the compatible type shared by all values in the specified columns, and coerce them to that type.
Parameters
----------
col_labels : list
List of column labels to infer and induce types over.
Returns
-------
PandasDataframe
A new PandasDataframe with the inferred schema.
"""
# Compute dtypes on the specified columns, and then set those dtypes on a new frame
new_cols = self.take_2d_labels_or_positional(col_labels=col_labels)
new_cols_dtypes = new_cols.tree_reduce(0, pandas.DataFrame.infer_objects).dtypes
new_dtypes = self.dtypes.copy()
new_dtypes[col_labels] = new_cols_dtypes
return self.__constructor__(
self._partitions,
self.copy_index_cache(copy_lengths=True),
self.copy_columns_cache(copy_lengths=True),
self._row_lengths_cache,
self._column_widths_cache,
new_dtypes,
pandas_backend=self._pandas_backend,
)
def join(
self,
axis: Union[int, Axis],
condition: Callable,
other: ModinDataframe,
join_type: Union[str, JoinType],
) -> PandasDataframe:
"""
Join this dataframe with the other.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the join on.
condition : callable
Function that determines which rows should be joined. The condition can be a
simple equality, e.g. "left.col1 == right.col1" or can be arbitrarily complex.
other : ModinDataframe
The other data to join with, i.e. the right dataframe.
join_type : string {"inner", "left", "right", "outer"} or modin.core.dataframe.base.utils.JoinType
The type of join to perform.
Returns
-------
PandasDataframe
A new PandasDataframe that is the result of applying the specified join over the two
dataframes.
Notes
-----
During the join, this dataframe is considered the left, while the other is
treated as the right.
Only inner joins, left outer, right outer, and full outer joins are currently supported.
Support for other join types (e.g. natural join) may be implemented in the future.
"""
pass
def rename(
self,
new_row_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,
new_col_labels: Optional[Union[Dict[Hashable, Hashable], Callable]] = None,
) -> PandasDataframe:
"""
Replace the row and column labels with the specified new labels.
Parameters
----------
new_row_labels : dictionary or callable, optional
Mapping or callable that relates old row labels to new labels.
new_col_labels : dictionary or callable, optional
Mapping or callable that relates old col labels to new labels.
Returns
-------
PandasDataframe
A new PandasDataframe with the new row and column labels.
"""
result = self.copy()
if new_row_labels is not None:
if callable(new_row_labels):
new_row_labels = result.index.map(new_row_labels)
result.index = new_row_labels
if new_col_labels is not None:
if callable(new_col_labels):
new_col_labels = result.columns.map(new_col_labels)
result.columns = new_col_labels
return result
def combine_and_apply(
self, func, new_index=None, new_columns=None, new_dtypes=None
):
"""
Combine all partitions into a single big one and apply the passed function to it.
Use this method with care as it collects all the data on the same worker,
it's only recommended to use this method on small or reduced datasets.
Parameters
----------
func : callable(pandas.DataFrame) -> pandas.DataFrame
A function to apply to the combined partition.
new_index : sequence, optional
Index of the result.
new_columns : sequence, optional
Columns of the result.
new_dtypes : dict-like, optional
Dtypes of the result.
Returns
-------
PandasDataframe
"""
if self._partitions.shape[1] > 1:
new_partitions = self._partition_mgr_cls.row_partitions(self._partitions)
new_partitions = np.array([[partition] for partition in new_partitions])
modin_frame = self.__constructor__(
new_partitions,
self.copy_index_cache(copy_lengths=True),
self.copy_columns_cache(),
self._row_lengths_cache,
[len(self.columns)] if self.has_materialized_columns else None,
self.copy_dtypes_cache(),
pandas_backend=self._pandas_backend,
)
else:
modin_frame = self
return modin_frame.apply_full_axis(
axis=0,
func=func,
new_index=new_index,
new_columns=new_columns,
dtypes=new_dtypes,
)
@lazy_metadata_decorator(apply_axis="both")
def _apply_func_to_range_partitioning(
self,
key_columns,
func,
ascending=True,
preserve_columns=False,
data=None,
data_key_columns=None,
level=None,
shuffle_func_cls=ShuffleSortFunctions,
**kwargs,
):
"""
Reshuffle data so it would be range partitioned and then apply the passed function row-wise.
Parameters
----------
key_columns : list of hashables
Columns to build the range partitioning for. Can't be specified along with `level`.
func : callable(pandas.DataFrame) -> pandas.DataFrame
Function to apply against partitions.
ascending : bool, default: True
Whether the range should be built in ascending or descending order.
preserve_columns : bool, default: False
If the columns cache should be preserved (specify this flag if `func` doesn't change column labels).
data : PandasDataframe, optional
Dataframe to range-partition along with the `self` frame. If specified, the `func` will recieve
a dataframe with an additional MultiIndex level in columns that separates `self` and `data`:
``df["grouper"] # self`` and ``df["data"] # data``.
data_key_columns : list of hashables, optional
Additional key columns from `data`. Will be combined with `key_columns`.
level : list of ints or labels, optional
Index level(s) to build the range partitioning for. Can't be specified along with `key_columns`.
shuffle_func_cls : cls, default: ShuffleSortFunctions
A class implementing ``modin.core.dataframe.pandas.utils.ShuffleFunctions`` to be used
as a shuffle function.
**kwargs : dict
Additional arguments to forward to the range builder function.
Returns
-------
PandasDataframe
A new dataframe.
"""
if data is not None:
# adding an extra MultiIndex level in order to separate `self grouper` from the `data`
# after concatenation
new_grouper_cols = pandas.MultiIndex.from_tuples(
[
("grouper", *col) if isinstance(col, tuple) else ("grouper", col)
for col in self.columns
]
)
grouper = self.copy()
grouper.columns = new_grouper_cols
new_data_cols = pandas.MultiIndex.from_tuples(
[
("data", *col) if isinstance(col, tuple) else ("data", col)
for col in data.columns
]
)
data = data.copy()
data.columns = new_data_cols
grouper = grouper.concat(axis=1, others=[data], how="right", sort=False)
# since original column names were modified, have to modify 'key_columns' as well
key_columns = [
("grouper", *col) if isinstance(col, tuple) else ("grouper", col)
for col in key_columns
]
if data_key_columns is None:
data_key_columns = []
else:
data_key_columns = [
("data", *col) if isinstance(col, tuple) else ("data", col)
for col in data_key_columns
]
key_columns += data_key_columns
else:
grouper = self
# If there's only one row partition can simply apply the function row-wise without the need to reshuffle
if grouper._partitions.shape[0] == 1:
result = grouper.apply_full_axis(
axis=1,
func=func,
new_columns=grouper.copy_columns_cache() if preserve_columns else None,
)
if preserve_columns:
result._set_axis_lengths_cache(grouper._column_widths_cache, axis=1)
return result
# don't want to inherit over-partitioning so doing this 'min' check
ideal_num_new_partitions = min(len(grouper._partitions), NPartitions.get())
m = len(grouper) / ideal_num_new_partitions
sampling_probability = (1 / m) * np.log(ideal_num_new_partitions * len(grouper))
# If this df is overpartitioned, we try to sample each partition with probability
# greater than 1, which leads to an error. In this case, we can do one of the following
# two things. If there is only enough rows for one partition, and we have only 1 column
# partition, we can just combine the overpartitioned df into one partition, and sort that
# partition. If there is enough data for more than one partition, we can tell the sorting
# algorithm how many partitions we want to end up with, so it samples and finds pivots
# according to that.
if sampling_probability >= 1:
from modin.config import MinRowPartitionSize
ideal_num_new_partitions = round(len(grouper) / MinRowPartitionSize.get())
if len(grouper) < MinRowPartitionSize.get() or ideal_num_new_partitions < 2:
# If the data is too small, we shouldn't try reshuffling/repartitioning but rather
# simply combine all partitions and apply the sorting to the whole dataframe
return grouper.combine_and_apply(func=func)
if ideal_num_new_partitions < len(grouper._partitions):
if len(grouper._partitions) % ideal_num_new_partitions == 0:
joining_partitions = np.split(
grouper._partitions, ideal_num_new_partitions
)
else:
step = round(len(grouper._partitions) / ideal_num_new_partitions)
joining_partitions = np.split(
grouper._partitions,
range(step, len(grouper._partitions), step),
)
new_partitions = np.array(
[
grouper._partition_mgr_cls.column_partitions(
ptn_grp, full_axis=False
)
for ptn_grp in joining_partitions
]
)
else:
new_partitions = grouper._partitions
else:
new_partitions = grouper._partitions
shuffling_functions = shuffle_func_cls(
grouper,
key_columns,
ascending[0] if is_list_like(ascending) else ascending,
ideal_num_new_partitions,
level=level,
**kwargs,
)
if key_columns:
# here we want to get indices of those partitions that hold the key columns
key_indices = grouper.columns.get_indexer_for(key_columns)
partition_indices = np.unique(
np.digitize(key_indices, np.cumsum(grouper.column_widths))
)
elif level is not None:
# each partition contains an index, so taking the first one
partition_indices = [0]
else:
raise ValueError("Must specify either 'level' or 'key_columns'")
new_partitions = grouper._partition_mgr_cls.shuffle_partitions(
new_partitions,
partition_indices,
shuffling_functions,
func,
)
result = grouper.__constructor__(new_partitions)
if preserve_columns:
result.set_columns_cache(grouper.copy_columns_cache())
# We perform the final steps of the sort on full axis partitions, so we know that the
# length of each partition is the full length of the dataframe.
if grouper.has_materialized_columns:
result._set_axis_lengths_cache([len(grouper.columns)], axis=1)
return result
@lazy_metadata_decorator(apply_axis="both")
def sort_by(
self,
axis: Union[int, Axis],
columns: Union[str, List[str]],
ascending: bool = True,
**kwargs,
) -> PandasDataframe:
"""
Logically reorder rows (columns if axis=1) lexicographically by the data in a column or set of columns.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to perform the sort over.
columns : string or list
Column label(s) to use to determine lexicographical ordering.
ascending : boolean, default: True
Whether to sort in ascending or descending order.
**kwargs : dict
Keyword arguments to pass when sorting partitions.
Returns
-------
PandasDataframe
A new PandasDataframe sorted into lexicographical order by the specified column(s).
"""
if not isinstance(columns, list):
columns = [columns]
def sort_function(df): # pragma: no cover
# When we do a sort on the result of Series.value_counts, we don't rename the index until
# after everything is done, which causes an error when sorting the partitions, since the
# index and the column share the same name, when in actuality, the index's name should be
# None. This fixes the indexes name beforehand in that case, so that the sort works.
index_renaming = None
if any(name in df.columns for name in df.index.names):
index_renaming = df.index.names
df.index = df.index.set_names([None] * len(df.index.names))
df = df.sort_values(by=columns, ascending=ascending, **kwargs)
if index_renaming is not None:
df.index = df.index.set_names(index_renaming)
return df
# If this df is empty, we don't want to try and shuffle or sort.
if len(self.get_axis(1)) == 0 or len(self) == 0:
return self.copy()
axis = Axis(axis)
if axis != Axis.ROW_WISE:
raise NotImplementedError(
f"Algebra sort only implemented row-wise. {axis.name} sort not implemented yet!"
)
result = self._apply_func_to_range_partitioning(
key_columns=[columns[0]],
func=sort_function,
ascending=ascending,
preserve_columns=True,
**kwargs,
)
result.set_dtypes_cache(self.copy_dtypes_cache())
if kwargs.get("ignore_index", False):
result.index = RangeIndex(len(self.get_axis(axis.value)))
# Since the strategy to pick our pivots involves random sampling
# we could end up picking poor pivots, leading to skew in our partitions.
# We should add a fix to check if there is skew in the partitions and rebalance
# them if necessary. Calling `rebalance_partitions` won't do this, since it only
# resolves the case where there isn't the right amount of partitions - not where
# there is skew across the lengths of partitions.
return result
@lazy_metadata_decorator(apply_axis="both")
def filter(self, axis: Union[Axis, int], condition: Callable) -> PandasDataframe:
"""
Filter data based on the function provided along an entire axis.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to filter over.
condition : callable(row|col) -> bool
The function to use for the filter. This function should filter the
data itself.
Returns
-------
PandasDataframe
A new filtered dataframe.
"""
axis = Axis(axis)
assert axis in (
Axis.ROW_WISE,
Axis.COL_WISE,
), "Axis argument to filter operator must be 0 (rows) or 1 (columns)"
new_partitions = self._partition_mgr_cls.map_axis_partitions(
axis.value, self._partitions, condition, keep_partitioning=True
)
new_axes, new_lengths = [0, 0], [0, 0]
new_axes[axis.value] = self.copy_axis_cache(axis.value, copy_lengths=True)
new_lengths[axis.value] = (
self._row_lengths_cache if axis.value == 0 else self._column_widths_cache
)
new_axes[axis.value ^ 1], new_lengths[axis.value ^ 1] = None, None
return self.__constructor__(
new_partitions,
*new_axes,
*new_lengths,
self.copy_dtypes_cache() if axis == Axis.COL_WISE else None,
pandas_backend=self._pandas_backend,
)
def filter_by_types(self, types: List[Hashable]) -> PandasDataframe:
"""
Allow the user to specify a type or set of types by which to filter the columns.
Parameters
----------
types : list
The types to filter columns by.
Returns
-------
PandasDataframe
A new PandasDataframe from the filter provided.
"""
return self.take_2d_labels_or_positional(
col_positions=[i for i, dtype in enumerate(self.dtypes) if dtype in types]
)
@lazy_metadata_decorator(apply_axis="both")
def explode(self, axis: Union[int, Axis], func: Callable) -> PandasDataframe:
"""
Explode list-like entries along an entire axis.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis specifying how to explode. If axis=1, explode according
to columns.
func : callable
The function to use to explode a single element.
Returns
-------
PandasFrame
A new filtered dataframe.
"""
axis = Axis(axis)
partitions = self._partition_mgr_cls.map_axis_partitions(
axis.value, self._partitions, func, keep_partitioning=True
)
if axis == Axis.COL_WISE:
new_index, row_lengths = self._compute_axis_labels_and_lengths(
0, partitions
)
new_columns, column_widths = self.columns, self._column_widths_cache
else:
new_index, row_lengths = self.index, self._row_lengths_cache
new_columns, column_widths = self._compute_axis_labels_and_lengths(
1, partitions
)
return self.__constructor__(
partitions,
new_index,
new_columns,
row_lengths,
column_widths,
pandas_backend=self._pandas_backend,
)
def combine(self) -> PandasDataframe:
"""
Create a single partition PandasDataframe from the partitions of the current dataframe.
Returns
-------
PandasDataframe
A single partition PandasDataframe.
"""
new_index = None
new_columns = None
if self._deferred_index:
new_index = self.index
if self._deferred_column:
new_columns = self.columns
partitions = self._partition_mgr_cls.combine(
self._partitions, new_index, new_columns
)
result = self.__constructor__(
partitions,
index=self.copy_index_cache(),
columns=self.copy_columns_cache(),
row_lengths=(
[sum(self._row_lengths_cache)]
if self._row_lengths_cache is not None
else None
),
column_widths=(
[sum(self._column_widths_cache)]
if self._column_widths_cache is not None
else None
),
dtypes=self.copy_dtypes_cache(),
pandas_backend=self._pandas_backend,
)
return result
@lazy_metadata_decorator(apply_axis="both")
def apply_full_axis(
self,
axis,
func,
new_index=None,
new_columns=None,
apply_indices=None,
enumerate_partitions: bool = False,
dtypes=None,
keep_partitioning=True,
num_splits=None,
sync_labels=True,
pass_axis_lengths_to_partitions=False,
) -> PandasDataframe:
"""
Perform a function across an entire axis.
Parameters
----------
axis : {0, 1}
The axis to apply over (0 - rows, 1 - columns).
func : callable
The function to apply.
new_index : list-like, optional
The index of the result. We may know this in advance,
and if not provided it must be computed.
new_columns : list-like, optional
The columns of the result. We may know this in
advance, and if not provided it must be computed.
apply_indices : list-like, optional
Indices of `axis ^ 1` to apply function over.
enumerate_partitions : bool, default: False
Whether pass partition index into applied `func` or not.
Note that `func` must be able to obtain `partition_idx` kwarg.
dtypes : list-like or scalar, optional
The data types of the result. This is an optimization
because there are functions that always result in a particular data
type, and allows us to avoid (re)computing it.
keep_partitioning : boolean, default: True
The flag to keep partition boundaries for Modin Frame if possible.
Setting it to True disables shuffling data from one partition to another in case the resulting
number of splits is equal to the initial number of splits.
num_splits : int, optional
The number of partitions to split the result into across the `axis`. If None, then the number
of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`
then the number of splits is preserved.
sync_labels : boolean, default: True
Synchronize external indexes (`new_index`, `new_columns`) with internal indexes.
This could be used when you're certain that the indices in partitions are equal to
the provided hints in order to save time on syncing them.
pass_axis_lengths_to_partitions : bool, default: False
Whether pass partition lengths along `axis ^ 1` to the kernel `func`.
Note that `func` must be able to obtain `df, *axis_lengths`.
Returns
-------
PandasDataframe
A new dataframe.
Notes
-----
The data shape may change as a result of the function.
"""
return self.broadcast_apply_full_axis(
axis=axis,
func=func,
new_index=new_index,
new_columns=new_columns,
apply_indices=apply_indices,
enumerate_partitions=enumerate_partitions,
dtypes=dtypes,
other=None,
keep_partitioning=keep_partitioning,
num_splits=num_splits,
sync_labels=sync_labels,
pass_axis_lengths_to_partitions=pass_axis_lengths_to_partitions,
)
@lazy_metadata_decorator(apply_axis="both")
def apply_full_axis_select_indices(
self,
axis,
func,
apply_indices=None,
numeric_indices=None,
new_index=None,
new_columns=None,
keep_remaining=False,
new_dtypes: Optional[Union[pandas.Series, ModinDtypes]] = None,
):
"""
Apply a function across an entire axis for a subset of the data.
Parameters
----------
axis : int
The axis to apply over.
func : callable
The function to apply.
apply_indices : list-like, optional
The labels to apply over.
numeric_indices : list-like, optional
The indices to apply over.
new_index : list-like, optional
The index of the result. We may know this in advance,
and if not provided it must be computed.
new_columns : list-like, optional
The columns of the result. We may know this in
advance, and if not provided it must be computed.
keep_remaining : boolean, default: False
Whether or not to drop the data that is not computed over.
new_dtypes : ModinDtypes or pandas.Series, optional
The data types of the result. This is an optimization
because there are functions that always result in a particular data
type, and allows us to avoid (re)computing it.
Returns
-------
PandasDataframe
A new dataframe.
"""
assert apply_indices is not None or numeric_indices is not None
# Convert indices to numeric indices
old_index = self.index if axis else self.columns
if apply_indices is not None:
numeric_indices = old_index.get_indexer_for(apply_indices)
# Get the indices for the axis being applied to (it is the opposite of axis
# being applied over)
dict_indices = self._get_dict_of_block_index(axis ^ 1, numeric_indices)
new_partitions = (
self._partition_mgr_cls.apply_func_to_select_indices_along_full_axis(
axis,
self._partitions,
func,
dict_indices,
keep_remaining=keep_remaining,
)
)
# TODO Infer columns and index from `keep_remaining` and `apply_indices`
if new_index is None:
new_index = self.index if axis == 1 else None
if new_columns is None:
new_columns = self.columns if axis == 0 else None
return self.__constructor__(
new_partitions,
new_index,
new_columns,
None,
None,
dtypes=new_dtypes,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def apply_select_indices(
self,
axis,
func,
apply_indices=None,
row_labels=None,
col_labels=None,
new_index=None,
new_columns=None,
new_dtypes: Optional[pandas.Series] = None,
keep_remaining=False,
item_to_distribute=no_default,
) -> PandasDataframe:
"""
Apply a function for a subset of the data.
Parameters
----------
axis : {0, 1}
The axis to apply over.
func : callable
The function to apply.
apply_indices : list-like, optional
The labels to apply over. Must be given if axis is provided.
row_labels : list-like, optional
The row labels to apply over. Must be provided with
`col_labels` to apply over both axes.
col_labels : list-like, optional
The column labels to apply over. Must be provided
with `row_labels` to apply over both axes.
new_index : list-like, optional
The index of the result, if known in advance.
new_columns : list-like, optional
The columns of the result, if known in advance.
new_dtypes : pandas.Series, optional
The dtypes of the result, if known in advance.
keep_remaining : boolean, default: False
Whether or not to drop the data that is not computed over.
item_to_distribute : np.ndarray or scalar, default: no_default
The item to split up so it can be applied over both axes.
Returns
-------
PandasDataframe
A new dataframe.
"""
# TODO Infer columns and index from `keep_remaining` and `apply_indices`
if new_index is None:
new_index = self.index if axis == 1 else None
if new_columns is None:
new_columns = self.columns if axis == 0 else None
if new_columns is not None and isinstance(new_dtypes, pandas.Series):
assert new_dtypes.index.equals(
new_columns
), f"{new_dtypes=} doesn't have the same columns as in {new_columns=}"
if axis is not None:
assert apply_indices is not None
# Convert indices to numeric indices
old_index = self.index if axis else self.columns
numeric_indices = old_index.get_indexer_for(apply_indices)
# Get indices being applied to (opposite of indices being applied over)
dict_indices = self._get_dict_of_block_index(axis ^ 1, numeric_indices)
new_partitions = self._partition_mgr_cls.apply_func_to_select_indices(
axis,
self._partitions,
func,
dict_indices,
keep_remaining=keep_remaining,
)
# Length objects for new object creation. This is shorter than if..else
# This object determines the lengths and widths based on the given
# parameters and builds a dictionary used in the constructor below. 0 gives
# the row lengths and 1 gives the column widths. Since the dimension of
# `axis` given may have changed, we currently just recompute it.
# TODO Determine lengths from current lengths if `keep_remaining=False`
lengths_objs = {
axis: (
[len(apply_indices)]
if not keep_remaining
else [self.row_lengths, self.column_widths][axis]
),
axis ^ 1: [self.row_lengths, self.column_widths][axis ^ 1],
}
return self.__constructor__(
new_partitions,
new_index,
new_columns,
lengths_objs[0],
lengths_objs[1],
new_dtypes,
pandas_backend=self._pandas_backend,
)
else:
# We are applying over both axes here, so make sure we have all the right
# variables set.
assert row_labels is not None and col_labels is not None
assert keep_remaining
assert item_to_distribute is not no_default
row_partitions_list = self._get_dict_of_block_index(0, row_labels).items()
col_partitions_list = self._get_dict_of_block_index(1, col_labels).items()
new_partitions = self._partition_mgr_cls.apply_func_to_indices_both_axis(
self._partitions,
func,
row_partitions_list,
col_partitions_list,
item_to_distribute,
# Passing caches instead of values in order to not trigger shapes recomputation
# if they are not used inside this function.
self._row_lengths_cache,
self._column_widths_cache,
)
return self.__constructor__(
new_partitions,
new_index,
new_columns,
self._row_lengths_cache,
self._column_widths_cache,
new_dtypes,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def broadcast_apply(
self,
axis,
func,
other,
join_type="left",
copartition=True,
labels="keep",
dtypes=None,
):
"""
Broadcast axis partitions of `other` to partitions of `self` and apply a function.
Parameters
----------
axis : {0, 1}
Axis to broadcast over.
func : callable
Function to apply.
other : PandasDataframe
Modin DataFrame to broadcast.
join_type : str, default: "left"
Type of join to apply.
copartition : bool, default: True
Whether to align indices/partitioning of the `self` and `other` frame.
Disabling this may save some time, however, you have to be 100% sure that
the indexing and partitioning are identical along the broadcasting axis,
this might be the case for example if `other` is a projection of the `self`
or vice-versa. If copartitioning is disabled and partitioning/indexing are
incompatible then you may end up with undefined behavior.
labels : {"keep", "replace", "drop"}, default: "keep"
Whether keep labels from `self` Modin DataFrame, replace them with labels
from joined DataFrame or drop altogether to make them be computed lazily later.
dtypes : "copy", pandas.Series or None, optional
Dtypes of the result. "copy" to keep old dtypes and None to compute them on demand.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
if copartition:
# Only sort the indices if they do not match
(
left_parts,
right_parts,
joined_index,
partition_sizes_along_axis,
) = self._copartition(
axis,
other,
join_type,
)
# unwrap list returned by `copartition`.
right_parts = right_parts[0]
else:
left_parts = self._partitions
right_parts = other._partitions
partition_sizes_along_axis, joined_index = self._get_axis_lengths_cache(
axis
), self.copy_axis_cache(axis)
new_frame = self._partition_mgr_cls.broadcast_apply(
axis, func, left_parts, right_parts
)
if isinstance(dtypes, str) and dtypes == "copy":
dtypes = self.copy_dtypes_cache()
def _pick_axis(get_axis, sizes_cache):
if labels == "keep":
return get_axis(), sizes_cache
if labels == "replace":
return joined_index, partition_sizes_along_axis
assert labels == "drop", f"Unexpected `labels`: {labels}"
return None, None
if axis == 0:
# Pass shape caches instead of values in order to not trigger shape computation.
new_index, new_row_lengths = _pick_axis(
self.copy_index_cache, self._row_lengths_cache
)
new_columns, new_column_widths = (
self.copy_columns_cache(),
self._column_widths_cache,
)
else:
new_index, new_row_lengths = (
self.copy_index_cache(),
self._row_lengths_cache,
)
new_columns, new_column_widths = _pick_axis(
self.copy_columns_cache, self._column_widths_cache
)
return self.__constructor__(
new_frame,
new_index,
new_columns,
new_row_lengths,
new_column_widths,
dtypes=dtypes,
pandas_backend=self._pandas_backend,
)
def _prepare_frame_to_broadcast(self, axis, indices, broadcast_all):
"""
Compute the indices to broadcast `self` considering `indices`.
Parameters
----------
axis : {0, 1}
Axis to broadcast along.
indices : dict
Dict of indices and internal indices of partitions where `self` must
be broadcasted.
broadcast_all : bool
Whether broadcast the whole axis of `self` frame or just a subset of it.
Returns
-------
dict
Dictionary with indices of partitions to broadcast.
Notes
-----
New dictionary of indices of `self` partitions represents that
you want to broadcast `self` at specified another partition named `other`. For example,
Dictionary {key: {key1: [0, 1], key2: [5]}} means, that in `other`[key] you want to
broadcast [self[key1], self[key2]] partitions and internal indices for `self` must be [[0, 1], [5]]
"""
if broadcast_all:
sizes = self.row_lengths if axis else self.column_widths
return {key: dict(enumerate(sizes)) for key in indices.keys()}
passed_len = 0
result_dict = {}
for part_num, internal in indices.items():
result_dict[part_num] = self._get_dict_of_block_index(
axis ^ 1, np.arange(passed_len, passed_len + len(internal))
)
passed_len += len(internal)
return result_dict
def _extract_partitions(self):
"""
Extract partitions if partitions are present.
If partitions are empty return a dummy partition with empty data but
index and columns of current dataframe.
Returns
-------
np.ndarray
NumPy array with extracted partitions.
"""
if self._partitions.size > 0:
return self._partitions
else:
dtypes = None
if self.has_materialized_dtypes:
dtypes = self.dtypes
return self._partition_mgr_cls.create_partition_from_metadata(
index=self.index, columns=self.columns, dtypes=dtypes
)
@lazy_metadata_decorator(apply_axis="both")
def broadcast_apply_select_indices(
self,
axis,
func,
other: PandasDataframe,
apply_indices=None,
numeric_indices=None,
keep_remaining=False,
broadcast_all=True,
new_index=None,
new_columns=None,
) -> PandasDataframe:
"""
Apply a function to select indices at specified axis and broadcast partitions of `other` Modin DataFrame.
Parameters
----------
axis : {0, 1}
Axis to apply function along.
func : callable
Function to apply.
other : PandasDataframe
Partitions of which should be broadcasted.
apply_indices : list, optional
List of labels to apply (if `numeric_indices` are not specified).
numeric_indices : list, optional
Numeric indices to apply (if `apply_indices` are not specified).
keep_remaining : bool, default: False
Whether drop the data that is not computed over or not.
broadcast_all : bool, default: True
Whether broadcast the whole axis of right frame to every
partition or just a subset of it.
new_index : pandas.Index, optional
Index of the result. We may know this in advance,
and if not provided it must be computed.
new_columns : pandas.Index, optional
Columns of the result. We may know this in advance,
and if not provided it must be computed.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
assert (
apply_indices is not None or numeric_indices is not None
), "Indices to apply must be specified!"
if other is None:
if apply_indices is None:
apply_indices = self.get_axis(axis)[numeric_indices]
return self.apply_select_indices(
axis=axis,
func=func,
apply_indices=apply_indices,
keep_remaining=keep_remaining,
new_index=new_index,
new_columns=new_columns,
)
if numeric_indices is None:
old_index = self.index if axis else self.columns
numeric_indices = old_index.get_indexer_for(apply_indices)
dict_indices = self._get_dict_of_block_index(axis ^ 1, numeric_indices)
broadcasted_dict = other._prepare_frame_to_broadcast(
axis, dict_indices, broadcast_all=broadcast_all
)
new_partitions = self._partition_mgr_cls.broadcast_apply_select_indices(
axis,
func,
self._partitions,
other._partitions,
dict_indices,
broadcasted_dict,
keep_remaining,
)
return self.__constructor__(
new_partitions,
index=new_index,
columns=new_columns,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def broadcast_apply_full_axis(
self,
axis,
func,
other,
new_index=None,
new_columns=None,
apply_indices=None,
enumerate_partitions=False,
dtypes=None,
keep_partitioning=True,
num_splits=None,
sync_labels=True,
pass_axis_lengths_to_partitions=False,
):
"""
Broadcast partitions of `other` Modin DataFrame and apply a function along full axis.
Parameters
----------
axis : {0, 1}
Axis to apply over (0 - rows, 1 - columns).
func : callable
Function to apply.
other : PandasDataframe or list
Modin DataFrame(s) to broadcast.
new_index : list-like, optional
Index of the result. We may know this in advance,
and if not provided it must be computed.
new_columns : list-like, optional
Columns of the result. We may know this in
advance, and if not provided it must be computed.
apply_indices : list-like, optional
Indices of `axis ^ 1` to apply function over.
enumerate_partitions : bool, default: False
Whether pass partition index into applied `func` or not.
Note that `func` must be able to obtain `partition_idx` kwarg.
dtypes : list-like or scalar, optional
Data types of the result. This is an optimization
because there are functions that always result in a particular data
type, and allows us to avoid (re)computing it.
keep_partitioning : boolean, default: True
The flag to keep partition boundaries for Modin Frame if possible.
Setting it to True disables shuffling data from one partition to another in case the resulting
number of splits is equal to the initial number of splits.
num_splits : int, optional
The number of partitions to split the result into across the `axis`. If None, then the number
of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`
then the number of splits is preserved.
sync_labels : boolean, default: True
Synchronize external indexes (`new_index`, `new_columns`) with internal indexes.
This could be used when you're certain that the indices in partitions are equal to
the provided hints in order to save time on syncing them.
pass_axis_lengths_to_partitions : bool, default: False
Whether pass partition lengths along `axis ^ 1` to the kernel `func`.
Note that `func` must be able to obtain `df, *axis_lengths`.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
if other is not None:
if not isinstance(other, list):
other = [other]
other = [o._extract_partitions() for o in other] if len(other) else None
if apply_indices is not None:
numeric_indices = self.get_axis(axis ^ 1).get_indexer_for(apply_indices)
apply_indices = self._get_dict_of_block_index(
axis ^ 1, numeric_indices
).keys()
apply_func_args = None
if pass_axis_lengths_to_partitions:
if axis == 0:
apply_func_args = (
self._column_widths_cache
if self._column_widths_cache is not None
else [part.width(materialize=False) for part in self._partitions[0]]
)
else:
apply_func_args = (
self._row_lengths_cache
if self._row_lengths_cache is not None
else [
part.length(materialize=False) for part in self._partitions.T[0]
]
)
new_partitions = self._partition_mgr_cls.broadcast_axis_partitions(
axis=axis,
left=self._partitions,
right=other,
apply_func=self._build_treereduce_func(axis, func),
apply_indices=apply_indices,
enumerate_partitions=enumerate_partitions,
keep_partitioning=keep_partitioning,
num_splits=num_splits,
apply_func_args=apply_func_args,
)
kw = {"row_lengths": None, "column_widths": None}
if isinstance(dtypes, str) and dtypes == "copy":
kw["dtypes"] = self.copy_dtypes_cache()
elif isinstance(dtypes, DtypesDescriptor):
kw["dtypes"] = ModinDtypes(dtypes)
elif dtypes is not None:
if isinstance(dtypes, (pandas.Series, ModinDtypes)):
kw["dtypes"] = dtypes.copy()
else:
if new_columns is None:
assert not is_list_like(dtypes)
dtype = pandas.api.types.pandas_dtype(dtypes)
kw["dtypes"] = ModinDtypes(DtypesDescriptor(remaining_dtype=dtype))
else:
kw["dtypes"] = (
pandas.Series(dtypes, index=new_columns)
if is_list_like(dtypes)
else pandas.Series(
[pandas.api.types.pandas_dtype(dtypes)] * len(new_columns),
index=new_columns,
)
)
is_index_materialized = ModinIndex.is_materialized_index(new_index)
is_columns_materialized = ModinIndex.is_materialized_index(new_columns)
if axis == 0:
if (
is_columns_materialized
and len(new_partitions.shape) > 1
and new_partitions.shape[1] == 1
):
kw["column_widths"] = [len(new_columns)]
elif axis == 1:
if is_index_materialized and new_partitions.shape[0] == 1:
kw["row_lengths"] = [len(new_index)]
if not keep_partitioning:
if kw["row_lengths"] is None and is_index_materialized:
if axis == 0:
kw["row_lengths"] = get_length_list(
axis_len=len(new_index),
num_splits=new_partitions.shape[0],
min_block_size=MinRowPartitionSize.get(),
)
elif axis == 1:
if self._row_lengths_cache is not None and len(new_index) == sum(
self._row_lengths_cache
):
kw["row_lengths"] = self._row_lengths_cache
if kw["column_widths"] is None and is_columns_materialized:
if axis == 1:
kw["column_widths"] = get_length_list(
axis_len=len(new_columns),
num_splits=new_partitions.shape[1],
min_block_size=MinColumnPartitionSize.get(),
)
elif axis == 0:
if self._column_widths_cache is not None and len(
new_columns
) == sum(self._column_widths_cache):
kw["column_widths"] = self._column_widths_cache
else:
if axis == 0:
if (
kw["row_lengths"] is None
and self._row_lengths_cache is not None
and is_index_materialized
and len(new_index) == sum(self._row_lengths_cache)
# to avoid problems that may arise when filtering empty dataframes
and all(r != 0 for r in self._row_lengths_cache)
):
kw["row_lengths"] = self._row_lengths_cache
elif axis == 1:
if (
kw["column_widths"] is None
and self._column_widths_cache is not None
and is_columns_materialized
and len(new_columns) == sum(self._column_widths_cache)
# to avoid problems that may arise when filtering empty dataframes
and all(w != 0 for w in self._column_widths_cache)
):
kw["column_widths"] = self._column_widths_cache
result = self.__constructor__(
new_partitions,
index=new_index,
columns=new_columns,
**kw,
pandas_backend=self._pandas_backend,
)
if sync_labels and new_index is not None:
result.synchronize_labels(axis=0)
if sync_labels and new_columns is not None:
result.synchronize_labels(axis=1)
return result
def _check_if_axes_identical(self, other: PandasDataframe, axis: int = 0) -> bool:
"""
Check whether indices/partitioning along the specified `axis` are identical when compared with `other`.
Parameters
----------
other : PandasDataframe
Dataframe to compare indices/partitioning with.
axis : int, default: 0
Returns
-------
bool
"""
if self.has_axis_cache(axis) and other.has_axis_cache(axis):
self_cache, other_cache = self._get_axis_cache(axis), other._get_axis_cache(
axis
)
equal_indices = self_cache.equals(other_cache)
if equal_indices:
equal_lengths = self_cache.compare_partition_lengths_if_possible(
other_cache
)
if isinstance(equal_lengths, bool):
return equal_lengths
return self._get_axis_lengths(axis) == other._get_axis_lengths(axis)
return False
return self.get_axis(axis).equals(
other.get_axis(axis)
) and self._get_axis_lengths(axis) == other._get_axis_lengths(axis)
def _copartition(
self, axis, other, how, sort=None, force_repartition=False, fill_value=None
):
"""
Copartition two Modin DataFrames.
Perform aligning of partitions, index and partition blocks.
Parameters
----------
axis : {0, 1}
Axis to copartition along (0 - rows, 1 - columns).
other : PandasDataframe
Other Modin DataFrame(s) to copartition against.
how : str
How to manage joining the index object ("left", "right", etc.).
sort : bool, default: None
Whether sort the joined index or not.
If ``None``, sort is defined in depend on labels equality along the axis.
force_repartition : bool, default: False
Whether force the repartitioning or not. By default,
this method will skip repartitioning if it is possible. This is because
reindexing is extremely inefficient. Because this method is used to
`join` or `append`, it is vital that the internal indices match.
fill_value : any, optional
Value to use for missing values.
Returns
-------
tuple
Tuple containing:
1) 2-d NumPy array of aligned left partitions
2) list of 2-d NumPy arrays of aligned right partitions
3) joined index along ``axis``, may be ``ModinIndex`` if not materialized
4) If materialized, list with sizes of partitions along axis that partitioning
was done on, otherwise ``None``. This list will be empty if and only if all
the frames are empty.
"""
if isinstance(other, type(self)):
other = [other]
if not force_repartition and all(
o._check_if_axes_identical(self, axis) for o in other
):
return (
self._partitions,
[o._partitions for o in other],
self.copy_axis_cache(axis, copy_lengths=True),
self._get_axis_lengths_cache(axis),
)
if sort is None:
sort = not all(self.get_axis(axis).equals(o.get_axis(axis)) for o in other)
self_index = self.get_axis(axis)
others_index = [o.get_axis(axis) for o in other]
joined_index, make_reindexer = self._join_index_objects(
axis, [self_index] + others_index, how, sort, fill_value
)
frames = [self] + other
non_empty_frames_idx = [
i for i, o in enumerate(frames) if o._partitions.size != 0
]
# If all frames are empty
if len(non_empty_frames_idx) == 0:
return (
self._partitions,
[o._partitions for o in other],
joined_index,
# There are no partition sizes because the resulting dataframe
# has no partitions.
[],
)
base_frame_idx = non_empty_frames_idx[0]
other_frames = frames[base_frame_idx + 1 :]
# Picking first non-empty frame
base_frame = frames[non_empty_frames_idx[0]]
base_index = base_frame.get_axis(axis)
# define conditions for reindexing and repartitioning `self` frame
do_reindex_base = not base_index.equals(joined_index)
do_repartition_base = force_repartition or do_reindex_base
# Perform repartitioning and reindexing for `base_frame` if needed.
# Also define length of base and frames. We will need to know the
# lengths for alignment.
if do_repartition_base:
reindexed_base = base_frame._partition_mgr_cls.map_axis_partitions(
axis,
base_frame._partitions,
make_reindexer(do_reindex_base, base_frame_idx),
)
if axis:
base_lengths = [obj.width() for obj in reindexed_base[0]]
else:
base_lengths = [obj.length() for obj in reindexed_base.T[0]]
else:
reindexed_base = base_frame._partitions
base_lengths = base_frame.column_widths if axis else base_frame.row_lengths
others_lengths = [o._get_axis_lengths(axis) for o in other_frames]
# define conditions for reindexing and repartitioning `other` frames
do_reindex_others = [
not o.get_axis(axis).equals(joined_index) for o in other_frames
]
do_repartition_others = [None] * len(other_frames)
for i in range(len(other_frames)):
do_repartition_others[i] = (
force_repartition
or do_reindex_others[i]
or others_lengths[i] != base_lengths
)
# perform repartitioning and reindexing for `other_frames` if needed
reindexed_other_list = [None] * len(other_frames)
for i in range(len(other_frames)):
if do_repartition_others[i]:
# indices of others frame start from `base_frame_idx` + 1
reindexed_other_list[i] = other_frames[
i
]._partition_mgr_cls.map_axis_partitions(
axis,
other_frames[i]._partitions,
make_reindexer(do_repartition_others[i], base_frame_idx + 1 + i),
lengths=base_lengths,
)
else:
reindexed_other_list[i] = other_frames[i]._partitions
reindexed_frames = (
[frames[i]._partitions for i in range(base_frame_idx)]
+ [reindexed_base]
+ reindexed_other_list
)
return (reindexed_frames[0], reindexed_frames[1:], joined_index, base_lengths)
@lazy_metadata_decorator(apply_axis="both")
def n_ary_op(
self,
op,
right_frames: list[PandasDataframe],
join_type="outer",
sort=None,
copartition_along_columns=True,
labels="replace",
dtypes: Optional[pandas.Series] = None,
) -> PandasDataframe:
"""
Perform an n-opary operation by joining with other Modin DataFrame(s).
Parameters
----------
op : callable
Function to apply after the join.
right_frames : list of PandasDataframe
Modin DataFrames to join with.
join_type : str, default: "outer"
Type of join to apply.
sort : bool, default: None
Whether to sort index and columns or not.
copartition_along_columns : bool, default: True
Whether to perform copartitioning along columns or not.
For some ops this isn't needed (e.g., `fillna`).
labels : {"replace", "drop"}, default: "replace"
Whether use labels from joined DataFrame or drop altogether to make
them be computed lazily later.
dtypes : pandas.Series, optional
Dtypes of the resultant dataframe, this argument will be
received if the resultant dtypes of n-opary operation is precomputed.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
left_parts, list_of_right_parts, joined_index, row_lengths = self._copartition(
0,
right_frames,
join_type,
sort=sort,
)
if copartition_along_columns:
new_left_frame = self.__constructor__(
left_parts,
joined_index,
self.copy_columns_cache(copy_lengths=True),
row_lengths,
self._column_widths_cache,
pandas_backend=self._pandas_backend,
)
new_right_frames = [
self.__constructor__(
right_parts,
joined_index,
right_frame.copy_columns_cache(copy_lengths=True),
row_lengths,
right_frame._column_widths_cache,
pandas_backend=self._pandas_backend,
)
for right_parts, right_frame in zip(list_of_right_parts, right_frames)
]
(
left_parts,
list_of_right_parts,
joined_columns,
column_widths,
) = new_left_frame._copartition(
1,
new_right_frames,
join_type,
sort=sort,
)
else:
joined_columns = self.copy_columns_cache(copy_lengths=True)
column_widths = self._column_widths_cache
new_frame = (
np.array([])
if len(left_parts) == 0
or any(len(right_parts) == 0 for right_parts in list_of_right_parts)
else self._partition_mgr_cls.n_ary_operation(
left_parts, op, list_of_right_parts
)
)
if labels == "drop":
joined_index = joined_columns = row_lengths = column_widths = None
return self.__constructor__(
new_frame,
joined_index,
joined_columns,
row_lengths,
column_widths,
dtypes,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def concat(
self,
axis: Union[int, Axis],
others: Union[PandasDataframe, List[PandasDataframe]],
how,
sort,
) -> PandasDataframe:
"""
Concatenate `self` with one or more other Modin DataFrames.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
Axis to concatenate over.
others : list
List of Modin DataFrames to concatenate with.
how : str
Type of join to use for the axis.
sort : bool
Whether sort the result or not.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
axis = Axis(axis)
new_widths = None
new_lengths = None
def _compute_new_widths():
widths = None
if self._column_widths_cache is not None and all(
o._column_widths_cache is not None for o in others
):
widths = self._column_widths_cache + [
width for o in others for width in o._column_widths_cache
]
return widths
# Fast path for equivalent columns and partitioning
if axis == Axis.ROW_WISE and all(
o._check_if_axes_identical(self, axis=1) for o in others
):
joined_index = self.copy_columns_cache(copy_lengths=True)
left_parts = self._partitions
right_parts = [o._partitions for o in others]
new_widths = self._column_widths_cache
elif axis == Axis.COL_WISE and all(
o._check_if_axes_identical(self, axis=0) for o in others
):
joined_index = self.copy_index_cache(copy_lengths=True)
left_parts = self._partitions
right_parts = [o._partitions for o in others]
new_lengths = self._row_lengths_cache
# we can only do this for COL_WISE because `concat` might rebalance partitions for ROW_WISE
new_widths = _compute_new_widths()
else:
(
left_parts,
right_parts,
joined_index,
partition_sizes_along_axis,
) = self._copartition(
axis.value ^ 1, others, how, sort=sort, force_repartition=False
)
if axis == Axis.COL_WISE:
new_lengths = partition_sizes_along_axis
new_widths = _compute_new_widths()
else:
new_widths = partition_sizes_along_axis
new_partitions, new_lengths2 = self._partition_mgr_cls.concat(
axis.value, left_parts, right_parts
)
if new_lengths is None:
new_lengths = new_lengths2
new_dtypes = None
new_index = None
new_columns = None
if axis == Axis.ROW_WISE:
if all(obj.has_materialized_index for obj in (self, *others)):
new_index = self.index.append([other.index for other in others])
new_columns = joined_index
frames = [self] + others
# TODO: should we wrap all `concat` call into "try except" block?
# `ModinDtypes.concat` can throw exception in case of duplicate values
new_dtypes = ModinDtypes.concat([frame._dtypes for frame in frames], axis=1)
# If we have already cached the length of each row in at least one
# of the row's partitions, we can build new_lengths for the new
# frame. Typically, if we know the length for any partition in a
# row, we know the length for the first partition in the row. So
# just check the lengths of the first column of partitions.
if not new_lengths:
new_lengths = []
if new_partitions.size > 0:
if all(
part._length_cache is not None for part in new_partitions.T[0]
):
new_lengths = self._get_lengths(new_partitions.T[0], axis)
else:
new_lengths = None
else:
if all(obj.has_materialized_columns for obj in (self, *others)):
new_columns = self.columns.append([other.columns for other in others])
new_index = joined_index
try:
new_dtypes = ModinDtypes.concat(
[self.copy_dtypes_cache()] + [o.copy_dtypes_cache() for o in others]
)
except NotImplementedError:
new_dtypes = None
# If we have already cached the width of each column in at least one
# of the column's partitions, we can build new_widths for the new
# frame. Typically, if we know the width for any partition in a
# column, we know the width for the first partition in the column.
# So just check the widths of the first row of partitions.
if not new_widths:
new_widths = []
if new_partitions.size > 0:
if all(part._width_cache is not None for part in new_partitions[0]):
new_widths = self._get_lengths(new_partitions[0], axis)
else:
new_widths = None
return self.__constructor__(
new_partitions,
new_index,
new_columns,
new_lengths,
new_widths,
new_dtypes,
pandas_backend=self._pandas_backend,
)
def _apply_func_to_range_partitioning_broadcast(
self,
right,
func,
key,
new_index=None,
new_columns=None,
new_dtypes: Optional[Union[ModinDtypes, pandas.Series]] = None,
):
"""
Apply `func` against two dataframes using range-partitioning implementation.
The method first builds range-partitioning for both dataframes using the data from
`self[key]`, after that, it applies `func` row-wise to `self` frame and
broadcasts row-parts of `right` to `self`.
Parameters
----------
right : PandasDataframe
func : callable(left : pandas.DataFrame, right : pandas.DataFrame) -> pandas.DataFrame
key : list of labels
Columns to use to build range-partitioning. Must present in both dataframes.
new_index : pandas.Index, optional
Index values to write to the result's cache.
new_columns : pandas.Index, optional
Column values to write to the result's cache.
new_dtypes : pandas.Series or ModinDtypes, optional
Dtype values to write to the result's cache.
Returns
-------
PandasDataframe
"""
if self._partitions.shape[0] == 1:
result = self.broadcast_apply_full_axis(
axis=1,
func=func,
new_columns=new_columns,
dtypes=new_dtypes,
other=right,
)
return result
if not isinstance(key, list):
key = [key]
shuffling_functions = ShuffleSortFunctions(
self,
key,
ascending=True,
ideal_num_new_partitions=self._partitions.shape[0],
)
# here we want to get indices of those partitions that hold the key columns
key_indices = self.columns.get_indexer_for(key)
partition_indices = np.unique(
np.digitize(key_indices, np.cumsum(self.column_widths))
)
new_partitions = self._partition_mgr_cls.shuffle_partitions(
self._partitions,
partition_indices,
shuffling_functions,
func,
right_partitions=right._partitions,
)
return self.__constructor__(
new_partitions,
index=new_index,
columns=new_columns,
dtypes=new_dtypes,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def groupby(
self,
axis: Union[int, Axis],
internal_by: List[str],
external_by: List[PandasDataframe],
by_positions: List[int],
operator: Callable,
result_schema: Optional[Dict[Hashable, type]] = None,
align_result_columns: bool = False,
series_groupby: bool = False,
add_missing_cats: bool = False,
**kwargs: dict,
) -> PandasDataframe:
"""
Generate groups based on values in the input column(s) and perform the specified operation on each.
Parameters
----------
axis : int or modin.core.dataframe.base.utils.Axis
The axis to apply the grouping over.
internal_by : list of strings
One or more column labels from the `self` dataframe to use for grouping.
external_by : list of PandasDataframes
PandasDataframes to group by (may be specified along with or without `internal_by`).
by_positions : list of ints
Specifies the order of grouping by `internal_by` and `external_by` columns.
Each element in `by_positions` specifies an index from either `external_by` or `internal_by`.
Indices for `external_by` are positive and start from 0. Indices for `internal_by` are negative
and start from -1 (so in order to convert them to a valid indices one should do ``-idx - 1``).
'''
by_positions = [0, -1, 1, -2, 2, 3]
internal_by = ["col1", "col2"]
external_by = [sr1, sr2, sr3, sr4]
df.groupby([sr1, "col1", sr2, "col2", sr3, sr4])
'''.
operator : callable(pandas.core.groupby.DataFrameGroupBy) -> pandas.DataFrame
The operation to carry out on each of the groups. The operator is another
algebraic operator with its own user-defined function parameter, depending
on the output desired by the user.
result_schema : dict, optional
Mapping from column labels to data types that represents the types of the output dataframe.
align_result_columns : bool, default: False
Whether to manually align columns between all the resulted row partitions.
This flag is helpful when dealing with UDFs as they can change the partition's shape
and labeling unpredictably, resulting in an invalid dataframe.
series_groupby : bool, default: False
Whether to convert a one-column DataFrame to a Series before performing groupby.
add_missing_cats : bool, default: False
Whether to add missing categories from `by` columns to the result.
**kwargs : dict
Additional arguments to pass to the ``df.groupby`` method (besides the 'by' argument).
Returns
-------
PandasDataframe
A new PandasDataframe containing the groupings specified, with the operator
applied to each group.
Notes
-----
No communication between groups is allowed in this algebra implementation.
The number of rows (columns if axis=1) returned by the user-defined function
passed to the groupby may be at most the number of rows in the group, and
may be as small as a single row.
Unlike the pandas API, an intermediate "GROUP BY" object is not present in this
algebra implementation.
"""
axis = Axis(axis)
if axis != Axis.ROW_WISE:
raise NotImplementedError(
f"Algebra groupby only implemented row-wise. {axis.name} axis groupby not implemented yet!"
)
has_external_grouper = len(external_by) > 0
skip_on_aligning_flag = "__skip_me_on_aligning__"
duplicated_suffix = "__duplicated_suffix__"
duplicated_pattern = r"_[\d]*__duplicated_suffix__"
kwargs["observed"] = True
level = kwargs.get("level")
if level is not None and not isinstance(level, list):
level = [level]
def apply_func(df): # pragma: no cover
if has_external_grouper:
external_grouper = df["grouper"]
external_grouper = [
# `df.groupby()` can only take a list of Series'es, so splitting
# the df into a list of individual Series'es
external_grouper.iloc[:, i]
for i in range(len(external_grouper.columns))
]
# renaming 'None' and duplicated names back to their original names
for obj in external_grouper:
if not isinstance(obj, pandas.Series):
continue
name = obj.name
if isinstance(name, str):
if name.startswith(MODIN_UNNAMED_SERIES_LABEL):
name = None
elif name.endswith(duplicated_suffix):
name = re.sub(duplicated_pattern, "", name)
elif isinstance(name, tuple):
if name[-1].endswith(duplicated_suffix):
name = (
*name[:-1],
re.sub(duplicated_pattern, "", name[-1]),
)
obj.name = name
df = df["data"]
else:
external_grouper = []
by = []
# restoring original order of 'by' columns
for idx in by_positions:
if idx >= 0:
by.append(external_grouper[idx])
else:
by.append(internal_by[-idx - 1])
if series_groupby:
df = df.squeeze(axis=1)
if kwargs.get("level") is not None:
assert len(by) == 0
# passing an empty list triggers an error
by = None
result = operator(df.groupby(by, **kwargs))
if align_result_columns and df.empty and result.empty:
# We want to align columns only of those frames that actually performed
# some groupby aggregation, if an empty frame was originally passed
# (an empty bin on reshuffling was created) then there were no groupby
# executed over this partition and so it has incorrect columns
# that shouldn't be considered on the aligning phase
result.attrs[skip_on_aligning_flag] = True
return result
if has_external_grouper:
grouper = (
external_by[0]
if len(external_by) == 1
else external_by[0].concat(
axis=1, others=external_by[1:], how="left", sort=False
)
)
new_grouper_cols = []
columns_were_changed = False
same_columns = {}
# duplicated names break range-partitioning mechanism, so renaming them.
# original names will be reverted in the actual groupby kernel
for col in grouper.columns:
suffix = same_columns.get(col)
if suffix is None:
same_columns[col] = 0
else:
same_columns[col] += 1
col = (
(*col[:-1], f"{col[-1]}_{suffix}{duplicated_suffix}")
if isinstance(col, tuple)
else f"{col}_{suffix}{duplicated_suffix}"
)
columns_were_changed = True
new_grouper_cols.append(col)
if columns_were_changed:
grouper.columns = pandas.Index(new_grouper_cols)
grouper_key_columns = grouper.columns
data = self
data_key_columns = internal_by
else:
grouper = self
grouper_key_columns = internal_by
data, data_key_columns = None, None
result = grouper._apply_func_to_range_partitioning(
key_columns=grouper_key_columns,
func=apply_func,
data=data,
data_key_columns=data_key_columns,
level=level,
)
# no need aligning columns if there's only one row partition
if add_missing_cats or align_result_columns and result._partitions.shape[0] > 1:
# FIXME: the current reshuffling implementation guarantees us that there's only one column
# partition in the result, so we should never hit this exception for now, however
# in the future, we might want to make this implementation more broader
if result._partitions.shape[1] > 1:
raise NotImplementedError(
"Aligning columns is not yet implemented for multiple column partitions."
)
# There're two implementations:
# 1. The first one work faster, but may stress the network a lot in cluster mode since
# it gathers all the dataframes in a single ray-kernel.
# 2. The second one works slower, but only gathers light pandas.Index objects,
# so there should be less stress on the network.
if add_missing_cats or not IsRayCluster.get():
if self.has_materialized_dtypes:
original_dtypes = pandas.Series(
{
# lazy proxies hold a reference to another modin's DataFrame which can be
# a problem during serialization, in this scenario we don't need actual
# categorical values, so a "category" string will be enough
name: (
"category"
if isinstance(dtype, LazyProxyCategoricalDtype)
else dtype
)
for name, dtype in self.dtypes.items()
}
)
else:
original_dtypes = None
def compute_aligned_columns(*dfs, initial_columns=None, by=None):
"""Take row partitions, filter empty ones, and return joined columns for them."""
if align_result_columns:
valid_dfs = [
df
for df in dfs
if not df.attrs.get(skip_on_aligning_flag, False)
]
if len(valid_dfs) == 0 and len(dfs) != 0:
valid_dfs = dfs
# Using '.concat()' on empty-slices instead of 'Index.join()'
# in order to get identical behavior to pandas when it joins
# results of different groups
combined_cols = pandas.concat(
[df.iloc[:0] for df in valid_dfs], axis=0, join="outer"
).columns
else:
combined_cols = dfs[0].columns
masks = None
if add_missing_cats:
masks, combined_cols = add_missing_categories_to_groupby(
dfs,
by,
operator,
initial_columns,
combined_cols,
is_udf_agg=align_result_columns,
kwargs=kwargs.copy(),
initial_dtypes=original_dtypes,
)
return (
(combined_cols, masks)
if align_result_columns
else (None, masks)
)
external_by_cols = [
None if col.startswith(MODIN_UNNAMED_SERIES_LABEL) else col
for obj in external_by
for col in obj.columns
]
by = []
# restoring original order of 'by' columns
for idx in by_positions:
if idx >= 0:
by.append(external_by_cols[idx])
else:
by.append(internal_by[-idx - 1])
# Passing all partitions to the 'compute_aligned_columns' kernel to get
# aligned columns
parts = result._partitions.flatten()
aligned_columns = parts[0].apply(
compute_aligned_columns,
*[part._data for part in parts[1:]],
initial_columns=pandas.Index(external_by_cols).append(self.columns),
by=by,
)
def apply_aligned(df, args, partition_idx):
combined_cols, mask = args
if mask is not None and mask.get(partition_idx) is not None:
values = mask[partition_idx]
original_names = df.index.names
# TODO: inserting 'values' based on 'searchsorted' result might be more efficient
# in cases of small amount of 'values'
df = pandas.concat([df, values])
if kwargs["sort"]:
df = df.sort_index(axis=0)
df.index.names = original_names
if combined_cols is not None:
df = df.reindex(columns=combined_cols)
return df
# Lazily applying aligned columns to partitions
new_partitions = self._partition_mgr_cls.lazy_map_partitions(
result._partitions,
apply_aligned,
func_args=(aligned_columns._data,),
enumerate_partitions=True,
)
else:
def join_cols(df, *cols):
"""Join `cols` and apply the joined columns to `df`."""
valid_cols = [
pandas.DataFrame(columns=col) for col in cols if col is not None
]
if len(valid_cols) == 0:
return df
# Using '.concat()' on empty-slices instead of 'Index.join()'
# in order to get identical behavior to pandas when it joins
# results of different groups
result_col = pandas.concat(valid_cols, axis=0, join="outer").columns
return df.reindex(columns=result_col)
# Getting futures for columns of non-empty partitions
cols = [
part.apply(
lambda df: (
None
if df.attrs.get(skip_on_aligning_flag, False)
else df.columns
)
)._data
for part in result._partitions.flatten()
]
# Lazily joining and applying the aligned columns
new_partitions = self._partition_mgr_cls.lazy_map_partitions(
result._partitions,
join_cols,
func_args=cols,
)
result = self.__constructor__(
new_partitions,
index=result.copy_index_cache(),
row_lengths=result._row_lengths_cache,
pandas_backend=self._pandas_backend,
)
if (
not result.has_materialized_index
and not has_external_grouper
and level is None
):
by_dtypes = ModinDtypes(self._dtypes).lazy_get(internal_by)
if by_dtypes.is_materialized:
new_index = ModinIndex(value=result, axis=0, dtypes=by_dtypes)
result.set_index_cache(new_index)
if result_schema is not None:
new_dtypes = pandas.Series(result_schema)
result.set_dtypes_cache(new_dtypes)
result.set_columns_cache(new_dtypes.index)
return result
@lazy_metadata_decorator(apply_axis="both")
def groupby_reduce(
self,
axis,
by,
map_func,
reduce_func,
new_index=None,
new_columns=None,
apply_indices=None,
):
"""
Groupby another Modin DataFrame dataframe and aggregate the result.
Parameters
----------
axis : {0, 1}
Axis to groupby and aggregate over.
by : PandasDataframe or None
A Modin DataFrame to group by.
map_func : callable
Map component of the aggregation.
reduce_func : callable
Reduce component of the aggregation.
new_index : pandas.Index, optional
Index of the result. We may know this in advance,
and if not provided it must be computed.
new_columns : pandas.Index, optional
Columns of the result. We may know this in advance,
and if not provided it must be computed.
apply_indices : list-like, optional
Indices of `axis ^ 1` to apply groupby over.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
by_parts = by if by is None else by._partitions
if by is None:
self._propagate_index_objs(axis=0)
if apply_indices is not None:
numeric_indices = self.get_axis(axis ^ 1).get_indexer_for(apply_indices)
apply_indices = list(
self._get_dict_of_block_index(axis ^ 1, numeric_indices).keys()
)
if by_parts is not None:
# inplace operation
if by_parts.shape[axis] != self._partitions.shape[axis]:
self._filter_empties(compute_metadata=False)
new_partitions = self._partition_mgr_cls.groupby_reduce(
axis, self._partitions, by_parts, map_func, reduce_func, apply_indices
)
return self.__constructor__(
new_partitions,
index=new_index,
columns=new_columns,
pandas_backend=self._pandas_backend,
)
@classmethod
def from_pandas(cls, df):
"""
Create a Modin DataFrame from a pandas DataFrame.
Parameters
----------
df : pandas.DataFrame
A pandas DataFrame.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
new_index = df.index
new_columns = df.columns
new_dtypes = df.dtypes
new_frame, pandas_backend, new_lengths, new_widths = (
cls._partition_mgr_cls.from_pandas(df, True)
)
return cls(
new_frame,
new_index,
new_columns,
new_lengths,
new_widths,
dtypes=new_dtypes,
pandas_backend=pandas_backend,
)
@classmethod
def from_arrow(cls, at):
"""
Create a Modin DataFrame from an Arrow Table.
Parameters
----------
at : pyarrow.table
Arrow Table.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
new_frame, pandas_backend, new_lengths, new_widths = (
cls._partition_mgr_cls.from_arrow(at, return_dims=True)
)
new_columns = Index.__new__(Index, data=at.column_names, dtype="O")
new_index = Index.__new__(RangeIndex, data=range(at.num_rows))
new_dtypes = pandas.Series(
[cls._arrow_type_to_dtype(col.type) for col in at.columns],
index=at.column_names,
)
return cls(
partitions=new_frame,
index=new_index,
columns=new_columns,
row_lengths=new_lengths,
column_widths=new_widths,
dtypes=new_dtypes,
pandas_backend=pandas_backend,
)
@classmethod
def _arrow_type_to_dtype(cls, arrow_type):
"""
Convert an arrow data type to a pandas data type.
Parameters
----------
arrow_type : arrow dtype
Arrow data type to be converted to a pandas data type.
Returns
-------
object
Any dtype compatible with pandas.
"""
import pyarrow
try:
# TODO: should we map arrow types to pyarrow-backed pandas types?
# It seems like this might help avoid the expense of transferring
# data between backends (numpy and pyarrow), but we need to be sure
# how this fits into the type inference system in pandas.
res = arrow_type.to_pandas_dtype()
# Conversion to pandas is not implemented for some arrow types,
# perform manual conversion for them:
except NotImplementedError:
if pyarrow.types.is_time(arrow_type):
res = np.dtype(datetime.time)
else:
raise
if not isinstance(res, (np.dtype, str)):
return np.dtype(res)
return res
@lazy_metadata_decorator(apply_axis="both")
def to_pandas(self):
"""
Convert this Modin DataFrame to a pandas DataFrame.
Returns
-------
pandas.DataFrame
"""
df = self._partition_mgr_cls.to_pandas(self._partitions)
if df.empty:
df = pandas.DataFrame(columns=self.columns, index=self.index)
if len(df.columns) and self.has_materialized_dtypes:
df = df.astype(self.dtypes)
else:
for axis, has_external_index in enumerate(
["has_materialized_index", "has_materialized_columns"]
):
# no need to check external and internal axes since in that case
# external axes will be computed from internal partitions
if getattr(self, has_external_index):
external_index = self.columns if axis else self.index
ErrorMessage.catch_bugs_and_request_email(
not df.axes[axis].equals(external_index),
f"Internal and external indices on axis {axis} do not match.",
)
# have to do this in order to assign some potentially missing metadata,
# the ones that were set to the external index but were never propagated
# into the internal ones
df = df.set_axis(axis=axis, labels=external_index, copy=False)
return df
def to_numpy(self, **kwargs):
"""
Convert this Modin DataFrame to a NumPy array.
Parameters
----------
**kwargs : dict
Additional keyword arguments to be passed in `to_numpy`.
Returns
-------
np.ndarray
"""
arr = self._partition_mgr_cls.to_numpy(self._partitions, **kwargs)
ErrorMessage.catch_bugs_and_request_email(
self.has_materialized_index
and len(arr) != len(self.index)
or self.has_materialized_columns
and len(arr[0]) != len(self.columns)
)
return arr
@lazy_metadata_decorator(apply_axis=None, transpose=True)
def transpose(self):
"""
Transpose the index and columns of this Modin DataFrame.
Reflect this Modin DataFrame over its main diagonal
by writing rows as columns and vice-versa.
Returns
-------
PandasDataframe
New Modin DataFrame.
"""
new_partitions = self._partition_mgr_cls.lazy_map_partitions(
self._partitions, lambda df: df.T
).T
if self.has_materialized_dtypes:
new_dtypes = pandas.Series(
np.full(len(self.index), find_common_type(self.dtypes.values)),
index=self.index,
)
else:
new_dtypes = None
return self.__constructor__(
new_partitions,
self.copy_columns_cache(copy_lengths=True),
self.copy_index_cache(copy_lengths=True),
self._column_widths_cache,
self._row_lengths_cache,
dtypes=new_dtypes,
pandas_backend=self._pandas_backend,
)
@lazy_metadata_decorator(apply_axis="both")
def finalize(self):
"""
Perform all deferred calls on partitions.
This makes `self` Modin Dataframe independent of a history of queries
that were used to build it.
"""
self._partition_mgr_cls.finalize(self._partitions)
def wait_computations(self):
"""Wait for all computations to complete without materializing data."""
self._partition_mgr_cls.wait_partitions(self._partitions.flatten())
def support_materialization_in_worker_process(self) -> bool:
"""
Whether it's possible to call function `to_pandas` during the pickling process, at the moment of recreating the object.
Returns
-------
bool
"""
return True
def __dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True):
"""
Get a Modin DataFrame that implements the dataframe exchange protocol.
See more about the protocol in https://data-apis.org/dataframe-protocol/latest/index.html.
Parameters
----------
nan_as_null : bool, default: False
A keyword intended for the consumer to tell the producer
to overwrite null values in the data with ``NaN`` (or ``NaT``).
This currently has no effect; once support for nullable extension
dtypes is added, this value should be propagated to columns.
allow_copy : bool, default: True
A keyword that defines whether or not the library is allowed
to make a copy of the data. For example, copying data would be necessary
if a library supports strided buffers, given that this protocol
specifies contiguous buffers. Currently, if the flag is set to ``False``
and a copy is needed, a ``RuntimeError`` will be raised.
Returns
-------
ProtocolDataframe
A dataframe object following the dataframe protocol specification.
"""
from modin.core.dataframe.pandas.interchange.dataframe_protocol.dataframe import (
PandasProtocolDataframe,
)
return PandasProtocolDataframe(
self, nan_as_null=nan_as_null, allow_copy=allow_copy
)
@classmethod
def from_interchange_dataframe(cls, df: ProtocolDataframe) -> PandasDataframe:
"""
Convert a DataFrame implementing the dataframe exchange protocol to a Core Modin Dataframe.
See more about the protocol in https://data-apis.org/dataframe-protocol/latest/index.html.
Parameters
----------
df : ProtocolDataframe
The DataFrame object supporting the dataframe exchange protocol.
Returns
-------
PandasDataframe
A new Core Modin Dataframe object.
"""
if type(df) is cls:
return df
if not hasattr(df, "__dataframe__"):
raise ValueError(
"`df` does not support DataFrame exchange protocol, i.e. `__dataframe__` method"
)
from modin.core.dataframe.pandas.interchange.dataframe_protocol.from_dataframe import (
from_dataframe_to_pandas,
)
ErrorMessage.default_to_pandas(message="`from_dataframe`")
pandas_df = from_dataframe_to_pandas(df)
return cls.from_pandas(pandas_df)
def case_when(self, caselist):
"""
Replace values where the conditions are True.
This is Series.case_when() implementation and, thus, it's designed to work
only with single-column DataFrames.
Parameters
----------
caselist : list of tuples
Returns
-------
PandasDataframe
"""
# The import is here to avoid an incorrect module initialization when running tests.
# This module is loaded before `pytest_configure()` is called. If `pytest_configure()`
# changes the engine, the `remote_function` decorator will not be valid.
from modin.core.execution.utils import remote_function
@remote_function
def remote_fn(df, name, caselist): # pragma: no cover
caselist = [
tuple(
(
data.squeeze(axis=1)
if isinstance(data, pandas.DataFrame)
else data
)
for data in case_tuple
)
for case_tuple in caselist
]
return pandas.DataFrame({name: df.squeeze(axis=1).case_when(caselist)})
cls = type(self)
use_map = True
is_trivial_idx = None
name = self.columns[0]
# Lists of modin frames: first for conditions, second for replacements
modin_lists = [[], []]
# Fill values for conditions and replacements respectively
fill_values = [True, None]
new_caselist = []
for case_tuple in caselist:
new_case = []
for data, modin_list, fill_value in zip(
case_tuple, modin_lists, fill_values
):
if isinstance(data, cls):
modin_list.append(data)
elif callable(data):
data = remote_function(data)
elif isinstance(data, pandas.Series):
use_map = False
if is_trivial_idx is None:
self_idx = self.index
length = len(self_idx)
is_trivial_idx = is_trivial_index(self_idx)
if is_trivial_idx and is_trivial_index(data.index):
data = data[:length]
diff = length - len(data)
if diff > 0:
data = pandas.concat(
[data, pandas.Series([fill_value] * diff)],
ignore_index=True,
)
else:
data = data.reindex(self_idx, fill_value=fill_value)
elif use_map and is_list_like(data):
use_map = False
new_case.append(data)
new_caselist.append(tuple(new_case))
if modin_lists[0] or modin_lists[1]:
# Copartition modin frames
use_map = False
columns = self.columns
column_widths = [1]
for modin_list, fill_value in zip(modin_lists, fill_values):
_, list_of_right_parts, joined_index, row_lengths = self._copartition(
Axis.ROW_WISE.value,
modin_list,
how="left",
sort=False,
fill_value=fill_value,
)
modin_list.clear()
modin_list.extend(
self.__constructor__(
part,
joined_index,
columns,
row_lengths,
column_widths,
pandas_backend=self._pandas_backend,
)
for part in list_of_right_parts
)
# Replace modin frames with copartitioned
caselist = new_caselist
new_caselist = []
for i in range(2):
modin_lists[i] = iter(modin_lists[i])
for case_tuple in caselist:
new_case = tuple(
next(modin_list) if isinstance(data, cls) else data
for data, modin_list in zip(case_tuple, modin_lists)
)
new_caselist.append(new_case)
# If all the conditions are callable and the replacements are either
# callable or scalar, use map().
if use_map:
return self.map(func=remote_fn, func_args=[name, new_caselist], lazy=True)
# Get the chunk of data corresponding the the specified partition
def map_data(
part_idx,
part_len,
data,
data_offset,
fill_value,
):
if isinstance(data, cls):
return data._partitions[part_idx][0]._data
if isinstance(data, pandas.Series):
return data[data_offset : data_offset + part_len]
return (
data[data_offset : data_offset + part_len]
if is_list_like(data)
else data
)
parts = [p[0] for p in self._partitions]
lengths = self.row_lengths
new_parts = []
data_offset = 0
# Split the data and apply the remote function to each partition
# with the corresponding chunk of data
for i, part, part_len in zip(range(len(parts)), parts, lengths):
cases = [
tuple(
map_data(i, part_len, data, data_offset, fill_value)
for data, fill_value in zip(c, (True, None))
)
for c in new_caselist
]
new_parts.append(
part.add_to_apply_calls(
remote_fn,
name,
cases,
length=part_len,
width=1,
)
)
data_offset += part_len
new_parts = np.array([[p] for p in new_parts])
return self.__constructor__(
new_parts,
columns=self.columns,
index=self.index,
row_lengths=lengths,
column_widths=[1],
pandas_backend=self._pandas_backend,
)
================================================
FILE: modin/core/dataframe/pandas/dataframe/utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Collection of algebra utility functions, used to shuffle data across partitions."""
import abc
from collections import namedtuple
from typing import TYPE_CHECKING, Callable, Optional, Union
import numpy as np
import pandas
from pandas._libs.tslibs import to_offset
from pandas.core.dtypes.common import is_list_like, is_numeric_dtype
from pandas.core.resample import _get_timestamp_range_edges
from modin.error_message import ErrorMessage
from modin.utils import _inherit_docstrings
if TYPE_CHECKING:
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
ColumnInfo = namedtuple("ColumnInfo", ["name", "pivots", "is_numeric"])
class ShuffleFunctions:
"""
Defines an interface to perform the sampling, quantiles picking, and the splitting stages for the range-partitioning building.
Parameters
----------
modin_frame : PandasDataframe
The frame to build the range-partitioning for.
columns : str or list of strings
The column/columns to use as a key.
ascending : bool
Whether the ranges should be in ascending or descending order.
ideal_num_new_partitions : int
The ideal number of new partitions.
**kwargs : dict
Additional keyword arguments.
"""
def __init__(
self, modin_frame, columns, ascending, ideal_num_new_partitions, **kwargs
):
pass
@abc.abstractmethod
def sample_fn(self, partition: pandas.DataFrame) -> pandas.DataFrame:
"""
Pick samples over the given partition.
Parameters
----------
partition : pandas.DataFrame
Returns
-------
pandas.DataFrame:
The samples for the partition.
"""
pass
@abc.abstractmethod
def pivot_fn(self, samples: "list[pandas.DataFrame]") -> int:
"""
Determine quantiles from the given samples and save it for the future ``.split_fn()`` calls.
Parameters
----------
samples : list of pandas.DataFrames
Returns
-------
int
The number of bins the ``.split_fn()`` will return.
"""
pass
@abc.abstractmethod
def split_fn(self, partition: pandas.DataFrame) -> "tuple[pandas.DataFrame, ...]":
"""
Split the given dataframe into the range-partitions defined by the preceding call of the ``.pivot_fn()``.
Parameters
----------
partition : pandas.DataFrame
Returns
-------
tuple of pandas.DataFrames
Notes
-----
In order to call this method you must call the ``.pivot_fn()`` first.
"""
pass
@_inherit_docstrings(ShuffleFunctions)
class ShuffleSortFunctions(ShuffleFunctions):
"""
Perform the sampling, quantiles picking, and the splitting stages for the range-partitioning building.
Parameters
----------
modin_frame : PandasDataframe
The frame to build the range-partitioning for.
columns : str, list of strings or None
The column/columns to use as a key. Can't be specified along with `level`.
ascending : bool
Whether the ranges should be in ascending or descending order.
ideal_num_new_partitions : int
The ideal number of new partitions.
level : list of strings or ints, or None
Index level(s) to use as a key. Can't be specified along with `columns`.
closed_on_right : bool, default: False
Whether to include the right limit in range-partitioning.
True: bins[i - 1] < x <= bins[i]
False: bins[i - 1] <= x < bins[i]
**kwargs : dict
Additional keyword arguments.
"""
def __init__(
self,
modin_frame: "PandasDataframe",
columns: Optional[Union[str, list]],
ascending: Union[list, bool],
ideal_num_new_partitions: int,
level: Optional[list[Union[str, int]]] = None,
closed_on_right: bool = False,
**kwargs: dict,
):
self.frame_len = len(modin_frame)
self.ideal_num_new_partitions = ideal_num_new_partitions
self.columns = columns if is_list_like(columns) else [columns]
self.ascending = ascending
self.kwargs = kwargs.copy()
self.level = level
self.columns_info = None
self.closed_on_right = closed_on_right
def sample_fn(self, partition: pandas.DataFrame) -> pandas.DataFrame:
if self.level is not None:
partition = self._index_to_df_zero_copy(partition, self.level)
else:
partition = partition[self.columns]
return self.pick_samples_for_quantiles(
partition, self.ideal_num_new_partitions, self.frame_len
)
def pivot_fn(self, samples: "list[pandas.DataFrame]") -> int:
key = self.kwargs.get("key", None)
samples = pandas.concat(samples, axis=0, copy=False)
columns_info: "list[ColumnInfo]" = []
number_of_groups = 1
cols = []
for i, col in enumerate(samples.columns):
num_pivots = int(self.ideal_num_new_partitions / number_of_groups)
if num_pivots < 2 and len(columns_info):
break
column_val = samples[col]
cols.append(col)
is_numeric = is_numeric_dtype(column_val.dtype)
# When we are not sorting numbers, we need our quantiles to not do arithmetic on the values
method = "linear" if is_numeric else "inverted_cdf"
pivots = self.pick_pivots_from_samples_for_sort(
column_val, num_pivots, method, key
)
columns_info.append(
ColumnInfo(
self.level[i] if self.level is not None else col,
pivots,
is_numeric,
)
)
number_of_groups *= len(pivots) + 1
self.columns_info = columns_info
return number_of_groups
def split_fn(
self,
partition: pandas.DataFrame,
) -> "tuple[pandas.DataFrame, ...]":
ErrorMessage.catch_bugs_and_request_email(
failure_condition=self.columns_info is None,
extra_log="The 'split_fn' doesn't have proper metadata, the probable reason is that it was called before 'pivot_fn'",
)
return self.split_partitions_using_pivots_for_sort(
partition,
self.columns_info,
self.ascending,
keys_are_index_levels=self.level is not None,
closed_on_right=self.closed_on_right,
**self.kwargs,
)
@staticmethod
def _find_quantiles(
df: Union[pandas.DataFrame, pandas.Series], quantiles: list, method: str
) -> np.ndarray:
"""
Find quantiles of a given dataframe using the specified method.
We use this method to provide backwards compatibility with NumPy versions < 1.23 (e.g. when
the user is using Modin in compat mode). This is basically a wrapper around `np.quantile` that
ensures we provide the correct `method` argument - i.e. if we are dealing with objects (which
may or may not support algebra), we do not want to use a method to find quantiles that will
involve algebra operations (e.g. mean) between the objects, since that may fail.
Parameters
----------
df : pandas.DataFrame or pandas.Series
The data to pick quantiles from.
quantiles : list[float]
The quantiles to compute.
method : str
The method to use. `linear` if dealing with numeric types, otherwise `inverted_cdf`.
Returns
-------
np.ndarray
A NumPy array with the quantiles of the data.
"""
if method == "linear":
# This is the default method for finding quantiles, so it does not need to be specified,
# which keeps backwards compatibility with older versions of NumPy that do not have a
# `method` keyword argument in np.quantile.
return np.unique(np.quantile(df, quantiles))
else:
try:
return np.unique(np.quantile(df, quantiles, method=method))
except Exception:
# In this case, we're dealing with an array of objects, but the current version of
# NumPy does not have a `method` kwarg. We need to use the older kwarg, `interpolation`
# instead.
return np.unique(np.quantile(df, quantiles, interpolation="lower"))
@staticmethod
def pick_samples_for_quantiles(
df: pandas.DataFrame,
num_partitions: int,
length: int,
) -> pandas.DataFrame:
"""
Pick samples over the given partition.
This function picks samples from the given partition using the TeraSort algorithm - each
value is sampled with probability 1 / m * ln(n * t) where m = total_length / num_partitions,
t = num_partitions, and n = total_length.
Parameters
----------
df : pandas.Dataframe
The masked dataframe to pick samples from.
num_partitions : int
The number of partitions.
length : int
The total length.
Returns
-------
pandas.DataFrame:
The samples for the partition.
Notes
-----
This sampling algorithm is inspired by TeraSort. You can find more information about TeraSort
and the sampling algorithm at https://www.cse.cuhk.edu.hk/~taoyf/paper/sigmod13-mr.pdf.
"""
m = length / num_partitions
probability = (1 / m) * np.log(num_partitions * length)
return df.sample(frac=probability)
def pick_pivots_from_samples_for_sort(
self,
samples: pandas.Series,
ideal_num_new_partitions: int,
method: str = "linear",
key: Optional[Callable] = None,
) -> np.ndarray:
"""
Determine quantiles from the given samples.
This function takes as input the quantiles calculated over all partitions from
`sample_func` defined above, and determines a final NPartitions.get() quantiles
to use to roughly sort the entire dataframe. It does so by collating all the samples
and computing NPartitions.get() quantiles for the overall set.
Parameters
----------
samples : pandas.Series
The samples computed by ``get_partition_quantiles_for_sort``.
ideal_num_new_partitions : int
The ideal number of new partitions.
method : str, default: linear
The method to use when picking quantiles.
key : Callable, default: None
The key to use on the samples when picking pivots.
Returns
-------
np.ndarray
A list of overall quantiles.
"""
samples = samples.to_numpy()
# We don't call `np.unique` on the samples, since if a quantile shows up in multiple
# partition's samples, this is probably an indicator of skew in the dataset, and we
# want our final partitions to take this into account.
if key is not None:
samples = key(samples)
# We don't want to pick very many quantiles if we have a very small dataframe.
num_quantiles = ideal_num_new_partitions
quantiles = [i / num_quantiles for i in range(1, num_quantiles)]
# If we only desire 1 partition, we need to ensure that we're not trying to find quantiles
# from an empty list of pivots.
if len(quantiles) > 0:
return self._find_quantiles(samples, quantiles, method)
return np.array([])
@staticmethod
def split_partitions_using_pivots_for_sort(
df: pandas.DataFrame,
columns_info: "list[ColumnInfo]",
ascending: bool,
keys_are_index_levels: bool = False,
closed_on_right: bool = False,
**kwargs: dict,
) -> "tuple[pandas.DataFrame, ...]":
"""
Split the given dataframe into the partitions specified by `pivots` in `columns_info`.
This function takes as input a row-axis partition, as well as the quantiles determined
by the `pivot_func` defined above. It then splits the input dataframe into NPartitions.get()
dataframes, with the elements in the i-th split belonging to the i-th partition, as determined
by the quantiles we're using.
Parameters
----------
df : pandas.Dataframe
The partition to split.
columns_info : list of ColumnInfo
Information regarding keys and pivots for range partitioning.
ascending : bool
The ascending flag.
keys_are_index_levels : bool, default: False
Whether `columns_info` describes index levels or actual columns from `df`.
closed_on_right : bool, default: False
Whether to include the right limit in range-partitioning.
True: bins[i - 1] < x <= bins[i]
False: bins[i - 1] <= x < bins[i]
**kwargs : dict
Additional keyword arguments.
Returns
-------
tuple[pandas.DataFrame]
A tuple of the splits from this partition.
"""
if len(columns_info) == 0:
# We can return the dataframe with zero changes if there were no pivots passed
return (df,)
key_data = (
ShuffleSortFunctions._index_to_df_zero_copy(
df, [col_info.name for col_info in columns_info]
)
if keys_are_index_levels
else df[[col_info.name for col_info in columns_info]]
)
na_index = key_data.isna().squeeze(axis=1)
if na_index.ndim == 2:
na_index = na_index.any(axis=1)
na_rows = df[na_index]
non_na_rows = df[~na_index]
def get_group(grp, key, df):
"""Get a group with the `key` from the `grp`, if it doesn't exist return an empty slice of `df`."""
try:
return grp.get_group(key)
except KeyError:
return pandas.DataFrame(index=df.index[:0], columns=df.columns).astype(
df.dtypes
)
groupby_codes = []
group_keys = []
for col_info in columns_info:
pivots = col_info.pivots
if len(pivots) == 0:
continue
# If `ascending=False` and we are dealing with a numeric dtype, we can pass in a reversed list
# of pivots, and `np.digitize` will work correctly. For object dtypes, we use `np.searchsorted`
# which breaks when we reverse the pivots.
if not ascending and col_info.is_numeric:
# `key` is already applied to `pivots` in the `pick_pivots_from_samples_for_sort` function.
pivots = pivots[::-1]
group_keys.append(range(len(pivots) + 1))
key = kwargs.pop("key", None)
cols_to_digitize = (
non_na_rows.index.get_level_values(col_info.name)
if keys_are_index_levels
else non_na_rows[col_info.name]
)
if key is not None:
cols_to_digitize = key(cols_to_digitize)
if cols_to_digitize.ndim == 2:
cols_to_digitize = cols_to_digitize.squeeze()
if col_info.is_numeric:
groupby_col = np.digitize(
cols_to_digitize, pivots, right=closed_on_right
)
# `np.digitize` returns results based off of the sort order of the pivots it is passed.
# When we only have one unique value in our pivots, `np.digitize` assumes that the pivots
# are sorted in ascending order, and gives us results based off of that assumption - so if
# we actually want to sort in descending order, we need to swap the new indices.
if not ascending and len(np.unique(pivots)) == 1:
groupby_col = len(pivots) - groupby_col
else:
groupby_col = np.searchsorted(
pivots,
cols_to_digitize,
side="left" if closed_on_right else "right",
)
# Since np.searchsorted requires the pivots to be in ascending order, if we want to sort
# in descending order, we need to swap the new indices.
if not ascending:
groupby_col = len(pivots) - groupby_col
groupby_codes.append(groupby_col)
if len(group_keys) == 0:
# We can return the dataframe with zero changes if there were no pivots passed
return (df,)
elif len(group_keys) == 1:
group_keys = group_keys[0]
else:
group_keys = pandas.MultiIndex.from_product(group_keys)
if len(non_na_rows) == 1:
groups = [
# taking an empty slice for an index's metadata
(
pandas.DataFrame(index=df.index[:0], columns=df.columns).astype(
df.dtypes
)
if key != groupby_codes[0]
else non_na_rows
)
for key in group_keys
]
else:
grouped = non_na_rows.groupby(groupby_codes)
groups = [get_group(grouped, key, df) for key in group_keys]
index_to_insert_na_vals = (
-1 if kwargs.get("na_position", "last") == "last" else 0
)
groups[index_to_insert_na_vals] = pandas.concat(
[groups[index_to_insert_na_vals], na_rows]
).astype(df.dtypes)
return tuple(groups)
@staticmethod
def _index_to_df_zero_copy(
df: pandas.DataFrame, levels: list[Union[str, int]]
) -> pandas.DataFrame:
"""
Convert index `level` of `df` to a ``pandas.DataFrame``.
Parameters
----------
df : pandas.DataFrame
levels : list of labels or ints
Index level to convert to a dataframe.
Returns
-------
pandas.DataFrame
The columns in the resulting dataframe use the same data arrays as the index levels
in the original `df`, so no copies.
"""
# calling 'df.index.to_frame()' creates a copy of the index, so doing the conversion manually
# to avoid the copy
data = {
(
df.index.names[lvl] if isinstance(lvl, int) else lvl
): df.index.get_level_values(lvl)
for lvl in levels
}
index_data = pandas.DataFrame(data, index=df.index, copy=False)
return index_data
@_inherit_docstrings(ShuffleSortFunctions)
class ShuffleResample(ShuffleSortFunctions):
def __init__(
self,
modin_frame: "PandasDataframe",
columns: Union[str, list],
ascending: Union[list, bool],
ideal_num_new_partitions: int,
resample_kwargs: dict,
**kwargs: dict,
):
resample_kwargs = resample_kwargs.copy()
rule = resample_kwargs.pop("rule")
if resample_kwargs["closed"] is None:
# this rule regarding the default value of 'closed' is inherited
# from pandas documentation for 'pandas.DataFrame.resample'
if rule in ("ME", "YE", "QE", "BME", "BA", "BQE", "W"):
resample_kwargs["closed"] = "right"
else:
resample_kwargs["closed"] = "left"
super().__init__(
modin_frame,
columns,
ascending,
ideal_num_new_partitions,
closed_on_right=resample_kwargs["closed"] == "right",
**kwargs,
)
resample_kwargs["freq"] = to_offset(rule)
self.resample_kwargs = resample_kwargs
@staticmethod
def pick_samples_for_quantiles(
df: pandas.DataFrame,
num_partitions: int,
length: int,
) -> pandas.DataFrame:
# to build proper bins we need min and max timestamp of the whole DatetimeIndex,
# so computing it in each partition
return pandas.concat([df.min().to_frame().T, df.max().to_frame().T])
def pick_pivots_from_samples_for_sort(
self,
samples: np.ndarray,
ideal_num_new_partitions: int,
method: str = "linear",
key: Optional[Callable] = None,
) -> np.ndarray:
if key is not None:
raise NotImplementedError(key)
max_value = samples.max()
first, last = _get_timestamp_range_edges(
samples.min(),
max_value,
self.resample_kwargs["freq"],
unit=samples.dt.unit,
closed=self.resample_kwargs["closed"],
origin=self.resample_kwargs["origin"],
offset=self.resample_kwargs["offset"],
)
all_bins = pandas.date_range(
start=first,
end=last,
freq=self.resample_kwargs["freq"],
ambiguous=True,
nonexistent="shift_forward",
unit=samples.dt.unit,
)
all_bins = self._adjust_bin_edges(
all_bins,
max_value,
freq=self.resample_kwargs["freq"],
closed=self.resample_kwargs["closed"],
)
# take pivot values with an even interval
step = 1 / ideal_num_new_partitions
bins = [
all_bins[int(len(all_bins) * i * step)]
for i in range(1, ideal_num_new_partitions)
]
return bins
def _adjust_bin_edges(
self,
binner: pandas.DatetimeIndex,
end_timestamp,
freq,
closed,
) -> pandas.DatetimeIndex:
"""
Adjust bin edges.
This function was copied & simplified from ``pandas.core.resample.TimeGrouper._adjuct_bin_edges()``.
Parameters
----------
binner : pandas.DatetimeIndex
end_timestamp : pandas.Timestamp
freq : str
closed : bool
Returns
-------
pandas.DatetimeIndex
"""
# Some hacks for > daily data, see pandas-dev/pandas#1471, pandas-dev/pandas#1458, pandas-dev/pandas#1483
if freq.name not in ("BME", "ME", "W") and freq.name.split("-")[0] not in (
"BQE",
"BYE",
"QE",
"YE",
"W",
):
return binner
# If the right end-point is on the last day of the month, roll forwards
# until the last moment of that day. Note that we only do this for offsets
# which correspond to the end of a super-daily period - "month start", for
# example, is excluded.
if closed == "right":
# GH 21459, GH 9119: Adjust the bins relative to the wall time
edges_dti = binner.tz_localize(None)
edges_dti = (
edges_dti
+ pandas.Timedelta(days=1, unit=edges_dti.unit).as_unit(edges_dti.unit)
- pandas.Timedelta(1, unit=edges_dti.unit).as_unit(edges_dti.unit)
)
binner = edges_dti.tz_localize(binner.tz)
# intraday values on last day
if binner[-2] > end_timestamp:
binner = binner[:-1]
return binner
@staticmethod
def split_partitions_using_pivots_for_sort(
df: pandas.DataFrame,
columns_info: "list[ColumnInfo]",
ascending: bool,
closed_on_right: bool = True,
**kwargs: dict,
) -> "tuple[pandas.DataFrame, ...]":
def add_attr(df, timestamp):
if "bin_bounds" in df.attrs:
df.attrs["bin_bounds"] = (*df.attrs["bin_bounds"], timestamp)
else:
df.attrs["bin_bounds"] = (timestamp,)
return df
result = ShuffleSortFunctions.split_partitions_using_pivots_for_sort(
df, columns_info, ascending, **kwargs
)
# it's required for each bin to know its bounds in order for resampling to work
# properly when down-sampling occurs. Reach here for an example:
# https://github.com/modin-project/modin/pull/7140#discussion_r1549246505
# We're writing the bounds as 'attrs' to avoid duplications in the final partition
for i, pivot in enumerate(columns_info[0].pivots):
add_attr(result[i], pivot - pandas.Timedelta(1, unit="ns"))
if i + 1 <= len(result):
add_attr(result[i + 1], pivot + pandas.Timedelta(1, unit="ns"))
return result
def lazy_metadata_decorator(apply_axis=None, axis_arg=-1, transpose=False):
"""
Lazily propagate metadata for the ``PandasDataframe``.
This decorator first adds the minimum required reindexing operations
to each partition's queue of functions to be lazily applied for
each PandasDataframe in the arguments by applying the function
run_f_on_minimally_updated_metadata. The decorator also sets the
flags for deferred metadata synchronization on the function result
if necessary.
Parameters
----------
apply_axis : str, default: None
The axes on which to apply the reindexing operations to the `self._partitions` lazily.
Case None: No lazy metadata propagation.
Case "both": Add reindexing operations on both axes to partition queue.
Case "opposite": Add reindexing operations complementary to given axis.
Case "rows": Add reindexing operations on row axis to partition queue.
axis_arg : int, default: -1
The index or column axis.
transpose : bool, default: False
Boolean for if a transpose operation is being used.
Returns
-------
Wrapped Function.
"""
def decorator(f):
from functools import wraps
@wraps(f)
def run_f_on_minimally_updated_metadata(self, *args, **kwargs):
from .dataframe import PandasDataframe
for obj in (
[self]
+ [o for o in args if isinstance(o, PandasDataframe)]
+ [v for v in kwargs.values() if isinstance(v, PandasDataframe)]
+ [
d
for o in args
if isinstance(o, list)
for d in o
if isinstance(d, PandasDataframe)
]
+ [
d
for _, o in kwargs.items()
if isinstance(o, list)
for d in o
if isinstance(d, PandasDataframe)
]
):
if apply_axis == "both":
if obj._deferred_index and obj._deferred_column:
obj._propagate_index_objs(axis=None)
elif obj._deferred_index:
obj._propagate_index_objs(axis=0)
elif obj._deferred_column:
obj._propagate_index_objs(axis=1)
elif apply_axis == "opposite":
if "axis" not in kwargs:
axis = args[axis_arg]
else:
axis = kwargs["axis"]
if axis == 0 and obj._deferred_column:
obj._propagate_index_objs(axis=1)
elif axis == 1 and obj._deferred_index:
obj._propagate_index_objs(axis=0)
elif apply_axis == "rows":
obj._propagate_index_objs(axis=0)
result = f(self, *args, **kwargs)
if apply_axis is None and not transpose:
result._deferred_index = self._deferred_index
result._deferred_column = self._deferred_column
elif apply_axis is None and transpose:
result._deferred_index = self._deferred_column
result._deferred_column = self._deferred_index
elif apply_axis == "opposite":
if axis == 0:
result._deferred_index = self._deferred_index
else:
result._deferred_column = self._deferred_column
elif apply_axis == "rows":
result._deferred_column = self._deferred_column
return result
return run_f_on_minimally_updated_metadata
return decorator
def add_missing_categories_to_groupby(
dfs,
by,
operator,
initial_columns,
combined_cols,
is_udf_agg,
kwargs,
initial_dtypes=None,
):
"""
Generate values for missing categorical values to be inserted into groupby result.
This function is used to emulate behavior of ``groupby(observed=False)`` parameter,
it takes groupby result that was computed using ``groupby(observed=True)``
and computes results for categorical values that are not presented in `dfs`.
Parameters
----------
dfs : list of pandas.DataFrames
Row partitions containing groupby results.
by : list of hashable
Column labels that were used to perform groupby.
operator : callable
Aggregation function that was used during groupby.
initial_columns : pandas.Index
Column labels of the original dataframe.
combined_cols : pandas.Index
Column labels of the groupby result.
is_udf_agg : bool
Whether ``operator`` is a UDF.
kwargs : dict
Parameters that were passed to ``groupby(by, **kwargs)``.
initial_dtypes : pandas.Series, optional
Dtypes of the original dataframe. If not specified, assume it's ``int64``.
Returns
-------
masks : dict[int, pandas.DataFrame]
Mapping between partition idx and a dataframe with results for missing categorical values
to insert to this partition.
new_combined_cols : pandas.Index
New column labels of the groupby result. If ``is_udf_agg is True``, then ``operator``
may change the resulted columns.
"""
kwargs["observed"] = False
new_combined_cols = combined_cols
### At first we need to compute missing categorical values
indices = [df.index for df in dfs]
# total_index contains all categorical values that resided in the result,
# missing values are computed differently depending on whether we're grouping
# on multiple groupers or not
total_index = indices[0].append(indices[1:])
if isinstance(total_index, pandas.MultiIndex):
if all(
not isinstance(level, pandas.CategoricalIndex)
for level in total_index.levels
):
return {}, new_combined_cols
missing_cats_dtype = {
name: (
level.dtype
if isinstance(level.dtype, pandas.CategoricalDtype)
# it's a bit confusing but we have to convert the remaining 'by' columns to categoricals
# in order to compute a proper fill value later in the code
else pandas.CategoricalDtype(level)
)
for level, name in zip(total_index.levels, total_index.names)
}
# if we're grouping on multiple groupers, then the missing categorical values is a
# carthesian product of (actual_missing_categorical_values X all_values_of_another_groupers)
complete_index = pandas.MultiIndex.from_product(
[
value.categories.astype(total_level.dtype)
for total_level, value in zip(
total_index.levels, missing_cats_dtype.values()
)
],
names=by,
)
missing_index = complete_index[~complete_index.isin(total_index)]
else:
if not isinstance(total_index, pandas.CategoricalIndex):
return {}, new_combined_cols
# if we're grouping on a single grouper then we simply compute the difference
# between categorical values in the result and the values defined in categorical dtype
missing_index = total_index.categories.difference(total_index.values)
missing_cats_dtype = {by[0]: pandas.CategoricalDtype(missing_index)}
missing_index.names = by
if len(missing_index) == 0:
return {}, new_combined_cols
### At this stage we want to get a fill_value for missing categorical values
if is_udf_agg and isinstance(total_index, pandas.MultiIndex):
# if grouping on multiple columns and aggregating with an UDF, then the
# fill value is always `np.nan`
missing_values = pandas.DataFrame({0: [np.nan]})
else:
# In case of a udf aggregation we're forced to run the operator against each
# missing category, as in theory it can return different results for each
# empty group. In other cases it's enough to run the operator against a single
# missing categorical and then broadcast the fill value to each missing value
if not is_udf_agg:
missing_cats_dtype = {
key: pandas.CategoricalDtype(value.categories[:1])
for key, value in missing_cats_dtype.items()
}
empty_df = pandas.DataFrame(columns=initial_columns)
# HACK: default 'object' dtype doesn't fit our needs, as most of the aggregations
# fail on a non-numeric columns, ideally, we need dtypes of the original dataframe,
# however, 'int64' also works fine here if the original schema is not available
empty_df = empty_df.astype(
"int64" if initial_dtypes is None else initial_dtypes
)
empty_df = empty_df.astype(missing_cats_dtype)
missing_values = operator(empty_df.groupby(by, **kwargs))
if is_udf_agg and not isinstance(total_index, pandas.MultiIndex):
missing_values = missing_values.drop(columns=by, errors="ignore")
new_combined_cols = pandas.concat(
[
pandas.DataFrame(columns=combined_cols),
missing_values.iloc[:0],
],
axis=0,
join="outer",
).columns
else:
# HACK: If the aggregation has failed, the result would be empty. Assuming the
# fill value to be `np.nan` here (this may not always be correct!!!)
fill_value = np.nan if len(missing_values) == 0 else missing_values.iloc[0, 0]
missing_values = pandas.DataFrame(
fill_value, index=missing_index, columns=combined_cols
)
# restoring original categorical dtypes for the indices (MultiIndex already have proper dtypes)
if not isinstance(missing_values.index, pandas.MultiIndex):
missing_values.index = missing_values.index.astype(total_index.dtype)
### Then we decide to which missing categorical values should go to which partition
if not kwargs["sort"]:
# If the result is allowed to be unsorted, simply insert all the missing
# categories to the last partition
mask = {len(indices) - 1: missing_values}
return mask, new_combined_cols
# If the result has to be sorted, we have to assign missing categoricals to proper partitions.
# For that purpose we define bins with corner values of each partition and then using either
# np.digitize or np.searchsorted find correct bins for each missing categorical value.
# Example: part0-> [0, 1, 2]; part1-> [3, 4, 10, 12]; part2-> [15, 17, 20, 100]
# bins -> [2, 12] # took last values of each partition excluding the last partition
# (every value that's matching 'x > part[-2][-1]' should go to the
# last partition, meaning that including the last value of the last
# partitions doesn't make sense)
# missing_cats -> [-2, 5, 6, 14, 21, 120]
# np.digitize(missing_cats, bins) -> [ 0, 1, 1, 2, 2, 2]
# ^-- mapping between values and partition idx to insert
bins = []
old_bins_to_new = {}
offset = 0
# building bins by taking last values of each partition excluding the last partition
for idx in indices[:-1]:
if len(idx) == 0:
# if a partition is empty, we can't use its values to define a bin, thus we simply
# skip it and remember the number of skipped partitions as an 'offset'
offset += 1
continue
# remember the number of skipped partitions before this bin, in order to restore original
# indexing at the end
old_bins_to_new[len(bins)] = offset
# for MultiIndices we always use the very first level for bins as using multiple levels
# doesn't affect the result
bins.append(idx[-1][0] if isinstance(idx, pandas.MultiIndex) else idx[-1])
old_bins_to_new[len(bins)] = offset
if len(bins) == 0:
# insert values to the first non-empty partition
return {old_bins_to_new.get(0, 0): missing_values}, new_combined_cols
# we used the very first level of MultiIndex to build bins, meaning that we also have
# to use values of the first index's level for 'digitize'
lvl_zero = (
missing_values.index.levels[0]
if isinstance(missing_values.index, pandas.MultiIndex)
else missing_values.index
)
if pandas.api.types.is_any_real_numeric_dtype(lvl_zero):
part_idx = np.digitize(lvl_zero, bins, right=True)
else:
part_idx = np.searchsorted(bins, lvl_zero)
### In the end we build a dictionary mapping partition index to a dataframe with missing categoricals
### to be inserted into this partition
masks = {}
if isinstance(total_index, pandas.MultiIndex):
for idx, values in pandas.RangeIndex(len(lvl_zero)).groupby(part_idx).items():
masks[idx] = missing_values[
pandas.Index(missing_values.index.codes[0]).isin(values)
]
else:
frame_idx = missing_values.index.to_frame()
for idx, values in lvl_zero.groupby(part_idx).items():
masks[idx] = missing_values[frame_idx.iloc[:, 0].isin(values)]
# Restore the original indexing by adding the amount of skipped missing partitions
masks = {key + old_bins_to_new[key]: value for key, value in masks.items()}
return masks, new_combined_cols
================================================
FILE: modin/core/dataframe/pandas/interchange/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe functionality related to data exchange protocols and optimized for pandas storage format."""
================================================
FILE: modin/core/dataframe/pandas/interchange/dataframe_protocol/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Base Modin Dataframe functionality related to the dataframe exchange protocol and optimized for pandas storage format.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
"""
================================================
FILE: modin/core/dataframe/pandas/interchange/dataframe_protocol/buffer.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Dataframe exchange protocol implementation.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
Notes
-----
- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and unsafe to
do in pure Python. It's more general but definitely less friendly than having
``to_arrow`` and ``to_numpy`` methods. So for the buffers which lack
``__dlpack__`` (e.g., because the column dtype isn't supported by DLPack),
this is worth looking at again.
"""
import enum
from typing import Tuple
import numpy as np
from modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (
ProtocolBuffer,
)
from modin.utils import _inherit_docstrings
@_inherit_docstrings(ProtocolBuffer)
class PandasProtocolBuffer(ProtocolBuffer):
"""
Data in the buffer is guaranteed to be contiguous in memory.
Note that there is no dtype attribute present, a buffer can be thought of
as simply a block of memory. However, if the column that the buffer is
attached to has a dtype that's supported by DLPack and ``__dlpack__`` is
implemented, then that dtype information will be contained in the return
value from ``__dlpack__``.
This distinction is useful to support both (a) data exchange via DLPack on a
buffer and (b) dtypes like variable-length strings which do not have a
fixed number of bytes per element.
Parameters
----------
x : np.ndarray
Data to be held by ``Buffer``.
allow_copy : bool, default: True
A keyword that defines whether or not the library is allowed
to make a copy of the data. For example, copying data would be necessary
if a library supports strided buffers, given that this protocol
specifies contiguous buffers. Currently, if the flag is set to ``False``
and a copy is needed, a ``RuntimeError`` will be raised.
"""
def __init__(self, x: np.ndarray, allow_copy: bool = True) -> None:
if not x.strides == (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is
# necessary. If that's not allowed, we need to raise an exception.
if allow_copy:
x = x.copy()
else:
raise RuntimeError(
"Exports cannot be zero-copy in the case "
+ "of a non-contiguous buffer"
)
# Store the numpy array in which the data resides as a private
# attribute, so we can use it to retrieve the public attributes
self._x = x
@property
def bufsize(self) -> int:
return self._x.size * self._x.dtype.itemsize
@property
def ptr(self) -> int:
return self._x.__array_interface__["data"][0]
def __dlpack__(self):
raise NotImplementedError("__dlpack__")
def __dlpack_device__(self) -> Tuple[enum.IntEnum, int]:
class Device(enum.IntEnum):
CPU = 1
return (Device.CPU, None)
def __repr__(self) -> str:
"""
Return a string representation for a particular ``PandasProtocolBuffer``.
Returns
-------
str
"""
return (
"Buffer("
+ str(
{
"bufsize": self.bufsize,
"ptr": self.ptr,
"device": self.__dlpack_device__()[0].name,
}
)
+ ")"
)
================================================
FILE: modin/core/dataframe/pandas/interchange/dataframe_protocol/column.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Dataframe exchange protocol implementation.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
Notes
-----
- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and unsafe to
do in pure Python. It's more general but definitely less friendly than having
``to_arrow`` and ``to_numpy`` methods. So for the buffers which lack
``__dlpack__`` (e.g., because the column dtype isn't supported by DLPack),
this is worth looking at again.
"""
from __future__ import annotations
from functools import cached_property
from typing import Any, Dict, Iterable, Optional, Tuple
import numpy as np
import pandas
from modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (
CategoricalDescription,
ProtocolColumn,
)
from modin.core.dataframe.base.interchange.dataframe_protocol.utils import (
ColumnNullType,
DTypeKind,
pandas_dtype_to_arrow_c,
)
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
from modin.utils import _inherit_docstrings
from .buffer import PandasProtocolBuffer
from .exception import NoOffsetsBuffer, NoValidityBuffer
_NO_VALIDITY_BUFFER = {
ColumnNullType.NON_NULLABLE: "This column is non-nullable so does not have a mask",
ColumnNullType.USE_NAN: "This column uses NaN as null so does not have a separate mask",
ColumnNullType.USE_SENTINEL: "This column uses a sentinel value so does not have a mask",
}
@_inherit_docstrings(ProtocolColumn)
class PandasProtocolColumn(ProtocolColumn):
"""
A column object, with only the methods and properties required by the interchange protocol defined.
A column can contain one or more chunks. Each chunk can contain up to three
buffers - a data buffer, a mask buffer (depending on null representation),
and an offsets buffer (if variable-size binary; e.g., variable-length strings).
TBD: Arrow has a separate "null" dtype, and has no separate mask concept.
Instead, it seems to use "children" for both columns with a bit mask,
and for nested dtypes. Unclear whether this is elegant or confusing.
This design requires checking the null representation explicitly.
The Arrow design requires checking:
1. the ARROW_FLAG_NULLABLE (for sentinel values)
2. if a column has two children, combined with one of those children
having a null dtype.
Making the mask concept explicit seems useful. One null dtype would
not be enough to cover both bit and byte masks, so that would mean
even more checking if we did it the Arrow way.
TBD: there's also the "chunk" concept here, which is implicit in Arrow as
multiple buffers per array (= column here). Semantically it may make
sense to have both: chunks were meant for example for lazy evaluation
of data which doesn't fit in memory, while multiple buffers per column
could also come from doing a selection operation on a single
contiguous buffer.
Given these concepts, one would expect chunks to be all of the same
size (say a 10,000 row dataframe could have 10 chunks of 1,000 rows),
while multiple buffers could have data-dependent lengths. Not an issue
in pandas if one column is backed by a single NumPy array, but in
Arrow it seems possible.
Are multiple chunks *and* multiple buffers per column necessary for
the purposes of this interchange protocol, or must producers either
reuse the chunk concept for this or copy the data?
Parameters
----------
column : PandasDataframe
A ``PandasDataframe`` object.
allow_copy : bool, default: True
A keyword that defines whether or not the library is allowed
to make a copy of the data. For example, copying data would be necessary
if a library supports strided buffers, given that this protocol
specifies contiguous buffers. Currently, if the flag is set to ``False``
and a copy is needed, a ``RuntimeError`` will be raised.
Notes
-----
This Column object can only be produced by ``__dataframe__``,
so doesn't need its own version or ``__column__`` protocol.
"""
def __init__(self, column: PandasDataframe, allow_copy: bool = True) -> None:
if not isinstance(column, PandasDataframe):
raise NotImplementedError(f"Columns of type {type(column)} not handled yet")
self._col = column
self._allow_copy = allow_copy
def size(self) -> int:
return len(self._col.index)
@property
def offset(self) -> int:
return 0
@cached_property
def dtype(self) -> Tuple[DTypeKind, int, str, str]:
dtype = self._col.dtypes.iloc[0]
if isinstance(dtype, pandas.CategoricalDtype):
pandas_series = self._col.to_pandas().squeeze(axis=1)
codes = pandas_series.values.codes
(
_,
bitwidth,
c_arrow_dtype_f_str,
_,
) = self._dtype_from_primitive_pandas_dtype(codes.dtype)
dtype_cache = (
DTypeKind.CATEGORICAL,
bitwidth,
c_arrow_dtype_f_str,
"=",
)
elif pandas.api.types.is_string_dtype(dtype):
dtype_cache = (DTypeKind.STRING, 8, pandas_dtype_to_arrow_c(dtype), "=")
else:
dtype_cache = self._dtype_from_primitive_pandas_dtype(dtype)
return dtype_cache
def _dtype_from_primitive_pandas_dtype(
self, dtype
) -> Tuple[DTypeKind, int, str, str]:
"""
Deduce dtype specific for the protocol from pandas dtype.
See `self.dtype` for details.
Parameters
----------
dtype : any
A pandas dtype.
Returns
-------
tuple
"""
_np_kinds = {
"i": DTypeKind.INT,
"u": DTypeKind.UINT,
"f": DTypeKind.FLOAT,
"b": DTypeKind.BOOL,
"M": DTypeKind.DATETIME,
}
kind = _np_kinds.get(dtype.kind, None)
if kind is None:
raise NotImplementedError(
f"Data type {dtype} not supported by the dataframe exchange protocol"
)
return (
kind,
dtype.itemsize * 8,
pandas_dtype_to_arrow_c(dtype),
dtype.byteorder,
)
@property
def describe_categorical(self) -> CategoricalDescription:
if self.dtype[0] != DTypeKind.CATEGORICAL:
raise TypeError(
"`describe_categorical only works on a column with "
+ "categorical dtype!"
)
pandas_series = self._col.to_pandas().squeeze(axis=1)
cat_frame = type(self._col).from_pandas(
pandas.DataFrame({"cat": pandas_series.cat.categories})
)
return {
"is_ordered": pandas_series.cat.ordered,
"is_dictionary": True,
"categories": PandasProtocolColumn(cat_frame, self._allow_copy),
}
@property
def describe_null(self) -> Tuple[int, Any]:
nulls = {
DTypeKind.FLOAT: (ColumnNullType.USE_NAN, None),
DTypeKind.DATETIME: (ColumnNullType.USE_NAN, None),
DTypeKind.INT: (ColumnNullType.NON_NULLABLE, None),
DTypeKind.UINT: (ColumnNullType.NON_NULLABLE, None),
DTypeKind.BOOL: (ColumnNullType.NON_NULLABLE, None),
# Null values for categoricals are stored as `-1` sentinel values
# in the category date (e.g., `col.values.codes` is int8 np.ndarray)
DTypeKind.CATEGORICAL: (ColumnNullType.USE_SENTINEL, -1),
# follow Arrow in using 1 as valid value and 0 for missing/null value
DTypeKind.STRING: (ColumnNullType.USE_BYTEMASK, 0),
}
kind = self.dtype[0]
try:
null, value = nulls[kind]
except KeyError:
raise NotImplementedError(f"Data type {kind} not yet supported")
return null, value
@cached_property
def null_count(self) -> int:
def map_func(df):
return df.isna()
def reduce_func(df):
return pandas.DataFrame(df.sum())
intermediate_df = self._col.tree_reduce(0, map_func, reduce_func)
# Set ``pandas.RangeIndex(1)`` to index and column labels because
# 1) We internally use `MODIN_UNNAMED_SERIES_LABEL` for labels of a reduced axis
# 2) The return value of `reduce_func` is a pandas DataFrame with
# index and column labels set to ``pandas.RangeIndex(1)``
# 3) We further use `to_pandas().squeeze()` to get an integer value of the null count.
# Otherwise, we get mismatching internal and external indices for both axes
intermediate_df.index = pandas.RangeIndex(1)
intermediate_df.columns = pandas.RangeIndex(1)
return intermediate_df.to_pandas().squeeze(axis=1).item()
@property
def metadata(self) -> Dict[str, Any]:
return {"modin.index": self._col.index}
def num_chunks(self) -> int:
return self._col._partitions.shape[0]
def get_chunks(
self, n_chunks: Optional[int] = None
) -> Iterable["PandasProtocolColumn"]:
cur_n_chunks = self.num_chunks()
n_rows = self.size()
if n_chunks is None or n_chunks == cur_n_chunks:
cum_row_lengths = np.cumsum([0] + self._col.row_lengths)
for i in range(len(cum_row_lengths) - 1):
yield PandasProtocolColumn(
self._col.take_2d_labels_or_positional(
row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),
col_positions=None,
),
allow_copy=self._col._allow_copy,
)
return
if n_chunks % cur_n_chunks != 0:
raise RuntimeError(
"The passed `n_chunks` must be a multiple of `self.num_chunks()`."
)
if n_chunks > n_rows:
raise RuntimeError(
"The passed `n_chunks` value is bigger than `self.num_rows()`."
)
chunksize = n_rows // n_chunks
new_lengths = [chunksize] * n_chunks
new_lengths[-1] = n_rows % n_chunks + new_lengths[-1]
new_partitions = self._col._partition_mgr_cls.map_axis_partitions(
0,
self._col._partitions,
lambda df: df,
keep_partitioning=False,
lengths=new_lengths,
)
new_df = self._col.__constructor__(
new_partitions,
self._col.index,
self._col.columns,
new_lengths,
self._col.column_widths,
)
cum_row_lengths = np.cumsum([0] + new_df.row_lengths)
for i in range(len(cum_row_lengths) - 1):
yield PandasProtocolColumn(
new_df.take_2d_labels_or_positional(
row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),
col_positions=None,
),
allow_copy=self._allow_copy,
)
def get_buffers(self) -> Dict[str, Any]:
buffers = {}
buffers["data"] = self._get_data_buffer()
try:
buffers["validity"] = self._get_validity_buffer()
except NoValidityBuffer:
buffers["validity"] = None
try:
buffers["offsets"] = self._get_offsets_buffer()
except NoOffsetsBuffer:
buffers["offsets"] = None
return buffers
_data_buffer_cache = None
def _get_data_buffer(
self,
) -> Tuple[PandasProtocolBuffer, Any]: # Any is for self.dtype tuple
"""
Return the buffer containing the data and the buffer's associated dtype.
Returns
-------
tuple
The data buffer.
"""
if self._data_buffer_cache is not None:
return self._data_buffer_cache
dtype = self.dtype
if dtype[0] in (
DTypeKind.INT,
DTypeKind.UINT,
DTypeKind.FLOAT,
DTypeKind.BOOL,
DTypeKind.DATETIME,
):
buffer = PandasProtocolBuffer(
self._col.to_numpy().flatten(), allow_copy=self._allow_copy
)
elif dtype[0] == DTypeKind.CATEGORICAL:
pandas_series = self._col.to_pandas().squeeze(axis=1)
codes = pandas_series.values.codes
buffer = PandasProtocolBuffer(codes, allow_copy=self._allow_copy)
dtype = self._dtype_from_primitive_pandas_dtype(codes.dtype)
elif dtype[0] == DTypeKind.STRING:
# Marshal the strings from a NumPy object array into a byte array
buf = self._col.to_numpy().flatten()
b = bytearray()
# TODO: this for-loop is slow; can be implemented in Cython/C/C++ later
for i in range(buf.size):
if type(buf[i]) is str:
b.extend(buf[i].encode(encoding="utf-8"))
# Convert the byte array to a pandas "buffer" using a NumPy array as the backing store
buffer = PandasProtocolBuffer(np.frombuffer(b, dtype="uint8"))
# Define the dtype for the returned buffer
dtype = (
DTypeKind.STRING,
8,
"u",
"=",
) # note: currently only support native endianness
else:
raise NotImplementedError(f"Data type {self._col.dtype[0]} not handled yet")
self._data_buffer_cache = (buffer, dtype)
return self._data_buffer_cache
_validity_buffer_cache = None
def _get_validity_buffer(self) -> Tuple[PandasProtocolBuffer, Any]:
"""
Get the validity buffer.
The buffer contains the mask values indicating
missing data and the buffer's associated dtype.
Returns
-------
tuple
The validity buffer.
Raises
------
``NoValidityBuffer`` if null representation is not a bit or byte mask.
"""
if self._validity_buffer_cache is not None:
return self._validity_buffer_cache
null, invalid = self.describe_null
if self.dtype[0] == DTypeKind.STRING:
# For now, have the mask array be comprised of bytes, rather than a bit array
buf = self._col.to_numpy().flatten()
# Determine the encoding for valid values
valid = invalid == 0
invalid = not valid
mask = np.empty(shape=(len(buf),), dtype=np.bool_)
for i, obj in enumerate(buf):
mask[i] = valid if isinstance(obj, str) else invalid
# Convert the mask array to a Pandas "buffer" using a NumPy array as the backing store
buffer = PandasProtocolBuffer(mask)
# Define the dtype of the returned buffer
dtype = (DTypeKind.BOOL, 8, "b", "=")
self._validity_buffer_cache = (buffer, dtype)
return self._validity_buffer_cache
try:
msg = _NO_VALIDITY_BUFFER[null]
except KeyError:
raise NotImplementedError("See self.describe_null")
raise NoValidityBuffer(msg)
_offsets_buffer_cache = None
def _get_offsets_buffer(self) -> Tuple[PandasProtocolBuffer, Any]:
"""
Get the offsets buffer.
The buffer contains the offset values for variable-size binary data
(e.g., variable-length strings) and the buffer's associated dtype.
Returns
-------
tuple
The offsets buffer.
Raises
------
``NoOffsetsBuffer`` if the data buffer does not have an associated offsets buffer.
"""
if self._offsets_buffer_cache is not None:
return self._offsets_buffer_cache
if self.dtype[0] == DTypeKind.STRING:
# For each string, we need to manually determine the next offset
values = self._col.to_numpy().flatten()
ptr = 0
offsets = [ptr] + [None] * len(values)
for i, v in enumerate(values):
# For missing values (in this case, `np.nan` values), we don't increment the pointer)
if type(v) is str:
b = v.encode(encoding="utf-8")
ptr += len(b)
offsets[i + 1] = ptr
# Convert the list of offsets to a NumPy array of signed 64-bit integers (note: Arrow allows the offsets array to be either `int32` or `int64`; here, we default to the latter)
buf = np.asarray(offsets, dtype="int64")
# Convert the offsets to a Pandas "buffer" using the NumPy array as the backing store
buffer = PandasProtocolBuffer(buf)
# Assemble the buffer dtype info
dtype = (
DTypeKind.INT,
64,
"l",
"=",
) # note: currently only support native endianness
else:
raise NoOffsetsBuffer(
"This column has a fixed-length dtype so does not have an offsets buffer"
)
self._offsets_buffer_cache = (buffer, dtype)
return self._offsets_buffer_cache
================================================
FILE: modin/core/dataframe/pandas/interchange/dataframe_protocol/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Dataframe exchange protocol implementation.
See more in https://data-apis.org/dataframe-protocol/latest/index.html.
Notes
-----
- Interpreting a raw pointer (as in ``Buffer.ptr``) is annoying and unsafe to
do in pure Python. It's more general but definitely less friendly than having
``to_arrow`` and ``to_numpy`` methods. So for the buffers which lack
``__dlpack__`` (e.g., because the column dtype isn't supported by DLPack),
this is worth looking at again.
"""
import collections
from typing import Any, Dict, Iterable, Optional, Sequence
import numpy as np
from modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (
ProtocolDataframe,
)
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
from modin.utils import _inherit_docstrings
from .column import PandasProtocolColumn
@_inherit_docstrings(ProtocolDataframe)
class PandasProtocolDataframe(ProtocolDataframe):
"""
A data frame class, with only the methods required by the interchange protocol defined.
Instances of this (private) class are returned from ``modin.pandas.DataFrame.__dataframe__``
as objects with the methods and attributes defined on this class.
A "data frame" represents an ordered collection of named columns.
A column's "name" must be a unique string. Columns may be accessed by name or by position.
This could be a public data frame class, or an object with the methods and
attributes defined on this DataFrame class could be returned from the
``__dataframe__`` method of a public data frame class in a library adhering
to the dataframe interchange protocol specification.
Parameters
----------
df : PandasDataframe
A ``PandasDataframe`` object.
nan_as_null : bool, default:False
A keyword intended for the consumer to tell the producer
to overwrite null values in the data with ``NaN`` (or ``NaT``).
This currently has no effect; once support for nullable extension
dtypes is added, this value should be propagated to columns.
allow_copy : bool, default: True
A keyword that defines whether or not the library is allowed
to make a copy of the data. For example, copying data would be necessary
if a library supports strided buffers, given that this protocol
specifies contiguous buffers. Currently, if the flag is set to ``False``
and a copy is needed, a ``RuntimeError`` will be raised.
"""
def __init__(
self,
df: PandasDataframe,
nan_as_null: bool = False,
allow_copy: bool = True,
) -> None:
self._df = df
self._nan_as_null = nan_as_null
self._allow_copy = allow_copy
def __dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True):
return PandasProtocolDataframe(
self._df, nan_as_null=nan_as_null, allow_copy=allow_copy
)
@property
def metadata(self) -> Dict[str, Any]:
return {"modin.index": self._df.index}
def num_columns(self) -> int:
return len(self._df.columns)
def num_rows(self) -> int:
return len(self._df.index)
def num_chunks(self) -> int:
return self._df._partitions.shape[0]
def column_names(self) -> Iterable[str]:
for col in self._df.columns:
yield col
def get_column(self, i: int) -> PandasProtocolColumn:
return PandasProtocolColumn(
self._df.take_2d_labels_or_positional(
row_positions=None, col_positions=[i]
),
allow_copy=self._allow_copy,
)
def get_column_by_name(self, name: str) -> PandasProtocolColumn:
return PandasProtocolColumn(
self._df.take_2d_labels_or_positional(
row_positions=None, col_labels=[name]
),
allow_copy=self._allow_copy,
)
def get_columns(self) -> Iterable[PandasProtocolColumn]:
for name in self._df.columns:
yield PandasProtocolColumn(
self._df.take_2d_labels_or_positional(
row_positions=None, col_labels=[name]
),
allow_copy=self._allow_copy,
)
def select_columns(self, indices: Sequence[int]) -> "PandasProtocolDataframe":
if not isinstance(indices, collections.abc.Sequence):
raise ValueError("`indices` is not a sequence")
return PandasProtocolDataframe(
self._df.take_2d_labels_or_positional(
row_positions=None, col_positions=indices
),
allow_copy=self._allow_copy,
)
def select_columns_by_name(self, names: Sequence[str]) -> "PandasProtocolDataframe":
if not isinstance(names, collections.abc.Sequence):
raise ValueError("`names` is not a sequence")
return PandasProtocolDataframe(
self._df.take_2d_labels_or_positional(row_positions=None, col_labels=names),
allow_copy=self._allow_copy,
)
def get_chunks(
self, n_chunks: Optional[int] = None
) -> Iterable["PandasProtocolDataframe"]:
cur_n_chunks = self.num_chunks()
n_rows = self.num_rows()
if n_chunks is None or n_chunks == cur_n_chunks:
cum_row_lengths = np.cumsum([0] + self._df.row_lengths)
for i in range(len(cum_row_lengths) - 1):
yield PandasProtocolDataframe(
self._df.take_2d_labels_or_positional(
row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),
col_positions=None,
),
allow_copy=self._allow_copy,
)
return
if n_chunks % cur_n_chunks != 0:
raise RuntimeError(
"The passed `n_chunks` must be a multiple of `self.num_chunks()`."
)
if n_chunks > n_rows:
raise RuntimeError(
"The passed `n_chunks` value is bigger than `self.num_rows()`."
)
chunksize = n_rows // n_chunks
new_lengths = [chunksize] * n_chunks
new_lengths[-1] = n_rows % n_chunks + new_lengths[-1]
new_partitions = self._df._partition_mgr_cls.map_axis_partitions(
0,
self._df._partitions,
lambda df: df,
keep_partitioning=False,
lengths=new_lengths,
)
new_df = self._df.__constructor__(
new_partitions,
self._df.index,
self._df.columns,
new_lengths,
self._df.column_widths,
)
cum_row_lengths = np.cumsum([0] + new_df.row_lengths)
for i in range(len(cum_row_lengths) - 1):
yield PandasProtocolDataframe(
new_df.take_2d_labels_or_positional(
row_positions=range(cum_row_lengths[i], cum_row_lengths[i + 1]),
col_positions=None,
),
allow_copy=self._allow_copy,
)
================================================
FILE: modin/core/dataframe/pandas/interchange/dataframe_protocol/exception.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Exceptions that can be caught by dataframe exchange protocol implementation for pandas storage format."""
class NoValidityBuffer(Exception):
"""Exception to be raised if there is no validity buffer for ``PandasProtocolColumn``."""
pass
class NoOffsetsBuffer(Exception):
"""Exception to be raised if there is no offsets buffer for ``PandasProtocolColumn``."""
pass
================================================
FILE: modin/core/dataframe/pandas/interchange/dataframe_protocol/from_dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses the functions building a ``pandas.DataFrame`` from a DataFrame exchange protocol object."""
import ctypes
import re
from typing import Any, Optional, Tuple, Union
import numpy as np
import pandas
from modin.core.dataframe.base.interchange.dataframe_protocol.dataframe import (
ProtocolBuffer,
ProtocolColumn,
ProtocolDataframe,
)
from modin.core.dataframe.base.interchange.dataframe_protocol.utils import (
ArrowCTypes,
ColumnNullType,
DTypeKind,
Endianness,
)
np_types_map = {
DTypeKind.INT: {8: np.int8, 16: np.int16, 32: np.int32, 64: np.int64},
DTypeKind.UINT: {8: np.uint8, 16: np.uint16, 32: np.uint32, 64: np.uint64},
DTypeKind.FLOAT: {32: np.float32, 64: np.float64},
# Consider bitmask to be a uint8 dtype to parse the bits later
DTypeKind.BOOL: {1: np.uint8, 8: bool},
}
def from_dataframe_to_pandas(df: ProtocolDataframe, n_chunks: Optional[int] = None):
"""
Build a ``pandas.DataFrame`` from an object supporting the DataFrame exchange protocol, i.e. `__dataframe__` method.
Parameters
----------
df : ProtocolDataframe
Object supporting the exchange protocol, i.e. `__dataframe__` method.
n_chunks : int, optional
Number of chunks to split `df`.
Returns
-------
pandas.DataFrame
"""
if not hasattr(df, "__dataframe__"):
raise ValueError("`df` does not support __dataframe__")
df = df.__dataframe__()
if isinstance(df, dict):
df = df["dataframe"]
pandas_dfs = []
for chunk in df.get_chunks(n_chunks):
pandas_df = protocol_df_chunk_to_pandas(chunk)
pandas_dfs.append(pandas_df)
pandas_df = pandas.concat(pandas_dfs, axis=0, ignore_index=True)
index_obj = df.metadata.get("modin.index", df.metadata.get("pandas.index", None))
if index_obj is not None:
pandas_df.index = index_obj
return pandas_df
def protocol_df_chunk_to_pandas(df):
"""
Convert exchange protocol chunk to ``pandas.DataFrame``.
Parameters
----------
df : ProtocolDataframe
Returns
-------
pandas.DataFrame
"""
# We need a dict of columns here, with each column being a NumPy array (at
# least for now, deal with non-NumPy dtypes later).
columns = dict()
buffers = [] # hold on to buffers, keeps memory alive
for name in df.column_names():
if not isinstance(name, str):
raise ValueError(f"Column {name} is not a string")
if name in columns:
raise ValueError(f"Column {name} is not unique")
col = df.get_column_by_name(name)
columns[name], buf = unpack_protocol_column(col)
buffers.append(buf)
pandas_df = pandas.DataFrame(columns)
pandas_df._buffers = buffers
return pandas_df
def unpack_protocol_column(
col: ProtocolColumn,
) -> Tuple[Union[np.ndarray, pandas.Series], Any]:
"""
Unpack an interchange protocol column to a pandas-ready column.
Parameters
----------
col : ProtocolColumn
Column to unpack.
Returns
-------
tuple
Tuple of resulting column (either an ndarray or a series) and the object
which keeps memory referenced by the column alive.
"""
dtype = col.dtype[0]
if dtype in (
DTypeKind.INT,
DTypeKind.UINT,
DTypeKind.FLOAT,
DTypeKind.BOOL,
):
return primitive_column_to_ndarray(col)
elif dtype == DTypeKind.CATEGORICAL:
return categorical_column_to_series(col)
elif dtype == DTypeKind.STRING:
return string_column_to_ndarray(col)
elif dtype == DTypeKind.DATETIME:
return datetime_column_to_ndarray(col)
else:
raise NotImplementedError(f"Data type {dtype} not handled yet")
def primitive_column_to_ndarray(col: ProtocolColumn) -> Tuple[np.ndarray, Any]:
"""
Convert a column holding one of the primitive dtypes (int, uint, float or bool) to a NumPy array.
Parameters
----------
col : ProtocolColumn
Returns
-------
tuple
Tuple of np.ndarray holding the data and the memory owner object that keeps the memory alive.
"""
buffers = col.get_buffers()
data_buff, data_dtype = buffers["data"]
data = buffer_to_ndarray(data_buff, data_dtype, col.offset, col.size())
data = set_nulls(data, col, buffers["validity"])
return data, buffers
def categorical_column_to_series(col: ProtocolColumn) -> Tuple[pandas.Series, Any]:
"""
Convert a column holding categorical data to a pandas Series.
Parameters
----------
col : ProtocolColumn
Returns
-------
tuple
Tuple of pandas.Series holding the data and the memory owner object that keeps the memory alive.
"""
cat_descr = col.describe_categorical
ordered, is_dict, categories = (
cat_descr["is_ordered"],
cat_descr["is_dictionary"],
cat_descr["categories"],
)
if not is_dict or categories is None:
raise NotImplementedError("Non-dictionary categoricals not supported yet")
buffers = col.get_buffers()
codes_buff, codes_dtype = buffers["data"]
codes = buffer_to_ndarray(codes_buff, codes_dtype, col.offset, col.size())
# Doing module in order to not get ``IndexError`` for out-of-bounds sentinel values in `codes`
cat_values, categories_buf = unpack_protocol_column(categories)
values = cat_values[codes % len(cat_values)]
cat = pandas.Categorical(values, categories=cat_values, ordered=ordered)
data = pandas.Series(cat)
data = set_nulls(data, col, buffers["validity"])
return data, [buffers, categories_buf]
def _inverse_null_buf(buf: np.ndarray, null_kind: ColumnNullType) -> np.ndarray:
"""
Inverse the boolean value of buffer storing either bit- or bytemask.
Parameters
----------
buf : np.ndarray
Buffer to inverse the boolean value for.
null_kind : {ColumnNullType.USE_BYTEMASK, ColumnNullType.USE_BITMASK}
How to treat the buffer.
Returns
-------
np.ndarray
Logically inversed buffer.
"""
if null_kind == ColumnNullType.USE_BITMASK:
return ~buf
assert (
null_kind == ColumnNullType.USE_BYTEMASK
), f"Unexpected null kind: {null_kind}"
# bytemasks use 0 for `False` and anything else for `True`, so convert to bool
# by direct comparison instead of bitwise reversal like we do for bitmasks
return buf == 0
def string_column_to_ndarray(col: ProtocolColumn) -> Tuple[np.ndarray, Any]:
"""
Convert a column holding string data to a NumPy array.
Parameters
----------
col : ProtocolColumn
Returns
-------
tuple
Tuple of np.ndarray holding the data and the memory owner object that keeps the memory alive.
"""
null_kind, sentinel_val = col.describe_null
if null_kind not in (
ColumnNullType.NON_NULLABLE,
ColumnNullType.USE_BITMASK,
ColumnNullType.USE_BYTEMASK,
):
raise NotImplementedError(
f"{null_kind} null kind is not yet supported for string columns."
)
buffers = col.get_buffers()
# Retrieve the data buffer containing the UTF-8 code units
data_buff, protocol_data_dtype = buffers["data"]
# We're going to reinterpret the buffer as uint8, so making sure we can do it safely
assert protocol_data_dtype[1] == 8 # bitwidth == 8
assert protocol_data_dtype[2] == ArrowCTypes.STRING # format_str == utf-8
# Convert the buffers to NumPy arrays, in order to go from STRING to an equivalent ndarray,
# we claim that the buffer is uint8 (i.e., a byte array)
data_dtype = (
DTypeKind.UINT,
8,
ArrowCTypes.UINT8,
Endianness.NATIVE,
)
# Specify zero offset as we don't want to chunk the string data
data = buffer_to_ndarray(data_buff, data_dtype, offset=0, length=col.size())
# Retrieve the offsets buffer containing the index offsets demarcating the beginning and end of each string
offset_buff, offset_dtype = buffers["offsets"]
# Offsets buffer contains start-stop positions of strings in the data buffer,
# meaning that it has more elements than in the data buffer, do `col.size() + 1` here
# to pass a proper offsets buffer size
offsets = buffer_to_ndarray(
offset_buff, offset_dtype, col.offset, length=col.size() + 1
)
null_pos = None
if null_kind in (ColumnNullType.USE_BITMASK, ColumnNullType.USE_BYTEMASK):
valid_buff, valid_dtype = buffers["validity"]
null_pos = buffer_to_ndarray(valid_buff, valid_dtype, col.offset, col.size())
if sentinel_val == 0:
null_pos = _inverse_null_buf(null_pos, null_kind)
# Assemble the strings from the code units
str_list = [None] * col.size()
for i in range(col.size()):
# Check for missing values
if null_pos is not None and null_pos[i]:
str_list[i] = np.nan
continue
# Extract a range of code units
units = data[offsets[i] : offsets[i + 1]]
# Convert the list of code units to bytes
str_bytes = bytes(units)
# Create the string
string = str_bytes.decode(encoding="utf-8")
# Add to our list of strings
str_list[i] = string
# Convert the string list to a NumPy array
return np.asarray(str_list, dtype="object"), buffers
def datetime_column_to_ndarray(col: ProtocolColumn) -> Tuple[np.ndarray, Any]:
"""
Convert a column holding DateTime data to a NumPy array.
Parameters
----------
col : ProtocolColumn
Returns
-------
tuple
Tuple of np.ndarray holding the data and the memory owner object that keeps the memory alive.
"""
buffers = col.get_buffers()
_, _, format_str, _ = col.dtype
dbuf, dtype = buffers["data"]
# Consider dtype being `uint` to get number of units passed since the 01.01.1970
data = buffer_to_ndarray(
dbuf,
(
DTypeKind.UINT,
dtype[1],
getattr(ArrowCTypes, f"UINT{dtype[1]}"),
Endianness.NATIVE,
),
col.offset,
col.size(),
)
def parse_format_str(format_str, data):
"""Parse datetime `format_str` to interpret the `data`."""
# timestamp 'ts{unit}:tz'
timestamp_meta = re.match(r"ts([smun]):(.*)", format_str)
if timestamp_meta:
unit, tz = timestamp_meta.group(1), timestamp_meta.group(2)
if tz != "":
raise NotImplementedError("Timezones are not supported yet")
if unit != "s":
# the format string describes only a first letter of the unit, add one extra
# letter to make the unit in numpy-style: 'm' -> 'ms', 'u' -> 'us', 'n' -> 'ns'
unit += "s"
data = data.astype(f"datetime64[{unit}]")
return data
# date 'td{Days/Ms}'
date_meta = re.match(r"td([Dm])", format_str)
if date_meta:
unit = date_meta.group(1)
if unit == "D":
# NumPy doesn't support DAY unit, so converting days to seconds
# (converting to uint64 to avoid overflow)
data = (data.astype(np.uint64) * (24 * 60 * 60)).astype("datetime64[s]")
elif unit == "m":
data = data.astype("datetime64[ms]")
else:
raise NotImplementedError(f"Date unit is not supported: {unit}")
return data
raise NotImplementedError(f"DateTime kind is not supported: {format_str}")
data = parse_format_str(format_str, data)
data = set_nulls(data, col, buffers["validity"])
return data, buffers
def buffer_to_ndarray(
buffer: ProtocolBuffer,
dtype: Tuple[DTypeKind, int, str, str],
offset: int = 0,
length: Optional[int] = None,
) -> np.ndarray:
"""
Build a NumPy array from the passed buffer.
Parameters
----------
buffer : ProtocolBuffer
Buffer to build a NumPy array from.
dtype : tuple
Data type of the buffer conforming protocol dtypes format.
offset : int, default: 0
Number of elements to offset from the start of the buffer.
length : int, optional
If the buffer is a bit-mask, specifies a number of bits to read
from the buffer. Has no effect otherwise.
Returns
-------
np.ndarray
Notes
-----
The returned array doesn't own the memory. A user of the function must keep the memory
owner object alive as long as the returned NumPy array is being used.
"""
kind, bit_width, _, _ = dtype
column_dtype = np_types_map.get(kind, {}).get(bit_width, None)
if column_dtype is None:
raise NotImplementedError(f"Convertion for {dtype} is not yet supported.")
# TODO: No DLPack yet, so need to construct a new ndarray from the data pointer
# and size in the buffer plus the dtype on the column. Use DLPack as NumPy supports
# it since https://github.com/numpy/numpy/pull/19083
ctypes_type = np.ctypeslib.as_ctypes_type(column_dtype)
data_pointer = ctypes.cast(
buffer.ptr + (offset * bit_width // 8), ctypes.POINTER(ctypes_type)
)
if bit_width == 1:
assert length is not None, "`length` must be specified for a bit-mask buffer."
arr = np.ctypeslib.as_array(data_pointer, shape=(buffer.bufsize,))
return bitmask_to_bool_ndarray(arr, length, first_byte_offset=offset % 8)
else:
return np.ctypeslib.as_array(
data_pointer, shape=(buffer.bufsize // (bit_width // 8),)
)
def bitmask_to_bool_ndarray(
bitmask: np.ndarray, mask_length: int, first_byte_offset: int = 0
) -> np.ndarray:
"""
Convert bit-mask to a boolean NumPy array.
Parameters
----------
bitmask : np.ndarray[uint8]
NumPy array of uint8 dtype representing the bitmask.
mask_length : int
Number of elements in the mask to interpret.
first_byte_offset : int, default: 0
Number of elements to offset from the start of the first byte.
Returns
-------
np.ndarray[bool]
"""
bytes_to_skip = first_byte_offset // 8
bitmask = bitmask[bytes_to_skip:]
first_byte_offset %= 8
bool_mask = np.zeros(mask_length, dtype=bool)
# Proccessing the first byte separately as it has its own offset
val = bitmask[0]
mask_idx = 0
bits_in_first_byte = min(8 - first_byte_offset, mask_length)
for j in range(bits_in_first_byte):
if val & (1 << (j + first_byte_offset)):
bool_mask[mask_idx] = True
mask_idx += 1
# `mask_length // 8` describes how many full bytes to process
for i in range((mask_length - bits_in_first_byte) // 8):
# doing `+ 1` as we already processed the first byte
val = bitmask[i + 1]
for j in range(8):
if val & (1 << j):
bool_mask[mask_idx] = True
mask_idx += 1
if len(bitmask) > 1:
# Processing reminder of last byte
val = bitmask[-1]
for j in range(len(bool_mask) - mask_idx):
if val & (1 << j):
bool_mask[mask_idx] = True
mask_idx += 1
return bool_mask
def set_nulls(
data: Union[np.ndarray, pandas.Series],
col: ProtocolColumn,
validity: Tuple[ProtocolBuffer, Tuple[DTypeKind, int, str, str]],
allow_modify_inplace: bool = True,
):
"""
Set null values for the data according to the column null kind.
Parameters
----------
data : np.ndarray or pandas.Series
Data to set nulls in.
col : ProtocolColumn
Column object that describes the `data`.
validity : tuple(ProtocolBuffer, dtype) or None
The return value of ``col.buffers()``. We do not access the ``col.buffers()``
here to not take the ownership of the memory of buffer objects.
allow_modify_inplace : bool, default: True
Whether to modify the `data` inplace when zero-copy is possible (True) or always
modify a copy of the `data` (False).
Returns
-------
np.ndarray or pandas.Series
Data with the nulls being set.
"""
null_kind, sentinel_val = col.describe_null
null_pos = None
if null_kind == ColumnNullType.USE_SENTINEL:
null_pos = data == sentinel_val
elif null_kind in (ColumnNullType.USE_BITMASK, ColumnNullType.USE_BYTEMASK):
valid_buff, valid_dtype = validity
null_pos = buffer_to_ndarray(valid_buff, valid_dtype, col.offset, col.size())
if sentinel_val == 0:
null_pos = _inverse_null_buf(null_pos, null_kind)
elif null_kind in (ColumnNullType.NON_NULLABLE, ColumnNullType.USE_NAN):
pass
else:
raise NotImplementedError(f"Null kind {null_kind} is not yet supported.")
if null_pos is not None and np.any(null_pos):
if not allow_modify_inplace:
data = data.copy()
try:
data[null_pos] = None
except TypeError:
# TypeError happens if the `data` dtype appears to be non-nullable in numpy notation
# (bool, int, uint), if such happens, cast the `data` to nullable float dtype.
data = data.astype(float)
data[null_pos] = None
return data
================================================
FILE: modin/core/dataframe/pandas/metadata/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Utilities and classes to handle work with metadata."""
from .dtypes import (
DtypesDescriptor,
LazyProxyCategoricalDtype,
ModinDtypes,
extract_dtype,
)
from .index import ModinIndex
__all__ = [
"ModinDtypes",
"ModinIndex",
"LazyProxyCategoricalDtype",
"DtypesDescriptor",
"extract_dtype",
]
================================================
FILE: modin/core/dataframe/pandas/metadata/dtypes.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module contains class ``ModinDtypes``."""
from __future__ import annotations
from typing import TYPE_CHECKING, Callable, Optional, Union
import pandas
from pandas._typing import DtypeObj, IndexLabel
from pandas.core.dtypes.cast import find_common_type
if TYPE_CHECKING:
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
from .index import ModinIndex
from modin.error_message import ErrorMessage
class DtypesDescriptor:
"""
Describes partial dtypes.
Parameters
----------
known_dtypes : dict[IndexLabel, DtypeObj] or pandas.Series, optional
Columns that we know dtypes for.
cols_with_unknown_dtypes : list[IndexLabel], optional
Column names that have unknown dtypes. If specified together with `remaining_dtype`, must describe all
columns with unknown dtypes, otherwise, the missing columns will be assigned to `remaining_dtype`.
If `cols_with_unknown_dtypes` is incomplete, you must specify `know_all_names=False`.
remaining_dtype : DtypeObj, optional
Dtype for columns that are not present neither in `known_dtypes` nor in `cols_with_unknown_dtypes`.
This parameter is intended to describe columns that we known dtypes for, but don't know their
names yet. Note, that this parameter DOESN'T describe dtypes for columns from `cols_with_unknown_dtypes`.
parent_df : PandasDataframe, optional
Dataframe object for which we describe dtypes. This dataframe will be used to compute
missing dtypes on ``.materialize()``.
columns_order : dict[int, IndexLabel], optional
Order of columns in the dataframe. If specified, must describe all the columns of the dataframe.
know_all_names : bool, default: True
Whether `known_dtypes` and `cols_with_unknown_dtypes` contain all column names for this dataframe besides those,
that are being described by `remaining_dtype`.
One can't pass `know_all_names=False` together with `remaining_dtype` as this creates ambiguity
on how to interpret missing columns (whether they belong to `remaining_dtype` or not).
_schema_is_known : bool, optional
Whether `known_dtypes` describe all columns in the dataframe. This parameter intended mostly
for internal use.
"""
def __init__(
self,
known_dtypes: Optional[Union[dict[IndexLabel, DtypeObj], pandas.Series]] = None,
cols_with_unknown_dtypes: Optional[list[IndexLabel]] = None,
remaining_dtype: Optional[DtypeObj] = None,
parent_df: Optional[PandasDataframe] = None,
columns_order: Optional[dict[int, IndexLabel]] = None,
know_all_names: bool = True,
_schema_is_known: Optional[bool] = None,
):
if not know_all_names and remaining_dtype is not None:
raise ValueError(
"It's not allowed to pass 'remaining_dtype' and 'know_all_names=False' at the same time."
)
# columns with known dtypes
self._known_dtypes: dict[IndexLabel, DtypeObj] = (
{} if known_dtypes is None else dict(known_dtypes)
)
if known_dtypes is not None and len(self._known_dtypes) != len(known_dtypes):
raise NotImplementedError(
"Duplicated column names are not yet supported by DtypesDescriptor"
)
# columns with unknown dtypes (they're not described by 'remaining_dtype')
if cols_with_unknown_dtypes is not None and len(
set(cols_with_unknown_dtypes)
) != len(cols_with_unknown_dtypes):
raise NotImplementedError(
"Duplicated column names are not yet supported by DtypesDescriptor"
)
self._cols_with_unknown_dtypes: list[IndexLabel] = (
[] if cols_with_unknown_dtypes is None else cols_with_unknown_dtypes
)
# whether 'known_dtypes' describe all columns in the dataframe
self._schema_is_known: Optional[bool] = _schema_is_known
if self._schema_is_known is None:
self._schema_is_known = False
if (
# if 'cols_with_unknown_dtypes' was explicitly specified as an empty list and
# we don't have any 'remaining_dtype', then we assume that 'known_dtypes' are complete
cols_with_unknown_dtypes is not None
and know_all_names
and remaining_dtype is None
and len(self._known_dtypes) > 0
):
self._schema_is_known = len(cols_with_unknown_dtypes) == 0
self._know_all_names: bool = know_all_names
# a common dtype for columns that are not present in 'known_dtypes' nor in 'cols_with_unknown_dtypes'
self._remaining_dtype: Optional[DtypeObj] = remaining_dtype
self._parent_df: Optional[PandasDataframe] = parent_df
if columns_order is None:
self._columns_order: Optional[dict[int, IndexLabel]] = None
# try to compute '._columns_order' using 'parent_df'
self.columns_order
else:
if remaining_dtype is not None:
raise ValueError(
"Passing 'columns_order' and 'remaining_dtype' is ambiguous. You have to manually "
+ "complete 'known_dtypes' using the information from 'columns_order' and 'remaining_dtype'."
)
elif not self._know_all_names:
raise ValueError(
"Passing 'columns_order' and 'know_all_names=False' is ambiguous. You have to manually "
+ "complete 'cols_with_unknown_dtypes' using the information from 'columns_order' "
+ "and pass 'know_all_names=True'."
)
elif len(columns_order) != (
len(self._cols_with_unknown_dtypes) + len(self._known_dtypes)
):
raise ValueError(
"The length of 'columns_order' doesn't match to 'known_dtypes' and 'cols_with_unknown_dtypes'"
)
self._columns_order: Optional[dict[int, IndexLabel]] = columns_order
def update_parent(self, new_parent: PandasDataframe):
"""
Set new parent dataframe.
Parameters
----------
new_parent : PandasDataframe
"""
self._parent_df = new_parent
LazyProxyCategoricalDtype.update_dtypes(self._known_dtypes, new_parent)
# try to compute '._columns_order' using 'new_parent'
self.columns_order
@property
def columns_order(self) -> Optional[dict[int, IndexLabel]]:
"""
Get order of columns for the described dataframe if available.
Returns
-------
dict[int, IndexLabel] or None
"""
if self._columns_order is not None:
return self._columns_order
if self._parent_df is None or not self._parent_df.has_materialized_columns:
return None
actual_columns = self._parent_df.columns
self._normalize_self_levels(actual_columns)
self._columns_order = {i: col for i, col in enumerate(actual_columns)}
# we got information about new columns and thus can potentially
# extend our knowledge about missing dtypes
if len(self._columns_order) > (
len(self._known_dtypes) + len(self._cols_with_unknown_dtypes)
):
new_cols = [
col
for col in self._columns_order.values()
if col not in self._known_dtypes
and col not in self._cols_with_unknown_dtypes
]
if self._remaining_dtype is not None:
self._known_dtypes.update(
{col: self._remaining_dtype for col in new_cols}
)
self._remaining_dtype = None
if len(self._cols_with_unknown_dtypes) == 0:
self._schema_is_known = True
else:
self._cols_with_unknown_dtypes.extend(new_cols)
self._know_all_names = True
return self._columns_order
def __repr__(self): # noqa: GL08
return (
f"DtypesDescriptor:\n\tknown dtypes: {self._known_dtypes};\n\t"
+ f"remaining dtype: {self._remaining_dtype};\n\t"
+ f"cols with unknown dtypes: {self._cols_with_unknown_dtypes};\n\t"
+ f"schema is known: {self._schema_is_known};\n\t"
+ f"has parent df: {self._parent_df is not None};\n\t"
+ f"columns order: {self._columns_order};\n\t"
+ f"know all names: {self._know_all_names}"
)
def __str__(self): # noqa: GL08
return self.__repr__()
def lazy_get(
self, ids: list[Union[IndexLabel, int]], numeric_index: bool = False
) -> DtypesDescriptor:
"""
Get dtypes descriptor for a subset of columns without triggering any computations.
Parameters
----------
ids : list of index labels or positional indexers
Columns for the subset.
numeric_index : bool, default: False
Whether `ids` are positional indixes or column labels to take.
Returns
-------
DtypesDescriptor
Descriptor that describes dtypes for columns specified in `ids`.
"""
if len(set(ids)) != len(ids):
raise NotImplementedError(
"Duplicated column names are not yet supported by DtypesDescriptor"
)
if numeric_index:
if self.columns_order is not None:
ids = [self.columns_order[i] for i in ids]
else:
raise ValueError(
"Can't lazily get columns by positional indixers if the columns order is unknown"
)
result = {}
unknown_cols = []
columns_order = {}
for i, col in enumerate(ids):
columns_order[i] = col
if col in self._cols_with_unknown_dtypes:
unknown_cols.append(col)
continue
dtype = self._known_dtypes.get(col)
if dtype is None and self._remaining_dtype is None:
unknown_cols.append(col)
elif dtype is None and self._remaining_dtype is not None:
result[col] = self._remaining_dtype
else:
result[col] = dtype
remaining_dtype = self._remaining_dtype if len(unknown_cols) != 0 else None
return DtypesDescriptor(
result,
unknown_cols,
remaining_dtype,
self._parent_df,
columns_order=columns_order,
)
def copy(self) -> DtypesDescriptor:
"""
Get a copy of this descriptor.
Returns
-------
DtypesDescriptor
"""
return type(self)(
# should access '.columns_order' first, as it may compute columns order
# and complete the metadata for 'self'
columns_order=(
None if self.columns_order is None else self.columns_order.copy()
),
known_dtypes=self._known_dtypes.copy(),
cols_with_unknown_dtypes=self._cols_with_unknown_dtypes.copy(),
remaining_dtype=self._remaining_dtype,
parent_df=self._parent_df,
know_all_names=self._know_all_names,
_schema_is_known=self._schema_is_known,
)
def set_index(self, new_index: Union[pandas.Index, ModinIndex]) -> DtypesDescriptor:
"""
Set new column names for this descriptor.
Parameters
----------
new_index : pandas.Index or ModinIndex
Returns
-------
DtypesDescriptor
New descriptor with updated column names.
Notes
-----
Calling this method on a descriptor that returns ``None`` for ``.columns_order``
will result into information lose.
"""
if len(new_index) != len(set(new_index)):
raise NotImplementedError(
"Duplicated column names are not yet supported by DtypesDescriptor"
)
if self.columns_order is None:
# we can't map new columns to old columns and lost all dtypes :(
return DtypesDescriptor(
cols_with_unknown_dtypes=new_index,
columns_order={i: col for i, col in enumerate(new_index)},
parent_df=self._parent_df,
know_all_names=True,
)
new_self = self.copy()
renamer = {old_c: new_index[i] for i, old_c in new_self.columns_order.items()}
new_self._known_dtypes = {
renamer[old_col]: value for old_col, value in new_self._known_dtypes.items()
}
new_self._cols_with_unknown_dtypes = [
renamer[old_col] for old_col in new_self._cols_with_unknown_dtypes
]
new_self._columns_order = {
i: renamer[old_col] for i, old_col in new_self._columns_order.items()
}
return new_self
def equals(self, other: DtypesDescriptor) -> bool:
"""
Compare two descriptors for equality.
Parameters
----------
other : DtypesDescriptor
Returns
-------
bool
"""
return (
self._known_dtypes == other._known_dtypes
and set(self._cols_with_unknown_dtypes)
== set(other._cols_with_unknown_dtypes)
and self._remaining_dtype == other._remaining_dtype
and self._schema_is_known == other._schema_is_known
and self.columns_order == other.columns_order
and self._know_all_names == other._know_all_names
)
@property
def is_materialized(self) -> bool:
"""
Whether this descriptor contains information about all dtypes in the dataframe.
Returns
-------
bool
"""
return self._schema_is_known
def _materialize_all_names(self):
"""Materialize missing column names."""
if self._know_all_names:
return
all_cols = self._parent_df.columns
self._normalize_self_levels(all_cols)
for col in all_cols:
if (
col not in self._known_dtypes
and col not in self._cols_with_unknown_dtypes
):
self._cols_with_unknown_dtypes.append(col)
self._know_all_names = True
def _materialize_cols_with_unknown_dtypes(self):
"""Compute dtypes for cols specified in `._cols_with_unknown_dtypes`."""
if (
len(self._known_dtypes) == 0
and len(self._cols_with_unknown_dtypes) == 0
and not self._know_all_names
):
# here we have to compute dtypes for all columns in the dataframe,
# so avoiding columns materialization by setting 'subset=None'
subset = None
else:
if not self._know_all_names:
self._materialize_all_names()
subset = self._cols_with_unknown_dtypes
if subset is None or len(subset) > 0:
self._known_dtypes.update(self._parent_df._compute_dtypes(subset))
self._know_all_names = True
self._cols_with_unknown_dtypes = []
def materialize(self):
"""Complete information about dtypes."""
if self.is_materialized:
return
if self._parent_df is None:
raise RuntimeError(
"It's not allowed to call '.materialize()' before '._parent_df' is specified."
)
self._materialize_cols_with_unknown_dtypes()
if self._remaining_dtype is not None:
cols = self._parent_df.columns
self._normalize_self_levels(cols)
self._known_dtypes.update(
{
col: self._remaining_dtype
for col in cols
if col not in self._known_dtypes
}
)
# we currently not guarantee for dtypes to be in a proper order:
# https://github.com/modin-project/modin/blob/8a332c1597c54d36f7ccbbd544e186b689f9ceb1/modin/pandas/test/utils.py#L644-L646
# so restoring the order only if it's possible
if self.columns_order is not None:
assert len(self.columns_order) == len(self._known_dtypes)
self._known_dtypes = {
self.columns_order[i]: self._known_dtypes[self.columns_order[i]]
for i in range(len(self.columns_order))
}
self._schema_is_known = True
self._remaining_dtype = None
self._parent_df = None
def to_series(self) -> pandas.Series:
"""
Convert descriptor to a pandas Series.
Returns
-------
pandas.Series
"""
self.materialize()
return pandas.Series(self._known_dtypes)
def get_dtypes_set(self) -> set[DtypeObj]:
"""
Get a set of dtypes from the descriptor.
Returns
-------
set[DtypeObj]
"""
if len(self._cols_with_unknown_dtypes) > 0 or not self._know_all_names:
self._materialize_cols_with_unknown_dtypes()
known_dtypes: set[DtypeObj] = set(self._known_dtypes.values())
if self._remaining_dtype is not None:
known_dtypes.add(self._remaining_dtype)
return known_dtypes
@classmethod
def _merge_dtypes(
cls, values: list[Union[DtypesDescriptor, pandas.Series, None]]
) -> DtypesDescriptor:
"""
Union columns described by ``values`` and compute common dtypes for them.
Parameters
----------
values : list of DtypesDescriptors, pandas.Series or Nones
Returns
-------
DtypesDescriptor
"""
known_dtypes = {}
cols_with_unknown_dtypes = []
know_all_names = True
dtypes_are_unknown = False
# index - joined column names, columns - dtypes taken from 'values'
# 0 1 2 3
# col1 int bool float int
# col2 int int int int
# colN bool bool bool int
dtypes_matrix = pandas.DataFrame()
for i, val in enumerate(values):
if isinstance(val, cls):
know_all_names &= val._know_all_names
dtypes = val._known_dtypes.copy()
dtypes.update({col: "unknown" for col in val._cols_with_unknown_dtypes})
if val._remaining_dtype is not None:
# we can't process remaining dtypes, so just discarding them
know_all_names = False
# setting a custom name to the Series to prevent duplicated names
# in the 'dtypes_matrix'
series = pandas.Series(dtypes, name=i)
dtypes_matrix = pandas.concat([dtypes_matrix, series], axis=1)
if not (val._know_all_names and val._remaining_dtype is None):
dtypes_matrix.fillna(
value={
# If we encountered a 'NaN' while 'val' describes all the columns, then
# it means, that the missing columns for this instance will be filled with NaNs (floats),
# otherwise, it may indicate missing columns that this 'val' has no info about,
# meaning that we shouldn't try computing a new dtype for this column,
# so marking it as 'unknown'
i: "unknown",
},
inplace=True,
)
elif isinstance(val, pandas.Series):
dtypes_matrix = pandas.concat([dtypes_matrix, val], axis=1)
elif val is None:
# one of the 'dtypes' is None, meaning that we wouldn't been infer a valid result dtype,
# however, we're continuing our loop so we would at least know the columns we're missing
# dtypes for
dtypes_are_unknown = True
know_all_names = False
else:
raise NotImplementedError(type(val))
if dtypes_are_unknown:
return DtypesDescriptor(
cols_with_unknown_dtypes=dtypes_matrix.index.tolist(),
know_all_names=know_all_names,
)
def combine_dtypes(row):
if (row == "unknown").any():
return "unknown"
row = row.fillna(pandas.api.types.pandas_dtype("float"))
return find_common_type(list(row.values))
dtypes = dtypes_matrix.apply(combine_dtypes, axis=1)
for col, dtype in dtypes.items():
if dtype == "unknown":
cols_with_unknown_dtypes.append(col)
else:
known_dtypes[col] = dtype
return DtypesDescriptor(
known_dtypes,
cols_with_unknown_dtypes,
remaining_dtype=None,
know_all_names=know_all_names,
)
@classmethod
def concat(
cls, values: list[Union[DtypesDescriptor, pandas.Series, None]], axis: int = 0
) -> DtypesDescriptor:
"""
Concatenate dtypes descriptors into a single descriptor.
Parameters
----------
values : list of DtypesDescriptors and pandas.Series
axis : int, default: 0
If ``axis == 0``: concatenate column names. This implements the logic of
how dtypes are combined on ``pd.concat([df1, df2], axis=1)``.
If ``axis == 1``: perform a union join for the column names described by
`values` and then find common dtypes for the columns appeared to be in
an intersection. This implements the logic of how dtypes are combined on
``pd.concat([df1, df2], axis=0).dtypes``.
Returns
-------
DtypesDescriptor
"""
if axis == 1:
return cls._merge_dtypes(values)
known_dtypes = {}
cols_with_unknown_dtypes = []
schema_is_known = True
# some default value to not mix it with 'None'
remaining_dtype = "default"
know_all_names = True
for val in values:
if isinstance(val, cls):
all_new_cols = (
list(val._known_dtypes.keys()) + val._cols_with_unknown_dtypes
)
if any(
col in known_dtypes or col in cols_with_unknown_dtypes
for col in all_new_cols
):
raise NotImplementedError(
"Duplicated column names are not yet supported by DtypesDescriptor"
)
know_all_names &= val._know_all_names
known_dtypes.update(val._known_dtypes)
cols_with_unknown_dtypes.extend(val._cols_with_unknown_dtypes)
if know_all_names:
if (
remaining_dtype == "default"
and val._remaining_dtype is not None
):
remaining_dtype = val._remaining_dtype
elif (
remaining_dtype != "default"
and val._remaining_dtype is not None
and remaining_dtype != val._remaining_dtype
):
remaining_dtype = None
know_all_names = False
else:
remaining_dtype = None
schema_is_known &= val._schema_is_known
elif isinstance(val, pandas.Series):
if any(
col in known_dtypes or col in cols_with_unknown_dtypes
for col in val.index
):
raise NotImplementedError(
"Duplicated column names are not yet supported by DtypesDescriptor"
)
known_dtypes.update(val)
elif val is None:
remaining_dtype = None
schema_is_known = False
know_all_names = False
else:
raise NotImplementedError(type(val))
return cls(
known_dtypes,
cols_with_unknown_dtypes,
None if remaining_dtype == "default" else remaining_dtype,
parent_df=None,
_schema_is_known=schema_is_known,
know_all_names=know_all_names,
)
@staticmethod
def _normalize_levels(columns, reference=None):
"""
Normalize levels of MultiIndex column names.
The function fills missing levels with empty strings as pandas do:
'''
>>> columns = ["a", ("l1", "l2"), ("l1a", "l2a", "l3a")]
>>> _normalize_levels(columns)
[("a", "", ""), ("l1", "l2", ""), ("l1a", "l2a", "l3a")]
>>> # with a reference
>>> idx = pandas.MultiIndex(...)
>>> idx.nlevels
4
>>> _normalize_levels(columns, reference=idx)
[("a", "", "", ""), ("l1", "l2", "", ""), ("l1a", "l2a", "l3a", "")]
'''
Parameters
----------
columns : sequence
Labels to normalize. If dictionary, will replace keys with normalized columns.
reference : pandas.Index, optional
An index to match the number of levels with. If reference is a MultiIndex, then the reference number
of levels should not be greater than the maximum number of levels in `columns`. If not specified,
the `columns` themselves become a `reference`.
Returns
-------
sequence
Column values with normalized levels.
dict[hashable, hashable]
Mapping from old column names to new names, only contains column names that
were changed.
Raises
------
ValueError
When the reference number of levels is greater than the maximum number of levels
in `columns`.
"""
if reference is None:
reference = columns
if isinstance(reference, pandas.Index):
max_nlevels = reference.nlevels
else:
max_nlevels = 1
for col in reference:
if isinstance(col, tuple):
max_nlevels = max(max_nlevels, len(col))
# if the reference is a regular flat index, then no actions are required (the result will be
# a flat index containing tuples of different lengths, this behavior fully matches pandas).
# Yes, this shortcut skips the 'if max_columns_nlevels > max_nlevels' below check on purpose.
if max_nlevels == 1:
return columns, {}
max_columns_nlevels = 1
for col in columns:
if isinstance(col, tuple):
max_columns_nlevels = max(max_columns_nlevels, len(col))
if max_columns_nlevels > max_nlevels:
raise ValueError(
f"The reference number of levels is greater than the maximum number of levels in columns: {max_columns_nlevels} > {max_nlevels}"
)
new_columns = []
old_to_new_mapping = {}
for col in columns:
old_col = col
if not isinstance(col, tuple):
col = (col,)
col = col + ("",) * (max_nlevels - len(col))
new_columns.append(col)
if old_col != col:
old_to_new_mapping[old_col] = col
return new_columns, old_to_new_mapping
def _normalize_self_levels(self, reference=None):
"""
Call ``self._normalize_levels()`` for known and unknown dtypes of this object.
Parameters
----------
reference : pandas.Index, optional
"""
_, old_to_new_mapping = self._normalize_levels(
self._known_dtypes.keys(), reference
)
for old_col, new_col in old_to_new_mapping.items():
value = self._known_dtypes.pop(old_col)
self._known_dtypes[new_col] = value
self._cols_with_unknown_dtypes, _ = self._normalize_levels(
self._cols_with_unknown_dtypes, reference
)
class ModinDtypes:
"""
A class that hides the various implementations of the dtypes needed for optimization.
Parameters
----------
value : pandas.Series, callable, DtypesDescriptor or ModinDtypes, optional
"""
def __init__(
self,
value: Optional[Union[Callable, pandas.Series, DtypesDescriptor, ModinDtypes]],
):
if callable(value) or isinstance(value, pandas.Series):
self._value = value
elif isinstance(value, DtypesDescriptor):
self._value = value.to_series() if value.is_materialized else value
elif isinstance(value, type(self)):
self._value = value.copy()._value
elif isinstance(value, None):
self._value = DtypesDescriptor()
else:
raise ValueError(f"ModinDtypes doesn't work with '{value}'")
def __repr__(self): # noqa: GL08
return f"ModinDtypes:\n\tvalue type: {type(self._value)};\n\tvalue:\n\t{self._value}"
def __str__(self): # noqa: GL08
return self.__repr__()
@property
def is_materialized(self) -> bool:
"""
Check if the internal representation is materialized.
Returns
-------
bool
"""
return isinstance(self._value, pandas.Series)
def get_dtypes_set(self) -> set[DtypeObj]:
"""
Get a set of dtypes from the descriptor.
Returns
-------
set[DtypeObj]
"""
if isinstance(self._value, DtypesDescriptor):
return self._value.get_dtypes_set()
if not self.is_materialized:
self.get()
return set(self._value.values)
def maybe_specify_new_frame_ref(self, new_parent: PandasDataframe) -> ModinDtypes:
"""
Set a new parent for the stored value if needed.
Parameters
----------
new_parent : PandasDataframe
Returns
-------
ModinDtypes
A copy of ``ModinDtypes`` with updated parent.
"""
new_self = self.copy()
if new_self.is_materialized:
LazyProxyCategoricalDtype.update_dtypes(new_self._value, new_parent)
return new_self
if isinstance(self._value, DtypesDescriptor):
new_self._value.update_parent(new_parent)
return new_self
return new_self
def lazy_get(self, ids: list, numeric_index: bool = False) -> ModinDtypes:
"""
Get new ``ModinDtypes`` for a subset of columns without triggering any computations.
Parameters
----------
ids : list of index labels or positional indexers
Columns for the subset.
numeric_index : bool, default: False
Whether `ids` are positional indixes or column labels to take.
Returns
-------
ModinDtypes
``ModinDtypes`` that describes dtypes for columns specified in `ids`.
"""
if isinstance(self._value, DtypesDescriptor):
res = self._value.lazy_get(ids, numeric_index)
return ModinDtypes(res)
elif callable(self._value):
new_self = self.copy()
old_value = new_self._value
new_self._value = lambda: (
old_value().iloc[ids] if numeric_index else old_value()[ids]
)
return new_self
ErrorMessage.catch_bugs_and_request_email(
failure_condition=not self.is_materialized
)
return ModinDtypes(self._value.iloc[ids] if numeric_index else self._value[ids])
@classmethod
def concat(cls, values: list, axis: int = 0) -> ModinDtypes:
"""
Concatenate dtypes.
Parameters
----------
values : list of DtypesDescriptors, pandas.Series, ModinDtypes and Nones
axis : int, default: 0
If ``axis == 0``: concatenate column names. This implements the logic of
how dtypes are combined on ``pd.concat([df1, df2], axis=1)``.
If ``axis == 1``: perform a union join for the column names described by
`values` and then find common dtypes for the columns appeared to be in
an intersection. This implements the logic of how dtypes are combined on
``pd.concat([df1, df2], axis=0).dtypes``.
Returns
-------
ModinDtypes
"""
preprocessed_vals = []
for val in values:
if isinstance(val, cls):
val = val._value
if isinstance(val, (DtypesDescriptor, pandas.Series)) or val is None:
preprocessed_vals.append(val)
else:
raise NotImplementedError(type(val))
try:
desc = DtypesDescriptor.concat(preprocessed_vals, axis=axis)
except NotImplementedError as e:
# 'DtypesDescriptor' doesn't support duplicated labels, however, if all values are pandas Series,
# we still can perform concatenation using pure pandas
if (
# 'pd.concat(axis=1)' fails on duplicated labels anyway, so doing this logic
# only in case 'axis=0'
axis == 0
and "duplicated" not in e.args[0].lower()
or not all(isinstance(val, pandas.Series) for val in values)
):
raise e
desc = pandas.concat(values)
return ModinDtypes(desc)
def set_index(self, new_index: Union[pandas.Index, ModinIndex]) -> ModinDtypes:
"""
Set new column names for stored dtypes.
Parameters
----------
new_index : pandas.Index or ModinIndex
Returns
-------
ModinDtypes
New ``ModinDtypes`` with updated column names.
"""
new_self = self.copy()
if self.is_materialized:
new_self._value.index = new_index
return new_self
elif callable(self._value):
old_val = new_self._value
new_self._value = lambda: old_val().set_axis(new_index)
return new_self
elif isinstance(new_self._value, DtypesDescriptor):
new_self._value = new_self._value.set_index(new_index)
return new_self
else:
raise NotImplementedError()
def get(self) -> pandas.Series:
"""
Get the materialized internal representation.
Returns
-------
pandas.Series
"""
if not self.is_materialized:
if callable(self._value):
self._value = self._value()
if self._value is None:
self._value = pandas.Series([])
elif isinstance(self._value, DtypesDescriptor):
self._value = self._value.to_series()
else:
raise NotImplementedError(type(self._value))
return self._value
def __len__(self):
"""
Redirect the 'len' request to the internal representation.
Returns
-------
int
Notes
-----
Executing this function materializes the data.
"""
if not self.is_materialized:
self.get()
return len(self._value)
def __reduce__(self):
"""
Serialize an object of this class.
Returns
-------
tuple
Notes
-----
The default implementation generates a recursion error. In a short:
during the construction of the object, `__getattr__` function is called, which
is not intended to be used in situations where the object is not initialized.
"""
return (self.__class__, (self._value,))
def __getattr__(self, name):
"""
Redirect access to non-existent attributes to the internal representation.
This is necessary so that objects of this class in most cases mimic the behavior
of the ``pandas.Series``. The main limitations of the current approach are type
checking and the use of this object where pandas dtypes are supposed to be used.
Parameters
----------
name : str
Attribute name.
Returns
-------
object
Attribute.
Notes
-----
Executing this function materializes the data.
"""
if not self.is_materialized:
self.get()
return self._value.__getattribute__(name)
def copy(self) -> ModinDtypes:
"""
Copy an object without materializing the internal representation.
Returns
-------
ModinDtypes
"""
idx_cache = self._value
if not callable(idx_cache):
idx_cache = idx_cache.copy()
return ModinDtypes(idx_cache)
def __getitem__(self, key): # noqa: GL08
if not self.is_materialized:
self.get()
return self._value.__getitem__(key)
def __setitem__(self, key, item): # noqa: GL08
if not self.is_materialized:
self.get()
self._value.__setitem__(key, item)
def __iter__(self): # noqa: GL08
if not self.is_materialized:
self.get()
return iter(self._value)
def __contains__(self, key): # noqa: GL08
if not self.is_materialized:
self.get()
return key in self._value
class LazyProxyCategoricalDtype(pandas.CategoricalDtype):
"""
A lazy proxy representing ``pandas.CategoricalDtype``.
Parameters
----------
categories : list-like, optional
ordered : bool, default: False
Notes
-----
Important note! One shouldn't use the class' constructor to instantiate a proxy instance,
it's intended only for compatibility purposes! In order to create a new proxy instance
use the appropriate class method `._build_proxy(...)`.
"""
def __init__(self, categories=None, ordered=False):
# These will be initialized later inside of the `._build_proxy()` method
self._parent, self._column_name, self._categories_val, self._materializer = (
None,
None,
None,
None,
)
super().__init__(categories, ordered)
@staticmethod
def update_dtypes(dtypes, new_parent):
"""
Update a parent for categorical proxies in a dtype object.
Parameters
----------
dtypes : dict-like
A dict-like object describing dtypes. The method will walk through every dtype
an update parents for categorical proxies inplace.
new_parent : object
"""
for key, value in dtypes.items():
if isinstance(value, LazyProxyCategoricalDtype):
dtypes[key] = value._update_proxy(new_parent, column_name=key)
def _update_proxy(self, parent, column_name):
"""
Create a new proxy, if either parent or column name are different.
Parameters
----------
parent : object
Source object to extract categories on demand.
column_name : str
Column name of the categorical column in the source object.
Returns
-------
pandas.CategoricalDtype or LazyProxyCategoricalDtype
"""
if self._is_materialized:
# The parent has been materialized, we don't need a proxy anymore.
return pandas.CategoricalDtype(self.categories, ordered=self._ordered)
elif parent is self._parent and column_name == self._column_name:
return self
else:
return self._build_proxy(parent, column_name, self._materializer)
@classmethod
def _build_proxy(cls, parent, column_name, materializer, dtype=None):
"""
Construct a lazy proxy.
Parameters
----------
parent : object
Source object to extract categories on demand.
column_name : str
Column name of the categorical column in the source object.
materializer : callable(parent, column_name) -> pandas.CategoricalDtype
A function to call in order to extract categorical values.
dtype : dtype, optional
The categories dtype.
Returns
-------
LazyProxyCategoricalDtype
"""
result = cls()
result._parent = parent
result._column_name = column_name
result._materializer = materializer
result._dtype = dtype
return result
def _get_dtype(self):
"""
Get the categories dtype.
Returns
-------
dtype
"""
if self._dtype is None:
self._dtype = self.categories.dtype
return self._dtype
def __reduce__(self):
"""
Serialize an object of this class.
Returns
-------
tuple
Notes
-----
This object is serialized into a ``pandas.CategoricalDtype`` as an actual proxy can't be
properly serialized because of the references it stores for its potentially distributed parent.
"""
return (pandas.CategoricalDtype, (self.categories, self.ordered))
@property
def _categories(self):
"""
Get materialized categorical values.
Returns
-------
pandas.Index
"""
if not self._is_materialized:
self._materialize_categories()
return self._categories_val
@_categories.setter
def _categories(self, categories):
"""
Set new categorical values.
Parameters
----------
categories : list-like
"""
self._categories_val = categories
self._parent = None # The parent is not required any more
self._materializer = None
self._dtype = None
@property
def _is_materialized(self) -> bool:
"""
Check whether categorical values were already materialized.
Returns
-------
bool
"""
return self._categories_val is not None
def _materialize_categories(self):
"""Materialize actual categorical values."""
ErrorMessage.catch_bugs_and_request_email(
failure_condition=self._parent is None,
extra_log="attempted to materialize categories with parent being 'None'",
)
categoricals = self._materializer(self._parent, self._column_name)
self._categories = categoricals.categories
self._ordered = categoricals.ordered
def get_categories_dtype(
cdt: Union[LazyProxyCategoricalDtype, pandas.CategoricalDtype],
) -> DtypeObj:
"""
Get the categories dtype.
Parameters
----------
cdt : LazyProxyCategoricalDtype or pandas.CategoricalDtype
Returns
-------
dtype
"""
return (
cdt._get_dtype()
if isinstance(cdt, LazyProxyCategoricalDtype)
else cdt.categories.dtype
)
def extract_dtype(value) -> DtypeObj | pandas.Series:
"""
Extract dtype(s) from the passed `value`.
Parameters
----------
value : object
Returns
-------
DtypeObj or pandas.Series of DtypeObj
"""
try:
dtype = pandas.api.types.pandas_dtype(value)
except (TypeError, ValueError):
dtype = pandas.Series(value).dtype
return dtype
================================================
FILE: modin/core/dataframe/pandas/metadata/index.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module contains class ModinIndex."""
import uuid
from typing import Optional
import pandas
from pandas.core.dtypes.common import is_list_like
from pandas.core.indexes.api import ensure_index
class ModinIndex:
"""
A class that hides the various implementations of the index needed for optimization.
Parameters
----------
value : sequence, PandasDataframe or callable() -> (pandas.Index, list of ints), optional
If a sequence passed this will be considered as the index values.
If a ``PandasDataframe`` passed then it will be used to lazily extract indices
when required, note that the `axis` parameter must be passed in this case.
If a callable passed then it's expected to return a pandas Index and a list of
partition lengths along the index axis.
If ``None`` was passed, the index will be considered an incomplete and will raise
a ``RuntimeError`` on an attempt of materialization. To complete the index object
you have to use ``.maybe_specify_new_frame_ref()`` method.
axis : int, optional
Specifies an axis the object represents, serves as an optional hint. This parameter
must be passed in case value is a ``PandasDataframe``.
dtypes : pandas.Series, optional
Materialized dtypes of index levels.
"""
def __init__(self, value=None, axis=None, dtypes: Optional[pandas.Series] = None):
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
self._is_default_callable = False
self._axis = axis
self._dtypes = dtypes
if callable(value):
self._value = value
elif isinstance(value, PandasDataframe):
assert axis is not None
self._value = self._get_default_callable(value, axis)
self._is_default_callable = True
elif value is None:
assert axis is not None
self._value = value
else:
self._value = ensure_index(value)
self._lengths_cache = None
# index/lengths ID's for faster comparison between other ModinIndex objects,
# these should be propagated to the copies of the index
self._index_id = uuid.uuid4()
self._lengths_id = uuid.uuid4()
def maybe_get_dtypes(self) -> Optional[pandas.Series]:
"""
Get index dtypes if available.
Returns
-------
pandas.Series or None
"""
if self._dtypes is not None:
return self._dtypes
if self.is_materialized:
self._dtypes = (
self._value.dtypes
if isinstance(self._value, pandas.MultiIndex)
else pandas.Series([self._value.dtype], index=[self._value.name])
)
return self._dtypes
return None
@staticmethod
def _get_default_callable(dataframe_obj, axis):
"""
Build a callable extracting index labels and partitions lengths for the specified axis.
Parameters
----------
dataframe_obj : PandasDataframe
axis : int
0 - extract indices, 1 - extract columns.
Returns
-------
callable() -> tuple(pandas.Index, list[ints])
"""
return lambda: dataframe_obj._compute_axis_labels_and_lengths(axis)
def maybe_specify_new_frame_ref(self, value, axis) -> "ModinIndex":
"""
Set a new reference for a frame used to lazily extract index labels if it's needed.
The method sets a new reference only if the indices are not yet materialized and
if a PandasDataframe was originally passed to construct this index (so the ModinIndex
object holds a reference to it). The reason the reference should be updated is that
we don't want to hold in memory those frames that are already not needed. Once the
reference is updated, the old frame will be garbage collected if there are no
more references to it.
Parameters
----------
value : PandasDataframe
New dataframe to reference.
axis : int
Axis to extract labels from.
Returns
-------
ModinIndex
New ModinIndex with the reference updated.
"""
if self._value is not None and (
not callable(self._value) or not self._is_default_callable
):
return self
new_index = self.copy(copy_lengths=True)
new_index._axis = axis
new_index._value = self._get_default_callable(value, new_index._axis)
# if the '._value' was 'None' initially, then the '_is_default_callable' flag was
# also being set to 'False', since now the '._value' is a default callable,
# so we want to ensure that the flag is set to 'True'
new_index._is_default_callable = True
return new_index
@property
def is_materialized(self) -> bool:
"""
Check if the internal representation is materialized.
Returns
-------
bool
"""
return self.is_materialized_index(self)
@classmethod
def is_materialized_index(cls, index) -> bool:
"""
Check if the passed object represents a materialized index.
Parameters
----------
index : object
An object to check.
Returns
-------
bool
"""
# importing here to avoid circular import issue
from modin.pandas.indexing import is_range_like
if isinstance(index, cls):
index = index._value
return is_list_like(index) or is_range_like(index) or isinstance(index, slice)
def get(self, return_lengths=False) -> pandas.Index:
"""
Get the materialized internal representation.
Parameters
----------
return_lengths : bool, default: False
In some cases, during the index calculation, it's possible to get
the lengths of the partitions. This flag allows this data to be used
for optimization.
Returns
-------
pandas.Index
"""
if not self.is_materialized:
if callable(self._value):
index, self._lengths_cache = self._value()
self._value = ensure_index(index)
elif self._value is None:
raise RuntimeError(
"It's not allowed to call '.materialize()' before '._value' is specified."
)
else:
raise NotImplementedError(type(self._value))
if return_lengths:
return self._value, self._lengths_cache
else:
return self._value
def equals(self, other: "ModinIndex") -> bool:
"""
Check equality of the index values.
Parameters
----------
other : ModinIndex
Returns
-------
bool
The result of the comparison.
"""
if self._index_id == other._index_id:
return True
if not self.is_materialized:
self.get()
if not other.is_materialized:
other.get()
return self._value.equals(other._value)
def compare_partition_lengths_if_possible(self, other: "ModinIndex"):
"""
Compare the partition lengths cache for the index being stored if possible.
The ``ModinIndex`` object may sometimes store the information about partition
lengths along the axis the index belongs to. If both `self` and `other` have
this information or it can be inferred from them, the method returns
a boolean - the result of the comparison, otherwise it returns ``None``
as an indication that the comparison cannot be made.
Parameters
----------
other : ModinIndex
Returns
-------
bool or None
The result of the comparison if both `self` and `other` contain
the lengths data, ``None`` otherwise.
"""
if self._lengths_id == other._lengths_id:
return True
can_extract_lengths_from_self = self._lengths_cache is not None or callable(
self._value
)
can_extract_lengths_from_other = other._lengths_cache is not None or callable(
other._value
)
if can_extract_lengths_from_self and can_extract_lengths_from_other:
return self.get(return_lengths=True)[1] == other.get(return_lengths=True)[1]
return None
def __len__(self):
"""
Redirect the 'len' request to the internal representation.
Returns
-------
int
Notes
-----
Executing this function materializes the data.
"""
if not self.is_materialized:
self.get()
return len(self._value)
def __reduce__(self):
"""
Serialize an object of this class.
Returns
-------
tuple
Notes
-----
The default implementation generates a recursion error. In a short:
during the construction of the object, `__getattr__` function is called, which
is not intended to be used in situations where the object is not initialized.
"""
return (
self.__class__,
(self._value, self._axis),
{
"_lengths_cache": self._lengths_cache,
"_index_id": self._index_id,
"_lengths_id": self._lengths_id,
"_is_default_callable": self._is_default_callable,
},
)
def __getitem__(self, key):
"""
Get an index value at the position of `key`.
Parameters
----------
key : int
Returns
-------
label
"""
if not self.is_materialized:
self.get()
return self._value[key]
def __getattr__(self, name):
"""
Redirect access to non-existent attributes to the internal representation.
This is necessary so that objects of this class in most cases mimic the behavior
of the ``pandas.Index``. The main limitations of the current approach are type
checking and the use of this object where pandas indexes are supposed to be used.
Parameters
----------
name : str
Attribute name.
Returns
-------
object
Attribute.
Notes
-----
Executing this function materializes the data.
"""
if not self.is_materialized:
self.get()
return self._value.__getattribute__(name)
def copy(self, copy_lengths=False) -> "ModinIndex":
"""
Copy an object without materializing the internal representation.
Parameters
----------
copy_lengths : bool, default: False
Whether to copy the stored partition lengths to the
new index object.
Returns
-------
ModinIndex
"""
idx_cache = self._value
if idx_cache is not None and not callable(idx_cache):
idx_cache = idx_cache.copy()
result = ModinIndex(idx_cache, axis=self._axis, dtypes=self._dtypes)
result._index_id = self._index_id
result._is_default_callable = self._is_default_callable
if copy_lengths:
result._lengths_cache = self._lengths_cache
result._lengths_id = self._lengths_id
return result
================================================
FILE: modin/core/dataframe/pandas/partitioning/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe classes related to its partitioning and optimized for pandas storage format."""
================================================
FILE: modin/core/dataframe/pandas/partitioning/axis_partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""The module defines base interface for an axis partition of a Modin DataFrame."""
import warnings
import numpy as np
import pandas
from modin.config import MinColumnPartitionSize, MinRowPartitionSize
from modin.core.dataframe.base.partitioning.axis_partition import (
BaseDataframeAxisPartition,
)
from modin.core.storage_formats.pandas.utils import (
generate_result_of_axis_func_pandas,
split_result_of_axis_func_pandas,
)
from .partition import PandasDataframePartition
class PandasDataframeAxisPartition(BaseDataframeAxisPartition):
"""
An abstract class is created to simplify and consolidate the code for axis partition that run pandas.
Because much of the code is similar, this allows us to reuse this code.
Parameters
----------
list_of_partitions : Union[list, PandasDataframePartition]
List of ``PandasDataframePartition`` and
``PandasDataframeAxisPartition`` objects, or a single
``PandasDataframePartition``.
get_ip : bool, default: False
Whether to get node IP addresses to conforming partitions or not.
full_axis : bool, default: True
Whether or not the axis partition encompasses the whole axis.
call_queue : list, optional
A list of tuples (callable, args, kwargs) that contains deferred calls.
length : the future's type or int, optional
Length, or reference to length, of wrapped ``pandas.DataFrame``.
width : the future's type or int, optional
Width, or reference to width, of wrapped ``pandas.DataFrame``.
"""
def __init__(
self,
list_of_partitions,
get_ip=False,
full_axis=True,
call_queue=None,
length=None,
width=None,
):
if isinstance(list_of_partitions, PandasDataframePartition):
list_of_partitions = [list_of_partitions]
self.full_axis = full_axis
self.call_queue = call_queue or []
self._length_cache = length
self._width_cache = width
# Check that all axis partition axes are the same in `list_of_partitions`
# We should never have mismatching axis in the current implementation. We add this
# defensive assertion to ensure that undefined behavior does not happen.
assert (
len(
set(
obj.axis
for obj in list_of_partitions
if isinstance(obj, PandasDataframeAxisPartition)
)
)
<= 1
)
self._list_of_constituent_partitions = list_of_partitions
# Defer computing _list_of_block_partitions because we might need to
# drain call queues for that.
self._list_of_block_partitions = None
@property
def list_of_blocks(self):
"""
Get the list of physical partition objects that compose this partition.
Returns
-------
list
A list of physical partition objects (``ray.ObjectRef``, ``distributed.Future`` e.g.).
"""
# Defer draining call queue (which is hidden in `partition.list_of_blocks` call) until we get the partitions.
# TODO Look into draining call queue at the same time as the task
return [
partition.list_of_blocks[0] for partition in self.list_of_block_partitions
]
@property
def list_of_block_partitions(self) -> list:
"""
Get the list of block partitions that compose this partition.
Returns
-------
List
A list of ``PandasDataframePartition``.
"""
if self._list_of_block_partitions is not None:
return self._list_of_block_partitions
self._list_of_block_partitions = []
# Extract block partitions from the block and axis partitions that
# constitute this partition.
for partition in self._list_of_constituent_partitions:
if isinstance(partition, PandasDataframeAxisPartition):
if partition.axis == self.axis:
# We are building an axis partition out of another
# axis partition `partition` that contains its own list
# of block partitions, partition.list_of_block_partitions.
# `partition` may have its own call queue, which has to be
# applied to the entire `partition` before we execute any
# further operations on its block parittions.
partition.drain_call_queue()
self._list_of_block_partitions.extend(
partition.list_of_block_partitions
)
else:
# If this axis partition is made of axis partitions
# for the other axes, squeeze such partitions into a single
# block so that this partition only holds a one-dimensional
# list of blocks. We could change this implementation to
# hold a 2-d list of blocks, but that would complicate the
# code quite a bit.
self._list_of_block_partitions.append(
partition.force_materialization().list_of_block_partitions[0]
)
else:
self._list_of_block_partitions.append(partition)
return self._list_of_block_partitions
@classmethod
def _get_drain_func(cls): # noqa: GL08
return PandasDataframeAxisPartition.drain
def drain_call_queue(self, num_splits=None):
"""
Execute all operations stored in this partition's call queue.
Parameters
----------
num_splits : int, default: None
The number of times to split the result object.
"""
if len(self.call_queue) == 0:
# this implicitly calls `drain_call_queue` for block partitions,
# which might have deferred call queues
_ = self.list_of_blocks
return
call_queue = self.call_queue
try:
# Clearing the queue before calling `.apply()` so it won't try to drain it repeatedly
self.call_queue = []
drained = self.apply(
self._get_drain_func(), num_splits=num_splits, call_queue=call_queue
)
except Exception:
# Restoring the call queue in case of an exception as it most likely wasn't drained
self.call_queue = call_queue
raise
if not isinstance(drained, list):
drained = [drained]
self._list_of_block_partitions = drained
def force_materialization(self, get_ip=False):
"""
Materialize partitions into a single partition.
Parameters
----------
get_ip : bool, default: False
Whether to get node ip address to a single partition or not.
Returns
-------
PandasDataframeAxisPartition
An axis partition containing only a single materialized partition.
"""
materialized = super().force_materialization(get_ip=get_ip)
self._list_of_block_partitions = materialized.list_of_block_partitions
return materialized
def apply(
self,
func,
*args,
num_splits=None,
other_axis_partition=None,
maintain_partitioning=True,
lengths=None,
manual_partition=False,
**kwargs,
):
"""
Apply a function to this axis partition along full axis.
Parameters
----------
func : callable
The function to apply.
*args : iterable
Positional arguments to pass to `func`.
num_splits : int, default: None
The number of times to split the result object.
other_axis_partition : PandasDataframeAxisPartition, default: None
Another `PandasDataframeAxisPartition` object to be applied
to func. This is for operations that are between two data sets.
maintain_partitioning : bool, default: True
Whether to keep the partitioning in the same
orientation as it was previously or not. This is important because we may be
operating on an individual AxisPartition and not touching the rest.
In this case, we have to return the partitioning to its previous
orientation (the lengths will remain the same). This is ignored between
two axis partitions.
lengths : iterable, default: None
The list of lengths to shuffle the object.
manual_partition : bool, default: False
If True, partition the result with `lengths`.
**kwargs : dict
Additional keywords arguments to be passed in `func`.
Returns
-------
list
A list of `PandasDataframePartition` objects.
"""
if not self.full_axis:
# If this is not a full axis partition, it already contains a subset of
# the full axis, so we shouldn't split the result further.
num_splits = 1
if len(self.call_queue) > 0:
self.drain_call_queue()
if num_splits is None:
num_splits = len(self.list_of_blocks)
if other_axis_partition is not None:
if not isinstance(other_axis_partition, list):
other_axis_partition = [other_axis_partition]
# (other_shape[i-1], other_shape[i]) will indicate slice
# to restore i-1 axis partition
other_shape = np.cumsum(
[0] + [len(o.list_of_blocks) for o in other_axis_partition]
)
return self._wrap_partitions(
self.deploy_func_between_two_axis_partitions(
self.axis,
func,
args,
kwargs,
num_splits,
len(self.list_of_blocks),
other_shape,
*tuple(
self.list_of_blocks
+ [
part
for axis_partition in other_axis_partition
for part in axis_partition.list_of_blocks
]
),
min_block_size=(
MinRowPartitionSize.get()
if self.axis == 0
else MinColumnPartitionSize.get()
),
)
)
result = self._wrap_partitions(
self.deploy_axis_func(
self.axis,
func,
args,
kwargs,
num_splits,
maintain_partitioning,
*self.list_of_blocks,
min_block_size=(
MinRowPartitionSize.get()
if self.axis == 0
else MinColumnPartitionSize.get()
),
lengths=lengths,
manual_partition=manual_partition,
)
)
if self.full_axis:
return result
else:
# If this is not a full axis partition, just take out the single split in the result.
return result[0]
def split(
self, split_func, num_splits, f_args=None, f_kwargs=None, extract_metadata=False
):
"""
Split axis partition into multiple partitions using the `split_func`.
Parameters
----------
split_func : callable(pandas.DataFrame) -> list[pandas.DataFrame]
A function that takes partition's content and split it into multiple chunks.
num_splits : int
The number of splits the `split_func` return.
f_args : iterable, optional
Positional arguments to pass to the `split_func`.
f_kwargs : dict, optional
Keyword arguments to pass to the `split_func`.
extract_metadata : bool, default: False
Whether to return metadata (length, width, ip) of the result. Passing `False` may relax
the load on object storage as the remote function would return X times fewer futures
(where X is the number of metadata values). Passing `False` makes sense for temporary
results where you know for sure that the metadata will never be requested.
Returns
-------
list
List of wrapped remote partition objects.
"""
f_args = tuple() if f_args is None else f_args
f_kwargs = {} if f_kwargs is None else f_kwargs
return self._wrap_partitions(
self.deploy_splitting_func(
self.axis,
split_func,
f_args,
f_kwargs,
num_splits,
*self.list_of_blocks,
extract_metadata=extract_metadata,
),
extract_metadata=extract_metadata,
)
@classmethod
def deploy_splitting_func(
cls,
axis,
split_func,
f_args,
f_kwargs,
num_splits,
*partitions,
extract_metadata=False,
):
"""
Deploy a splitting function along a full axis.
Parameters
----------
axis : {0, 1}
The axis to perform the function along.
split_func : callable(pandas.DataFrame) -> list[pandas.DataFrame]
The function to perform.
f_args : list or tuple
Positional arguments to pass to `split_func`.
f_kwargs : dict
Keyword arguments to pass to `split_func`.
num_splits : int
The number of splits the `split_func` return.
*partitions : iterable
All partitions that make up the full axis (row or column).
extract_metadata : bool, default: False
Whether to return metadata (length, width, ip) of the result. Note that `True` value
is not supported in `PandasDataframeAxisPartition` class.
Returns
-------
list
A list of pandas DataFrames.
"""
dataframe = pandas.concat(list(partitions), axis=axis, copy=False)
# to reduce peak memory consumption
del partitions
return split_func(dataframe, *f_args, **f_kwargs)
@classmethod
def deploy_axis_func(
cls,
axis,
func,
f_args,
f_kwargs,
num_splits,
maintain_partitioning,
*partitions,
min_block_size,
lengths=None,
manual_partition=False,
return_generator=False,
):
"""
Deploy a function along a full axis.
Parameters
----------
axis : {0, 1}
The axis to perform the function along.
func : callable
The function to perform.
f_args : list or tuple
Positional arguments to pass to ``func``.
f_kwargs : dict
Keyword arguments to pass to ``func``.
num_splits : int
The number of splits to return (see `split_result_of_axis_func_pandas`).
maintain_partitioning : bool
If True, keep the old partitioning if possible.
If False, create a new partition layout.
*partitions : iterable
All partitions that make up the full axis (row or column).
min_block_size : int
Minimum number of rows/columns in a single split.
lengths : list, optional
The list of lengths to shuffle the object.
manual_partition : bool, default: False
If True, partition the result with `lengths`.
return_generator : bool, default: False
Return a generator from the function, set to `True` for Ray backend
as Ray remote functions can return Generators.
Returns
-------
list | Generator
A list or generator of pandas DataFrames.
"""
len_partitions = len(partitions)
lengths_partitions = [len(part) for part in partitions]
widths_partitions = [len(part.columns) for part in partitions]
dataframe = pandas.concat(list(partitions), axis=axis, copy=False)
# to reduce peak memory consumption
del partitions
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=FutureWarning)
try:
result = func(dataframe, *f_args, **f_kwargs)
except ValueError as err:
if "assignment destination is read-only" in str(err):
result = func(dataframe.copy(), *f_args, **f_kwargs)
else:
raise err
# to reduce peak memory consumption
del dataframe
if num_splits == 1:
# If we're not going to split the result, we don't need to specify
# split lengths.
lengths = None
elif manual_partition:
# The split function is expecting a list
lengths = list(lengths)
# We set lengths to None so we don't use the old lengths for the resulting partition
# layout. This is done if the number of splits is changing or we are told not to
# keep the old partitioning.
elif num_splits != len_partitions or not maintain_partitioning:
lengths = None
else:
if axis == 0:
lengths = lengths_partitions
if sum(lengths) != len(result):
lengths = None
else:
lengths = widths_partitions
if sum(lengths) != len(result.columns):
lengths = None
if return_generator:
return generate_result_of_axis_func_pandas(
axis,
num_splits,
result,
min_block_size,
lengths,
)
else:
return split_result_of_axis_func_pandas(
axis, num_splits, result, min_block_size, lengths
)
@classmethod
def deploy_func_between_two_axis_partitions(
cls,
axis,
func,
f_args,
f_kwargs,
num_splits,
len_of_left,
other_shape,
*partitions,
min_block_size,
return_generator=False,
):
"""
Deploy a function along a full axis between two data sets.
Parameters
----------
axis : {0, 1}
The axis to perform the function along.
func : callable
The function to perform.
f_args : list or tuple
Positional arguments to pass to ``func``.
f_kwargs : dict
Keyword arguments to pass to ``func``.
num_splits : int
The number of splits to return (see `split_result_of_axis_func_pandas`).
len_of_left : int
The number of values in `partitions` that belong to the left data set.
other_shape : np.ndarray
The shape of right frame in terms of partitions, i.e.
(other_shape[i-1], other_shape[i]) will indicate slice to restore i-1 axis partition.
*partitions : iterable
All partitions that make up the full axis (row or column) for both data sets.
min_block_size : int
Minimum number of rows/columns in a single split.
return_generator : bool, default: False
Return a generator from the function, set to `True` for Ray backend
as Ray remote functions can return Generators.
Returns
-------
list | Generator
A list or generator of pandas DataFrames.
"""
lt_frame = pandas.concat(partitions[:len_of_left], axis=axis, copy=False)
rt_parts = partitions[len_of_left:]
# to reduce peak memory consumption
del partitions
# reshaping flattened `rt_parts` array into a frame with shape `other_shape`
combined_axis = [
pandas.concat(
rt_parts[other_shape[i - 1] : other_shape[i]],
axis=axis,
copy=False,
)
for i in range(1, len(other_shape))
]
# to reduce peak memory consumption
del rt_parts
rt_frame = pandas.concat(combined_axis, axis=axis ^ 1, copy=False)
# to reduce peak memory consumption
del combined_axis
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=FutureWarning)
result = func(lt_frame, rt_frame, *f_args, **f_kwargs)
# to reduce peak memory consumption
del lt_frame, rt_frame
if return_generator:
return generate_result_of_axis_func_pandas(
axis,
num_splits,
result,
min_block_size,
)
else:
return split_result_of_axis_func_pandas(
axis,
num_splits,
result,
min_block_size,
)
@classmethod
def drain(cls, df: pandas.DataFrame, call_queue: list):
"""
Execute all operations stored in the call queue on the pandas object (helper function).
Parameters
----------
df : pandas.DataFrame
call_queue : list
Call queue that needs to be executed on pandas DataFrame.
Returns
-------
pandas.DataFrame
"""
for func, args, kwargs in call_queue:
df = func(df, *args, **kwargs)
return df
def mask(self, row_indices, col_indices):
"""
Create (synchronously) a mask that extracts the indices provided.
Parameters
----------
row_indices : list-like, slice or label
The row labels for the rows to extract.
col_indices : list-like, slice or label
The column labels for the columns to extract.
Returns
-------
PandasDataframeAxisPartition
A new ``PandasDataframeAxisPartition`` object, materialized.
"""
return (
self.force_materialization()
.list_of_block_partitions[0]
.mask(row_indices, col_indices)
)
def to_pandas(self):
"""
Convert the data in this partition to a ``pandas.DataFrame``.
Returns
-------
pandas DataFrame.
"""
return self.force_materialization().list_of_block_partitions[0].to_pandas()
def to_numpy(self):
"""
Convert the data in this partition to a ``numpy.array``.
Returns
-------
NumPy array.
"""
return self.force_materialization().list_of_block_partitions[0].to_numpy()
_length_cache = None
def length(self, materialize=True):
"""
Get the length of this partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
int
The length of the partition.
"""
if self._length_cache is None:
if self.axis == 0:
self._length_cache = sum(
obj.length() for obj in self.list_of_block_partitions
)
else:
self._length_cache = self.list_of_block_partitions[0].length(
materialize
)
return self._length_cache
_width_cache = None
def width(self, materialize=True):
"""
Get the width of this partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
int
The width of the partition.
"""
if self._width_cache is None:
if self.axis == 1:
self._width_cache = sum(
obj.width() for obj in self.list_of_block_partitions
)
else:
self._width_cache = self.list_of_block_partitions[0].width(materialize)
return self._width_cache
def wait(self):
"""Wait completing computations on the object wrapped by the partition."""
pass
def add_to_apply_calls(self, func, *args, length=None, width=None, **kwargs):
"""
Add a function to the call queue.
Parameters
----------
func : callable or a future type
Function to be added to the call queue.
*args : iterable
Additional positional arguments to be passed in `func`.
length : A future type or int, optional
Length, or reference to it, of wrapped ``pandas.DataFrame``.
width : A future type or int, optional
Width, or reference to it, of wrapped ``pandas.DataFrame``.
**kwargs : dict
Additional keyword arguments to be passed in `func`.
Returns
-------
PandasDataframeAxisPartition
A new ``PandasDataframeAxisPartition`` object.
"""
return type(self)(
self.list_of_block_partitions,
full_axis=self.full_axis,
call_queue=self.call_queue + [[func, args, kwargs]],
length=length,
width=width,
)
================================================
FILE: modin/core/dataframe/pandas/partitioning/partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""The module defines base interface for a partition of a Modin DataFrame."""
from __future__ import annotations
import logging
import uuid
from abc import ABC
from copy import copy
from functools import cached_property
import pandas
from pandas.api.types import is_scalar
from modin.core.storage_formats.pandas.utils import length_fn_pandas, width_fn_pandas
from modin.logging import ClassLogger, get_logger
from modin.logging.config import LogLevel
from modin.pandas.indexing import compute_sliced_len
class PandasDataframePartition(
ABC, ClassLogger, modin_layer="BLOCK-PARTITION", log_level=LogLevel.DEBUG
): # pragma: no cover
"""
An abstract class that is base for any partition class of ``pandas`` storage format.
The class providing an API that has to be overridden by child classes.
"""
_length_cache = None
_width_cache = None
_identity_cache = None
_data = None
execution_wrapper = None
# these variables are intentionally initialized at runtime
# so as not to initialize the engine during import
_iloc_func = None
def __init__(self):
if type(self)._iloc_func is None:
# Places `_iloc` function into the storage to speed up
# remote function calls and caches the result.
# It also postpones engine initialization, which happens
# implicitly when `execution_wrapper.put` is called.
if self.execution_wrapper is not None:
type(self)._iloc_func = staticmethod(
self.execution_wrapper.put(self._iloc)
)
else:
type(self)._iloc_func = staticmethod(self._iloc)
@cached_property
def __constructor__(self) -> type[PandasDataframePartition]:
"""
Create a new instance of this object.
Returns
-------
PandasDataframePartition
New instance of pandas partition.
"""
return type(self)
def get(self):
"""
Get the object wrapped by this partition.
Returns
-------
object
The object that was wrapped by this partition.
Notes
-----
This is the opposite of the classmethod `put`.
E.g. if you assign `x = PandasDataframePartition.put(1)`, `x.get()` should
always return 1.
"""
log = get_logger()
self._is_debug(log) and log.debug(f"ENTER::Partition.get::{self._identity}")
self.drain_call_queue()
result = self.execution_wrapper.materialize(self._data)
self._is_debug(log) and log.debug(f"EXIT::Partition.get::{self._identity}")
return result
@property
def list_of_blocks(self):
"""
Get the list of physical partition objects that compose this partition.
Returns
-------
list
A list of physical partition objects (``ray.ObjectRef``, ``distributed.Future`` e.g.).
"""
# Defer draining call queue until we get the partitions.
# TODO Look into draining call queue at the same time as the task
self.drain_call_queue()
return [self._data]
def apply(self, func, *args, **kwargs):
"""
Apply a function to the object wrapped by this partition.
Parameters
----------
func : callable
Function to apply.
*args : iterable
Additional positional arguments to be passed in `func`.
**kwargs : dict
Additional keyword arguments to be passed in `func`.
Returns
-------
PandasDataframePartition
New `PandasDataframePartition` object.
Notes
-----
It is up to the implementation how `kwargs` are handled. They are
an important part of many implementations. As of right now, they
are not serialized.
"""
pass
def add_to_apply_calls(self, func, *args, length=None, width=None, **kwargs):
"""
Add a function to the call queue.
Parameters
----------
func : callable
Function to be added to the call queue.
*args : iterable
Additional positional arguments to be passed in `func`.
length : reference or int, optional
Length, or reference to length, of wrapped ``pandas.DataFrame``.
width : reference or int, optional
Width, or reference to width, of wrapped ``pandas.DataFrame``.
**kwargs : dict
Additional keyword arguments to be passed in `func`.
Returns
-------
PandasDataframePartition
New `PandasDataframePartition` object with the function added to the call queue.
Notes
-----
This function will be executed when `apply` is called. It will be executed
in the order inserted; apply's func operates the last and return.
"""
return self.__constructor__(
self._data,
call_queue=self.call_queue + [[func, args, kwargs]],
length=length,
width=width,
)
def drain_call_queue(self):
"""Execute all operations stored in the call queue on the object wrapped by this partition."""
pass
def wait(self):
"""Wait for completion of computations on the object wrapped by the partition."""
pass
def to_pandas(self):
"""
Convert the object wrapped by this partition to a ``pandas.DataFrame``.
Returns
-------
pandas.DataFrame
Notes
-----
If the underlying object is a pandas DataFrame, this will likely
only need to call `get`.
"""
dataframe = self.get()
assert isinstance(dataframe, (pandas.DataFrame, pandas.Series))
return dataframe
def to_numpy(self, **kwargs):
"""
Convert the object wrapped by this partition to a NumPy array.
Parameters
----------
**kwargs : dict
Additional keyword arguments to be passed in ``to_numpy``.
Returns
-------
np.ndarray
Notes
-----
If the underlying object is a pandas DataFrame, this will return
a 2D NumPy array.
"""
return self.apply(lambda df: df.to_numpy(**kwargs)).get()
@staticmethod
def _iloc(df, row_labels, col_labels): # noqa: RT01, PR01
"""Perform `iloc` on dataframes wrapped in partitions (helper function)."""
return df.iloc[row_labels, col_labels]
def mask(self, row_labels, col_labels):
"""
Lazily create a mask that extracts the indices provided.
Parameters
----------
row_labels : list-like, slice or label
The row labels for the rows to extract.
col_labels : list-like, slice or label
The column labels for the columns to extract.
Returns
-------
PandasDataframePartition
New `PandasDataframePartition` object.
"""
def is_full_axis_mask(index, axis_length):
"""Check whether `index` mask grabs `axis_length` amount of elements."""
if isinstance(index, slice):
return index == slice(None) or (
isinstance(axis_length, int)
and compute_sliced_len(index, axis_length) == axis_length
)
return (
hasattr(index, "__len__")
and isinstance(axis_length, int)
and len(index) == axis_length
)
row_labels = [row_labels] if is_scalar(row_labels) else row_labels
col_labels = [col_labels] if is_scalar(col_labels) else col_labels
if is_full_axis_mask(row_labels, self._length_cache) and is_full_axis_mask(
col_labels, self._width_cache
):
return copy(self)
new_obj = self.add_to_apply_calls(self._iloc_func, row_labels, col_labels)
def try_recompute_cache(indices, previous_cache):
"""Compute new axis-length cache for the masked frame based on its previous cache."""
if not isinstance(indices, slice):
return len(indices)
if not isinstance(previous_cache, int):
return None
return compute_sliced_len(indices, previous_cache)
new_obj._length_cache = try_recompute_cache(row_labels, self._length_cache)
new_obj._width_cache = try_recompute_cache(col_labels, self._width_cache)
return new_obj
@classmethod
def put(cls, obj):
"""
Put an object into a store and wrap it with partition object.
Parameters
----------
obj : object
An object to be put.
Returns
-------
PandasDataframePartition
New `PandasDataframePartition` object.
"""
pass
@classmethod
def preprocess_func(cls, func):
"""
Preprocess a function before an `apply` call.
Parameters
----------
func : callable
Function to preprocess.
Returns
-------
callable
An object that can be accepted by `apply`.
Notes
-----
This is a classmethod because the definition of how to preprocess
should be class-wide. Also, we may want to use this before we
deploy a preprocessed function to multiple `PandasDataframePartition`
objects.
"""
pass
@classmethod
def _length_extraction_fn(cls):
"""
Return the function that computes the length of the object wrapped by this partition.
Returns
-------
callable
The function that computes the length of the object wrapped by this partition.
"""
return length_fn_pandas
@classmethod
def _width_extraction_fn(cls):
"""
Return the function that computes the width of the object wrapped by this partition.
Returns
-------
callable
The function that computes the width of the object wrapped by this partition.
"""
return width_fn_pandas
def length(self, materialize=True):
"""
Get the length of the object wrapped by this partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
int or its Future
The length of the object.
"""
if self._length_cache is None:
self._length_cache = self.apply(self._length_extraction_fn()).get()
return self._length_cache
def width(self, materialize=True):
"""
Get the width of the object wrapped by the partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
int or its Future
The width of the object.
"""
if self._width_cache is None:
self._width_cache = self.apply(self._width_extraction_fn()).get()
return self._width_cache
@property
def _identity(self):
"""
Calculate identifier on request for debug logging mode.
Returns
-------
str
"""
if self._identity_cache is None:
self._identity_cache = uuid.uuid4().hex
return self._identity_cache
def split(self, split_func, num_splits, *args):
"""
Split the object wrapped by the partition into multiple partitions.
Parameters
----------
split_func : Callable[pandas.DataFrame, List[Any]] -> List[pandas.DataFrame]
The function that will split this partition into multiple partitions. The list contains
pivots to split by, and will have the same dtype as the major column we are shuffling on.
num_splits : int
The number of resulting partitions (may be empty).
*args : List[Any]
Arguments to pass to ``split_func``.
Returns
-------
list
A list of partitions.
"""
log = get_logger()
self._is_debug(log) and log.debug(f"ENTER::Partition.split::{self._identity}")
self._is_debug(log) and log.debug(f"SUBMIT::_split_df::{self._identity}")
outputs = self.execution_wrapper.deploy(
split_func, [self._data] + list(args), num_returns=num_splits
)
self._is_debug(log) and log.debug(f"EXIT::Partition.split::{self._identity}")
return [self.__constructor__(output) for output in outputs]
@classmethod
def empty(cls):
"""
Create a new partition that wraps an empty pandas DataFrame.
Returns
-------
PandasDataframePartition
New `PandasDataframePartition` object.
"""
return cls.put(pandas.DataFrame(), 0, 0)
def _is_debug(self, logger=None):
"""
Check that the logger is set to debug mode.
Parameters
----------
logger : logging.logger, optional
Logger obtained from Modin's `get_logger` utility.
Explicit transmission of this parameter can be used in the case
when within the context of `_is_debug` call there was already
`get_logger` call. This is an optimization.
Returns
-------
bool
"""
if logger is None:
logger = get_logger()
return logger.isEnabledFor(logging.DEBUG)
================================================
FILE: modin/core/dataframe/pandas/partitioning/partition_manager.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module holding base PartitionManager class - the thing that tracks partitions across the distribution.
The manager also allows manipulating the data - running functions at each partition, shuffle over the distribution, etc.
"""
import os
import warnings
from abc import ABC
from functools import wraps
from typing import TYPE_CHECKING, Optional
import numpy as np
import pandas
from pandas._libs.lib import no_default
from modin.config import (
BenchmarkMode,
CpuCount,
DynamicPartitioning,
Engine,
MinColumnPartitionSize,
MinRowPartitionSize,
NPartitions,
PersistentPickle,
ProgressBar,
)
from modin.core.dataframe.pandas.utils import create_pandas_df_from_partitions
from modin.core.storage_formats.pandas.utils import compute_chunksize
from modin.error_message import ErrorMessage
from modin.logging import ClassLogger
from modin.logging.config import LogLevel
from modin.pandas.utils import get_pandas_backend
if TYPE_CHECKING:
from modin.core.dataframe.pandas.dataframe.utils import ShuffleFunctions
def wait_computations_if_benchmark_mode(func):
"""
Make sure a `func` finished its computations in benchmark mode.
Parameters
----------
func : callable
A function that should be performed in syncronous mode.
Returns
-------
callable
Wrapped function that executes eagerly (if benchmark mode) or original `func`.
Notes
-----
`func` should return NumPy array with partitions.
"""
@wraps(func)
def wait(cls, *args, **kwargs):
"""Wait for computation results."""
result = func(cls, *args, **kwargs)
if BenchmarkMode.get():
if isinstance(result, tuple):
partitions = result[0]
else:
partitions = result
# When partitions have a deferred call queue, calling
# partition.wait() on each partition serially will serially kick
# off each deferred computation and wait for each partition to
# finish before kicking off the next one. Instead, we want to
# serially kick off all the deferred computations so that they can
# all run asynchronously, then wait on all the results.
cls.finalize(partitions)
# The partition manager invokes the relevant .wait() method under
# the hood, which should wait in parallel for all computations to finish
cls.wait_partitions(partitions.flatten())
return result
return wait
class PandasDataframePartitionManager(
ClassLogger, ABC, modin_layer="PARTITION-MANAGER", log_level=LogLevel.DEBUG
):
"""
Base class for managing the dataframe data layout and operators across the distribution of partitions.
Partition class is the class to use for storing each partition.
Each partition must extend the `PandasDataframePartition` class.
"""
_partition_class = None
# Column partitions class is the class to use to create the column partitions.
_column_partitions_class = None
# Row partitions class is the class to use to create the row partitions.
_row_partition_class = None
_execution_wrapper = None
@classmethod
def materialize_futures(cls, input_list):
"""
Materialize all futures in the input list.
Parameters
----------
input_list : list
The list that has to be manipulated.
Returns
-------
list
A new list with materialized objects.
"""
# Do nothing if input_list is None or [].
if input_list is None:
return None
filtered_list = []
filtered_idx = []
for idx, item in enumerate(input_list):
if cls._execution_wrapper.is_future(item):
filtered_idx.append(idx)
filtered_list.append(item)
filtered_list = cls._execution_wrapper.materialize(filtered_list)
result = input_list.copy()
for idx, item in zip(filtered_idx, filtered_list):
result[idx] = item
return result
@classmethod
def preprocess_func(cls, map_func):
"""
Preprocess a function to be applied to `PandasDataframePartition` objects.
Parameters
----------
map_func : callable
The function to be preprocessed.
Returns
-------
callable
The preprocessed version of the `map_func` provided.
Notes
-----
Preprocessing does not require any specific format, only that the
`PandasDataframePartition.apply` method will recognize it (for the subclass
being used).
If your `PandasDataframePartition` objects assume that a function provided
is serialized or wrapped or in some other format, this is the place
to add that logic. It is possible that this can also just return
`map_func` if the `apply` method of the `PandasDataframePartition` object
you are using does not require any modification to a given function.
"""
if cls._execution_wrapper.is_future(map_func):
return map_func # Has already been preprocessed
old_value = PersistentPickle.get()
# When performing a function with Modin objects, it is more profitable to
# do the conversion to pandas once on the main process than several times
# on worker processes. Details: https://github.com/modin-project/modin/pull/6673/files#r1391086755
# For Dask, otherwise there may be an error: `coroutine 'Client._gather' was never awaited`
need_update = not PersistentPickle.get() and Engine.get() != "Dask"
if need_update:
PersistentPickle.put(True)
try:
result = cls._partition_class.preprocess_func(map_func)
finally:
if need_update:
PersistentPickle.put(old_value)
return result
# END Abstract Methods
@classmethod
def create_partition_from_metadata(
cls, dtypes: Optional[pandas.Series] = None, **metadata
):
"""
Create NumPy array of partitions that holds an empty dataframe with given metadata.
Parameters
----------
dtypes : pandas.Series, optional
Column dtypes.
Upon creating a pandas DataFrame from `metadata` we call `astype` since
pandas doesn't allow to pass a list of dtypes directly in the constructor.
**metadata : dict
Metadata that has to be wrapped in a partition.
Returns
-------
np.ndarray
A NumPy 2D array of a single partition which contains the data.
"""
metadata_dataframe = pandas.DataFrame(**metadata)
if dtypes is not None:
metadata_dataframe = metadata_dataframe.astype(dtypes)
return np.array([[cls._partition_class.put(metadata_dataframe)]])
@classmethod
def column_partitions(cls, partitions, full_axis=True):
"""
Get the list of `BaseDataframeAxisPartition` objects representing column-wise partitions.
Parameters
----------
partitions : list-like
List of (smaller) partitions to be combined to column-wise partitions.
full_axis : bool, default: True
Whether or not this partition contains the entire column axis.
Returns
-------
list
A list of `BaseDataframeAxisPartition` objects.
Notes
-----
Each value in this list will be an `BaseDataframeAxisPartition` object.
`BaseDataframeAxisPartition` is located in `axis_partition.py`.
"""
if not isinstance(partitions, list):
partitions = [partitions]
return [
cls._column_partitions_class(col, full_axis=full_axis)
for frame in partitions
for col in frame.T
]
@classmethod
def row_partitions(cls, partitions):
"""
List of `BaseDataframeAxisPartition` objects representing row-wise partitions.
Parameters
----------
partitions : list-like
List of (smaller) partitions to be combined to row-wise partitions.
Returns
-------
list
A list of `BaseDataframeAxisPartition` objects.
Notes
-----
Each value in this list will an `BaseDataframeAxisPartition` object.
`BaseDataframeAxisPartition` is located in `axis_partition.py`.
"""
if not isinstance(partitions, list):
partitions = [partitions]
return [cls._row_partition_class(row) for frame in partitions for row in frame]
@classmethod
def axis_partition(cls, partitions, axis, full_axis: bool = True):
"""
Logically partition along given axis (columns or rows).
Parameters
----------
partitions : list-like
List of partitions to be combined.
axis : {0, 1}
0 for column partitions, 1 for row partitions.
full_axis : bool, default: True
Whether or not this partition contains the entire column axis.
Returns
-------
list
A list of `BaseDataframeAxisPartition` objects.
"""
make_column_partitions = axis == 0
if not full_axis and not make_column_partitions:
raise NotImplementedError(
(
"Row partitions must contain the entire axis. We don't "
+ "support virtual partitioning for row partitions yet."
)
)
return (
cls.column_partitions(partitions)
if make_column_partitions
else cls.row_partitions(partitions)
)
@classmethod
def groupby_reduce(
cls, axis, partitions, by, map_func, reduce_func, apply_indices=None
):
"""
Groupby data using the `map_func` provided along the `axis` over the `partitions` then reduce using `reduce_func`.
Parameters
----------
axis : {0, 1}
Axis to groupby over.
partitions : NumPy 2D array
Partitions of the ModinFrame to groupby.
by : NumPy 2D array
Partitions of 'by' to broadcast.
map_func : callable
Map function.
reduce_func : callable,
Reduce function.
apply_indices : list of ints, default: None
Indices of `axis ^ 1` to apply function over.
Returns
-------
NumPy array
Partitions with applied groupby.
"""
if apply_indices is not None:
partitions = (
partitions[apply_indices] if axis else partitions[:, apply_indices]
)
if by is not None:
# need to make sure that the partitioning of the following objects
# coincides in the required axis, because `partition_manager.broadcast_apply`
# doesn't call `_copartition` unlike `modin_frame.broadcast_apply`
assert partitions.shape[axis] == by.shape[axis], (
f"the number of partitions along {axis=} is not equal: "
+ f"{partitions.shape[axis]} != {by.shape[axis]}"
)
mapped_partitions = cls.broadcast_apply(
axis, map_func, left=partitions, right=by
)
else:
mapped_partitions = cls.map_partitions(partitions, map_func)
# Assuming, that the output will not be larger than the input,
# keep the current number of partitions.
num_splits = min(len(partitions), NPartitions.get())
return cls.map_axis_partitions(
axis,
mapped_partitions,
reduce_func,
enumerate_partitions=True,
num_splits=num_splits,
)
@classmethod
@wait_computations_if_benchmark_mode
def broadcast_apply_select_indices(
cls,
axis,
apply_func,
left,
right,
left_indices,
right_indices,
keep_remaining=False,
):
"""
Broadcast the `right` partitions to `left` and apply `apply_func` to selected indices.
Parameters
----------
axis : {0, 1}
Axis to apply and broadcast over.
apply_func : callable
Function to apply.
left : NumPy 2D array
Left partitions.
right : NumPy 2D array
Right partitions.
left_indices : list-like
Indices to apply function to.
right_indices : dictionary of indices of right partitions
Indices that you want to bring at specified left partition, for example
dict {key: {key1: [0, 1], key2: [5]}} means that in left[key] you want to
broadcast [right[key1], right[key2]] partitions and internal indices
for `right` must be [[0, 1], [5]].
keep_remaining : bool, default: False
Whether or not to keep the other partitions.
Some operations may want to drop the remaining partitions and
keep only the results.
Returns
-------
NumPy array
An array of partition objects.
Notes
-----
Your internal function must take these kwargs:
[`internal_indices`, `other`, `internal_other_indices`] to work correctly!
"""
if not axis:
partitions_for_apply = left.T
right = right.T
else:
partitions_for_apply = left
[obj.drain_call_queue() for row in right for obj in row]
def get_partitions(index):
"""Grab required partitions and indices from `right` and `right_indices`."""
must_grab = right_indices[index]
partitions_list = np.array([right[i] for i in must_grab.keys()])
indices_list = list(must_grab.values())
return {"other": partitions_list, "internal_other_indices": indices_list}
new_partitions = np.array(
[
(
partitions_for_apply[i]
if i not in left_indices
else cls._apply_func_to_list_of_partitions_broadcast(
apply_func,
partitions_for_apply[i],
internal_indices=left_indices[i],
**get_partitions(i),
)
)
for i in range(len(partitions_for_apply))
if i in left_indices or keep_remaining
]
)
if not axis:
new_partitions = new_partitions.T
return new_partitions
@classmethod
@wait_computations_if_benchmark_mode
def base_broadcast_apply(cls, axis, apply_func, left, right):
"""
Broadcast the `right` partitions to `left` and apply `apply_func` function.
Parameters
----------
axis : {0, 1}
Axis to apply and broadcast over.
apply_func : callable
Function to apply.
left : np.ndarray
NumPy array of left partitions.
right : np.ndarray
NumPy array of right partitions.
Returns
-------
np.ndarray
NumPy array of result partition objects.
Notes
-----
This will often be overridden by implementations. It materializes the
entire partitions of the right and applies them to the left through `apply`.
"""
def map_func(df, *others):
other = (
pandas.concat(others, axis=axis ^ 1) if len(others) > 1 else others[0]
)
# to reduce peak memory consumption
del others
return apply_func(df, other)
map_func = cls.preprocess_func(map_func)
rt_axis_parts = cls.axis_partition(right, axis ^ 1)
return np.array(
[
[
part.apply(
map_func,
*(
rt_axis_parts[col_idx].list_of_blocks
if axis
else rt_axis_parts[row_idx].list_of_blocks
),
)
for col_idx, part in enumerate(left[row_idx])
]
for row_idx in range(len(left))
]
)
@classmethod
@wait_computations_if_benchmark_mode
def broadcast_axis_partitions(
cls,
axis,
apply_func,
left,
right,
keep_partitioning=False,
num_splits=None,
apply_indices=None,
broadcast_all=True,
enumerate_partitions=False,
lengths=None,
apply_func_args=None,
**kwargs,
):
"""
Broadcast the `right` partitions to `left` and apply `apply_func` along full `axis`.
Parameters
----------
axis : {0, 1}
Axis to apply and broadcast over.
apply_func : callable
Function to apply.
left : NumPy 2D array
Left partitions.
right : NumPy 2D array
Right partitions.
keep_partitioning : boolean, default: False
The flag to keep partition boundaries for Modin Frame if possible.
Setting it to True disables shuffling data from one partition to another in case the resulting
number of splits is equal to the initial number of splits.
num_splits : int, optional
The number of partitions to split the result into across the `axis`. If None, then the number
of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`
then the number of splits is preserved.
apply_indices : list of ints, default: None
Indices of `axis ^ 1` to apply function over.
broadcast_all : bool, default: True
Whether or not to pass all right axis partitions to each of the left axis partitions.
enumerate_partitions : bool, default: False
Whether or not to pass partition index into `apply_func`.
Note that `apply_func` must be able to accept `partition_idx` kwarg.
lengths : list of ints, default: None
The list of lengths to shuffle the object. Note:
1. Passing `lengths` omits the `num_splits` parameter as the number of splits
will now be inferred from the number of integers present in `lengths`.
2. When passing lengths you must explicitly specify `keep_partitioning=False`.
apply_func_args : list-like, optional
Positional arguments to pass to the `func`.
**kwargs : dict
Additional options that could be used by different engines.
Returns
-------
NumPy array
An array of partition objects.
"""
ErrorMessage.catch_bugs_and_request_email(
failure_condition=keep_partitioning and lengths is not None,
extra_log=f"`keep_partitioning` must be set to `False` when passing `lengths`. Got: {keep_partitioning=} | {lengths=}",
)
# Since we are already splitting the DataFrame back up after an
# operation, we will just use this time to compute the number of
# partitions as best we can right now.
if keep_partitioning and num_splits is None:
num_splits = len(left) if axis == 0 else len(left.T)
elif lengths:
num_splits = len(lengths)
elif num_splits is None:
num_splits = NPartitions.get()
else:
ErrorMessage.catch_bugs_and_request_email(
failure_condition=not isinstance(num_splits, int),
extra_log=f"Expected `num_splits` to be an integer, got: {type(num_splits)} | {num_splits=}",
)
preprocessed_map_func = cls.preprocess_func(apply_func)
left_partitions = cls.axis_partition(left, axis)
right_partitions = None if right is None else cls.axis_partition(right, axis)
# For mapping across the entire axis, we don't maintain partitioning because we
# may want to line to partitioning up with another BlockPartitions object. Since
# we don't need to maintain the partitioning, this gives us the opportunity to
# load-balance the data as well.
kw = {
"num_splits": num_splits,
"maintain_partitioning": keep_partitioning,
}
if lengths:
kw["lengths"] = lengths
kw["manual_partition"] = True
if apply_indices is None:
apply_indices = np.arange(len(left_partitions))
result_blocks = np.array(
[
left_partitions[i].apply(
preprocessed_map_func,
*(apply_func_args if apply_func_args else []),
other_axis_partition=(
right_partitions if broadcast_all else right_partitions[i]
),
**kw,
**({"partition_idx": idx} if enumerate_partitions else {}),
**kwargs,
)
for idx, i in enumerate(apply_indices)
]
)
# If we are mapping over columns, they are returned to use the same as
# rows, so we need to transpose the returned 2D NumPy array to return
# the structure to the correct order.
return result_blocks.T if not axis else result_blocks
@classmethod
@wait_computations_if_benchmark_mode
def base_map_partitions(
cls,
partitions,
map_func,
func_args=None,
func_kwargs=None,
):
"""
Apply `map_func` to every partition in `partitions`.
Parameters
----------
partitions : NumPy 2D array
Partitions housing the data of Modin Frame.
map_func : callable
Function to apply.
func_args : iterable, optional
Positional arguments for the 'map_func'.
func_kwargs : dict, optional
Keyword arguments for the 'map_func'.
Returns
-------
NumPy array
An array of partitions
"""
preprocessed_map_func = cls.preprocess_func(map_func)
return np.array(
[
[
part.apply(
preprocessed_map_func,
*func_args if func_args is not None else (),
**func_kwargs if func_kwargs is not None else {},
)
for part in row_of_parts
]
for row_of_parts in partitions
]
)
@classmethod
@wait_computations_if_benchmark_mode
def broadcast_apply(
cls,
axis,
apply_func,
left,
right,
):
"""
Broadcast the `right` partitions to `left` and apply `apply_func` function using different approaches to achieve the best performance.
Parameters
----------
axis : {0, 1}
Axis to apply and broadcast over.
apply_func : callable
Function to apply.
left : np.ndarray
NumPy array of left partitions.
right : np.ndarray
NumPy array of right partitions.
Returns
-------
np.ndarray
NumPy array of result partition objects.
"""
if not DynamicPartitioning.get():
# block-wise broadcast
new_partitions = cls.base_broadcast_apply(
axis,
apply_func,
left,
right,
)
else:
# The dynamic partitioning behavior of `broadcast_apply` differs from that of `map_partitions`,
# since the columnar approach for `broadcast_apply` results in slowdown.
# axis-wise broadcast
new_partitions = cls.broadcast_axis_partitions(
axis=axis ^ 1,
left=left,
right=right,
apply_func=apply_func,
broadcast_all=False,
keep_partitioning=True,
)
return new_partitions
@classmethod
@wait_computations_if_benchmark_mode
def map_partitions(
cls,
partitions,
map_func,
func_args=None,
func_kwargs=None,
):
"""
Apply `map_func` to `partitions` using different approaches to achieve the best performance.
Parameters
----------
partitions : NumPy 2D array
Partitions housing the data of Modin Frame.
map_func : callable
Function to apply.
func_args : iterable, optional
Positional arguments for the 'map_func'.
func_kwargs : dict, optional
Keyword arguments for the 'map_func'.
Returns
-------
NumPy array
An array of partitions
"""
if not DynamicPartitioning.get():
# block-wise map
new_partitions = cls.base_map_partitions(
partitions, map_func, func_args, func_kwargs
)
else:
# axis-wise map
# we choose an axis for a combination of partitions
# whose size is closer to the number of CPUs
if abs(partitions.shape[0] - CpuCount.get()) < abs(
partitions.shape[1] - CpuCount.get()
):
axis = 1
else:
axis = 0
column_splits = CpuCount.get() // partitions.shape[1]
if axis == 0 and column_splits > 1:
# splitting by parts of columnar partitions
new_partitions = cls.map_partitions_joined_by_column(
partitions, column_splits, map_func, func_args, func_kwargs
)
else:
# splitting by full axis partitions
new_partitions = cls.map_axis_partitions(
axis,
partitions,
lambda df: map_func(
df,
*(func_args if func_args is not None else ()),
**(func_kwargs if func_kwargs is not None else {}),
),
keep_partitioning=True,
)
return new_partitions
@classmethod
@wait_computations_if_benchmark_mode
def lazy_map_partitions(
cls,
partitions,
map_func,
func_args=None,
func_kwargs=None,
enumerate_partitions=False,
):
"""
Apply `map_func` to every partition in `partitions` *lazily*.
Parameters
----------
partitions : NumPy 2D array
Partitions of Modin Frame.
map_func : callable
Function to apply.
func_args : iterable, optional
Positional arguments for the 'map_func'.
func_kwargs : dict, optional
Keyword arguments for the 'map_func'.
enumerate_partitions : bool, default: False
Returns
-------
NumPy array
An array of partitions
"""
preprocessed_map_func = cls.preprocess_func(map_func)
return np.array(
[
[
part.add_to_apply_calls(
preprocessed_map_func,
*(tuple() if func_args is None else func_args),
**func_kwargs if func_kwargs is not None else {},
**({"partition_idx": i} if enumerate_partitions else {}),
)
for part in row
]
for i, row in enumerate(partitions)
]
)
@classmethod
def map_axis_partitions(
cls,
axis,
partitions,
map_func,
keep_partitioning=False,
num_splits=None,
lengths=None,
enumerate_partitions=False,
**kwargs,
):
"""
Apply `map_func` to every partition in `partitions` along given `axis`.
Parameters
----------
axis : {0, 1}
Axis to perform the map across (0 - index, 1 - columns).
partitions : NumPy 2D array
Partitions of Modin Frame.
map_func : callable
Function to apply.
keep_partitioning : boolean, default: False
The flag to keep partition boundaries for Modin Frame if possible.
Setting it to True disables shuffling data from one partition to another in case the resulting
number of splits is equal to the initial number of splits.
num_splits : int, optional
The number of partitions to split the result into across the `axis`. If None, then the number
of splits will be infered automatically. If `num_splits` is None and `keep_partitioning=True`
then the number of splits is preserved.
lengths : list of ints, default: None
The list of lengths to shuffle the object. Note:
1. Passing `lengths` omits the `num_splits` parameter as the number of splits
will now be inferred from the number of integers present in `lengths`.
2. When passing lengths you must explicitly specify `keep_partitioning=False`.
enumerate_partitions : bool, default: False
Whether or not to pass partition index into `map_func`.
Note that `map_func` must be able to accept `partition_idx` kwarg.
**kwargs : dict
Additional options that could be used by different engines.
Returns
-------
NumPy array
An array of new partitions for Modin Frame.
Notes
-----
This method should be used in the case when `map_func` relies on
some global information about the axis.
"""
return cls.broadcast_axis_partitions(
axis=axis,
left=partitions,
apply_func=map_func,
keep_partitioning=keep_partitioning,
num_splits=num_splits,
right=None,
lengths=lengths,
enumerate_partitions=enumerate_partitions,
**kwargs,
)
@classmethod
def map_partitions_joined_by_column(
cls,
partitions,
column_splits,
map_func,
map_func_args=None,
map_func_kwargs=None,
):
"""
Combine several blocks by column into one virtual partition and apply "map_func" to them.
Parameters
----------
partitions : NumPy 2D array
Partitions of Modin Frame.
column_splits : int
The number of splits by column.
map_func : callable
Function to apply.
map_func_args : iterable, optional
Positional arguments for the 'map_func'.
map_func_kwargs : dict, optional
Keyword arguments for the 'map_func'.
Returns
-------
NumPy array
An array of new partitions for Modin Frame.
"""
if column_splits < 1:
raise ValueError(
"The value of columns_splits must be greater than or equal to 1."
)
# step cannot be less than 1
step = max(partitions.shape[0] // column_splits, 1)
preprocessed_map_func = cls.preprocess_func(map_func)
result = np.empty(partitions.shape, dtype=object)
for i in range(
0,
partitions.shape[0],
step,
):
partitions_subset = partitions[i : i + step]
# This is necessary when ``partitions.shape[0]`` is not divisible
# by `column_splits` without a remainder.
actual_step = len(partitions_subset)
kw = {
"num_splits": actual_step,
}
joined_column_partitions = cls.column_partitions(partitions_subset)
for j in range(partitions.shape[1]):
result[i : i + actual_step, j] = joined_column_partitions[j].apply(
preprocessed_map_func,
*map_func_args if map_func_args is not None else (),
**kw,
**map_func_kwargs if map_func_kwargs is not None else {},
)
return result
@classmethod
def concat(cls, axis, left_parts, right_parts):
"""
Concatenate the blocks of partitions with another set of blocks.
Parameters
----------
axis : int
The axis to concatenate to.
left_parts : np.ndarray
NumPy array of partitions to concatenate with.
right_parts : np.ndarray or list
NumPy array of partitions to be concatenated.
Returns
-------
np.ndarray
A new NumPy array with concatenated partitions.
list[int] or None
Row lengths if possible to compute it.
Notes
-----
Assumes that the blocks are already the same shape on the
dimension being concatenated. A ValueError will be thrown if this
condition is not met.
"""
# TODO: Possible change is `isinstance(right_parts, list)`
if type(right_parts) is list:
# `np.array` with partitions of empty ModinFrame has a shape (0,)
# but `np.concatenate` can concatenate arrays only if its shapes at
# specified axis are equals, so filtering empty frames to avoid concat error
right_parts = [o for o in right_parts if o.size != 0]
to_concat = (
[left_parts] + right_parts if left_parts.size != 0 else right_parts
)
result = (
np.concatenate(to_concat, axis=axis) if len(to_concat) else left_parts
)
else:
result = np.append(left_parts, right_parts, axis=axis)
if axis == 0:
return cls.rebalance_partitions(result)
else:
return result, None
@classmethod
def to_pandas(cls, partitions):
"""
Convert NumPy array of PandasDataframePartition to pandas DataFrame.
Parameters
----------
partitions : np.ndarray
NumPy array of PandasDataframePartition.
Returns
-------
pandas.DataFrame
A pandas DataFrame
"""
return create_pandas_df_from_partitions(
cls.get_objects_from_partitions(partitions.flatten()), partitions.shape
)
@classmethod
def to_numpy(cls, partitions, **kwargs):
"""
Convert NumPy array of PandasDataframePartition to NumPy array of data stored within `partitions`.
Parameters
----------
partitions : np.ndarray
NumPy array of PandasDataframePartition.
**kwargs : dict
Keyword arguments for PandasDataframePartition.to_numpy function.
Returns
-------
np.ndarray
A NumPy array.
"""
return np.block(
[[block.to_numpy(**kwargs) for block in row] for row in partitions]
)
@classmethod
def split_pandas_df_into_partitions(
cls, df, row_chunksize, col_chunksize, update_bar
):
"""
Split given pandas DataFrame according to the row/column chunk sizes into distributed partitions.
Parameters
----------
df : pandas.DataFrame
row_chunksize : int
col_chunksize : int
update_bar : callable(x) -> x
Function that updates a progress bar.
Returns
-------
2D np.ndarray[PandasDataframePartition]
"""
put_func = cls._partition_class.put
# even a full-axis slice can cost something (https://github.com/pandas-dev/pandas/issues/55202)
# so we try not to do it if unnecessary.
if col_chunksize >= len(df.columns):
col_parts = [df]
else:
col_parts = [
df.iloc[:, i : i + col_chunksize]
for i in range(0, len(df.columns), col_chunksize)
]
parts = [
[
update_bar(
put_func(col_part.iloc[i : i + row_chunksize]),
)
for col_part in col_parts
]
for i in range(0, len(df), row_chunksize)
]
return np.array(parts)
@classmethod
@wait_computations_if_benchmark_mode
def from_pandas(cls, df, return_dims=False):
"""
Return the partitions from pandas.DataFrame.
Parameters
----------
df : pandas.DataFrame
A pandas.DataFrame.
return_dims : bool, default: False
If it's True, return as (np.ndarray, row_lengths, col_widths),
else np.ndarray.
Returns
-------
(np.ndarray, backend) or (np.ndarray, backend, row_lengths, col_widths)
A NumPy array with partitions (with dimensions or not).
"""
num_splits = NPartitions.get()
min_row_block_size = MinRowPartitionSize.get()
min_column_block_size = MinColumnPartitionSize.get()
row_chunksize = compute_chunksize(df.shape[0], num_splits, min_row_block_size)
col_chunksize = compute_chunksize(
df.shape[1], num_splits, min_column_block_size
)
bar_format = (
"{l_bar}{bar}{r_bar}"
if os.environ.get("DEBUG_PROGRESS_BAR", "False") == "True"
else "{desc}: {percentage:3.0f}%{bar} Elapsed time: {elapsed}, estimated remaining time: {remaining}"
)
if ProgressBar.get():
with warnings.catch_warnings():
warnings.simplefilter("ignore")
try:
from tqdm.autonotebook import tqdm as tqdm_notebook
except ImportError:
raise ImportError("Please pip install tqdm to use the progress bar")
rows = max(1, round(len(df) / row_chunksize))
cols = max(1, round(len(df.columns) / col_chunksize))
update_count = rows * cols
pbar = tqdm_notebook(
total=round(update_count),
desc="Distributing Dataframe",
bar_format=bar_format,
)
else:
pbar = None
def update_bar(f):
if ProgressBar.get():
pbar.update(1)
return f
parts = cls.split_pandas_df_into_partitions(
df, row_chunksize, col_chunksize, update_bar
)
backend = get_pandas_backend(df.dtypes)
if ProgressBar.get():
pbar.close()
if not return_dims:
return parts, backend
else:
row_lengths = [
(
row_chunksize
if i + row_chunksize < len(df)
else len(df) % row_chunksize or row_chunksize
)
for i in range(0, len(df), row_chunksize)
]
col_widths = [
(
col_chunksize
if i + col_chunksize < len(df.columns)
else len(df.columns) % col_chunksize or col_chunksize
)
for i in range(0, len(df.columns), col_chunksize)
]
return parts, backend, row_lengths, col_widths
@classmethod
def from_arrow(cls, at, return_dims=False):
"""
Return the partitions from Apache Arrow (PyArrow).
Parameters
----------
at : pyarrow.table
Arrow Table.
return_dims : bool, default: False
If it's True, return as (np.ndarray, row_lengths, col_widths),
else np.ndarray.
Returns
-------
(np.ndarray, backend) or (np.ndarray, backend, row_lengths, col_widths)
A NumPy array with partitions (with dimensions or not).
"""
return cls.from_pandas(at.to_pandas(), return_dims=return_dims)
@classmethod
def get_objects_from_partitions(cls, partitions):
"""
Get the objects wrapped by `partitions` (in parallel if supported).
Parameters
----------
partitions : np.ndarray
NumPy array with ``PandasDataframePartition``-s.
Returns
-------
list
The objects wrapped by `partitions`.
"""
if hasattr(cls, "_execution_wrapper"):
# more efficient parallel implementation
for idx, part in enumerate(partitions):
if hasattr(part, "force_materialization"):
partitions[idx] = part.force_materialization()
assert all(
[len(partition.list_of_blocks) == 1 for partition in partitions]
), "Implementation assumes that each partition contains a single block."
return cls._execution_wrapper.materialize(
[partition.list_of_blocks[0] for partition in partitions]
)
return [partition.get() for partition in partitions]
@classmethod
def wait_partitions(cls, partitions):
"""
Wait on the objects wrapped by `partitions`, without materializing them.
This method will block until all computations in the list have completed.
Parameters
----------
partitions : np.ndarray
NumPy array with ``PandasDataframePartition``-s.
Notes
-----
This method should be implemented in a more efficient way for engines that supports
waiting on objects in parallel.
"""
for partition in partitions:
partition.wait()
@classmethod
def get_indices(cls, axis, partitions, index_func=None):
"""
Get the internal indices stored in the partitions.
Parameters
----------
axis : {0, 1}
Axis to extract the labels over.
partitions : np.ndarray
NumPy array with PandasDataframePartition's.
index_func : callable, default: None
The function to be used to extract the indices.
Returns
-------
pandas.Index
A pandas Index object.
list of pandas.Index
The list of internal indices for each partition.
Notes
-----
These are the global indices of the object. This is mostly useful
when you have deleted rows/columns internally, but do not know
which ones were deleted.
"""
if index_func is None:
index_func = lambda df: df.axes[axis] # noqa: E731
ErrorMessage.catch_bugs_and_request_email(not callable(index_func))
func = cls.preprocess_func(index_func)
target = partitions.T if axis == 0 else partitions
if len(target):
new_idx = [idx.apply(func) for idx in target[0]]
new_idx = cls.get_objects_from_partitions(new_idx)
else:
new_idx = [pandas.Index([])]
# filter empty indexes in case there are multiple partitions
total_idx = list(filter(len, new_idx))
if len(total_idx) > 0:
# TODO FIX INFORMATION LEAK!!!!1!!1!!
total_idx = total_idx[0].append(total_idx[1:])
else:
# Meaning that all partitions returned a zero-length index,
# in this case, we return an index of any partition to preserve
# the index's metadata
total_idx = new_idx[0]
return total_idx, new_idx
@classmethod
def _apply_func_to_list_of_partitions_broadcast(
cls, func, partitions, other, **kwargs
):
"""
Apply a function to a list of remote partitions.
`other` partitions will be broadcasted to `partitions`
and `func` will be applied.
Parameters
----------
func : callable
The func to apply.
partitions : np.ndarray
The partitions to which the `func` will apply.
other : np.ndarray
The partitions to be broadcasted to `partitions`.
**kwargs : dict
Keyword arguments for PandasDataframePartition.apply function.
Returns
-------
list
A list of PandasDataframePartition objects.
"""
preprocessed_func = cls.preprocess_func(func)
return [
obj.apply(preprocessed_func, other=[o.get() for o in broadcasted], **kwargs)
for obj, broadcasted in zip(partitions, other.T)
]
@classmethod
def _apply_func_to_list_of_partitions(cls, func, partitions, **kwargs):
"""
Apply a function to a list of remote partitions.
Parameters
----------
func : callable
The func to apply.
partitions : np.ndarray
The partitions to which the `func` will apply.
**kwargs : dict
Keyword arguments for PandasDataframePartition.apply function.
Returns
-------
list
A list of PandasDataframePartition objects.
Notes
-----
This preprocesses the `func` first before applying it to the partitions.
"""
preprocessed_func = cls.preprocess_func(func)
return [obj.apply(preprocessed_func, **kwargs) for obj in partitions]
@classmethod
def combine(cls, partitions, new_index=None, new_columns=None):
"""
Convert a NumPy 2D array of partitions to a NumPy 2D array of a single partition.
Parameters
----------
partitions : np.ndarray
The partitions which have to be converted to a single partition.
new_index : pandas.Index, optional
Index for propagation into internal partitions.
Optimization allowing to do this in one remote kernel.
new_columns : pandas.Index, optional
Columns for propagation into internal partitions.
Optimization allowing to do this in one remote kernel.
Returns
-------
np.ndarray
A NumPy 2D array of a single partition.
"""
if partitions.size <= 1 and new_index is None and new_columns is None:
return partitions
def to_pandas_remote(df, partition_shape, *dfs):
"""Copy of ``cls.to_pandas()`` method adapted for a remote function."""
return create_pandas_df_from_partitions(
(df,) + dfs,
partition_shape,
called_from_remote=True,
new_index=new_index,
new_columns=new_columns,
)
preprocessed_func = cls.preprocess_func(to_pandas_remote)
partition_shape = partitions.shape
partitions_flattened = partitions.flatten()
for idx, part in enumerate(partitions_flattened):
if hasattr(part, "force_materialization"):
partitions_flattened[idx] = part.force_materialization()
partition_refs = [
partition.list_of_blocks[0] for partition in partitions_flattened[1:]
]
combined_partition = partitions.flat[0].apply(
preprocessed_func, partition_shape, *partition_refs
)
return np.array([combined_partition]).reshape(1, -1)
@classmethod
@wait_computations_if_benchmark_mode
def apply_func_to_select_indices(
cls, axis, partitions, func, indices, keep_remaining=False
):
"""
Apply a function to select indices.
Parameters
----------
axis : {0, 1}
Axis to apply the `func` over.
partitions : np.ndarray
The partitions to which the `func` will apply.
func : callable
The function to apply to these indices of partitions.
indices : dict
The indices to apply the function to.
keep_remaining : bool, default: False
Whether or not to keep the other partitions. Some operations
may want to drop the remaining partitions and keep
only the results.
Returns
-------
np.ndarray
A NumPy array with partitions.
Notes
-----
Your internal function must take a kwarg `internal_indices` for
this to work correctly. This prevents information leakage of the
internal index to the external representation.
"""
if partitions.size == 0:
return np.array([[]])
# Handling dictionaries has to be done differently, but we still want
# to figure out the partitions that need to be applied to, so we will
# store the dictionary in a separate variable and assign `indices` to
# the keys to handle it the same as we normally would.
if isinstance(func, dict):
dict_func = func
else:
dict_func = None
if not axis:
partitions_for_apply = partitions.T
else:
partitions_for_apply = partitions
# We may have a command to perform different functions on different
# columns at the same time. We attempt to handle this as efficiently as
# possible here. Functions that use this in the dictionary format must
# accept a keyword argument `func_dict`.
if dict_func is not None:
if not keep_remaining:
result = np.array(
[
cls._apply_func_to_list_of_partitions(
func,
partitions_for_apply[o_idx],
func_dict={
i_idx: dict_func[i_idx]
for i_idx in list_to_apply
if i_idx >= 0
},
)
for o_idx, list_to_apply in indices.items()
]
)
else:
result = np.array(
[
(
partitions_for_apply[i]
if i not in indices
else cls._apply_func_to_list_of_partitions(
func,
partitions_for_apply[i],
func_dict={
idx: dict_func[idx]
for idx in indices[i]
if idx >= 0
},
)
)
for i in range(len(partitions_for_apply))
]
)
else:
if not keep_remaining:
# We are passing internal indices in here. In order for func to
# actually be able to use this information, it must be able to take in
# the internal indices. This might mean an iloc in the case of Pandas
# or some other way to index into the internal representation.
result = np.array(
[
cls._apply_func_to_list_of_partitions(
func,
partitions_for_apply[idx],
internal_indices=list_to_apply,
)
for idx, list_to_apply in indices.items()
]
)
else:
# The difference here is that we modify a subset and return the
# remaining (non-updated) blocks in their original position.
result = np.array(
[
(
partitions_for_apply[i]
if i not in indices
else cls._apply_func_to_list_of_partitions(
func,
partitions_for_apply[i],
internal_indices=indices[i],
)
)
for i in range(len(partitions_for_apply))
]
)
return result.T if not axis else result
@classmethod
@wait_computations_if_benchmark_mode
def apply_func_to_select_indices_along_full_axis(
cls, axis, partitions, func, indices, keep_remaining=False
):
"""
Apply a function to a select subset of full columns/rows.
Parameters
----------
axis : {0, 1}
The axis to apply the function over.
partitions : np.ndarray
The partitions to which the `func` will apply.
func : callable
The function to apply.
indices : list-like
The global indices to apply the func to.
keep_remaining : bool, default: False
Whether or not to keep the other partitions.
Some operations may want to drop the remaining partitions and
keep only the results.
Returns
-------
np.ndarray
A NumPy array with partitions.
Notes
-----
This should be used when you need to apply a function that relies
on some global information for the entire column/row, but only need
to apply a function to a subset.
For your func to operate directly on the indices provided,
it must use `internal_indices` as a keyword argument.
"""
if partitions.size == 0:
return np.array([[]])
# Handling dictionaries has to be done differently, but we still want
# to figure out the partitions that need to be applied to, so we will
# store the dictionary in a separate variable and assign `indices` to
# the keys to handle it the same as we normally would.
if isinstance(func, dict):
dict_func = func
else:
dict_func = None
preprocessed_func = cls.preprocess_func(func)
# Since we might be keeping the remaining blocks that are not modified,
# we have to also keep the block_partitions object in the correct
# direction (transpose for columns).
if not keep_remaining:
selected_partitions = partitions.T if not axis else partitions
selected_partitions = np.array([selected_partitions[i] for i in indices])
selected_partitions = (
selected_partitions.T if not axis else selected_partitions
)
else:
selected_partitions = partitions
if not axis:
partitions_for_apply = cls.column_partitions(selected_partitions)
partitions_for_remaining = partitions.T
else:
partitions_for_apply = cls.row_partitions(selected_partitions)
partitions_for_remaining = partitions
# We may have a command to perform different functions on different
# columns at the same time. We attempt to handle this as efficiently as
# possible here. Functions that use this in the dictionary format must
# accept a keyword argument `func_dict`.
if dict_func is not None:
if not keep_remaining:
result = np.array(
[
part.apply(
preprocessed_func,
func_dict={idx: dict_func[idx] for idx in indices[i]},
)
for i, part in zip(indices, partitions_for_apply)
]
)
else:
result = np.array(
[
(
partitions_for_remaining[i]
if i not in indices
else cls._apply_func_to_list_of_partitions(
preprocessed_func,
partitions_for_apply[i],
func_dict={idx: dict_func[idx] for idx in indices[i]},
)
)
for i in range(len(partitions_for_apply))
]
)
else:
if not keep_remaining:
# See notes in `apply_func_to_select_indices`
result = np.array(
[
part.apply(preprocessed_func, internal_indices=indices[i])
for i, part in zip(indices, partitions_for_apply)
]
)
else:
# See notes in `apply_func_to_select_indices`
result = np.array(
[
(
partitions_for_remaining[i]
if i not in indices
else partitions_for_apply[i].apply(
preprocessed_func, internal_indices=indices[i]
)
)
for i in range(len(partitions_for_remaining))
]
)
return result.T if not axis else result
@classmethod
@wait_computations_if_benchmark_mode
def apply_func_to_indices_both_axis(
cls,
partitions,
func,
row_partitions_list,
col_partitions_list,
item_to_distribute=no_default,
row_lengths=None,
col_widths=None,
):
"""
Apply a function along both axes.
Parameters
----------
partitions : np.ndarray
The partitions to which the `func` will apply.
func : callable
The function to apply.
row_partitions_list : iterable of tuples
Iterable of tuples, containing 2 values:
1. Integer row partition index.
2. Internal row indexer of this partition.
col_partitions_list : iterable of tuples
Iterable of tuples, containing 2 values:
1. Integer column partition index.
2. Internal column indexer of this partition.
item_to_distribute : np.ndarray or scalar, default: no_default
The item to split up so it can be applied over both axes.
row_lengths : list of ints, optional
Lengths of partitions for every row. If not specified this information
is extracted from partitions itself.
col_widths : list of ints, optional
Widths of partitions for every column. If not specified this information
is extracted from partitions itself.
Returns
-------
np.ndarray
A NumPy array with partitions.
Notes
-----
For your func to operate directly on the indices provided,
it must use `row_internal_indices`, `col_internal_indices` as keyword
arguments.
"""
partition_copy = partitions.copy()
row_position_counter = 0
if row_lengths is None:
row_lengths = [None] * len(row_partitions_list)
if col_widths is None:
col_widths = [None] * len(col_partitions_list)
def compute_part_size(indexer, remote_part, part_idx, axis):
"""Compute indexer length along the specified axis for the passed partition."""
if isinstance(indexer, slice):
shapes_container = row_lengths if axis == 0 else col_widths
part_size = shapes_container[part_idx]
if part_size is None:
part_size = (
remote_part.length() if axis == 0 else remote_part.width()
)
shapes_container[part_idx] = part_size
indexer = range(*indexer.indices(part_size))
return len(indexer)
for row_idx, row_values in enumerate(row_partitions_list):
row_blk_idx, row_internal_idx = row_values
col_position_counter = 0
row_offset = 0
for col_idx, col_values in enumerate(col_partitions_list):
col_blk_idx, col_internal_idx = col_values
remote_part = partition_copy[row_blk_idx, col_blk_idx]
row_offset = compute_part_size(
row_internal_idx, remote_part, row_idx, axis=0
)
col_offset = compute_part_size(
col_internal_idx, remote_part, col_idx, axis=1
)
if item_to_distribute is not no_default:
if isinstance(item_to_distribute, np.ndarray):
item = item_to_distribute[
row_position_counter : row_position_counter + row_offset,
col_position_counter : col_position_counter + col_offset,
]
else:
item = item_to_distribute
item = {"item": item}
else:
item = {}
block_result = remote_part.add_to_apply_calls(
func,
row_internal_indices=row_internal_idx,
col_internal_indices=col_internal_idx,
**item,
)
partition_copy[row_blk_idx, col_blk_idx] = block_result
col_position_counter += col_offset
row_position_counter += row_offset
return partition_copy
@classmethod
@wait_computations_if_benchmark_mode
def n_ary_operation(cls, left, func, right: list):
r"""
Apply an n-ary operation to multiple ``PandasDataframe`` objects.
This method assumes that all the partitions of the dataframes in left
and right have the same dimensions. For each position i, j in each
dataframe's partitions, the result has a partition at (i, j) whose data
is func(left_partitions[i,j], \*each_right_partitions[i,j]).
Parameters
----------
left : np.ndarray
The partitions of left ``PandasDataframe``.
func : callable
The function to apply.
right : list of np.ndarray
The list of partitions of other ``PandasDataframe``.
Returns
-------
np.ndarray
A NumPy array with new partitions.
"""
func = cls.preprocess_func(func)
def get_right_block(right_partitions, row_idx, col_idx):
partition = right_partitions[row_idx][col_idx]
blocks = partition.list_of_blocks
"""
NOTE:
Currently we do one remote call per right virtual partition to
materialize the partitions' blocks, then another remote call to do
the n_ary operation. we could get better performance if we
assembled the other partition within the remote `apply` call, by
passing the partition in as `other_axis_partition`. However,
passing `other_axis_partition` requires some extra care that would
complicate the code quite a bit:
- block partitions don't know how to deal with `other_axis_partition`
- the right axis partition's axis could be different from the axis
of the corresponding left partition
- there can be multiple other_axis_partition because this is an n-ary
operation and n can be > 2.
So for now just do the materialization in a separate remote step.
"""
if len(blocks) > 1:
partition.force_materialization()
assert len(partition.list_of_blocks) == 1
return partition.list_of_blocks[0]
return np.array(
[
[
part.apply(
func,
*(
get_right_block(right_partitions, row_idx, col_idx)
for right_partitions in right
),
)
for col_idx, part in enumerate(left[row_idx])
]
for row_idx in range(len(left))
]
)
@classmethod
def finalize(cls, partitions):
"""
Perform all deferred calls on partitions.
Parameters
----------
partitions : np.ndarray
Partitions of Modin Dataframe on which all deferred calls should be performed.
"""
[part.drain_call_queue() for row in partitions for part in row]
@classmethod
def rebalance_partitions(cls, partitions):
"""
Rebalance a 2-d array of partitions if we are using ``PandasOnRay`` or ``PandasOnDask`` executions.
For all other executions, the partitions are returned unchanged.
Rebalance the partitions by building a new array
of partitions out of the original ones so that:
- If all partitions have a length, each new partition has roughly the same number of rows.
- Otherwise, each new partition spans roughly the same number of old partitions.
Parameters
----------
partitions : np.ndarray
The 2-d array of partitions to rebalance.
Returns
-------
np.ndarray
A NumPy array with the same; or new, rebalanced, partitions, depending on the execution
engine and storage format.
list[int] or None
Row lengths if possible to compute it.
"""
# We rebalance when the ratio of the number of existing partitions to
# the ideal number of partitions is larger than this threshold. The
# threshold is a heuristic that may need to be tuned for performance.
max_excess_of_num_partitions = 1.5
num_existing_partitions = partitions.shape[0]
ideal_num_new_partitions = NPartitions.get()
if (
num_existing_partitions
<= ideal_num_new_partitions * max_excess_of_num_partitions
):
return partitions, None
# If any partition has an unknown length, give each axis partition
# roughly the same number of row partitions. We use `_length_cache` here
# to avoid materializing any unmaterialized lengths.
if any(
partition._length_cache is None for row in partitions for partition in row
):
# We need each partition to go into an axis partition, but the
# number of axis partitions may not evenly divide the number of
# partitions.
chunk_size = compute_chunksize(
num_existing_partitions, ideal_num_new_partitions, min_block_size=1
)
new_partitions = np.array(
[
cls.column_partitions(
partitions[i : i + chunk_size],
full_axis=False,
)
for i in range(
0,
num_existing_partitions,
chunk_size,
)
]
)
return new_partitions, None
# If we know the number of rows in every partition, then we should try
# instead to give each new partition roughly the same number of rows.
new_partitions = []
# `start` is the index of the first existing partition that we want to
# put into the current new partition.
start = 0
total_rows = sum(part.length() for part in partitions[:, 0])
ideal_partition_size = compute_chunksize(
total_rows, ideal_num_new_partitions, min_block_size=1
)
for _ in range(ideal_num_new_partitions):
# We might pick up old partitions too quickly and exhaust all of them.
if start >= len(partitions):
break
# `stop` is the index of the last existing partition so far that we
# want to put into the current new partition.
stop = start
partition_size = partitions[start][0].length()
# Add existing partitions into the current new partition until the
# number of rows in the new partition hits `ideal_partition_size`.
while stop < len(partitions) and partition_size < ideal_partition_size:
stop += 1
if stop < len(partitions):
partition_size += partitions[stop][0].length()
# If the new partition is larger than we want, split the last
# current partition that it contains into two partitions, where
# the first partition has just enough rows to make the current
# new partition have length `ideal_partition_size`, and the second
# partition has the remainder.
if partition_size > ideal_partition_size * max_excess_of_num_partitions:
prev_length = sum(row[0].length() for row in partitions[start:stop])
new_last_partition_size = ideal_partition_size - prev_length
partitions = np.insert(
partitions,
stop + 1,
[
obj.mask(slice(new_last_partition_size, None), slice(None))
for obj in partitions[stop]
],
0,
)
# TODO: explicit `_length_cache` computing may be avoided after #4903 is merged
for obj in partitions[stop + 1]:
obj._length_cache = partition_size - (
prev_length + new_last_partition_size
)
partitions[stop, :] = [
obj.mask(slice(None, new_last_partition_size), slice(None))
for obj in partitions[stop]
]
# TODO: explicit `_length_cache` computing may be avoided after #4903 is merged
for obj in partitions[stop]:
obj._length_cache = new_last_partition_size
# The new virtual partitions are not `full_axis`, even if they
# happen to span all rows in the dataframe, because they are
# meant to be the final partitions of the dataframe. They've
# already been split up correctly along axis 0, but using the
# default full_axis=True would cause partition.apply() to split
# its result along axis 0.
new_partitions.append(
cls.column_partitions(partitions[start : stop + 1], full_axis=False)
)
start = stop + 1
new_partitions = np.array(new_partitions)
lengths = [part.length() for part in new_partitions[:, 0]]
return new_partitions, lengths
@classmethod
@wait_computations_if_benchmark_mode
def shuffle_partitions(
cls,
partitions,
index,
shuffle_functions: "ShuffleFunctions",
final_shuffle_func,
right_partitions=None,
):
"""
Return shuffled partitions.
Parameters
----------
partitions : np.ndarray
The 2-d array of partitions to shuffle.
index : int or list of ints
The index(es) of the column partitions corresponding to the partitions that contain the column to sample.
shuffle_functions : ShuffleFunctions
An object implementing the functions that we will be using to perform this shuffle.
final_shuffle_func : Callable(pandas.DataFrame) -> pandas.DataFrame
Function that shuffles the data within each new partition.
right_partitions : np.ndarray, optional
Partitions to broadcast to `self` partitions. If specified, the method builds range-partitioning
for `right_partitions` basing on bins calculated for `partitions`, then performs broadcasting.
Returns
-------
np.ndarray
A list of row-partitions that have been shuffled.
"""
# Mask the partition that contains the column that will be sampled.
masked_partitions = partitions[:, index]
# Sample each partition
sample_func = cls.preprocess_func(shuffle_functions.sample_fn)
if masked_partitions.ndim == 1:
samples = [partition.apply(sample_func) for partition in masked_partitions]
else:
samples = [
cls._row_partition_class(row_part, full_axis=False).apply(sample_func)
for row_part in masked_partitions
]
# Get each sample to pass in to the pivot function
samples = cls.get_objects_from_partitions(samples)
num_bins = shuffle_functions.pivot_fn(samples)
# Convert our list of block partitions to row partitions. We need to create full-axis
# row partitions since we need to send the whole partition to the split step as otherwise
# we wouldn't know how to split the block partitions that don't contain the shuffling key.
row_partitions = cls.row_partitions(partitions)
if num_bins > 1:
# Gather together all of the sub-partitions
split_row_partitions = np.array(
[
partition.split(
shuffle_functions.split_fn,
num_splits=num_bins,
# The partition's metadata will never be accessed for the split partitions,
# thus no need to compute it.
extract_metadata=False,
)
for partition in row_partitions
]
).T
if right_partitions is None:
# We need to convert every partition that came from the splits into a column partition.
return np.array(
[
[
cls._column_partitions_class(
row_partition, full_axis=False
).apply(final_shuffle_func)
]
for row_partition in split_row_partitions
]
)
right_row_parts = cls.row_partitions(right_partitions)
right_split_row_partitions = np.array(
[
partition.split(
shuffle_functions.split_fn,
num_splits=num_bins,
extract_metadata=False,
)
for partition in right_row_parts
]
).T
return np.array(
[
cls._column_partitions_class(row_partition, full_axis=False).apply(
final_shuffle_func,
other_axis_partition=cls._column_partitions_class(
right_row_partitions
),
)
for right_row_partitions, row_partition in zip(
right_split_row_partitions, split_row_partitions
)
]
)
else:
# If there are not pivots we can simply apply the function row-wise
if right_partitions is None:
return np.array(
[row_part.apply(final_shuffle_func) for row_part in row_partitions]
)
right_row_parts = cls.row_partitions(right_partitions)
return np.array(
[
row_part.apply(
final_shuffle_func, other_axis_partition=right_row_part
)
for right_row_part, row_part in zip(right_row_parts, row_partitions)
]
)
================================================
FILE: modin/core/dataframe/pandas/utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Collection of utility functions for the PandasDataFrame."""
import pandas
from pandas.api.types import union_categoricals
from modin.error_message import ErrorMessage
def concatenate(dfs, copy=True):
"""
Concatenate pandas DataFrames with saving 'category' dtype.
All dataframes' columns must be equal to each other.
Parameters
----------
dfs : list
List of pandas DataFrames to concatenate.
copy : bool, default: True
Make explicit copy when creating dataframe.
Returns
-------
pandas.DataFrame
A pandas DataFrame.
"""
for df in dfs:
assert df.columns.equals(dfs[0].columns)
for i in dfs[0].columns.get_indexer_for(dfs[0].select_dtypes("category").columns):
columns = [df.iloc[:, i] for df in dfs]
all_categorical_parts_are_empty = None
has_non_categorical_parts = False
for col in columns:
if isinstance(col.dtype, pandas.CategoricalDtype):
if all_categorical_parts_are_empty is None:
all_categorical_parts_are_empty = len(col) == 0
continue
all_categorical_parts_are_empty &= len(col) == 0
else:
has_non_categorical_parts = True
# 'union_categoricals' raises an error if some of the passed values don't have categorical dtype,
# if it happens, we only want to continue when all parts with categorical dtypes are actually empty.
# This can happen if there were an aggregation that discards categorical dtypes and that aggregation
# doesn't properly do so for empty partitions
if has_non_categorical_parts and all_categorical_parts_are_empty:
continue
union = union_categoricals(columns)
for df in dfs:
df.isetitem(
i, pandas.Categorical(df.iloc[:, i], categories=union.categories)
)
# `ValueError: buffer source array is read-only` if copy==False
if len(dfs) == 1 and copy:
# concat doesn't make a copy if len(dfs) == 1,
# so do it explicitly
return dfs[0].copy()
return pandas.concat(dfs, copy=copy)
def create_pandas_df_from_partitions(
partition_data,
partition_shape,
called_from_remote=False,
new_index=None,
new_columns=None,
):
"""
Convert partition data of multiple dataframes to a single dataframe.
Parameters
----------
partition_data : list
List of pandas DataFrames or list of Object references holding pandas DataFrames.
partition_shape : int or tuple
Shape of the partitions NumPy array.
called_from_remote : bool, default: False
Flag used to check if explicit copy should be done in concat.
new_index : pandas.Index, optional
Index for propagation into internal partitions.
Optimization allowing to do this in one remote kernel.
new_columns : pandas.Index, optional
Columns for propagation into internal partitions.
Optimization allowing to do this in one remote kernel.
Returns
-------
pandas.DataFrame
A pandas DataFrame.
"""
if all(
isinstance(obj, (pandas.DataFrame, pandas.Series)) for obj in partition_data
):
height, width, *_ = tuple(partition_shape) + (0,)
# restore 2d array
objs = iter(partition_data)
partition_data = [[next(objs) for _ in range(width)] for __ in range(height)]
else:
# Partitions do not always contain pandas objects.
# This implementation comes from the fact that calling `partition.get`
# function is not always equivalent to `partition.to_pandas`.
partition_data = [[obj.to_pandas() for obj in part] for part in partition_data]
if all(isinstance(part, pandas.Series) for row in partition_data for part in row):
axis = 0
elif all(
isinstance(part, pandas.DataFrame) for row in partition_data for part in row
):
axis = 1
else:
ErrorMessage.catch_bugs_and_request_email(True)
def is_part_empty(part):
return part.empty and (
not isinstance(part, pandas.DataFrame) or (len(part.columns) == 0)
)
df_rows = [
pandas.concat([part for part in row], axis=axis, copy=False)
for row in partition_data
if not all(is_part_empty(part) for part in row)
]
# to reduce peak memory consumption
del partition_data
if len(df_rows) == 0:
res = pandas.DataFrame()
else:
res = concatenate(df_rows, copy=not called_from_remote)
if new_index is not None:
res.index = new_index
if new_columns is not None:
res.columns = new_columns
return res
================================================
FILE: modin/core/execution/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to execution engines supported."""
================================================
FILE: modin/core/execution/dask/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Dask execution engine."""
================================================
FILE: modin/core/execution/dask/common/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Common utilities for Dask execution engine."""
from .engine_wrapper import DaskWrapper
from .utils import initialize_dask
__all__ = [
"initialize_dask",
"DaskWrapper",
]
================================================
FILE: modin/core/execution/dask/common/engine_wrapper.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses class responsible for execution of remote operations."""
from collections import UserDict
import pandas
from dask.distributed import wait
from distributed import Future
from distributed.client import default_client
from distributed.worker import get_worker
def get_dask_client():
"""
Get the Dask client, reusing the worker's client if execution is on a Dask worker.
Returns
-------
distributed.Client
The Dask client.
"""
try:
client = default_client()
except ValueError:
# We ought to be in a worker process
worker = get_worker()
client = worker.client
return client
def _deploy_dask_func(func, *args, return_pandas_df=None, **kwargs): # pragma: no cover
"""
Wrap `func` to ease calling it remotely.
Parameters
----------
func : callable
A local function that we want to call remotely.
*args : iterable
Positional arguments to pass to `func` when calling remotely.
return_pandas_df : bool, optional
Whether to convert the result of `func` to a pandas DataFrame or not.
**kwargs : dict
Keyword arguments to pass to `func` when calling remotely.
Returns
-------
distributed.Future or list
Dask identifier of the result being put into distributed memory.
"""
result = func(*args, **kwargs)
if return_pandas_df and not isinstance(result, pandas.DataFrame):
result = pandas.DataFrame(result)
return result
class DaskWrapper:
"""The class responsible for execution of remote operations."""
@classmethod
def deploy(
cls,
func,
f_args=None,
f_kwargs=None,
return_pandas_df=None,
num_returns=1,
pure=True,
):
"""
Deploy a function in a worker process.
Parameters
----------
func : callable or distributed.Future
Function to be deployed in a worker process.
f_args : list or tuple, optional
Positional arguments to pass to ``func``.
f_kwargs : dict, optional
Keyword arguments to pass to ``func``.
return_pandas_df : bool, optional
Whether to convert the result of `func` to a pandas DataFrame or not.
num_returns : int, default: 1
The number of returned objects.
pure : bool, default: True
Whether or not `func` is pure. See `Client.submit` for details.
Returns
-------
list
The result of ``func`` split into parts in accordance with ``num_returns``.
"""
client = get_dask_client()
args = [] if f_args is None else f_args
kwargs = {} if f_kwargs is None else f_kwargs
if callable(func):
remote_task_future = client.submit(func, *args, pure=pure, **kwargs)
else:
# for the case where type(func) is distributed.Future
remote_task_future = client.submit(
_deploy_dask_func,
func,
*args,
pure=pure,
return_pandas_df=return_pandas_df,
**kwargs,
)
if num_returns != 1:
return [
client.submit(lambda tup, i: tup[i], remote_task_future, i)
for i in range(num_returns)
]
return remote_task_future
@classmethod
def is_future(cls, item):
"""
Check if the item is a Future.
Parameters
----------
item : distributed.Future or object
Future or object to check.
Returns
-------
boolean
If the value is a future.
"""
return isinstance(item, Future)
@classmethod
def materialize(cls, future):
"""
Materialize data matching `future` object.
Parameters
----------
future : distributed.Future or list
Future object of list of future objects whereby data needs to be materialized.
Returns
-------
Any
An object(s) from the distributed memory.
"""
client = get_dask_client()
return client.gather(future)
@classmethod
def put(cls, data, **kwargs):
"""
Put data into distributed memory.
Parameters
----------
data : list, dict, or object
Data to scatter out to workers. Output type matches input type.
**kwargs : dict
Additional keyword arguments to be passed in `Client.scatter`.
Returns
-------
List, dict, iterator, or queue of futures matching the type of input.
"""
if isinstance(data, dict):
# there is a bug that looks similar to https://github.com/dask/distributed/issues/3965;
# to avoid this we could change behaviour for serialization:
#
# vs
# {'sep': , \
# 'delimiter': ...
data = UserDict(data)
client = get_dask_client()
return client.scatter(data, **kwargs)
@classmethod
def wait(cls, obj_ids, num_returns=None):
"""
Wait on the objects without materializing them (blocking operation).
Parameters
----------
obj_ids : list, scalar
num_returns : int, optional
"""
if not isinstance(obj_ids, list):
obj_ids = [obj_ids]
if num_returns is None:
num_returns = len(obj_ids)
if num_returns == len(obj_ids):
wait(obj_ids, return_when="ALL_COMPLETED")
else:
# Dask doesn't natively support `num_returns` as int.
# `wait` function doesn't always return only one finished future,
# so a simple loop is not enough here
done, not_done = wait(obj_ids, return_when="FIRST_COMPLETED")
while len(done) < num_returns and (i := 0 < num_returns):
extra_done, not_done = wait(not_done, return_when="FIRST_COMPLETED")
done.update(extra_done)
i += 1
================================================
FILE: modin/core/execution/dask/common/utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses utility function to initialize Dask environment."""
import os
from modin.config import (
CIAWSAccessKeyID,
CIAWSSecretAccessKey,
CpuCount,
DaskThreadsPerWorker,
GithubCI,
Memory,
NPartitions,
)
from modin.core.execution.utils import set_env
def initialize_dask():
"""Initialize Dask environment."""
from distributed.client import default_client
from distributed.worker import get_worker
try:
# Check if running within a Dask worker process
get_worker()
# If the above line does not raise an error, we are in a worker process
# and should not create a new client
return
except ValueError:
# Not in a Dask worker, proceed to check for or create a client
pass
try:
client = default_client()
def _disable_warnings():
import warnings
warnings.simplefilter("ignore", category=FutureWarning)
client.run(_disable_warnings)
except ValueError:
from distributed import Client
num_cpus = CpuCount.get()
threads_per_worker = DaskThreadsPerWorker.get()
memory_limit = Memory.get()
worker_memory_limit = memory_limit // num_cpus if memory_limit else "auto"
# when the client is initialized, environment variables are inherited
with set_env(PYTHONWARNINGS="ignore::FutureWarning"):
client = Client(
n_workers=num_cpus,
threads_per_worker=threads_per_worker,
memory_limit=worker_memory_limit,
)
if GithubCI.get():
# set these keys to run tests that write to the mock s3 service. this seems
# to be the way to pass environment variables to the workers:
# https://jacobtomlinson.dev/posts/2021/bio-for-2021/
access_key = CIAWSAccessKeyID.get()
aws_secret = CIAWSSecretAccessKey.get()
client.run(
lambda: os.environ.update(
{
"AWS_ACCESS_KEY_ID": access_key,
"AWS_SECRET_ACCESS_KEY": aws_secret,
}
)
)
num_cpus = len(client.ncores())
NPartitions._put(num_cpus)
CpuCount._put(num_cpus)
================================================
FILE: modin/core/execution/dask/implementations/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Dask execution engine and optimized for specific storage formats."""
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Dask execution engine and optimized for pandas storage format."""
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/dataframe/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe class optimized for pandas on Dask execution."""
from .dataframe import PandasOnDaskDataframe
__all__ = ["PandasOnDaskDataframe"]
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/dataframe/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses class that implements ``PandasDataframe``."""
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
from modin.utils import _inherit_docstrings
from ..partitioning.partition_manager import PandasOnDaskDataframePartitionManager
class PandasOnDaskDataframe(PandasDataframe):
"""
The class implements the interface in ``PandasDataframe``.
Parameters
----------
partitions : np.ndarray
A 2D NumPy array of partitions.
index : sequence
The index for the dataframe. Converted to a pandas.Index.
columns : sequence
The columns object for the dataframe. Converted to a pandas.Index.
row_lengths : list, optional
The length of each partition in the rows. The "height" of
each of the block partitions. Is computed if not provided.
column_widths : list, optional
The width of each partition in the columns. The "width" of
each of the block partitions. Is computed if not provided.
dtypes : pandas.Series, optional
The data types for the dataframe columns.
pandas_backend : {"pyarrow", None}, optional
Backend used by pandas. None - means default NumPy backend.
"""
_partition_mgr_cls = PandasOnDaskDataframePartitionManager
@classmethod
def reconnect(cls, address, attributes): # noqa: GL08
# The main goal is to configure the client for the worker process
# using the address passed by the custom `__reduce__` function
try:
from distributed import default_client
default_client()
except ValueError:
from distributed import Client
# setup `default_client` for worker process
_ = Client(address)
obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj
def __reduce__(self): # noqa: GL08
from distributed import default_client
address = default_client().scheduler_info()["address"]
return self.reconnect, (address, self.__dict__)
@property
@_inherit_docstrings(PandasDataframe.engine)
def engine(self) -> str:
return "Dask"
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/io/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base IO classes optimized for pandas on Dask execution."""
from .io import PandasOnDaskIO
__all__ = [
"PandasOnDaskIO",
]
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses class that implements ``BaseIO`` using Dask as an execution engine."""
import numpy as np
from distributed.client import default_client
from modin.core.execution.dask.common import DaskWrapper
from modin.core.execution.dask.implementations.pandas_on_dask.dataframe import (
PandasOnDaskDataframe,
)
from modin.core.execution.dask.implementations.pandas_on_dask.partitioning import (
PandasOnDaskDataframePartition,
)
from modin.core.io import (
BaseIO,
CSVDispatcher,
ExcelDispatcher,
FeatherDispatcher,
FWFDispatcher,
JSONDispatcher,
ParquetDispatcher,
SQLDispatcher,
)
from modin.core.storage_formats.pandas.parsers import (
PandasCSVParser,
PandasExcelParser,
PandasFeatherParser,
PandasFWFParser,
PandasJSONParser,
PandasParquetParser,
PandasSQLParser,
)
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
from modin.distributed.dataframe.pandas.partitions import (
from_partitions,
unwrap_partitions,
)
from modin.experimental.core.io import (
ExperimentalCSVGlobDispatcher,
ExperimentalCustomTextDispatcher,
ExperimentalGlobDispatcher,
ExperimentalSQLDispatcher,
)
from modin.experimental.core.storage_formats.pandas.parsers import (
ExperimentalCustomTextParser,
ExperimentalPandasCSVGlobParser,
ExperimentalPandasJsonParser,
ExperimentalPandasParquetParser,
ExperimentalPandasPickleParser,
ExperimentalPandasXmlParser,
)
from modin.pandas.series import Series
from modin.utils import MODIN_UNNAMED_SERIES_LABEL
class PandasOnDaskIO(BaseIO):
"""The class implements interface in ``BaseIO`` using Dask as an execution engine."""
frame_cls = PandasOnDaskDataframe
frame_partition_cls = PandasOnDaskDataframePartition
query_compiler_cls = PandasQueryCompiler
build_args = dict(
frame_cls=PandasOnDaskDataframe,
frame_partition_cls=PandasOnDaskDataframePartition,
query_compiler_cls=PandasQueryCompiler,
base_io=BaseIO,
)
def __make_read(*classes, build_args=build_args):
# used to reduce code duplication
return type("", (DaskWrapper, *classes), build_args).read
def __make_write(*classes, build_args=build_args):
# used to reduce code duplication
return type("", (DaskWrapper, *classes), build_args).write
read_csv = __make_read(PandasCSVParser, CSVDispatcher)
read_fwf = __make_read(PandasFWFParser, FWFDispatcher)
read_json = __make_read(PandasJSONParser, JSONDispatcher)
read_parquet = __make_read(PandasParquetParser, ParquetDispatcher)
to_parquet = __make_write(ParquetDispatcher)
# Blocked on pandas-dev/pandas#12236. It is faster to default to pandas.
# read_hdf = __make_read(PandasHDFParser, HDFReader)
read_feather = __make_read(PandasFeatherParser, FeatherDispatcher)
read_sql = __make_read(PandasSQLParser, SQLDispatcher)
to_sql = __make_write(SQLDispatcher)
read_excel = __make_read(PandasExcelParser, ExcelDispatcher)
# experimental methods that don't exist in pandas
read_csv_glob = __make_read(
ExperimentalPandasCSVGlobParser, ExperimentalCSVGlobDispatcher
)
read_parquet_glob = __make_read(
ExperimentalPandasParquetParser, ExperimentalGlobDispatcher
)
to_parquet_glob = __make_write(
ExperimentalGlobDispatcher,
build_args={**build_args, "base_write": BaseIO.to_parquet},
)
read_json_glob = __make_read(
ExperimentalPandasJsonParser, ExperimentalGlobDispatcher
)
to_json_glob = __make_write(
ExperimentalGlobDispatcher,
build_args={**build_args, "base_write": BaseIO.to_json},
)
read_xml_glob = __make_read(ExperimentalPandasXmlParser, ExperimentalGlobDispatcher)
to_xml_glob = __make_write(
ExperimentalGlobDispatcher,
build_args={**build_args, "base_write": BaseIO.to_xml},
)
read_pickle_glob = __make_read(
ExperimentalPandasPickleParser, ExperimentalGlobDispatcher
)
to_pickle_glob = __make_write(
ExperimentalGlobDispatcher,
build_args={**build_args, "base_write": BaseIO.to_pickle},
)
read_custom_text = __make_read(
ExperimentalCustomTextParser, ExperimentalCustomTextDispatcher
)
read_sql_distributed = __make_read(
ExperimentalSQLDispatcher, build_args={**build_args, "base_read": read_sql}
)
del __make_read # to not pollute class namespace
del __make_write # to not pollute class namespace
@classmethod
def from_dask(cls, dask_obj):
"""
Create a Modin `query_compiler` from a Dask DataFrame.
Parameters
----------
dask_obj : dask.dataframe.DataFrame
The Dask DataFrame to convert from.
Returns
-------
BaseQueryCompiler
QueryCompiler containing data from the Dask DataFrame.
"""
client = default_client()
dask_fututures = client.compute(dask_obj.to_delayed())
modin_df = from_partitions(dask_fututures, axis=0)._query_compiler
return modin_df
@classmethod
def to_dask(cls, modin_obj):
"""
Convert a Modin DataFrame/Series to a Dask DataFrame/Series.
Parameters
----------
modin_obj : modin.pandas.DataFrame, modin.pandas.Series
The Modin DataFrame/Series to convert.
Returns
-------
dask.dataframe.DataFrame or dask.dataframe.Series
Converted object with type depending on input.
"""
from dask.dataframe import from_delayed
partitions = unwrap_partitions(modin_obj, axis=0)
# partiotions must be converted to pandas Series
if isinstance(modin_obj, Series):
client = default_client()
def df_to_series(df):
series = df[df.columns[0]]
if df.columns[0] == MODIN_UNNAMED_SERIES_LABEL:
series.name = None
return series
partitions = [client.submit(df_to_series, part) for part in partitions]
return from_delayed(partitions)
@classmethod
def from_map(cls, func, iterable, *args, **kwargs):
"""
Create a Modin `query_compiler` from a map function.
This method will construct a Modin `query_compiler` split by row partitions.
The number of row partitions matches the number of elements in the iterable object.
Parameters
----------
func : callable
Function to map across the iterable object.
iterable : Iterable
An iterable object.
*args : tuple
Positional arguments to pass in `func`.
**kwargs : dict
Keyword arguments to pass in `func`.
Returns
-------
BaseQueryCompiler
QueryCompiler containing data returned by map function.
"""
func = cls.frame_cls._partition_mgr_cls.preprocess_func(func)
partitions = np.array(
[
[
cls.frame_partition_cls(
DaskWrapper.deploy(
func,
f_args=(obj,) + args,
f_kwargs=kwargs,
return_pandas_df=True,
)
)
]
for obj in iterable
]
)
return cls.query_compiler_cls(cls.frame_cls(partitions))
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/partitioning/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe classes related to its partitioning and optimized for pandas on Dask execution."""
from .partition import PandasOnDaskDataframePartition
from .partition_manager import PandasOnDaskDataframePartitionManager
from .virtual_partition import (
PandasOnDaskDataframeColumnPartition,
PandasOnDaskDataframeRowPartition,
PandasOnDaskDataframeVirtualPartition,
)
__all__ = [
"PandasOnDaskDataframePartition",
"PandasOnDaskDataframePartitionManager",
"PandasOnDaskDataframeVirtualPartition",
"PandasOnDaskDataframeColumnPartition",
"PandasOnDaskDataframeRowPartition",
]
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses class that wraps data (block partition) and its metadata."""
import pandas
from distributed import Future
from distributed.utils import get_ip
from modin.core.dataframe.pandas.partitioning.partition import PandasDataframePartition
from modin.core.execution.dask.common import DaskWrapper
from modin.logging import get_logger
from modin.pandas.indexing import compute_sliced_len
class PandasOnDaskDataframePartition(PandasDataframePartition):
"""
The class implements the interface in ``PandasDataframePartition``.
Parameters
----------
data : distributed.Future
A reference to pandas DataFrame that need to be wrapped with this class.
length : distributed.Future or int, optional
Length or reference to it of wrapped pandas DataFrame.
width : distributed.Future or int, optional
Width or reference to it of wrapped pandas DataFrame.
ip : distributed.Future or str, optional
Node IP address or reference to it that holds wrapped pandas DataFrame.
call_queue : list, optional
Call queue that needs to be executed on wrapped pandas DataFrame.
"""
execution_wrapper = DaskWrapper
def __init__(self, data, length=None, width=None, ip=None, call_queue=None):
super().__init__()
assert isinstance(data, Future)
self._data = data
if call_queue is None:
call_queue = []
self.call_queue = call_queue
self._length_cache = length
self._width_cache = width
self._ip_cache = ip
log = get_logger()
self._is_debug(log) and log.debug(
"Partition ID: {}, Height: {}, Width: {}, Node IP: {}".format(
self._identity,
str(self._length_cache),
str(self._width_cache),
str(self._ip_cache),
)
)
def apply(self, func, *args, **kwargs):
"""
Apply a function to the object wrapped by this partition.
Parameters
----------
func : callable or distributed.Future
A function to apply.
*args : iterable
Additional positional arguments to be passed in `func`.
**kwargs : dict
Additional keyword arguments to be passed in `func`.
Returns
-------
PandasOnDaskDataframePartition
A new ``PandasOnDaskDataframePartition`` object.
Notes
-----
The keyword arguments are sent as a dictionary.
"""
log = get_logger()
self._is_debug(log) and log.debug(f"ENTER::Partition.apply::{self._identity}")
call_queue = self.call_queue + [[func, args, kwargs]]
if len(call_queue) > 1:
self._is_debug(log) and log.debug(
f"SUBMIT::_apply_list_of_funcs::{self._identity}"
)
futures = self.execution_wrapper.deploy(
func=apply_list_of_funcs,
f_args=(call_queue, self._data),
num_returns=2,
pure=False,
)
else:
# We handle `len(call_queue) == 1` in a different way because
# this improves performance a bit.
func, f_args, f_kwargs = call_queue[0]
futures = self.execution_wrapper.deploy(
func=apply_func,
f_args=(self._data, func, *f_args),
f_kwargs=f_kwargs,
num_returns=2,
pure=False,
)
self._is_debug(log) and log.debug(f"SUBMIT::_apply_func::{self._identity}")
self._is_debug(log) and log.debug(f"EXIT::Partition.apply::{self._identity}")
return self.__constructor__(futures[0], ip=futures[1])
def drain_call_queue(self):
"""Execute all operations stored in the call queue on the object wrapped by this partition."""
log = get_logger()
self._is_debug(log) and log.debug(
f"ENTER::Partition.drain_call_queue::{self._identity}"
)
if len(self.call_queue) == 0:
return
call_queue = self.call_queue
if len(call_queue) > 1:
self._is_debug(log) and log.debug(
f"SUBMIT::_apply_list_of_funcs::{self._identity}"
)
futures = self.execution_wrapper.deploy(
func=apply_list_of_funcs,
f_args=(call_queue, self._data),
num_returns=2,
pure=False,
)
else:
# We handle `len(call_queue) == 1` in a different way because
# this improves performance a bit.
func, f_args, f_kwargs = call_queue[0]
self._is_debug(log) and log.debug(f"SUBMIT::_apply_func::{self._identity}")
futures = self.execution_wrapper.deploy(
func=apply_func,
f_args=(self._data, func, *f_args),
f_kwargs=f_kwargs,
num_returns=2,
pure=False,
)
self._data = futures[0]
self._ip_cache = futures[1]
self._is_debug(log) and log.debug(
f"EXIT::Partition.drain_call_queue::{self._identity}"
)
self.call_queue = []
def wait(self):
"""Wait completing computations on the object wrapped by the partition."""
self.drain_call_queue()
self.execution_wrapper.wait(self._data)
def mask(self, row_labels, col_labels):
"""
Lazily create a mask that extracts the indices provided.
Parameters
----------
row_labels : list-like, slice or label
The row labels for the rows to extract.
col_labels : list-like, slice or label
The column labels for the columns to extract.
Returns
-------
PandasOnDaskDataframePartition
A new ``PandasOnDaskDataframePartition`` object.
"""
log = get_logger()
self._is_debug(log) and log.debug(f"ENTER::Partition.mask::{self._identity}")
new_obj = super().mask(row_labels, col_labels)
if isinstance(row_labels, slice) and isinstance(self._length_cache, Future):
if row_labels == slice(None):
# fast path - full axis take
new_obj._length_cache = self._length_cache
else:
new_obj._length_cache = self.execution_wrapper.deploy(
func=compute_sliced_len, f_args=(row_labels, self._length_cache)
)
if isinstance(col_labels, slice) and isinstance(self._width_cache, Future):
if col_labels == slice(None):
# fast path - full axis take
new_obj._width_cache = self._width_cache
else:
new_obj._width_cache = self.execution_wrapper.deploy(
func=compute_sliced_len, f_args=(col_labels, self._width_cache)
)
self._is_debug(log) and log.debug(f"EXIT::Partition.mask::{self._identity}")
return new_obj
def __copy__(self):
"""
Create a copy of this partition.
Returns
-------
PandasOnDaskDataframePartition
A copy of this partition.
"""
return self.__constructor__(
self._data,
length=self._length_cache,
width=self._width_cache,
ip=self._ip_cache,
call_queue=self.call_queue,
)
@classmethod
def put(cls, obj):
"""
Put an object into distributed memory and wrap it with partition object.
Parameters
----------
obj : any
An object to be put.
Returns
-------
PandasOnDaskDataframePartition
A new ``PandasOnDaskDataframePartition`` object.
"""
return cls(
cls.execution_wrapper.put(obj, hash=False),
len(obj.index),
len(obj.columns),
)
@classmethod
def preprocess_func(cls, func):
"""
Preprocess a function before an ``apply`` call.
Parameters
----------
func : callable
The function to preprocess.
Returns
-------
callable
An object that can be accepted by ``apply``.
"""
return cls.execution_wrapper.put(func, hash=False, broadcast=True)
def length(self, materialize=True):
"""
Get the length of the object wrapped by this partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
int or distributed.Future
The length of the object.
"""
if self._length_cache is None:
self._length_cache = self.apply(len)._data
if isinstance(self._length_cache, Future) and materialize:
self._length_cache = self.execution_wrapper.materialize(self._length_cache)
return self._length_cache
def width(self, materialize=True):
"""
Get the width of the object wrapped by the partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
int or distributed.Future
The width of the object.
"""
if self._width_cache is None:
self._width_cache = self.apply(lambda df: len(df.columns))._data
if isinstance(self._width_cache, Future) and materialize:
self._width_cache = self.execution_wrapper.materialize(self._width_cache)
return self._width_cache
def ip(self, materialize=True):
"""
Get the node IP address of the object wrapped by this partition.
Parameters
----------
materialize : bool, default: True
Whether to forcibly materialize the result into an integer. If ``False``
was specified, may return a future of the result if it hasn't been
materialized yet.
Returns
-------
str
IP address of the node that holds the data.
"""
if self._ip_cache is None:
self._ip_cache = self.apply(lambda df: pandas.DataFrame([]))._ip_cache
if materialize and isinstance(self._ip_cache, Future):
self._ip_cache = self.execution_wrapper.materialize(self._ip_cache)
return self._ip_cache
def apply_func(partition, func, *args, **kwargs):
"""
Execute a function on the partition in a worker process.
Parameters
----------
partition : pandas.DataFrame
A pandas DataFrame the function needs to be executed on.
func : callable
The function to perform.
*args : list
Positional arguments to pass to ``func``.
**kwargs : dict
Keyword arguments to pass to ``func``.
Returns
-------
pandas.DataFrame
The resulting pandas DataFrame.
str
The node IP address of the worker process.
Notes
-----
Directly passing a call queue entry (i.e. a list of [func, args, kwargs]) instead of
destructuring it causes a performance penalty.
"""
result = func(partition, *args, **kwargs)
return result, get_ip()
def apply_list_of_funcs(call_queue, partition):
"""
Execute all operations stored in the call queue on the partition in a worker process.
Parameters
----------
call_queue : list
A call queue of ``[func, args, kwargs]`` triples that needs to be executed on the partition.
partition : pandas.DataFrame
A pandas DataFrame the call queue needs to be executed on.
Returns
-------
pandas.DataFrame
The resulting pandas DataFrame.
str
The node IP address of the worker process.
"""
for func, f_args, f_kwargs in call_queue:
partition = func(partition, *f_args, **f_kwargs)
return partition, get_ip()
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses class that implements ``PandasDataframePartitionManager``."""
from modin.core.dataframe.pandas.partitioning.partition_manager import (
PandasDataframePartitionManager,
)
from modin.core.execution.dask.common import DaskWrapper
from .partition import PandasOnDaskDataframePartition
from .virtual_partition import (
PandasOnDaskDataframeColumnPartition,
PandasOnDaskDataframeRowPartition,
)
class PandasOnDaskDataframePartitionManager(PandasDataframePartitionManager):
"""The class implements the interface in `PandasDataframePartitionManager`."""
# This object uses PandasOnDaskDataframePartition objects as the underlying store.
_partition_class = PandasOnDaskDataframePartition
_column_partitions_class = PandasOnDaskDataframeColumnPartition
_row_partition_class = PandasOnDaskDataframeRowPartition
_execution_wrapper = DaskWrapper
@classmethod
def wait_partitions(cls, partitions):
"""
Wait on the objects wrapped by `partitions` in parallel, without materializing them.
This method will block until all computations in the list have completed.
Parameters
----------
partitions : np.ndarray
NumPy array with ``PandasDataframePartition``-s.
"""
cls._execution_wrapper.wait(
[block for partition in partitions for block in partition.list_of_blocks]
)
================================================
FILE: modin/core/execution/dask/implementations/pandas_on_dask/partitioning/virtual_partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses classes responsible for storing a virtual partition and applying a function to it."""
import pandas
from distributed.utils import get_ip
from modin.core.dataframe.pandas.partitioning.axis_partition import (
PandasDataframeAxisPartition,
)
from modin.core.execution.dask.common import DaskWrapper
from modin.utils import _inherit_docstrings
from .partition import PandasOnDaskDataframePartition
class PandasOnDaskDataframeVirtualPartition(PandasDataframeAxisPartition):
"""
The class implements the interface in ``PandasDataframeAxisPartition``.
Parameters
----------
list_of_partitions : Union[list, PandasOnDaskDataframePartition]
List of ``PandasOnDaskDataframePartition`` and
``PandasOnDaskDataframeVirtualPartition`` objects, or a single
``PandasOnDaskDataframePartition``.
get_ip : bool, default: False
Whether to get node IP addresses of conforming partitions or not.
full_axis : bool, default: True
Whether or not the virtual partition encompasses the whole axis.
call_queue : list, optional
A list of tuples (callable, args, kwargs) that contains deferred calls.
length : distributed.Future or int, optional
Length, or reference to length, of wrapped ``pandas.DataFrame``.
width : distributed.Future or int, optional
Width, or reference to width, of wrapped ``pandas.DataFrame``.
"""
axis = None
_PARTITIONS_METADATA_LEN = 3 # (length, width, ip)
partition_type = PandasOnDaskDataframePartition
@property
def list_of_ips(self):
"""
Get the IPs holding the physical objects composing this partition.
Returns
-------
List
A list of IPs as ``distributed.Future`` or str.
"""
# Defer draining call queue until we get the ip address
result = [None] * len(self.list_of_block_partitions)
for idx, partition in enumerate(self.list_of_block_partitions):
partition.drain_call_queue()
result[idx] = partition.ip(materialize=False)
return result
@classmethod
@_inherit_docstrings(PandasDataframeAxisPartition.deploy_splitting_func)
def deploy_splitting_func(
cls,
axis,
func,
f_args,
f_kwargs,
num_splits,
*partitions,
extract_metadata=False,
):
return DaskWrapper.deploy(
func=_deploy_dask_func,
f_args=(
PandasDataframeAxisPartition.deploy_splitting_func,
axis,
func,
f_args,
f_kwargs,
num_splits,
*partitions,
),
f_kwargs={"extract_metadata": extract_metadata},
num_returns=(
num_splits * (1 + cls._PARTITIONS_METADATA_LEN)
if extract_metadata
else num_splits
),
pure=False,
)
@classmethod
def deploy_axis_func(
cls,
axis,
func,
f_args,
f_kwargs,
num_splits,
maintain_partitioning,
*partitions,
min_block_size,
lengths=None,
manual_partition=False,
):
"""
Deploy a function along a full axis.
Parameters
----------
axis : {0, 1}
The axis to perform the function along.
func : callable
The function to perform.
f_args : list or tuple
Positional arguments to pass to ``func``.
f_kwargs : dict
Keyword arguments to pass to ``func``.
num_splits : int
The number of splits to return (see `split_result_of_axis_func_pandas`).
maintain_partitioning : bool
If True, keep the old partitioning if possible.
If False, create a new partition layout.
*partitions : iterable
All partitions that make up the full axis (row or column).
min_block_size : int
Minimum number of rows/columns in a single split.
lengths : iterable, default: None
The list of lengths to shuffle the partition into.
manual_partition : bool, default: False
If True, partition the result with `lengths`.
Returns
-------
list
A list of distributed.Future.
"""
result_num_splits = len(lengths) if lengths else num_splits
return DaskWrapper.deploy(
func=_deploy_dask_func,
f_args=(
PandasDataframeAxisPartition.deploy_axis_func,
axis,
func,
f_args,
f_kwargs,
num_splits,
maintain_partitioning,
*partitions,
),
f_kwargs={
"min_block_size": min_block_size,
"lengths": lengths,
"manual_partition": manual_partition,
},
num_returns=result_num_splits * (1 + cls._PARTITIONS_METADATA_LEN),
pure=False,
)
@classmethod
def deploy_func_between_two_axis_partitions(
cls,
axis,
func,
f_args,
f_kwargs,
num_splits,
len_of_left,
other_shape,
*partitions,
min_block_size,
):
"""
Deploy a function along a full axis between two data sets.
Parameters
----------
axis : {0, 1}
The axis to perform the function along.
func : callable
The function to perform.
f_args : list or tuple
Positional arguments to pass to ``func``.
f_kwargs : dict
Keyword arguments to pass to ``func``.
num_splits : int
The number of splits to return (see `split_result_of_axis_func_pandas`).
len_of_left : int
The number of values in `partitions` that belong to the left data set.
other_shape : np.ndarray
The shape of right frame in terms of partitions, i.e.
(other_shape[i-1], other_shape[i]) will indicate slice to restore i-1 axis partition.
*partitions : iterable
All partitions that make up the full axis (row or column) for both data sets.
min_block_size : int
Minimum number of rows/columns in a single split.
Returns
-------
list
A list of distributed.Future.
"""
return DaskWrapper.deploy(
func=_deploy_dask_func,
f_args=(
PandasDataframeAxisPartition.deploy_func_between_two_axis_partitions,
axis,
func,
f_args,
f_kwargs,
num_splits,
len_of_left,
other_shape,
*partitions,
),
f_kwargs={
"min_block_size": min_block_size,
},
num_returns=num_splits * (1 + cls._PARTITIONS_METADATA_LEN),
pure=False,
)
def wait(self):
"""Wait completing computations on the object wrapped by the partition."""
self.drain_call_queue()
DaskWrapper.wait(self.list_of_blocks)
@_inherit_docstrings(PandasOnDaskDataframeVirtualPartition)
class PandasOnDaskDataframeColumnPartition(PandasOnDaskDataframeVirtualPartition):
axis = 0
@_inherit_docstrings(PandasOnDaskDataframeVirtualPartition)
class PandasOnDaskDataframeRowPartition(PandasOnDaskDataframeVirtualPartition):
axis = 1
def _deploy_dask_func(
deployer,
axis,
f_to_deploy,
f_args,
f_kwargs,
*args,
extract_metadata=True,
**kwargs,
):
"""
Execute a function on an axis partition in a worker process.
This is ALWAYS called on either ``PandasDataframeAxisPartition.deploy_axis_func``
or ``PandasDataframeAxisPartition.deploy_func_between_two_axis_partitions``, which both
serve to deploy another dataframe function on a Dask worker process.
Parameters
----------
deployer : callable
A `PandasDataFrameAxisPartition.deploy_*` method that will call `deploy_f`.
axis : {0, 1}
The axis to perform the function along.
f_to_deploy : callable or RayObjectID
The function to deploy.
f_args : list or tuple
Positional arguments to pass to ``f_to_deploy``.
f_kwargs : dict
Keyword arguments to pass to ``f_to_deploy``.
*args : list
Positional arguments to pass to ``func``.
extract_metadata : bool, default: True
Whether to return metadata (length, width, ip) of the result. Passing `False` may relax
the load on object storage as the remote function would return 4 times fewer futures.
Passing `False` makes sense for temporary results where you know for sure that the
metadata will never be requested.
**kwargs : dict
Keyword arguments to pass to ``func``.
Returns
-------
list
The result of the function ``func`` and metadata for it.
"""
result = deployer(axis, f_to_deploy, f_args, f_kwargs, *args, **kwargs)
if not extract_metadata:
return result
ip = get_ip()
if isinstance(result, pandas.DataFrame):
return result, len(result), len(result.columns), ip
elif all(isinstance(r, pandas.DataFrame) for r in result):
return [i for r in result for i in [r, len(r), len(r.columns), ip]]
else:
return [i for r in result for i in [r, None, None, ip]]
================================================
FILE: modin/core/execution/dispatching/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to dispatching to specific execution."""
================================================
FILE: modin/core/execution/dispatching/factories/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Factories responsible for dispatching to specific execution."""
================================================
FILE: modin/core/execution/dispatching/factories/dispatcher.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Contain IO dispatcher class.
Dispatcher routes the work to execution-specific functions.
"""
from typing import Union
from pandas._libs.lib import NoDefault, no_default
from modin.config import Backend, Engine, IsExperimental, StorageFormat
from modin.core.execution.dispatching.factories import factories
from modin.core.storage_formats.base import BaseQueryCompiler
from modin.utils import _inherit_docstrings
class FactoryNotFoundError(AttributeError):
"""
``FactoryNotFound`` exception class.
Raise when no matching factory could be found.
"""
pass
class StubIoEngine(object):
"""
IO-Engine that does nothing more than raise NotImplementedError when any method is called.
Parameters
----------
factory_name : str
Factory name, which will be reflected in error messages.
Notes
-----
Used for testing purposes.
"""
def __init__(self, factory_name=""):
self.factory_name = factory_name or "Unknown"
def __getattr__(self, name):
"""
Return a function that raises `NotImplementedError` for the `name` method.
Parameters
----------
name : str
Method name to indicate in `NotImplementedError`.
Returns
-------
callable
"""
def stub(*args, **kw):
raise NotImplementedError(
f"Method {self.factory_name}.{name} is not implemented"
)
return stub
class StubFactory(factories.BaseFactory):
"""
Factory that does nothing more than raise NotImplementedError when any method is called.
Notes
-----
Used for testing purposes.
"""
io_cls = StubIoEngine()
@classmethod
def set_failing_name(cls, factory_name):
"""
Fill in `.io_cls` class attribute with ``StubIoEngine`` engine.
Parameters
----------
factory_name : str
Name to pass to the ``StubIoEngine`` constructor.
"""
cls.io_cls = StubIoEngine(factory_name)
return cls
class FactoryDispatcher(object):
"""
Class that routes IO-work to the factories.
This class is responsible for keeping selected factory up-to-date and dispatching
calls of IO-functions to its actual execution-specific implementations.
"""
__factory: factories.BaseFactory = None
@classmethod
def get_factory(cls) -> factories.BaseFactory:
"""Get current factory."""
if cls.__factory is None:
from modin.pandas import _initialize_engine
Engine.subscribe(
lambda engine_parameter: _initialize_engine(engine_parameter.get())
)
Backend.subscribe(cls._update_factory)
return_value = cls.__factory
return return_value
@classmethod
def _get_prepared_factory_for_backend(cls, backend) -> factories.BaseFactory:
"""
Get factory for the specified backend.
Parameters
----------
backend : str
Backend name.
Returns
-------
factories.BaseFactory
Factory for the specified backend.
"""
execution = Backend.get_execution_for_backend(backend)
from modin.pandas import _initialize_engine
_initialize_engine(execution.engine)
factory_name = f"{execution.storage_format}On{execution.engine}Factory"
experimental_factory_name = "Experimental" + factory_name
try:
factory = getattr(factories, factory_name, None) or getattr(
factories, experimental_factory_name
)
except AttributeError:
if not IsExperimental.get():
# allow missing factories in experimental mode only
msg = (
"Cannot find neither factory {} nor experimental factory {}. "
+ "Potential reason might be incorrect environment variable value for "
+ f"{StorageFormat.varname} or {Engine.varname}"
)
raise FactoryNotFoundError(
msg.format(factory_name, experimental_factory_name)
)
factory = StubFactory.set_failing_name(factory_name)
else:
try:
factory.prepare()
except ModuleNotFoundError as err:
raise ModuleNotFoundError(
f"Make sure all required packages are installed: {str(err)}"
) from err
return factory
@classmethod
def _update_factory(cls, *args):
"""
Update and prepare factory with a new one specified via Modin config.
Parameters
----------
*args : iterable
This parameters serves the compatibility purpose.
Does not affect the result.
"""
cls.__factory = cls._get_prepared_factory_for_backend(Backend.get())
@classmethod
def from_pandas(
cls, df, backend: Union[str, NoDefault] = no_default
) -> BaseQueryCompiler:
"""
Create a Modin query compiler from a pandas DataFrame.
Parameters
----------
df : pandas.DataFrame
The pandas DataFrame to convert.
backend : str or NoDefault, default: NoDefault
The backend to use for the resulting query compiler. If NoDefault,
use the current global default ``Backend`` from the Modin config.
Returns
-------
BaseQueryCompiler
A Modin query compiler that wraps the input pandas DataFrame.
"""
return (
cls.get_factory()
if backend is no_default
else cls._get_prepared_factory_for_backend(backend)
)._from_pandas(df)
@classmethod
@_inherit_docstrings(factories.BaseFactory._from_arrow)
def from_arrow(cls, at):
return cls.get_factory()._from_arrow(at)
@classmethod
@_inherit_docstrings(factories.BaseFactory._from_non_pandas)
def from_non_pandas(cls, *args, **kwargs):
return cls.get_factory()._from_non_pandas(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._from_interchange_dataframe)
def from_interchange_dataframe(cls, *args, **kwargs):
return cls.get_factory()._from_interchange_dataframe(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._from_ray)
def from_ray(cls, ray_obj):
return cls.get_factory()._from_ray(ray_obj)
@classmethod
@_inherit_docstrings(factories.BaseFactory._from_dask)
def from_dask(cls, dask_obj):
return cls.get_factory()._from_dask(dask_obj)
@classmethod
@_inherit_docstrings(factories.BaseFactory._from_map)
def from_map(cls, func, iterable, *args, **kwargs):
return cls.get_factory()._from_map(func, iterable, *args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_parquet)
def read_parquet(cls, **kwargs):
return cls.get_factory()._read_parquet(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_csv)
def read_csv(cls, **kwargs):
return cls.get_factory()._read_csv(**kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_csv_glob)
def read_csv_glob(cls, **kwargs):
return cls.get_factory()._read_csv_glob(**kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_pickle_glob)
def read_pickle_glob(cls, **kwargs):
return cls.get_factory()._read_pickle_glob(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_json)
def read_json(cls, **kwargs):
return cls.get_factory()._read_json(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_gbq)
def read_gbq(cls, **kwargs):
return cls.get_factory()._read_gbq(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_html)
def read_html(cls, **kwargs):
return cls.get_factory()._read_html(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_clipboard)
def read_clipboard(cls, **kwargs):
return cls.get_factory()._read_clipboard(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_excel)
def read_excel(cls, **kwargs):
return cls.get_factory()._read_excel(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_hdf)
def read_hdf(cls, **kwargs):
return cls.get_factory()._read_hdf(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_feather)
def read_feather(cls, **kwargs):
return cls.get_factory()._read_feather(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_stata)
def read_stata(cls, **kwargs):
return cls.get_factory()._read_stata(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_sas)
def read_sas(cls, **kwargs): # pragma: no cover
return cls.get_factory()._read_sas(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_pickle)
def read_pickle(cls, **kwargs):
return cls.get_factory()._read_pickle(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_sql)
def read_sql(cls, **kwargs):
return cls.get_factory()._read_sql(**kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_sql_distributed)
def read_sql_distributed(cls, **kwargs):
return cls.get_factory()._read_sql_distributed(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_fwf)
def read_fwf(cls, **kwargs):
return cls.get_factory()._read_fwf(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_sql_table)
def read_sql_table(cls, **kwargs):
return cls.get_factory()._read_sql_table(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_sql_query)
def read_sql_query(cls, **kwargs):
return cls.get_factory()._read_sql_query(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._read_spss)
def read_spss(cls, **kwargs):
return cls.get_factory()._read_spss(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_sql)
def to_sql(cls, *args, **kwargs):
return cls.get_factory()._to_sql(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_pickle)
def to_pickle(cls, *args, **kwargs):
return cls.get_factory()._to_pickle(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._to_pickle_glob)
def to_pickle_glob(cls, *args, **kwargs):
return cls.get_factory()._to_pickle_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_parquet_glob)
def read_parquet_glob(cls, *args, **kwargs):
return cls.get_factory()._read_parquet_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._to_parquet_glob)
def to_parquet_glob(cls, *args, **kwargs):
return cls.get_factory()._to_parquet_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_json_glob)
def read_json_glob(cls, *args, **kwargs):
return cls.get_factory()._read_json_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._to_json_glob)
def to_json_glob(cls, *args, **kwargs):
return cls.get_factory()._to_json_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_xml_glob)
def read_xml_glob(cls, *args, **kwargs):
return cls.get_factory()._read_xml_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._to_xml_glob)
def to_xml_glob(cls, *args, **kwargs):
return cls.get_factory()._to_xml_glob(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.PandasOnRayFactory._read_custom_text)
def read_custom_text(cls, **kwargs):
return cls.get_factory()._read_custom_text(**kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_csv)
def to_csv(cls, *args, **kwargs):
return cls.get_factory()._to_csv(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_json)
def to_json(cls, *args, **kwargs):
return cls.get_factory()._to_json(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_json)
def to_json_series(cls, *args, **kwargs):
return cls.get_factory()._to_json_series(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_xml)
def to_xml(cls, *args, **kwargs):
return cls.get_factory()._to_xml(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_parquet)
def to_parquet(cls, *args, **kwargs):
return cls.get_factory()._to_parquet(*args, **kwargs)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_ray)
def to_ray(cls, modin_obj):
return cls.get_factory()._to_ray(modin_obj)
@classmethod
@_inherit_docstrings(factories.BaseFactory._to_dask)
def to_dask(cls, modin_obj):
return cls.get_factory()._to_dask(modin_obj)
================================================
FILE: modin/core/execution/dispatching/factories/factories.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module contains Factories for all of the supported Modin executions.
Factory is a bridge between calls of IO function from high-level API and its
actual implementation in the execution, bound to that factory. Each execution is represented
with a Factory class.
"""
import re
import typing
import warnings
import pandas
from pandas.util._decorators import doc
from modin.core.io import BaseIO
from modin.core.storage_formats.pandas.native_query_compiler import NativeQueryCompiler
from modin.utils import get_current_execution
_doc_abstract_factory_class = """
Abstract {role} factory which allows to override the IO module easily.
This class is responsible for dispatching calls of IO-functions to its
actual execution-specific implementations.
Attributes
----------
io_cls : BaseIO
IO module class of the underlying execution. The place to dispatch calls to.
"""
_doc_factory_class = """
Factory of {execution_name} execution.
This class is responsible for dispatching calls of IO-functions to its
actual execution-specific implementations.
Attributes
----------
io_cls : {execution_name}IO
IO module class of the underlying execution. The place to dispatch calls to.
"""
_doc_factory_prepare_method = """
Initialize Factory.
Fills in `.io_cls` class attribute with {io_module_name} lazily.
"""
_doc_io_method_raw_template = """
Build query compiler from {source}.
Parameters
----------
{params}
Returns
-------
QueryCompiler
Query compiler of the selected storage format.
"""
_doc_io_method_template = (
_doc_io_method_raw_template
+ """
See Also
--------
modin.pandas.{method}
"""
)
_doc_io_method_all_params = """*args : args
Arguments to pass to the QueryCompiler builder method.
**kwargs : kwargs
Arguments to pass to the QueryCompiler builder method."""
_doc_io_method_kwargs_params = """**kwargs : kwargs
Arguments to pass to the QueryCompiler builder method."""
types_dictionary = {"pandas": {"category": pandas.CategoricalDtype}}
supported_executions = (
"PandasOnRay",
"PandasOnUnidist",
"PandasOnDask",
)
class FactoryInfo(typing.NamedTuple):
"""
Structure that stores information about factory.
Parameters
----------
engine : str
Name of underlying execution engine.
partition : str
Name of the partition format.
experimental : bool
Whether underlying engine is experimental-only.
"""
engine: str
partition: str
experimental: bool
class NotRealFactory(Exception):
"""
``NotRealFactory`` exception class.
Raise when no matching factory could be found.
"""
pass
@doc(_doc_abstract_factory_class, role="")
class BaseFactory(object):
io_cls: typing.Type[BaseIO] = None # The module where the I/O functionality exists.
@classmethod
def get_info(cls) -> FactoryInfo:
"""
Get information about current factory.
Notes
-----
It parses factory name, so it must be conformant with how ``FactoryDispatcher``
class constructs factory names.
"""
try:
experimental, partition, engine = re.match(
r"^(Experimental)?(.*)On(.*)Factory$", cls.__name__
).groups()
except AttributeError:
raise NotRealFactory()
return FactoryInfo(
engine=engine, partition=partition, experimental=bool(experimental)
)
@classmethod
@doc(
_doc_factory_prepare_method,
io_module_name="an underlying execution's IO-module",
)
def prepare(cls):
raise NotImplementedError("Subclasses of BaseFactory must implement prepare")
@classmethod
@doc(
_doc_io_method_template,
source="pandas DataFrame",
params="df : pandas.DataFrame",
method="io.from_pandas",
)
def _from_pandas(cls, df):
return cls.io_cls.from_pandas(df)
@classmethod
@doc(
_doc_io_method_template,
source="Arrow Table",
params="at : pyarrow.Table",
method="io.from_arrow",
)
def _from_arrow(cls, at):
return cls.io_cls.from_arrow(at)
@classmethod
@doc(
_doc_io_method_template,
source="a non-pandas object (dict, list, np.array etc...)",
params=_doc_io_method_all_params,
method="io.from_non_pandas",
)
def _from_non_pandas(cls, *args, **kwargs):
return cls.io_cls.from_non_pandas(*args, **kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a DataFrame object supporting exchange protocol `__dataframe__()`",
params=_doc_io_method_all_params,
method="io.from_interchange_dataframe",
)
def _from_interchange_dataframe(cls, *args, **kwargs):
return cls.io_cls.from_interchange_dataframe(*args, **kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a Ray Dataset",
params="ray_obj : ray.data.Dataset",
method="modin.core.execution.ray.implementations.pandas_on_ray.io.PandasOnRayIO.from_ray",
)
def _from_ray(cls, ray_obj):
return cls.io_cls.from_ray(ray_obj)
@classmethod
@doc(
_doc_io_method_template,
source="a Dask DataFrame",
params="dask_obj : dask.dataframe.DataFrame",
method="modin.core.execution.dask.implementations.pandas_on_dask.io.PandasOnDaskIO.from_dask",
)
def _from_dask(cls, dask_obj):
return cls.io_cls.from_dask(dask_obj)
@classmethod
def _from_map(cls, func, iterable, *args, **kwargs):
"""
Create a Modin `query_compiler` from a map function.
This method will construct a Modin `query_compiler` split by row partitions.
The number of row partitions matches the number of elements in the iterable object.
Parameters
----------
func : callable
Function to map across the iterable object.
iterable : Iterable
An iterable object.
*args : tuple
Positional arguments to pass in `func`.
**kwargs : dict
Keyword arguments to pass in `func`.
Returns
-------
BaseQueryCompiler
QueryCompiler containing data returned by map function.
"""
return cls.io_cls.from_map(func, iterable, *args, **kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a Parquet file",
params=_doc_io_method_kwargs_params,
method="read_parquet",
)
def _read_parquet(cls, **kwargs):
return cls.io_cls.read_parquet(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a CSV file",
params=_doc_io_method_kwargs_params,
method="read_csv",
)
def _read_csv(cls, **kwargs):
return cls.io_cls.read_csv(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a JSON file",
params=_doc_io_method_kwargs_params,
method="read_json",
)
def _read_json(cls, **kwargs):
return cls.io_cls.read_json(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a Google BigQuery",
params=_doc_io_method_kwargs_params,
method="read_gbq",
)
def _read_gbq(cls, **kwargs):
return cls.io_cls.read_gbq(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="an HTML document",
params=_doc_io_method_kwargs_params,
method="read_html",
)
def _read_html(cls, **kwargs):
return cls.io_cls.read_html(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="clipboard",
params=_doc_io_method_kwargs_params,
method="read_clipboard",
)
def _read_clipboard(cls, **kwargs): # pragma: no cover
return cls.io_cls.read_clipboard(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="an Excel file",
params=_doc_io_method_kwargs_params,
method="read_excel",
)
def _read_excel(cls, **kwargs):
return cls.io_cls.read_excel(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="an HDFStore",
params=_doc_io_method_kwargs_params,
method="read_hdf",
)
def _read_hdf(cls, **kwargs):
return cls.io_cls.read_hdf(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a feather-format object",
params=_doc_io_method_kwargs_params,
method="read_feather",
)
def _read_feather(cls, **kwargs):
return cls.io_cls.read_feather(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a Stata file",
params=_doc_io_method_kwargs_params,
method="read_stata",
)
def _read_stata(cls, **kwargs):
return cls.io_cls.read_stata(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a SAS file",
params=_doc_io_method_kwargs_params,
method="read_sas",
)
def _read_sas(cls, **kwargs): # pragma: no cover
return cls.io_cls.read_sas(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a pickled Modin or pandas DataFrame",
params=_doc_io_method_kwargs_params,
method="read_pickle",
)
def _read_pickle(cls, **kwargs):
return cls.io_cls.read_pickle(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a SQL query or database table",
params=_doc_io_method_kwargs_params,
method="read_sql",
)
def _read_sql(cls, **kwargs):
return cls.io_cls.read_sql(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a table of fixed-width formatted lines",
params=_doc_io_method_kwargs_params,
method="read_fwf",
)
def _read_fwf(cls, **kwargs):
return cls.io_cls.read_fwf(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a SQL database table",
params=_doc_io_method_kwargs_params,
method="read_sql_table",
)
def _read_sql_table(cls, **kwargs):
return cls.io_cls.read_sql_table(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="a SQL query",
params=_doc_io_method_kwargs_params,
method="read_sql_query",
)
def _read_sql_query(cls, **kwargs):
return cls.io_cls.read_sql_query(**kwargs)
@classmethod
@doc(
_doc_io_method_template,
source="an SPSS file",
params=_doc_io_method_kwargs_params,
method="read_spss",
)
def _read_spss(cls, **kwargs):
return cls.io_cls.read_spss(**kwargs)
@classmethod
def _to_sql(cls, *args, **kwargs):
"""
Write query compiler content to a SQL database.
Parameters
----------
*args : args
Arguments to the writer method.
**kwargs : kwargs
Arguments to the writer method.
"""
return cls.io_cls.to_sql(*args, **kwargs)
@classmethod
def _to_pickle(cls, *args, **kwargs):
"""
Pickle query compiler object.
Parameters
----------
*args : args
Arguments to the writer method.
**kwargs : kwargs
Arguments to the writer method.
"""
return cls.io_cls.to_pickle(*args, **kwargs)
@classmethod
def _to_csv(cls, *args, **kwargs):
"""
Write query compiler content to a CSV file.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
return cls.io_cls.to_csv(*args, **kwargs)
@classmethod
def _to_json(cls, *args, **kwargs):
"""
Write query compiler content to a JSON file.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
return cls.io_cls.to_json(*args, **kwargs)
@classmethod
def _to_json_series(cls, *args, **kwargs):
"""
Write query compiler content of a Series to a JSON file.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
return cls.io_cls.to_json_series(*args, **kwargs)
@classmethod
def _to_xml(cls, *args, **kwargs):
"""
Write query compiler content to a XML file.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
return cls.io_cls.to_xml(*args, **kwargs)
@classmethod
def _to_parquet(cls, *args, **kwargs):
"""
Write query compiler content to a parquet file.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
return cls.io_cls.to_parquet(*args, **kwargs)
@classmethod
def _to_ray(cls, modin_obj):
"""
Write query compiler content to a Ray Dataset.
Parameters
----------
modin_obj : modin.pandas.DataFrame, modin.pandas.Series
The Modin DataFrame/Series to write.
Returns
-------
ray.data.Dataset
A Ray Dataset object.
Notes
-----
Modin DataFrame/Series can only be converted to a Ray Dataset if Modin uses a Ray engine.
"""
return cls.io_cls.to_ray(modin_obj)
@classmethod
def _to_dask(cls, modin_obj):
"""
Write query compiler content to a Dask DataFrame/Series.
Parameters
----------
modin_obj : modin.pandas.DataFrame, modin.pandas.Series
The Modin DataFrame/Series to write.
Returns
-------
dask.dataframe.DataFrame or dask.dataframe.Series
A Dask DataFrame/Series object.
Notes
-----
Modin DataFrame/Series can only be converted to a Dask DataFrame/Series if Modin uses a Dask engine.
"""
return cls.io_cls.to_dask(modin_obj)
# experimental methods that don't exist in pandas
@classmethod
@doc(
_doc_io_method_raw_template,
source="CSV files",
params=_doc_io_method_kwargs_params,
)
def _read_csv_glob(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_read_csv_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.read_csv_glob(**kwargs)
@classmethod
@doc(
_doc_io_method_raw_template,
source="Pickle files",
params=_doc_io_method_kwargs_params,
)
def _read_pickle_glob(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_read_pickle_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.read_pickle_glob(**kwargs)
@classmethod
@doc(
_doc_io_method_raw_template,
source="SQL files",
params=_doc_io_method_kwargs_params,
)
def _read_sql_distributed(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
extra_parameters = (
"partition_column",
"lower_bound",
"upper_bound",
"max_sessions",
)
if any(
param in kwargs and kwargs[param] is not None
for param in extra_parameters
):
warnings.warn(
f"Distributed read_sql() was only implemented for {', '.join(supported_executions)} executions."
)
for param in extra_parameters:
del kwargs[param]
return cls.io_cls.read_sql(**kwargs)
return cls.io_cls.read_sql_distributed(**kwargs)
@classmethod
@doc(
_doc_io_method_raw_template,
source="Custom text files",
params=_doc_io_method_kwargs_params,
)
def _read_custom_text(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_read_custom_text()` is not implemented for {current_execution} execution."
)
return cls.io_cls.read_custom_text(**kwargs)
@classmethod
def _to_pickle_glob(cls, *args, **kwargs):
"""
Distributed pickle query compiler object.
Parameters
----------
*args : args
Arguments to the writer method.
**kwargs : kwargs
Arguments to the writer method.
"""
# TODO(https://github.com/modin-project/modin/issues/7429): Use
# frame-level execution instead of the global, default execution.
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_to_pickle_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.to_pickle_glob(*args, **kwargs)
@classmethod
@doc(
_doc_io_method_raw_template,
source="Parquet files",
params=_doc_io_method_kwargs_params,
)
def _read_parquet_glob(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_read_parquet_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.read_parquet_glob(**kwargs)
@classmethod
def _to_parquet_glob(cls, *args, **kwargs):
"""
Write query compiler content to several parquet files.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_to_parquet_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.to_parquet_glob(*args, **kwargs)
@classmethod
@doc(
_doc_io_method_raw_template,
source="Json files",
params=_doc_io_method_kwargs_params,
)
def _read_json_glob(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_read_json_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.read_json_glob(**kwargs)
@classmethod
def _to_json_glob(cls, *args, **kwargs):
"""
Write query compiler content to several json files.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_to_json_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.to_json_glob(*args, **kwargs)
@classmethod
@doc(
_doc_io_method_raw_template,
source="XML files",
params=_doc_io_method_kwargs_params,
)
def _read_xml_glob(cls, **kwargs):
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_read_xml_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.read_xml_glob(**kwargs)
@classmethod
def _to_xml_glob(cls, *args, **kwargs):
"""
Write query compiler content to several XML files.
Parameters
----------
*args : args
Arguments to pass to the writer method.
**kwargs : kwargs
Arguments to pass to the writer method.
"""
current_execution = get_current_execution()
if current_execution not in supported_executions:
raise NotImplementedError(
f"`_to_xml_glob()` is not implemented for {current_execution} execution."
)
return cls.io_cls.to_xml_glob(*args, **kwargs)
@doc(_doc_factory_class, execution_name="PandasOnRay")
class PandasOnRayFactory(BaseFactory):
@classmethod
@doc(_doc_factory_prepare_method, io_module_name="``PandasOnRayIO``")
def prepare(cls):
from modin.core.execution.ray.implementations.pandas_on_ray.io import (
PandasOnRayIO,
)
cls.io_cls = PandasOnRayIO
@doc(_doc_factory_class, execution_name="PandasOnPython")
class PandasOnPythonFactory(BaseFactory):
@classmethod
@doc(_doc_factory_prepare_method, io_module_name="``PandasOnPythonIO``")
def prepare(cls):
from modin.core.execution.python.implementations.pandas_on_python.io import (
PandasOnPythonIO,
)
cls.io_cls = PandasOnPythonIO
@doc(_doc_factory_class, execution_name="PandasOnDask")
class PandasOnDaskFactory(BaseFactory):
@classmethod
@doc(_doc_factory_prepare_method, io_module_name="``PandasOnDaskIO``")
def prepare(cls):
from modin.core.execution.dask.implementations.pandas_on_dask.io import (
PandasOnDaskIO,
)
cls.io_cls = PandasOnDaskIO
@doc(_doc_factory_class, execution_name="PandasOnUnidist")
class PandasOnUnidistFactory(BaseFactory):
@classmethod
@doc(_doc_factory_prepare_method, io_module_name="``PandasOnUnidistIO``")
def prepare(cls):
from modin.core.execution.unidist.implementations.pandas_on_unidist.io import (
PandasOnUnidistIO,
)
cls.io_cls = PandasOnUnidistIO
class NativeIO(BaseIO):
"""
I/O class for native pandas execution.
This class inherits the default function implementations from the
``BaseIO`` parent class.
"""
_should_warn_on_default_to_pandas: bool = False
query_compiler_cls = NativeQueryCompiler
@doc(_doc_factory_class, execution_name="NativeOnNative")
class NativeOnNativeFactory(BaseFactory):
@classmethod
@doc(_doc_factory_prepare_method, io_module_name="`NativeIO`")
def prepare(cls):
cls.io_cls = NativeIO
================================================
FILE: modin/core/execution/modin_aqp.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
The module for working with displaying progress bars for Modin execution engines.
Modin Automatic Query Progress (AQP).
"""
import inspect
import os
import threading
import time
import warnings
from modin.config import Engine, ProgressBar
progress_bars = {}
bar_lock = threading.Lock()
def call_progress_bar(result_parts, line_no):
"""
Attach a progress bar to given `result_parts`.
The progress bar is expected to be shown in a Jupyter Notebook cell.
Parameters
----------
result_parts : list of list of object refs (futures)
Objects which are being computed for which progress is requested.
line_no : int
Line number in the call stack which we're displaying progress for.
"""
with warnings.catch_warnings():
warnings.simplefilter("ignore")
try:
from tqdm.autonotebook import tqdm as tqdm_notebook
except ImportError:
raise ImportError("Please pip install tqdm to use the progress bar")
from IPython import get_ipython
try:
cell_no = get_ipython().execution_count
# This happens if we are not in ipython or jupyter.
# No progress bar is supported in that case.
except AttributeError:
return
pbar_id = f"{cell_no}-{line_no}"
futures = [
block
for row in result_parts
for partition in row
for block in partition.list_of_blocks
]
bar_format = (
"{l_bar}{bar}{r_bar}"
if "DEBUG_PROGRESS_BAR" in os.environ
and os.environ["DEBUG_PROGRESS_BAR"] == "True"
else "{desc}: {percentage:3.0f}%{bar} Elapsed time: {elapsed}, estimated remaining time: {remaining}"
)
bar_lock.acquire()
if pbar_id in progress_bars:
if hasattr(progress_bars[pbar_id], "container"):
if hasattr(progress_bars[pbar_id].container.children[0], "max"):
index = 0
else:
index = 1
progress_bars[pbar_id].container.children[index].max = progress_bars[
pbar_id
].container.children[index].max + len(futures)
progress_bars[pbar_id].total = progress_bars[pbar_id].total + len(futures)
progress_bars[pbar_id].refresh()
else:
progress_bars[pbar_id] = tqdm_notebook(
total=len(futures),
desc="Estimated completion of line " + str(line_no),
bar_format=bar_format,
)
bar_lock.release()
threading.Thread(target=_show_time_updates, args=(progress_bars[pbar_id],)).start()
# TODO(https://github.com/modin-project/modin/issues/7429): Use
# frame-level engine config.
modin_engine = Engine.get()
engine_wrapper = None
if modin_engine == "Ray":
from modin.core.execution.ray.common.engine_wrapper import RayWrapper
engine_wrapper = RayWrapper
elif modin_engine == "Unidist":
from modin.core.execution.unidist.common.engine_wrapper import UnidistWrapper
engine_wrapper = UnidistWrapper
else:
raise NotImplementedError(
f"ProgressBar feature is not supported for {modin_engine} engine."
)
for i in range(1, len(futures) + 1):
engine_wrapper.wait(futures, num_returns=i)
progress_bars[pbar_id].update(1)
progress_bars[pbar_id].refresh()
if progress_bars[pbar_id].n == progress_bars[pbar_id].total:
progress_bars[pbar_id].close()
def display_time_updates(bar):
"""
Start displaying the progress `bar` in a notebook.
Parameters
----------
bar : tqdm.tqdm
The progress bar wrapper to display in a notebook cell.
"""
threading.Thread(target=_show_time_updates, args=(bar,)).start()
def _show_time_updates(p_bar):
"""
Refresh displayed progress bar `p_bar` periodically until it is complete.
Parameters
----------
p_bar : tqdm.tqdm
The progress bar wrapper being displayed to refresh.
"""
while p_bar.total > p_bar.n:
time.sleep(1)
if p_bar.total > p_bar.n:
p_bar.refresh()
def progress_bar_wrapper(f):
"""
Wrap computation function inside a progress bar.
Spawns another thread which displays a progress bar showing
estimated completion time.
Parameters
----------
f : callable
The name of the function to be wrapped.
Returns
-------
callable
Decorated version of `f` which reports progress.
"""
from functools import wraps
@wraps(f)
def magic(*args, **kwargs):
result_parts = f(*args, **kwargs)
if ProgressBar.get():
current_frame = inspect.currentframe()
function_name = None
while function_name != "":
(
filename,
line_number,
function_name,
lines,
index,
) = inspect.getframeinfo(current_frame)
current_frame = current_frame.f_back
t = threading.Thread(
target=call_progress_bar,
args=(result_parts, line_number),
)
t.start()
# We need to know whether or not we are in a jupyter notebook
from IPython import get_ipython
try:
ipy_str = str(type(get_ipython()))
if "zmqshell" not in ipy_str:
t.join()
except Exception:
pass
return result_parts
return magic
================================================
FILE: modin/core/execution/python/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Python execution engine."""
================================================
FILE: modin/core/execution/python/common/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Python execution engine."""
from .engine_wrapper import PythonWrapper
__all__ = ["PythonWrapper"]
================================================
FILE: modin/core/execution/python/common/engine_wrapper.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Python execution engine."""
class PythonWrapper:
"""Python engine wrapper serving for the compatibility purpose with other engines."""
@classmethod
def deploy(cls, func, f_args=None, f_kwargs=None, num_returns=1):
"""
Run the passed function.
Parameters
----------
func : callable
f_args : sequence, optional
Positional arguments to pass to the `func`.
f_kwargs : dict, optional
Keyword arguments to pass to the `func`.
num_returns : int, default: 1
Number of return values from the `func`.
Returns
-------
object
Returns the result of the `func`.
"""
args = [] if f_args is None else f_args
kwargs = {} if f_kwargs is None else f_kwargs
return func(*args, **kwargs)
@classmethod
def is_future(cls, item):
"""
Check if the item is a Future.
Parameters
----------
item : object
Returns
-------
boolean
Always return false.
"""
return False
@classmethod
def materialize(cls, obj_id):
"""
Get the data from the data storage.
The method only serves for the compatibility purpose, what it actually
does is just return the passed value as is.
Parameters
----------
obj_id : object
Returns
-------
object
The passed `obj_id` itself.
"""
return obj_id
@classmethod
def put(cls, data, **kwargs):
"""
Put data into the data storage.
The method only serves for the compatibility purpose, what it actually
does is just return the passed value as is.
Parameters
----------
data : object
**kwargs : dict
Returns
-------
object
The passed `data` itself.
"""
return data
================================================
FILE: modin/core/execution/python/implementations/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Python execution engine and optimized for specific storage formats."""
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Python execution engine and optimized for pandas storage format."""
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/dataframe/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe class optimized for pandas on Python execution."""
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/dataframe/dataframe.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
Module contains class ``PandasOnPythonDataframe``.
``PandasOnPythonDataframe`` is dataframe class with pandas storage format and Python engine.
"""
from modin.core.dataframe.pandas.dataframe.dataframe import PandasDataframe
from modin.utils import _inherit_docstrings
from ..partitioning.partition_manager import PandasOnPythonDataframePartitionManager
class PandasOnPythonDataframe(PandasDataframe):
"""
Class for dataframes with pandas storage format and Python engine.
``PandasOnPythonDataframe`` doesn't implement any specific interfaces,
all functionality is inherited from the ``PandasDataframe`` class.
Parameters
----------
partitions : np.ndarray
A 2D NumPy array of partitions.
index : sequence
The index for the dataframe. Converted to a ``pandas.Index``.
columns : sequence
The columns object for the dataframe. Converted to a ``pandas.Index``.
row_lengths : list, optional
The length of each partition in the rows. The "height" of
each of the block partitions. Is computed if not provided.
column_widths : list, optional
The width of each partition in the columns. The "width" of
each of the block partitions. Is computed if not provided.
dtypes : pandas.Series, optional
The data types for the dataframe columns.
pandas_backend : {"pyarrow", None}, optional
Backend used by pandas. None - means default NumPy backend.
"""
_partition_mgr_cls = PandasOnPythonDataframePartitionManager
@property
@_inherit_docstrings(PandasDataframe.engine)
def engine(self) -> str:
return "Python"
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/io/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base IO classes optimized for pandas on Python execution."""
from .io import PandasOnPythonIO
__all__ = [
"PandasOnPythonIO",
]
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/io/io.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module for housing IO classes with pandas storage format and Python engine."""
from modin.core.execution.python.implementations.pandas_on_python.dataframe.dataframe import (
PandasOnPythonDataframe,
)
from modin.core.io import BaseIO
from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
class PandasOnPythonIO(BaseIO):
"""
Class for storing IO functions operating on pandas storage format and Python engine.
Inherits default function implementations from ``BaseIO`` parent class.
"""
frame_cls = PandasOnPythonDataframe
query_compiler_cls = PandasQueryCompiler
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/partitioning/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Base Modin Dataframe classes related to its partitioning and optimized for pandas on Python execution."""
from .partition import PandasOnPythonDataframePartition
from .partition_manager import PandasOnPythonDataframePartitionManager
from .virtual_partition import (
PandasOnPythonDataframeAxisPartition,
PandasOnPythonDataframeColumnPartition,
PandasOnPythonDataframeRowPartition,
)
__all__ = [
"PandasOnPythonDataframePartition",
"PandasOnPythonDataframePartitionManager",
"PandasOnPythonDataframeAxisPartition",
"PandasOnPythonDataframeColumnPartition",
"PandasOnPythonDataframeRowPartition",
]
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/partitioning/partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""The module defines interface for a partition with pandas storage format and Python engine."""
import warnings
from modin.core.dataframe.pandas.partitioning.partition import PandasDataframePartition
from modin.core.execution.python.common import PythonWrapper
class PandasOnPythonDataframePartition(PandasDataframePartition):
"""
Partition class with interface for pandas storage format and Python engine.
Class holds the data and metadata for a single partition and implements
methods of parent abstract class ``PandasDataframePartition``.
Parameters
----------
data : pandas.DataFrame
``pandas.DataFrame`` that should be wrapped with this class.
length : int, optional
Length of `data` (number of rows in the input dataframe).
width : int, optional
Width of `data` (number of columns in the input dataframe).
call_queue : list, optional
Call queue of the partition (list with entities that should be called
before partition materialization).
Notes
-----
Objects of this class are treated as immutable by partition manager
subclasses. There is no logic for updating in-place.
"""
execution_wrapper = PythonWrapper
def __init__(self, data, length=None, width=None, call_queue=None):
super().__init__()
if hasattr(data, "copy"):
data = data.copy()
self._data = data
if call_queue is None:
call_queue = []
self.call_queue = call_queue
self._length_cache = length
self._width_cache = width
def get(self):
"""
Flush the `call_queue` and return copy of the data.
Returns
-------
pandas.DataFrame
Copy of DataFrame that was wrapped by this partition.
Notes
-----
Since this object is a simple wrapper, just return the copy of data.
"""
self.drain_call_queue()
return self._data.copy() if hasattr(self._data, "copy") else self._data
def apply(self, func, *args, **kwargs):
"""
Apply a function to the object wrapped by this partition.
Parameters
----------
func : callable
Function to apply.
*args : iterable
Additional positional arguments to be passed in `func`.
**kwargs : dict
Additional keyword arguments to be passed in `func`.
Returns
-------
PandasOnPythonDataframePartition
New ``PandasOnPythonDataframePartition`` object.
"""
def call_queue_closure(data, call_queue):
"""
Apply callables from `call_queue` on copy of the `data` and return the result.
Parameters
----------
data : pandas.DataFrame or pandas.Series
Data to use for computations.
call_queue : array-like
Array with callables and it's kwargs to be applied to the `data`.
Returns
-------
pandas.DataFrame or pandas.Series
"""
result = data.copy()
for func, f_args, f_kwargs in call_queue:
try:
result = func(result, *f_args, **f_kwargs)
except Exception as err:
self.call_queue = []
raise err
return result
self._data = call_queue_closure(self._data, self.call_queue)
self.call_queue = []
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=FutureWarning)
return self.__constructor__(func(self._data.copy(), *args, **kwargs))
def drain_call_queue(self):
"""Execute all operations stored in the call queue on the object wrapped by this partition."""
if len(self.call_queue) == 0:
return
self.apply(lambda x: x)
def wait(self):
"""
Wait for completion of computations on the object wrapped by the partition.
Internally will be done by flushing the call queue.
"""
self.drain_call_queue()
@classmethod
def put(cls, obj):
"""
Create partition containing `obj`.
Parameters
----------
obj : pandas.DataFrame
DataFrame to be put into the new partition.
Returns
-------
PandasOnPythonDataframePartition
New ``PandasOnPythonDataframePartition`` object.
"""
return cls(obj.copy(), len(obj.index), len(obj.columns))
@classmethod
def preprocess_func(cls, func):
"""
Preprocess a function before an ``apply`` call.
Parameters
----------
func : callable
Function to preprocess.
Returns
-------
callable
An object that can be accepted by ``apply``.
Notes
-----
No special preprocessing action is required, so unmodified
`func` will be returned.
"""
return func
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/partitioning/partition_manager.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module houses class for managing partitions with pandas storage format and Python engine."""
from modin.core.dataframe.pandas.partitioning.partition_manager import (
PandasDataframePartitionManager,
)
from modin.core.execution.python.common import PythonWrapper
from .partition import PandasOnPythonDataframePartition
from .virtual_partition import (
PandasOnPythonDataframeColumnPartition,
PandasOnPythonDataframeRowPartition,
)
class PandasOnPythonDataframePartitionManager(PandasDataframePartitionManager):
"""
Class for managing partitions with pandas storage format and Python engine.
Inherits all functionality from ``PandasDataframePartitionManager`` base class.
"""
_partition_class = PandasOnPythonDataframePartition
_column_partitions_class = PandasOnPythonDataframeColumnPartition
_row_partition_class = PandasOnPythonDataframeRowPartition
_execution_wrapper = PythonWrapper
================================================
FILE: modin/core/execution/python/implementations/pandas_on_python/partitioning/virtual_partition.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""The module defines interface for a virtual partition with pandas storage format and python engine."""
from modin.core.dataframe.pandas.partitioning.axis_partition import (
PandasDataframeAxisPartition,
)
from modin.utils import _inherit_docstrings
from .partition import PandasOnPythonDataframePartition
class PandasOnPythonDataframeAxisPartition(PandasDataframeAxisPartition):
"""
Class defines axis partition interface with pandas storage format and Python engine.
Inherits functionality from ``PandasDataframeAxisPartition`` class.
Parameters
----------
list_of_partitions : Union[list, PandasOnPythonDataframePartition]
List of ``PandasOnPythonDataframePartition`` and
``PandasOnPythonDataframeVirtualPartition`` objects, or a single
``PandasOnPythonDataframePartition``.
get_ip : bool, default: False
Whether to get node IP addresses to conforming partitions or not.
full_axis : bool, default: True
Whether or not the virtual partition encompasses the whole axis.
call_queue : list, optional
A list of tuples (callable, args, kwargs) that contains deferred calls.
length : int, optional
Length, or reference to length, of wrapped ``pandas.DataFrame``.
width : int, optional
Width, or reference to width, of wrapped ``pandas.DataFrame``.
"""
partition_type = PandasOnPythonDataframePartition
@_inherit_docstrings(PandasOnPythonDataframeAxisPartition)
class PandasOnPythonDataframeColumnPartition(PandasOnPythonDataframeAxisPartition):
axis = 0
@_inherit_docstrings(PandasOnPythonDataframeAxisPartition)
class PandasOnPythonDataframeRowPartition(PandasOnPythonDataframeAxisPartition):
axis = 1
================================================
FILE: modin/core/execution/ray/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin's functionality related to Ray execution engine."""
================================================
FILE: modin/core/execution/ray/common/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Common utilities for Ray execution engine."""
from .engine_wrapper import MaterializationHook, RayWrapper, SignalActor
from .utils import initialize_ray
__all__ = [
"initialize_ray",
"RayWrapper",
"MaterializationHook",
"SignalActor",
]
================================================
FILE: modin/core/execution/ray/common/deferred_execution.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Module with classes and utilities for deferred remote execution in Ray workers."""
from enum import Enum
from itertools import islice
from typing import (
Any,
Callable,
Dict,
Generator,
Iterable,
List,
Optional,
Tuple,
Union,
)
import pandas
import ray
from ray._private.services import get_node_ip_address
from modin.config import RayTaskCustomResources
from modin.core.execution.ray.common import MaterializationHook, RayWrapper
from modin.logging import get_logger
ObjectRefType = Union[ray.ObjectRef, None]
ObjectRefOrListType = Union[ObjectRefType, List[ObjectRefType]]
ListOrTuple = (list, tuple)
class DeferredExecution:
"""
Deferred execution task.
This class represents a single node in the execution tree. The input is either
an object reference or another node on which this node depends.
The output is calculated by the specified Callable.
If the input is a DeferredExecution node, it is executed first and the execution
output is used as the input for this one. All the executions are performed in a
single batch (i.e. using a single remote call) and the results are saved in all
the nodes that have multiple subscribers.
Parameters
----------
data : ObjectRefType or DeferredExecution
The execution input.
func : callable or ObjectRefType
A function to be executed.
args : list or tuple
Additional positional arguments to be passed in `func`.
kwargs : dict
Additional keyword arguments to be passed in `func`.
num_returns : int, optional
The number of the return values.
Attributes
----------
data : ObjectRefType or DeferredExecution
The execution input.
func : callable or ObjectRefType
A function to be executed.
args : list or tuple
Additional positional arguments to be passed in `func`.
kwargs : dict
Additional keyword arguments to be passed in `func`.
num_returns : int
The number of the return values.
flat_args : bool
True means that there are no lists or DeferredExecution objects in `args`.
In this case, no arguments processing is performed and `args` is passed
to the remote method as is.
flat_kwargs : bool
The same as `flat_args` but for the `kwargs` values.
"""
def __init__(
self,
data: Union[
ObjectRefType,
"DeferredExecution",
List[Union[ObjectRefType, "DeferredExecution"]],
],
func: Union[Callable, ObjectRefType],
args: Union[List[Any], Tuple[Any]],
kwargs: Dict[str, Any],
num_returns=1,
):
if isinstance(data, DeferredExecution):
data.subscribe()
self.data = data
self.func = func
self.args = args
self.kwargs = kwargs
self.num_returns = num_returns
self.flat_args = self._flat_args(args)
self.flat_kwargs = self._flat_args(kwargs.values())
self.subscribers = 0
@classmethod
def _flat_args(cls, args: Iterable):
"""
Check if the arguments list is flat and subscribe to all `DeferredExecution` objects.
Parameters
----------
args : Iterable
Returns
-------
bool
"""
flat = True
for arg in args:
if isinstance(arg, DeferredExecution):
flat = False
arg.subscribe()
elif isinstance(arg, ListOrTuple):
flat = False
cls._flat_args(arg)
return flat
def exec(
self,
) -> Tuple[ObjectRefOrListType, Union["MetaList", List], Union[int, List[int]]]:
"""
Execute this task, if required.
Returns
-------
tuple
The execution result, MetaList, containing the length, width and
the worker's ip address (the last value in the list) and the values
offset in the list. I.e. length = meta_list[offset],
width = meta_list[offset + 1], ip = meta_list[-1].
"""
if self.has_result:
return self.data, self.meta, self.meta_offset
if (
not isinstance(self.data, DeferredExecution)
and self.flat_args
and self.flat_kwargs
and self.num_returns == 1
):
result, length, width, ip = remote_exec_func.options(
resources=RayTaskCustomResources.get()
).remote(self.func, self.data, *self.args, **self.kwargs)
meta = MetaList([length, width, ip])
self._set_result(result, meta, 0)
return result, meta, 0
# If there are no subscribers, we still need the result here. We don't need to decrement
# it back. After the execution, the result is saved and the counter has no effect.
self.subscribers += 2
consumers, output = self._deconstruct()
# The last result is the MetaList, so adding +1 here.
num_returns = sum(c.num_returns for c in consumers) + 1
results = self._remote_exec_chain(num_returns, *output)
meta = MetaList(results.pop())
meta_offset = 0
results = iter(results)
for de in consumers:
if de.num_returns == 1:
de._set_result(next(results), meta, meta_offset)
meta_offset += 2
else:
res = list(islice(results, num_returns))
offsets = list(range(0, 2 * num_returns, 2))
de._set_result(res, meta, offsets)
meta_offset += 2 * num_returns
return self.data, self.meta, self.meta_offset
@property
def has_result(self):
"""
Return true if this task has already been executed and the result is set.
Returns
-------
bool
"""
return not hasattr(self, "func")
def subscribe(self):
"""
Increment the `subscribers` counter.
Subscriber is any instance that could trigger the execution of this task.
In case of a multiple subscribers, the execution could be triggerred multiple
times. To prevent the multiple executions, the execution result is returned
from the worker and saved in this instance. Subsequent calls to `execute()`
return the previously saved result.
"""
self.subscribers += 1
def unsubscribe(self):
"""Decrement the `subscribers` counter."""
self.subscribers -= 1
assert self.subscribers >= 0
def _deconstruct(self) -> Tuple[List["DeferredExecution"], List[Any]]:
"""
Convert the specified execution tree to a flat list.
This is required for the automatic Ray object references
materialization before passing the list to a Ray worker.
The format of the list is the following:
sequence<
| 
In the GIFs below, Modin (left) and pandas (right) perform *the same pandas operations* on a 2GB dataset. The only difference between the two notebook examples is the import statement.
### Installation
#### From PyPI
Modin can be installed with `pip` on Linux, Windows and MacOS:
```bash
pip install "modin[all]" # (Recommended) Install Modin with Ray and Dask engines.
```
If you want to install Modin with a specific engine, we recommend:
```bash
pip install "modin[ray]" # Install Modin dependencies and Ray.
pip install "modin[dask]" # Install Modin dependencies and Dask.
pip install "modin[mpi]" # Install Modin dependencies and MPI through unidist.
```
To get Modin on MPI through unidist (as of unidist 0.5.0) fully working
it is required to have a working MPI implementation installed beforehand.
Otherwise, installation of `modin[mpi]` may fail. Refer to
[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)
section of the unidist documentation for more details about installation.
**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray` instead of `ray[default]`.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing `ray[default]` along with `modin[ray]`.
Modin automatically detects which engine(s) you have installed and uses that for scheduling computation.
#### From conda-forge
Installing from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`
will install Modin and three engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask) and
[MPI through unidist](https://github.com/modin-project/unidist).
```bash
conda install -c conda-forge modin-all
```
Each engine can also be installed individually (and also as a combination of several engines):
```bash
conda install -c conda-forge modin-ray # Install Modin dependencies and Ray.
conda install -c conda-forge modin-dask # Install Modin dependencies and Dask.
conda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.
```
**Note:** Since Modin 0.30.0 we use a reduced set of Ray dependencies: `ray-core` instead of `ray-default`.
This means that the dashboard and cluster launcher are no longer installed by default.
If you need those, consider installing `ray-default` along with `modin-ray`.
Refer to
[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)
section of the unidist documentation for more details on how to install a specific MPI implementation to run on.
To speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:
```bash
conda install -n base conda-libmamba-solver
```
and then use it during istallation either like:
```bash
conda install -c conda-forge modin-ray --experimental-solver=libmamba
```
or starting from conda 22.11 and libmamba solver 22.12 versions:
```bash
conda install -c conda-forge modin-ray --solver=libmamba
```
#### Choosing a Compute Engine
If you want to choose a specific compute engine to run on, you can set the environment
variable `MODIN_ENGINE` and Modin will do computation with that engine:
```bash
export MODIN_ENGINE=ray # Modin will use Ray
export MODIN_ENGINE=dask # Modin will use Dask
export MODIN_ENGINE=unidist # Modin will use Unidist
```
If you want to choose the Unidist engine, you should set the additional environment
variable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:
```bash
export UNIDIST_BACKEND=mpi # Unidist will use MPI backend
```
This can also be done within a notebook/interpreter before you import Modin:
```python
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put("ray") # Modin will use Ray
modin_cfg.Engine.put("dask") # Modin will use Dask
modin_cfg.Engine.put('unidist') # Modin will use Unidist
unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend
```
_Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior._
#### Which engine should I use?
On Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required
to use either of these engines as Modin abstracts away all of the complexity, so feel
free to pick either!
### Pandas API Coverage
| 
| 
The `modin.pandas` DataFrame is an extremely light-weight parallel DataFrame.
Modin transparently distributes the data and computation so that you can continue using the same pandas API
while working with more data faster. Because it is so light-weight,
Modin provides speed-ups of up to 4x on a laptop with 4 physical cores.
In pandas, you are only able to use one core at a time when you are doing computation of
any kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a
traditionally synchronous task like `read_csv`, we see large speedups by efficiently
distributing the work across your entire machine.
```python
import modin.pandas as pd
df = pd.read_csv("my_dataset.csv")
```
#### Modin can handle the datasets that pandas can't
Often data scientists have to switch between different tools
for operating on datasets of different sizes. Processing large dataframes with pandas
is slow, and pandas does not support working with dataframes that are too large to fit
into the available memory. As a result, pandas workflows that work well
for prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size
of your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably
work with hundreds of GBs without worrying about substantial slowdown or memory errors.
With [cluster](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html)
and [out of core](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html)
support, Modin is a DataFrame library with both great single-node performance and high
scalability in a cluster.
#### Modin Architecture
We designed [Modin's architecture](https://modin.readthedocs.io/en/latest/development/architecture.html)
to be modular so we can plug in different components as they develop and improve:
### Other Resources
#### Getting Started with Modin
- [Documentation](https://modin.readthedocs.io/en/latest/)
- [10-min Quickstart Guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html)
- [Examples and Tutorials](https://modin.readthedocs.io/en/latest/getting_started/examples.html)
- [Videos and Blogposts](https://modin.readthedocs.io/en/latest/getting_started/examples.html#talks-podcasts)
- [Benchmarking Modin](https://modin.readthedocs.io/en/latest/usage_guide/benchmarking.html)
#### Modin Community
- [Slack](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA)
- [Twitter](https://twitter.com/modin_project)
- [Mailing List](https://groups.google.com/g/modin-dev)
- [GitHub Issues](https://github.com/modin-project/modin/issues)
- [StackOverflow](https://stackoverflow.com/questions/tagged/modin)
#### Learn More about Modin
- [Frequently Asked Questions (FAQs)](https://modin.readthedocs.io/en/latest/getting_started/faq.html)
- [Troubleshooting Guide](https://modin.readthedocs.io/en/latest/getting_started/troubleshooting.html)
- [Development Guide](https://modin.readthedocs.io/en/latest/development/index.html)
- Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:
- [Flexible Rule-Based Decomposition and Metadata Independence in Modin](https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) (VLDB 2021)
- [Dataframe Systems: Theory, Architecture, and Implementation](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.pdf) (PhD Dissertation 2021)
- [Towards Scalable Dataframe Systems](https://arxiv.org/pdf/2001.00888.pdf) (VLDB 2020)
#### Getting Involved
***`modin.pandas` is currently under active development. Requests and contributions are welcome!***
For more information on how to contribute to Modin, check out the
[Modin Contribution Guide](https://modin.readthedocs.io/en/latest/development/contributing.html).
### License
[Apache License 2.0](LICENSE)
================================================
FILE: asv_bench/README.md
================================================
# Modin ASV benchmarks
## Here are some scenarios in which [ASV](https://asv.readthedocs.io/en/stable/index.html) can be used:
* Check the impact of the new patch on the performance of a certain set of operations:
`asv continuous -f 1.05 src/main HEAD -b TimeGroupBy --launch-method=spawn`
* Check for presence of errors inside of benchmarks after changing them or writing new ones:
`asv run --quick --show-stderr --python=same --launch-method=spawn`
* Run entire benchmark suite to get the current times:
`asv run --launch-method=spawn`
* Check the range of commits for performance degradation:
```
asv run [start_hash]..[end_hash] --launch-method=spawn
asv publish
asv preview
```
For more consistent results, you may need to use the following parameters which
description is in [ASV docs](https://asv.readthedocs.io/en/stable/benchmarks.html?highlight=sample_time#timing-benchmarks):
* `-a sample_time=1`
* `-a warmup_time=1`
### Notes about using Modin on Ray with Asv:
* `--launch-method=forkserver` is not working;
* Each set of parameters for each test is launched in its own process, which brings
a large overhead, since for each process redis server and other necessary processes
from ray initialization are started and destroyed.
## Adding new benchmark
Basic information on writing benchmarks is present [in ASV documentation](https://asv.readthedocs.io/en/stable/writing_benchmarks.html)
Benchmarks from `benchmarks/benchmarks.py`, `benchmarks/scalability/scalability_benchmarks.py` or `benchmarks/io/csv.py`
could be used as a starting point.
Requirements:
* the benchmark should be able to run both on Modin and on Pandas when the appropriate value
of the environment variable `MODIN_ASV_USE_IMPL` is selected.
* the size of the benchmark dataset should depend on the environment variable `MODIN_TEST_DATASET_SIZE`.
## Changing existing benchmark
It should be remembered that the hash calculated from the benchmark source code is used to display the results.
When changing the benchmark, the old results will no longer be displayed in the dashboard. In general, this is the correct
behavior so as not to get a situation when incomparable numbers are displayed in the dashboard.
But it should be noted that there could be changes in the source code when it is still correct to compare
the "before" and "after" versions, for example, name of a variable changed, comment added, etc.
In this case you must either run a new version of the benchmark for all the commits ever accounted for or manually change
the hash in the corresponding result files.
## Pipeline for displaying results in a dashboard
Step 1: checking benchmarks for validity, runs in PRs CI.
During the test, the benchmarks are run once on small data.
The implementation can be found in `test-asv-benchmarks` job of [ci.yml](https://github.com/modin-project/modin/blob/main/.github/workflows/ci.yml)
Step 2: running benchmarks with saving the results in [modin-bench@master](https://github.com/modin-project/modin-bench).
The launch takes place on internal server using specific TeamCity configuration.
The description of the server can be found in the ["Benchmark list"](https://modin.org/modin-bench/#summarylist?sort=0&dir=asc) tab,
on the left when you hover the mouse over the machine name.
This step starts as scheduled (now every half hour), subject to the presence of new commits in the Modin `main` branch.
Command to run benchmarks: `asv run HASHFILE:hashfile.txt --show-stderr --machine xeon-e5 --launch-method=spawn`.
In the file `hashfile.txt` is the last modin commit hash.
Writing to a `modin-bench@master` triggers 3 step of the pipeline.
Step 3: converting the results to html representation, which is saved in [modin-bench@gh-pages](https://github.com/modin-project/modin-bench)
The implementation can be found in `deploy-gh-pages` job of [push.yml](https://github.com/modin-project/modin-bench/blob/master/.github/workflows/push.yml)
Basic actions for step 2:
* setup environment variable:
* export MODIN_TEST_DATASET=Big
* export MODIN_CPUS=44
* setup git client
* prepare json file with machine description
* This file should be placed in the user's home directory.
* ASV does not always automatically create the file with the description of the machine correctly (e.g. due to being run in a container).
It is recommended to create a file using [asv machine](https://asv.readthedocs.io/en/stable/commands.html?highlight=machine%20description#asv-machine) command, and manually check the result.
[Example](https://github.com/modin-project/modin-bench/blob/master/results/xeon-e5/machine.json)
* copy old result to folder where new result will appear
(conflict resolution will be performed by ASV itself instead of git)
* push performance result to modin-bench repository
================================================
FILE: asv_bench/asv.conf.dask.json
================================================
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,
// The name of the project being benchmarked
"project": "modin",
// The project's homepage
"project_url": "https://modin.readthedocs.io/",
// The URL or local path of the source code repository for the
// project being benchmarked
"repo": "..",
// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["main"],
// Customizable commands for building, installing, and
// uninstalling the project. See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[dask]"],
// The tool to use to create environments. May be "conda",
// "virtualenv" or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",
// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
"install_timeout": 6000,
// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/modin-project/modin/commit/",
// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.9"],
// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": ".asv/results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": ".asv/html",
}
================================================
FILE: asv_bench/asv.conf.json
================================================
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,
// The name of the project being benchmarked
"project": "modin",
// The project's homepage
"project_url": "https://modin.readthedocs.io/",
// The URL or local path of the source code repository for the
// project being benchmarked
"repo": "..",
// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["main"],
// Customizable commands for building, installing, and
// uninstalling the project. See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[ray]"],
// The tool to use to create environments. May be "conda",
// "virtualenv" or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",
// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
"install_timeout": 6000,
// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/modin-project/modin/commit/",
// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.9"],
// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": ".asv/results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": ".asv/html",
}
================================================
FILE: asv_bench/asv.conf.unidist.json
================================================
{
// The version of the config file format. Do not change, unless
// you know what you are doing.
"version": 1,
// The name of the project being benchmarked
"project": "modin",
// The project's homepage
"project_url": "https://modin.readthedocs.io/",
// The URL or local path of the source code repository for the
// project being benchmarked
"repo": "..",
// List of branches to benchmark. If not provided, defaults to "master"
// (for git) or "default" (for mercurial).
"branches": ["main"],
// Customizable commands for building, installing, and
// uninstalling the project. See asv.conf.json documentation.
"install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[unidist]"],
// The tool to use to create environments. May be "conda",
// "virtualenv" or other value depending on the plugins in use.
// If missing or the empty string, the tool will be automatically
// determined by looking for tools on the PATH environment
// variable.
"environment_type": "conda",
// timeout in seconds for installing any dependencies in environment
// defaults to 10 min
"install_timeout": 6000,
// the base URL to show a commit for the project.
"show_commit_url": "https://github.com/modin-project/modin/commit/",
// The Pythons you'd like to test against. If not provided, defaults
// to the current version of Python used to run `asv`.
"pythons": ["3.9"],
// The list of conda channel names to be searched for benchmark
// dependency packages in the specified order
"conda_channels": ["conda-forge", "defaults"],
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env"
"env_dir": ".asv/env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": ".asv/results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": ".asv/html",
}
================================================
FILE: asv_bench/benchmarks/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin benchmarks."""
================================================
FILE: asv_bench/benchmarks/benchmarks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""General Modin benchmarks."""
# define `MODIN_CPUS` env var to control the number of partitions
# it should be defined before modin.pandas import (in case of using os.environ)
# define `MODIN_ASV_USE_IMPL` env var to choose library for using in performance
# measurements
import math
import numpy as np
from .utils import (
GROUPBY_NGROUPS,
IMPL,
RAND_HIGH,
RAND_LOW,
execute,
gen_nan_data,
generate_dataframe,
get_benchmark_shapes,
random_booleans,
random_columns,
random_string,
translator_groupby_ngroups,
)
class BaseTimeGroupBy:
def setup(self, shape, ngroups=5, groupby_ncols=1):
ngroups = translator_groupby_ngroups(ngroups, shape)
self.df, self.groupby_columns = generate_dataframe(
"int",
*shape,
RAND_LOW,
RAND_HIGH,
groupby_ncols,
count_groups=ngroups,
)
class TimeGroupByMultiColumn(BaseTimeGroupBy):
param_names = ["shape", "ngroups", "groupby_ncols"]
params = [
get_benchmark_shapes("TimeGroupByMultiColumn"),
GROUPBY_NGROUPS,
[6],
]
def time_groupby_agg_quan(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).agg("quantile"))
def time_groupby_agg_mean(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).apply(lambda df: df.mean()))
class TimeGroupByDefaultAggregations(BaseTimeGroupBy):
param_names = ["shape", "ngroups"]
params = [
get_benchmark_shapes("TimeGroupByDefaultAggregations"),
GROUPBY_NGROUPS,
]
def time_groupby_count(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).count())
def time_groupby_size(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).size())
def time_groupby_sum(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).sum())
def time_groupby_mean(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).mean())
class TimeGroupByDictionaryAggregation(BaseTimeGroupBy):
param_names = ["shape", "ngroups", "operation_type"]
params = [
get_benchmark_shapes("TimeGroupByDictionaryAggregation"),
GROUPBY_NGROUPS,
["reduce", "aggregation"],
]
operations = {
"reduce": ["sum", "count", "prod"],
"aggregation": ["quantile", "std", "median"],
}
def setup(self, shape, ngroups, operation_type):
super().setup(shape, ngroups)
self.cols_to_agg = self.df.columns[1:4]
operations = self.operations[operation_type]
self.agg_dict = {
c: operations[i % len(operations)] for i, c in enumerate(self.cols_to_agg)
}
def time_groupby_dict_agg(self, *args, **kwargs):
execute(self.df.groupby(by=self.groupby_columns).agg(self.agg_dict))
class TimeJoin:
param_names = ["shapes", "how", "sort"]
params = [
get_benchmark_shapes("TimeJoin"),
["left", "inner"],
[False],
]
def setup(self, shapes, how, sort):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_join(self, shapes, how, sort):
# join dataframes on index to get the predictable shape
execute(self.df1.join(self.df2, how=how, lsuffix="left_", sort=sort))
class TimeJoinStringIndex:
param_names = ["shapes", "sort"]
params = [
get_benchmark_shapes("TimeJoinStringIndex"),
[True, False],
]
def setup(self, shapes, sort):
assert shapes[0] % 100 == 0, "implementation restriction"
level1 = IMPL.Index([f"i-{i}" for i in range(10)], dtype=object).values
level2 = IMPL.Index(
[f"i-{i}" for i in range(shapes[0] // 100)], dtype=object
).values
codes1 = np.arange(10).repeat(shapes[0] // 100)
codes2 = np.tile(np.arange(shapes[0] // 100), 10)
index2 = IMPL.MultiIndex(levels=[level1, level2], codes=[codes1, codes2])
self.df_multi = IMPL.DataFrame(
np.random.randn(len(index2), 4), index=index2, columns=["A", "B", "C", "D"]
)
self.key1 = np.tile(level1.take(codes1), 10)
self.key2 = np.tile(level2.take(codes2), 10)
self.df = generate_dataframe("int", *shapes, RAND_LOW, RAND_HIGH)
# just to keep source shape
self.df = self.df.drop(columns=self.df.columns[-2:])
self.df["key1"] = self.key1
self.df["key2"] = self.key2
execute(self.df)
self.df_key1 = IMPL.DataFrame(
np.random.randn(len(level1), 4), index=level1, columns=["A", "B", "C", "D"]
)
self.df_key2 = IMPL.DataFrame(
np.random.randn(len(level2), 4), index=level2, columns=["A", "B", "C", "D"]
)
def time_join_dataframe_index_multi(self, shapes, sort):
execute(self.df.join(self.df_multi, on=["key1", "key2"], sort=sort))
def time_join_dataframe_index_single_key_bigger(self, shapes, sort):
execute(self.df.join(self.df_key2, on="key2", sort=sort))
def time_join_dataframe_index_single_key_small(self, shapes, sort):
execute(self.df.join(self.df_key1, on="key1", sort=sort))
class TimeMergeDefault:
param_names = ["shapes", "how", "sort"]
params = [
get_benchmark_shapes("TimeMergeDefault"),
["left", "inner"],
[True, False],
]
def setup(self, shapes, how, sort):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_merge(self, shapes, how, sort):
execute(IMPL.merge(self.df1, self.df2, how=how, sort=sort))
class TimeMerge:
param_names = ["shapes", "how", "sort"]
params = [
get_benchmark_shapes("TimeMerge"),
["left", "inner"],
[True, False],
]
def setup(self, shapes, how, sort):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_merge(self, shapes, how, sort):
# merge dataframes by index to get the predictable shape
execute(
self.df1.merge(
self.df2, left_index=True, right_index=True, how=how, sort=sort
)
)
def time_merge_dataframe_empty_right(self, shapes, how, sort):
# Getting an empty dataframe using `iloc` should be very fast,
# so the impact on the time of the merge operation should be negligible.
execute(IMPL.merge(self.df1, self.df2.iloc[:0], how=how, sort=sort))
def time_merge_dataframe_empty_left(self, shapes, how, sort):
# Getting an empty dataframe using `iloc` should be very fast,
# so the impact on the time of the merge operation should be negligible.
execute(IMPL.merge(self.df1.iloc[:0], self.df2, how=how, sort=sort))
class TimeMergeCategoricals:
param_names = ["shapes", "data_type"]
params = [
get_benchmark_shapes("MergeCategoricals"),
["object", "category"],
]
def setup(self, shapes, data_type):
assert len(shapes) == 2
assert shapes[1] == 2
size = (shapes[0],)
self.left = IMPL.DataFrame(
{
"X": np.random.choice(range(0, 10), size=size),
"Y": np.random.choice(["one", "two", "three"], size=size),
}
)
self.right = IMPL.DataFrame(
{
"X": np.random.choice(range(0, 10), size=size),
"Z": np.random.choice(["jjj", "kkk", "sss"], size=size),
}
)
if data_type == "category":
self.left = self.left.assign(Y=self.left["Y"].astype("category"))
execute(self.left)
self.right = self.right.assign(Z=self.right["Z"].astype("category"))
execute(self.right)
def time_merge_categoricals(self, shapes, data_type):
execute(IMPL.merge(self.left, self.right, on="X"))
class TimeConcat:
param_names = ["shapes", "how", "axis", "ignore_index"]
params = [
get_benchmark_shapes("TimeConcat"),
["inner", "outer"],
[0, 1],
[True, False],
]
def setup(self, shapes, how, axis, ignore_index):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
def time_concat(self, shapes, how, axis, ignore_index):
execute(
IMPL.concat(
[self.df1, self.df2], axis=axis, join=how, ignore_index=ignore_index
)
)
class TimeBinaryOp:
param_names = ["shapes", "binary_op", "axis"]
params = [
get_benchmark_shapes("TimeBinaryOp"),
["mul"],
[0, 1],
]
def setup(self, shapes, binary_op, axis):
self.df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
self.df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
self.op = getattr(self.df1, binary_op)
def time_binary_op(self, shapes, binary_op, axis):
execute(self.op(self.df2, axis=axis))
class TimeBinaryOpSeries:
param_names = ["shapes", "binary_op"]
params = [
get_benchmark_shapes("TimeBinaryOpSeries"),
["mul"],
]
def setup(self, shapes, binary_op):
df1 = generate_dataframe("int", *shapes[0], RAND_LOW, RAND_HIGH)
df2 = generate_dataframe("int", *shapes[1], RAND_LOW, RAND_HIGH)
self.series1 = df1[df1.columns[0]]
self.series2 = df2[df2.columns[0]]
self.op = getattr(self.series1, binary_op)
execute(self.series1)
execute(self.series2)
def time_binary_op_series(self, shapes, binary_op):
execute(self.op(self.series2))
class BaseTimeSetItem:
param_names = ["shape", "item_length", "loc", "is_equal_indices"]
@staticmethod
def get_loc(df, loc, axis, item_length):
locs_dict = {
"zero": 0,
"middle": len(df.axes[axis]) // 2,
"last": len(df.axes[axis]) - 1,
}
base_loc = locs_dict[loc]
range_based_loc = np.arange(
base_loc, min(len(df.axes[axis]), base_loc + item_length)
)
return (
(df.axes[axis][base_loc], base_loc)
if len(range_based_loc) == 1
else (df.axes[axis][range_based_loc], range_based_loc)
)
def setup(self, shape, item_length, loc, is_equal_indices):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH).copy()
self.loc, self.iloc = self.get_loc(
self.df, loc, item_length=item_length, axis=1
)
self.item = self.df[self.loc] + 1
self.item_raw = self.item.to_numpy()
if not is_equal_indices:
self.item.index = reversed(self.item.index)
class TimeSetItem(BaseTimeSetItem):
params = [
get_benchmark_shapes("TimeSetItem"),
[1],
["zero", "middle", "last"],
[True, False],
]
def time_setitem_qc(self, *args, **kwargs):
self.df[self.loc] = self.item
execute(self.df)
def time_setitem_raw(self, *args, **kwargs):
self.df[self.loc] = self.item_raw
execute(self.df)
class TimeInsert(BaseTimeSetItem):
params = [
get_benchmark_shapes("TimeInsert"),
[1],
["zero", "middle", "last"],
[True, False],
]
def time_insert_qc(self, *args, **kwargs):
self.df.insert(loc=self.iloc, column=random_string(), value=self.item)
execute(self.df)
def time_insert_raw(self, *args, **kwargs):
self.df.insert(loc=self.iloc, column=random_string(), value=self.item_raw)
execute(self.df)
class TimeArithmetic:
param_names = ["shape", "axis"]
params = [
get_benchmark_shapes("TimeArithmetic"),
[0, 1],
]
def setup(self, shape, axis):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
def time_sum(self, shape, axis):
execute(self.df.sum(axis=axis))
def time_count(self, shape, axis):
execute(self.df.count(axis=axis))
def time_median(self, shape, axis):
execute(self.df.median(axis=axis))
def time_nunique(self, shape, axis):
execute(self.df.nunique(axis=axis))
def time_apply(self, shape, axis):
execute(self.df.apply(lambda df: df.sum(), axis=axis))
def time_mean(self, shape, axis):
execute(self.df.mean(axis=axis))
def time_mode(self, shape, axis):
execute(self.df.mode(axis=axis))
def time_add(self, shape, axis):
execute(self.df.add(2, axis=axis))
def time_mul(self, shape, axis):
execute(self.df.mul(2, axis=axis))
def time_mod(self, shape, axis):
execute(self.df.mod(2, axis=axis))
def time_abs(self, shape, axis):
execute(self.df.abs())
def time_aggregate(self, shape, axis):
execute(self.df.aggregate(lambda df: df.sum(), axis=axis))
def time_is_in(self, shape, axis):
execute(self.df.isin([0, 2]))
def time_transpose(self, shape, axis):
execute(self.df.transpose())
class TimeSortValues:
param_names = ["shape", "columns_number", "ascending_list"]
params = [
get_benchmark_shapes("TimeSortValues"),
[1, 2, 10, 100],
[False, True],
]
def setup(self, shape, columns_number, ascending_list):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.columns = random_columns(self.df.columns, columns_number)
self.ascending = (
random_booleans(columns_number)
if ascending_list
else bool(random_booleans(1)[0])
)
def time_sort_values(self, shape, columns_number, ascending_list):
execute(self.df.sort_values(self.columns, ascending=self.ascending))
class TimeDrop:
param_names = ["shape", "axis", "drop_ncols"]
params = [
get_benchmark_shapes("TimeDrop"),
[0, 1],
[1, 0.8],
]
def setup(self, shape, axis, drop_ncols):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
drop_count = (
int(len(self.df.axes[axis]) * drop_ncols)
if isinstance(drop_ncols, float)
else drop_ncols
)
self.labels = self.df.axes[axis][:drop_count]
def time_drop(self, shape, axis, drop_ncols):
execute(self.df.drop(self.labels, axis=axis))
class TimeHead:
param_names = ["shape", "head_count"]
params = [
get_benchmark_shapes("TimeHead"),
[5, 0.8],
]
def setup(self, shape, head_count):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.head_count = (
int(head_count * len(self.df.index))
if isinstance(head_count, float)
else head_count
)
def time_head(self, shape, head_count):
execute(self.df.head(self.head_count))
class TimeTail:
param_names = ["shape", "tail_count"]
params = [
get_benchmark_shapes("TimeTail"),
[5, 0.8],
]
def setup(self, shape, tail_count):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.tail_count = (
int(tail_count * len(self.df.index))
if isinstance(tail_count, float)
else tail_count
)
def time_tail(self, shape, tail_count):
execute(self.df.tail(self.tail_count))
class TimeExplode:
param_names = ["shape"]
params = [
get_benchmark_shapes("TimeExplode"),
]
def setup(self, shape):
self.df = generate_dataframe(
"int", *shape, RAND_LOW, RAND_HIGH, gen_unique_key=True
)
def time_explode(self, shape):
execute(self.df.explode("col1"))
class TimeFillnaSeries:
param_names = ["value_type", "shape", "limit"]
params = [
["scalar", "dict", "Series"],
get_benchmark_shapes("TimeFillnaSeries"),
[None, 0.8],
]
def setup(self, value_type, shape, limit):
self.series = gen_nan_data(*shape)
if value_type == "scalar":
self.value = 18.19
elif value_type == "dict":
self.value = {k: k * 1.23 for k in range(shape[0])}
elif value_type == "Series":
self.value = IMPL.Series(
[k * 1.23 for k in range(shape[0])], index=IMPL.RangeIndex(shape[0])
)
else:
assert False
limit = int(limit * shape[0]) if limit else None
self.kw = {"value": self.value, "limit": limit}
def time_fillna(self, value_type, shape, limit):
execute(self.series.fillna(**self.kw))
def time_fillna_inplace(self, value_type, shape, limit):
self.series.fillna(inplace=True, **self.kw)
execute(self.series)
class TimeFillnaDataFrame:
param_names = ["value_type", "shape", "limit"]
params = [
["scalar", "dict", "DataFrame", "Series"],
get_benchmark_shapes("TimeFillnaDataFrame"),
[None, 0.8],
]
def setup(self, value_type, shape, limit):
self.df = gen_nan_data(*shape)
columns = self.df.columns
if value_type == "scalar":
self.value = 18.19
elif value_type == "dict":
self.value = {k: i * 1.23 for i, k in enumerate(columns)}
elif value_type == "Series":
self.value = IMPL.Series(
[i * 1.23 for i in range(len(columns))], index=columns
)
elif value_type == "DataFrame":
self.value = IMPL.DataFrame(
{
k: [i + j * 1.23 for j in range(shape[0])]
for i, k in enumerate(columns)
},
index=IMPL.RangeIndex(shape[0]),
columns=columns,
)
else:
assert False
limit = int(limit * shape[0]) if limit else None
self.kw = {"value": self.value, "limit": limit}
def time_fillna(self, value_type, shape, limit):
execute(self.df.fillna(**self.kw))
def time_fillna_inplace(self, value_type, shape, limit):
self.df.fillna(inplace=True, **self.kw)
execute(self.df)
class BaseTimeValueCounts:
def setup(self, shape, ngroups=5, subset=1):
ngroups = translator_groupby_ngroups(ngroups, shape)
self.df, self.subset = generate_dataframe(
"int",
*shape,
RAND_LOW,
RAND_HIGH,
groupby_ncols=subset,
count_groups=ngroups,
)
class TimeValueCountsFrame(BaseTimeValueCounts):
param_names = ["shape", "ngroups", "subset"]
params = [
get_benchmark_shapes("TimeValueCountsFrame"),
GROUPBY_NGROUPS,
[2, 10],
]
def time_value_counts(self, *args, **kwargs):
execute(self.df.value_counts(subset=self.subset))
class TimeValueCountsSeries(BaseTimeValueCounts):
param_names = ["shape", "ngroups", "bins"]
params = [
get_benchmark_shapes("TimeValueCountsSeries"),
GROUPBY_NGROUPS,
[None, 3],
]
def setup(self, shape, ngroups, bins):
super().setup(ngroups=ngroups, shape=shape)
self.df = self.df[self.subset[0]]
def time_value_counts(self, shape, ngroups, bins):
execute(self.df.value_counts(bins=bins))
class TimeIndexing:
param_names = ["shape", "indexer_type"]
params = [
get_benchmark_shapes("TimeIndexing"),
[
"bool_array",
"bool_series",
"scalar",
"slice",
"continuous_slice",
"numpy_array_take_all_values",
"python_list_take_10_values",
"function",
],
]
indexer_getters = {
"bool_array": lambda df: np.array([False, True] * (len(df) // 2)),
# This boolean-Series is a projection of the source frame, it shouldn't
# be reimported or triggered to execute:
"bool_series": lambda df: df.iloc[:, 0] > 50,
"scalar": lambda df: len(df) // 2,
"slice": lambda df: slice(0, len(df), 2),
"continuous_slice": lambda df: slice(len(df) // 2),
"numpy_array_take_all_values": lambda df: np.arange(len(df)),
"python_list_take_10_values": lambda df: list(range(min(10, len(df)))),
"function": lambda df: (lambda df: df.index[::-2]),
}
def setup(self, shape, indexer_type):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.indexer = self.indexer_getters[indexer_type](self.df)
if isinstance(self.indexer, (IMPL.Series, IMPL.DataFrame)):
# HACK: Triggering `dtypes` meta-data computation in advance,
# so it won't affect the `loc/iloc` time:
self.indexer.dtypes
def time_iloc(self, shape, indexer_type):
# Pandas doesn't implement `df.iloc[series boolean_mask]` and raises an exception on it.
# Replacing this with the semantically equivalent construction:
if indexer_type != "bool_series":
execute(self.df.iloc[self.indexer])
else:
execute(self.df[self.indexer])
def time_loc(self, shape, indexer_type):
execute(self.df.loc[self.indexer])
class TimeIndexingColumns:
param_names = ["shape"]
params = [get_benchmark_shapes("TimeIndexing")]
def setup(self, shape):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
self.numeric_indexer = [0, 1]
self.labels_indexer = self.df.columns[self.numeric_indexer].tolist()
def time_iloc(self, shape):
execute(self.df.iloc[:, self.numeric_indexer])
def time_loc(self, shape):
execute(self.df.loc[:, self.labels_indexer])
def time___getitem__(self, shape):
execute(self.df[self.labels_indexer])
class TimeMultiIndexing:
param_names = ["shape"]
params = [get_benchmark_shapes("TimeMultiIndexing")]
def setup(self, shape):
df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
index = IMPL.MultiIndex.from_product(
[df.index[: shape[0] // 2], ["bar", "foo"]]
)
columns = IMPL.MultiIndex.from_product(
[df.columns[: shape[1] // 2], ["buz", "fuz"]]
)
df.index = index
df.columns = columns
self.df = df.sort_index(axis=1)
def time_multiindex_loc(self, shape):
execute(
self.df.loc[
self.df.index[2] : self.df.index[-2],
self.df.columns[2] : self.df.columns[-2],
]
)
class TimeResetIndex:
param_names = ["shape", "drop", "level"]
params = [
get_benchmark_shapes("TimeResetIndex"),
[False, True],
[None, "level_1"],
]
def setup(self, shape, drop, level):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
if level:
index = IMPL.MultiIndex.from_product(
[self.df.index[: shape[0] // 2], ["bar", "foo"]],
names=["level_1", "level_2"],
)
self.df.index = index
def time_reset_index(self, shape, drop, level):
execute(self.df.reset_index(drop=drop, level=level))
class TimeAstype:
param_names = ["shape", "dtype", "astype_ncolumns"]
params = [
get_benchmark_shapes("TimeAstype"),
["float64", "category"],
["one", "all"],
]
def setup(self, shape, dtype, astype_ncolumns):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
if astype_ncolumns == "all":
self.astype_arg = dtype
elif astype_ncolumns == "one":
self.astype_arg = {"col1": dtype}
else:
raise ValueError(f"astype_ncolumns: {astype_ncolumns} isn't supported")
def time_astype(self, shape, dtype, astype_ncolumns):
execute(self.df.astype(self.astype_arg))
class TimeDescribe:
param_names = ["shape"]
params = [
get_benchmark_shapes("TimeDescribe"),
]
def setup(self, shape):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
def time_describe(self, shape):
execute(self.df.describe())
class TimeProperties:
param_names = ["shape"]
params = [
get_benchmark_shapes("TimeProperties"),
]
def setup(self, shape):
self.df = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH)
def time_shape(self, shape):
return self.df.shape
def time_columns(self, shape):
return self.df.columns
def time_index(self, shape):
return self.df.index
class TimeIndexingNumericSeries:
param_names = ["shape", "dtype", "index_structure"]
params = [
get_benchmark_shapes("TimeIndexingNumericSeries"),
(np.int64, np.uint64, np.float64),
("unique_monotonic_inc", "nonunique_monotonic_inc"),
]
def setup(self, shape, dtype, index_structure):
N = shape[0]
indices = {
"unique_monotonic_inc": IMPL.Index(range(N), dtype=dtype),
"nonunique_monotonic_inc": IMPL.Index(
list(range(N // 100)) + [(N // 100) - 1] + list(range(N // 100, N - 1)),
dtype=dtype,
),
}
self.data = IMPL.Series(np.random.rand(N), index=indices[index_structure])
self.array = np.arange(N // 2)
self.index_to_query = N // 2
self.array_list = self.array.tolist()
execute(self.data)
def time_getitem_scalar(self, shape, index, index_structure):
# not calling execute as execute function fails for scalar
self.data[self.index_to_query]
def time_getitem_slice(self, shape, index, index_structure):
execute(self.data[: self.index_to_query])
def time_getitem_list_like(self, shape, index, index_structure):
execute(self.data[[self.index_to_query]])
def time_getitem_array(self, shape, index, index_structure):
execute(self.data[self.array])
def time_getitem_lists(self, shape, index, index_structure):
execute(self.data[self.array_list])
def time_iloc_array(self, shape, index, index_structure):
execute(self.data.iloc[self.array])
def time_iloc_list_like(self, shape, index, index_structure):
execute(self.data.iloc[[self.index_to_query]])
def time_iloc_scalar(self, shape, index, index_structure):
# not calling execute as execute function fails for scalar
self.data.iloc[self.index_to_query]
def time_iloc_slice(self, shape, index, index_structure):
execute(self.data.iloc[: self.index_to_query])
def time_loc_array(self, shape, index, index_structure):
execute(self.data.loc[self.array])
def time_loc_list_like(self, shape, index, index_structure):
execute(self.data.loc[[self.index_to_query]])
def time_loc_scalar(self, shape, index, index_structure):
self.data.loc[self.index_to_query]
def time_loc_slice(self, shape, index, index_structure):
execute(self.data.loc[: self.index_to_query])
class TimeReindex:
param_names = ["shape"]
params = [get_benchmark_shapes("TimeReindex")]
def setup(self, shape):
rows, cols = shape
rng = IMPL.date_range(start="1/1/1970", periods=rows, freq="1min")
self.df = IMPL.DataFrame(
np.random.rand(rows, cols), index=rng, columns=range(cols)
)
self.df["foo"] = "bar"
self.rng_subset = IMPL.Index(rng[::2])
self.df2 = IMPL.DataFrame(
index=range(rows), data=np.random.rand(rows, cols), columns=range(cols)
)
level1 = IMPL.Index(
[f"i-{i}" for i in range(rows // 10)], dtype=object
).values.repeat(10)
level2 = np.tile(
IMPL.Index([f"i-{i}" for i in range(10)], dtype=object).values, rows // 10
)
index = IMPL.MultiIndex.from_arrays([level1, level2])
self.s = IMPL.Series(np.random.randn(rows), index=index)
self.s_subset = self.s[::2]
self.s_subset_no_cache = self.s[::2].copy()
mi = IMPL.MultiIndex.from_product([rng[: len(rng) // 10], range(10)])
self.s2 = IMPL.Series(np.random.randn(len(mi)), index=mi)
self.s2_subset = self.s2[::2].copy()
execute(self.df), execute(self.df2)
execute(self.s), execute(self.s_subset)
execute(self.s2), execute(self.s2_subset)
execute(self.s_subset_no_cache)
def time_reindex_dates(self, shape):
execute(self.df.reindex(self.rng_subset))
def time_reindex_columns(self, shape):
execute(self.df2.reindex(columns=self.df.columns[1:5]))
def time_reindex_multiindex_with_cache(self, shape):
# MultiIndex._values gets cached (pandas specific)
execute(self.s.reindex(self.s_subset.index))
def time_reindex_multiindex_no_cache(self, shape):
# Copy to avoid MultiIndex._values getting cached (pandas specific)
execute(self.s.reindex(self.s_subset_no_cache.index.copy()))
def time_reindex_multiindex_no_cache_dates(self, shape):
# Copy to avoid MultiIndex._values getting cached (pandas specific)
execute(self.s2_subset.reindex(self.s2.index.copy()))
class TimeReindexMethod:
params = [
get_benchmark_shapes("TimeReindexMethod"),
["pad", "backfill"],
[IMPL.date_range, IMPL.period_range],
]
param_names = ["shape", "method", "constructor"]
def setup(self, shape, method, constructor):
N = shape[0]
self.idx = constructor("1/1/2000", periods=N, freq="1min")
self.ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]
execute(self.ts)
def time_reindex_method(self, shape, method, constructor):
execute(self.ts.reindex(self.idx, method=method))
class TimeFillnaMethodSeries:
params = [get_benchmark_shapes("TimeFillnaMethodSeries"), ["pad", "backfill"]]
param_names = ["shape", "method"]
def setup(self, shape, method):
N = shape[0]
self.idx = IMPL.date_range("1/1/2000", periods=N, freq="1min")
ts = IMPL.Series(np.random.randn(N), index=self.idx)[::2]
self.ts_reindexed = ts.reindex(self.idx)
self.ts_float32 = self.ts_reindexed.astype("float32")
execute(self.ts_reindexed), execute(self.ts_float32)
def time_reindexed(self, shape, method):
execute(self.ts_reindexed.fillna(method=method))
def time_float_32(self, shape, method):
execute(self.ts_float32.fillna(method=method))
class TimeFillnaMethodDataframe:
params = [get_benchmark_shapes("TimeFillnaMethodDataframe"), ["pad", "backfill"]]
param_names = ["shape", "method"]
def setup(self, shape, method):
self.idx = IMPL.date_range("1/1/2000", periods=shape[0], freq="1min")
df_ts = IMPL.DataFrame(np.random.randn(*shape), index=self.idx)[::2]
self.df_ts_reindexed = df_ts.reindex(self.idx)
self.df_ts_float32 = self.df_ts_reindexed.astype("float32")
execute(self.df_ts_reindexed), execute(self.df_ts_float32)
def time_reindexed(self, shape, method):
execute(self.df_ts_reindexed.fillna(method=method))
def time_float_32(self, shape, method):
execute(self.df_ts_float32.fillna(method=method))
class TimeLevelAlign:
params = [get_benchmark_shapes("TimeLevelAlign")]
param_names = ["shapes"]
def setup(self, shapes):
rows, cols = shapes[0]
rows_sqrt = round(math.sqrt(rows))
# the new number of rows may differ from the requested (slightly, so ok)
rows = rows_sqrt * rows_sqrt
self.index = IMPL.MultiIndex(
levels=[np.arange(10), np.arange(rows_sqrt), np.arange(rows_sqrt)],
codes=[
np.arange(10).repeat(rows),
np.tile(np.arange(rows_sqrt).repeat(rows_sqrt), 10),
np.tile(np.tile(np.arange(rows_sqrt), rows_sqrt), 10),
],
)
self.df1 = IMPL.DataFrame(
np.random.randn(len(self.index), cols), index=self.index
)
self.df2 = IMPL.DataFrame(np.random.randn(*shapes[1]))
execute(self.df1), execute(self.df2)
def time_align_level(self, shapes):
left, right = self.df1.align(self.df2, level=1, copy=False)
execute(left), execute(right)
def time_reindex_level(self, shapes):
# `reindex` returns the same result here as `align`.
# Approximately the same performance is expected.
execute(self.df2.reindex(self.index, level=1))
class TimeDropDuplicatesDataframe:
params = [get_benchmark_shapes("TimeDropDuplicatesDataframe")]
param_names = ["shape"]
def setup(self, shape):
rows, cols = shape
N = rows // 10
K = 10
data = {}
# dataframe would have cols-1 keys(strings) and one value(int) column
for col in range(cols - 1):
data["key" + str(col + 1)] = IMPL.Index(
[f"i-{i}" for i in range(N)], dtype=object
).values.repeat(K)
data["value"] = np.random.randn(N * K)
self.df = IMPL.DataFrame(data)
execute(self.df)
def time_drop_dups(self, shape):
execute(self.df.drop_duplicates(self.df.columns[:-1]))
def time_drop_dups_inplace(self, shape):
self.df.drop_duplicates(self.df.columns[:-1], inplace=True)
execute(self.df)
class TimeDropDuplicatesSeries:
params = [get_benchmark_shapes("TimeDropDuplicatesSeries")]
param_names = ["shape"]
def setup(self, shape):
rows = shape[0]
self.series = IMPL.Series(
np.tile(
IMPL.Index([f"i-{i}" for i in range(rows // 10)], dtype=object).values,
10,
)
)
execute(self.series)
def time_drop_dups(self, shape):
execute(self.series.drop_duplicates())
def time_drop_dups_string(self, shape):
self.series.drop_duplicates(inplace=True)
execute(self.series)
class TimeDatetimeAccessor:
params = [get_benchmark_shapes("TimeDatetimeAccessor")]
param_names = ["shape"]
def setup(self, shape):
self.series = IMPL.Series(
IMPL.timedelta_range("1 days", periods=shape[0], freq="h")
)
execute(self.series)
def time_dt_accessor(self, shape):
execute(self.series.dt)
def time_timedelta_days(self, shape):
execute(self.series.dt.days)
def time_timedelta_seconds(self, shape):
execute(self.series.dt.seconds)
class BaseCategories:
def setup(self, shape):
rows = shape[0]
arr = [f"s{i:04d}" for i in np.random.randint(0, rows // 10, size=rows)]
self.ts = IMPL.Series(arr).astype("category")
execute(self.ts)
class TimeSetCategories(BaseCategories):
params = [get_benchmark_shapes("TimeSetCategories")]
param_names = ["shape"]
def time_set_categories(self, shape):
execute(self.ts.cat.set_categories(self.ts.cat.categories[::2]))
class TimeRemoveCategories(BaseCategories):
params = [get_benchmark_shapes("TimeRemoveCategories")]
param_names = ["shape"]
def time_remove_categories(self, shape):
execute(self.ts.cat.remove_categories(self.ts.cat.categories[::2]))
class BaseReshape:
def setup(self, shape):
rows, cols = shape
k = 10
arrays = [
np.arange(rows // k).repeat(k),
np.roll(np.tile(np.arange(rows // k), k), 25),
]
index = IMPL.MultiIndex.from_arrays(arrays)
self.df = IMPL.DataFrame(np.random.randn(rows, cols), index=index)
execute(self.df)
class TimeStack(BaseReshape):
params = [get_benchmark_shapes("TimeStack")]
param_names = ["shape"]
def setup(self, shape):
super().setup(shape)
self.udf = self.df.unstack(1)
execute(self.udf)
def time_stack(self, shape):
execute(self.udf.stack())
class TimeUnstack(BaseReshape):
params = [get_benchmark_shapes("TimeUnstack")]
param_names = ["shape"]
def time_unstack(self, shape):
execute(self.df.unstack(1))
class TimeReplace:
params = [get_benchmark_shapes("TimeReplace")]
param_names = ["shape"]
def setup(self, shape):
rows, cols = shape
self.to_replace = {i: getattr(IMPL, "Timestamp")(i) for i in range(rows)}
self.df = IMPL.DataFrame(np.random.randint(rows, size=(rows, cols)))
execute(self.df)
def time_replace(self, shape):
execute(self.df.replace(self.to_replace))
class TimeGroups:
params = [get_benchmark_shapes("TimeGroups")]
param_names = ["shape"]
def setup(self, shape):
self.series = IMPL.Series(np.random.randint(0, 100, size=shape[0]))
execute(self.series)
# returns a pretty dict thus not calling execute
def time_series_groups(self, shape):
self.series.groupby(self.series).groups
# returns a dict thus not calling execute
def time_series_indices(self, shape):
self.series.groupby(self.series).indices
class TimeRepr:
params = [get_benchmark_shapes("TimeRepr")]
param_names = ["shape"]
def setup(self, shape):
self.df = IMPL.DataFrame(np.random.randn(*shape))
execute(self.df)
# returns a string thus not calling execute
def time_repr(self, shape):
repr(self.df)
class TimeMaskBool:
params = [get_benchmark_shapes("TimeMaskBool")]
param_names = ["shape"]
def setup(self, shape):
self.df = IMPL.DataFrame(np.random.randn(*shape))
self.mask = self.df < 0
execute(self.df), execute(self.mask)
def time_frame_mask(self, shape):
execute(self.df.mask(self.mask))
class TimeIsnull:
params = [get_benchmark_shapes("TimeIsnull")]
param_names = ["shape"]
def setup(self, shape):
sample = np.array([np.nan, 1.0])
data = np.random.choice(sample, (shape[0], shape[1]))
self.df = IMPL.DataFrame(data)
execute(self.df)
def time_isnull(self, shape):
execute(IMPL.isnull(self.df))
class TimeDropna:
params = (["all", "any"], [0, 1], get_benchmark_shapes("TimeDropna"))
param_names = ["how", "axis", "shape"]
def setup(self, how, axis, shape):
row, col = shape
self.df = IMPL.DataFrame(np.random.randn(row, col))
self.df.iloc[row // 20 : row // 10, col // 3 : col // 2] = np.nan
self.df["foo"] = "bar"
execute(self.df)
def time_dropna(self, how, axis, shape):
execute(self.df.dropna(how=how, axis=axis))
class TimeEquals:
params = [get_benchmark_shapes("TimeEquals")]
param_names = ["shape"]
def setup(self, shape):
self.df = IMPL.DataFrame(np.random.randn(*shape))
self.df.iloc[-1, -1] = np.nan
execute(self.df)
# returns a boolean thus not calling execute
def time_frame_float_equal(self, shape):
self.df.equals(self.df)
from .utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/io/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""IO Modin benchmarks."""
================================================
FILE: asv_bench/benchmarks/io/csv.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
import numpy as np
from ..utils import (
ASV_USE_IMPL,
IMPL,
RAND_HIGH,
RAND_LOW,
execute,
generate_dataframe,
get_benchmark_shapes,
get_shape_id,
prepare_io_data,
)
class BaseReadCsv:
# test data file should be created only once
def setup_cache(self, test_filename="io_test_file"):
test_filenames = prepare_io_data(
test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)
)
return test_filenames
def setup(self, test_filenames, shape, *args, **kwargs):
# ray init
if ASV_USE_IMPL == "modin":
IMPL.DataFrame([])
self.shape_id = get_shape_id(shape)
class TimeReadCsvSkiprows(BaseReadCsv):
shapes = get_benchmark_shapes("TimeReadCsvSkiprows")
skiprows_mapping = {
"lambda_even_rows": lambda x: x % 2,
"range_uniform": np.arange(1, shapes[0][0] // 10),
"range_step2": np.arange(1, shapes[0][0], 2),
}
data_type = "str_int"
param_names = ["shape", "skiprows"]
params = [
shapes,
[None, "lambda_even_rows", "range_uniform", "range_step2"],
]
def setup(self, test_filenames, shape, skiprows):
super().setup(test_filenames, shape, skiprows)
self.skiprows = self.skiprows_mapping[skiprows] if skiprows else None
def time_skiprows(self, test_filenames, shape, skiprows):
execute(IMPL.read_csv(test_filenames[self.shape_id], skiprows=self.skiprows))
class TimeReadCsvTrueFalseValues(BaseReadCsv):
data_type = "true_false_int"
param_names = ["shape"]
params = [get_benchmark_shapes("TimeReadCsvTrueFalseValues")]
def time_true_false_values(self, test_filenames, shape):
execute(
IMPL.read_csv(
test_filenames[self.shape_id],
true_values=["Yes", "true"],
false_values=["No", "false"],
),
)
class TimeReadCsvNamesDtype:
shapes = get_benchmark_shapes("TimeReadCsvNamesDtype")
_dtypes_params = ["Int64", "Int64_Timestamp"]
_timestamp_columns = ["col1", "col2"]
param_names = ["shape", "names", "dtype"]
params = [
shapes,
["array-like"],
_dtypes_params,
]
def _get_file_id(self, shape, dtype):
return get_shape_id(shape) + dtype
def _add_timestamp_columns(self, df):
df = df.copy()
date_column = IMPL.date_range("2000", periods=df.shape[0], freq="ms")
for col in self._timestamp_columns:
df[col] = date_column
return df
def setup_cache(self, test_filename="io_test_file_csv_names_dtype"):
# filenames with a metadata of saved dataframes
cache = {}
for shape in self.shapes:
for dtype in self._dtypes_params:
df = generate_dataframe(
"int", *shape, RAND_LOW, RAND_HIGH, impl="pandas"
)
if dtype == "Int64_Timestamp":
df = self._add_timestamp_columns(df)
file_id = self._get_file_id(shape, dtype)
cache[file_id] = (
f"{test_filename}_{file_id}.csv",
df.columns.to_list(),
df.dtypes.to_dict(),
)
df.to_csv(cache[file_id][0], index=False)
return cache
def setup(self, cache, shape, names, dtype):
# ray init
if ASV_USE_IMPL == "modin":
IMPL.DataFrame([])
file_id = self._get_file_id(shape, dtype)
self.filename, self.names, self.dtype = cache[file_id]
self.parse_dates = None
if dtype == "Int64_Timestamp":
# cached version of dtype should not change
self.dtype = self.dtype.copy()
for col in self._timestamp_columns:
del self.dtype[col]
self.parse_dates = self._timestamp_columns
def time_read_csv_names_dtype(self, cache, shape, names, dtype):
execute(
IMPL.read_csv(
self.filename,
names=self.names,
header=0,
dtype=self.dtype,
parse_dates=self.parse_dates,
)
)
from ..utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/io/parquet.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
from ..utils import (
ASV_USE_IMPL,
IMPL,
execute,
get_benchmark_shapes,
get_shape_id,
prepare_io_data_parquet,
)
class TimeReadParquet:
shapes = get_benchmark_shapes("TimeReadParquet")
data_type = "str_int"
param_names = ["shape"]
params = [
shapes,
]
# test data file should be created only once
def setup_cache(self, test_filename="io_test_file"):
test_filenames = prepare_io_data_parquet(
test_filename, self.data_type, get_benchmark_shapes(self.__class__.__name__)
)
return test_filenames
def setup(self, test_filenames, shape):
# ray init
if ASV_USE_IMPL == "modin":
IMPL.DataFrame([])
self.shape_id = get_shape_id(shape)
def time_read_parquet(self, test_filenames, shape):
execute(
IMPL.read_parquet(
test_filenames[self.shape_id],
)
)
from ..utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/scalability/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Benchmarks measuring how Modin performance scales when MODIN_CPUS are changed."""
================================================
FILE: asv_bench/benchmarks/scalability/scalability_benchmarks.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""These benchmarks are supposed to be run only for modin, since they do not make sense for pandas."""
import modin.pandas as pd
try:
from modin.pandas.io import from_pandas
except ImportError:
from modin.pandas.utils import from_pandas
try:
from modin.pandas.io import to_numpy, to_pandas
except ImportError:
try:
from modin.utils import to_numpy, to_pandas
except ImportError:
# This provides compatibility with older versions of the Modin, allowing us to test old commits.
from modin.pandas.utils import to_pandas
import pandas
from ..utils import (
RAND_HIGH,
RAND_LOW,
execute,
gen_data,
generate_dataframe,
get_benchmark_shapes,
)
class TimeFromPandas:
param_names = ["shape", "cpus"]
params = [
get_benchmark_shapes("TimeFromPandas"),
[4, 16, 32],
]
def setup(self, shape, cpus):
self.data = pandas.DataFrame(gen_data("int", *shape, RAND_LOW, RAND_HIGH))
from modin.config import NPartitions
NPartitions.get = lambda: cpus
# trigger ray init
pd.DataFrame([])
def time_from_pandas(self, shape, cpus):
execute(from_pandas(self.data))
class TimeToPandas:
param_names = ["shape", "cpus"]
params = [
get_benchmark_shapes("TimeToPandas"),
[4, 16, 32],
]
def setup(self, shape, cpus):
from modin.config import NPartitions
NPartitions.get = lambda: cpus
self.data = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH, impl="modin")
def time_to_pandas(self, shape, cpus):
# to_pandas is already synchronous
to_pandas(self.data)
class TimeToNumPy:
param_names = ["shape", "cpus"]
params = [
get_benchmark_shapes("TimeToNumPy"),
[4, 16, 32],
]
def setup(self, shape, cpus):
from modin.config import NPartitions
NPartitions.get = lambda: cpus
self.data = generate_dataframe("int", *shape, RAND_LOW, RAND_HIGH, impl="modin")
def time_to_numpy(self, shape, cpus):
# to_numpy is already synchronous
to_numpy(self.data)
from ..utils import setup # noqa: E402, F401
================================================
FILE: asv_bench/benchmarks/utils/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Modin benchmarks utils."""
from .common import (
IMPL,
execute,
gen_data,
gen_nan_data,
generate_dataframe,
get_shape_id,
prepare_io_data,
prepare_io_data_parquet,
random_booleans,
random_columns,
random_string,
setup,
translator_groupby_ngroups,
)
from .compatibility import ASV_USE_IMPL, ASV_USE_STORAGE_FORMAT
from .data_shapes import GROUPBY_NGROUPS, RAND_HIGH, RAND_LOW, get_benchmark_shapes
__all__ = [
"ASV_USE_IMPL",
"ASV_USE_STORAGE_FORMAT",
"RAND_LOW",
"RAND_HIGH",
"GROUPBY_NGROUPS",
"get_benchmark_shapes",
"IMPL",
"execute",
"get_shape_id",
"gen_data",
"gen_nan_data",
"generate_dataframe",
"prepare_io_data",
"prepare_io_data_parquet",
"random_string",
"random_columns",
"random_booleans",
"translator_groupby_ngroups",
"setup",
]
================================================
FILE: asv_bench/benchmarks/utils/common.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""
The module contains the functionality that is used when benchmarking Modin commits.
In the case of using utilities from the main Modin code, there is a chance that when
benchmarking old commits, the utilities changed, which in turn can unexpectedly affect
the performance results, hence some utility functions are duplicated here.
"""
import logging
import uuid
from typing import Optional, Union
import numpy as np
import pandas
import modin.pandas
from .compatibility import ASV_DATASET_SIZE, ASV_USE_ENGINE, ASV_USE_IMPL
from .data_shapes import RAND_HIGH, RAND_LOW
POSSIBLE_IMPL = {
"modin": modin.pandas,
"pandas": pandas,
}
IMPL = POSSIBLE_IMPL[ASV_USE_IMPL]
def translator_groupby_ngroups(groupby_ngroups: Union[str, int], shape: tuple) -> int:
"""
Translate a string representation of the number of groups, into a number.
Parameters
----------
groupby_ngroups : str or int
Number of groups that will be used in `groupby` operation.
shape : tuple
Same as pandas.Dataframe.shape.
Returns
-------
int
"""
if ASV_DATASET_SIZE == "big":
if groupby_ngroups == "huge_amount_groups":
return min(shape[0] // 2, 5000)
return groupby_ngroups
else:
return groupby_ngroups
class weakdict(dict): # noqa: GL08
__slots__ = ("__weakref__",)
data_cache = dict()
dataframes_cache = dict()
def gen_nan_data(nrows: int, ncols: int) -> dict:
"""
Generate nan data with caching.
The generated data are saved in the dictionary and on a subsequent call,
if the keys match, saved data will be returned. Therefore, we need
to carefully monitor the changing of saved data and make its copy if needed.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
Returns
-------
modin.pandas.DataFrame or pandas.DataFrame or modin.pandas.Series or pandas.Series
DataFrame or Series with shape (nrows, ncols) or (nrows,), respectively.
"""
cache_key = (ASV_USE_IMPL, nrows, ncols)
if cache_key in data_cache:
return data_cache[cache_key]
logging.info("Generating nan data {} rows and {} columns".format(nrows, ncols))
if ncols > 1:
columns = [f"col{x}" for x in range(ncols)]
data = IMPL.DataFrame(np.nan, index=IMPL.RangeIndex(nrows), columns=columns)
elif ncols == 1:
data = IMPL.Series(np.nan, index=IMPL.RangeIndex(nrows))
else:
assert False, "Number of columns (ncols) should be >= 1"
data_cache[cache_key] = data
return data
def gen_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:
"""
Generate int data.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
"""
data = {
"col{}".format(i): np.random.randint(rand_low, rand_high, size=(nrows))
for i in range(ncols)
}
return data
def gen_str_int_data(nrows: int, ncols: int, rand_low: int, rand_high: int) -> dict:
"""
Generate int data and string data.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
One of the columns with string values.
"""
data = gen_int_data(nrows, ncols, rand_low, rand_high).copy()
# convert values in arbitary column to string type
key = list(data.keys())[0]
data[key] = [f"str_{x}" for x in data[key]]
return data
def gen_true_false_int_data(nrows, ncols, rand_low, rand_high):
"""
Generate int data and string data "true" and "false" values.
Parameters
----------
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
One half of the columns with integer values, another half - with "true" and
"false" string values.
"""
data = gen_int_data(nrows // 2, ncols // 2, rand_low, rand_high)
data_true_false = {
"tf_col{}".format(i): np.random.choice(
["Yes", "true", "No", "false"], size=(nrows - nrows // 2)
)
for i in range(ncols - ncols // 2)
}
data.update(data_true_false)
return data
def gen_data(
data_type: str,
nrows: int,
ncols: int,
rand_low: int,
rand_high: int,
) -> dict:
"""
Generate data with caching.
The generated data are saved in the dictionary and on a subsequent call,
if the keys match, saved data will be returned. Therefore, we need
to carefully monitor the changing of saved data and make its copy if needed.
Parameters
----------
data_type : {"int", "str_int", "true_false_int"}
Type of data generation.
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
Returns
-------
dict
Number of keys - `ncols`, each of them store np.ndarray of `nrows` length.
Notes
-----
Returned data type depends on the `data_type` parameter in the next way:
- `data_type`=="int" - all columns will be contain only integer values;
- `data_type`=="str_int" some of the columns will be of string type;
- `data_type`=="true_false_int" half of the columns will be filled with
string values representing "true" and "false" values and another half - with
integers.
"""
type_to_generator = {
"int": gen_int_data,
"str_int": gen_str_int_data,
"true_false_int": gen_true_false_int_data,
}
cache_key = (data_type, nrows, ncols, rand_low, rand_high)
if cache_key in data_cache:
return data_cache[cache_key]
logging.info(
"Generating {} data {} rows and {} columns [{}-{}]".format(
data_type, nrows, ncols, rand_low, rand_high
)
)
assert data_type in type_to_generator
data_generator = type_to_generator[data_type]
data = data_generator(nrows, ncols, rand_low, rand_high)
data_cache[cache_key] = weakdict(data)
return data
def generate_dataframe(
data_type: str,
nrows: int,
ncols: int,
rand_low: int,
rand_high: int,
groupby_ncols: Optional[int] = None,
count_groups: Optional[int] = None,
gen_unique_key: bool = False,
cache_prefix: str = None,
impl: str = None,
) -> Union[modin.pandas.DataFrame, pandas.DataFrame]:
"""
Generate DataFrame with caching.
The generated dataframes are saved in the dictionary and on a subsequent call,
if the keys match, one of the saved dataframes will be returned. Therefore, we need
to carefully monitor that operations that change the dataframe work with its copy.
Parameters
----------
data_type : str
Type of data generation;
supported types: {"int", "str_int"}.
nrows : int
Number of rows.
ncols : int
Number of columns.
rand_low : int
Low bound for random generator.
rand_high : int
High bound for random generator.
groupby_ncols : int, default: None
Number of columns for which `groupby` will be called in the future;
to get more stable performance results, we need to have the same number of values
in each group every benchmarking time.
count_groups : int, default: None
Count of groups in groupby columns.
gen_unique_key : bool, default: False
Generate `col1` column where all elements are unique.
cache_prefix : str, optional
Prefix to add to the cache key of the requested frame.
impl : str, optional
Implementation used to create the dataframe;
supported implemetations: {"modin", "pandas"}.
Returns
-------
modin.pandas.DataFrame or pandas.DataFrame [and list]
Notes
-----
The list of groupby columns names returns when groupby columns are generated.
"""
assert not (
(groupby_ncols is None) ^ (count_groups is None)
), "You must either specify both parameters 'groupby_ncols' and 'count_groups' or none of them."
if groupby_ncols and count_groups:
ncols -= groupby_ncols
if impl is None:
impl = ASV_USE_IMPL
cache_key = (
impl,
data_type,
nrows,
ncols,
rand_low,
rand_high,
groupby_ncols,
count_groups,
gen_unique_key,
)
if cache_prefix is not None:
cache_key = (cache_prefix, *cache_key)
if cache_key in dataframes_cache:
return dataframes_cache[cache_key]
logging.info(
"Allocating {} DataFrame {}: {} rows and {} columns [{}-{}]".format(
impl, data_type, nrows, ncols, rand_low, rand_high
)
)
data = gen_data(data_type, nrows, ncols, rand_low, rand_high)
if groupby_ncols and count_groups:
groupby_columns = [f"groupby_col{x}" for x in range(groupby_ncols)]
for groupby_col in groupby_columns:
data[groupby_col] = np.tile(np.arange(count_groups), nrows // count_groups)
if gen_unique_key:
data["col1"] = np.arange(nrows)
df = POSSIBLE_IMPL[impl].DataFrame(data)
if groupby_ncols and count_groups:
dataframes_cache[cache_key] = df, groupby_columns
return df, groupby_columns
dataframes_cache[cache_key] = df
return df
def random_string() -> str:
"""
Create a 36-character random string.
Returns
-------
str
"""
return str(uuid.uuid4())
def random_columns(df_columns: list, columns_number: int) -> list:
"""
Pick sublist of random columns from a given sequence.
Parameters
----------
df_columns : list
Columns to choose from.
columns_number : int
How many columns to pick.
Returns
-------
list
"""
return list(np.random.choice(df_columns, size=columns_number))
def random_booleans(number: int) -> list:
"""
Create random list of booleans with `number` elements.
Parameters
----------
number : int
Count of booleans in result list.
Returns
-------
list
"""
return list(np.random.choice([True, False], size=number))
def execute(df: Union[modin.pandas.DataFrame, pandas.DataFrame]):
"""
Make sure the calculations are finished.
Parameters
----------
df : modin.pandas.DataFrame or pandas.Datarame
DataFrame to be executed.
"""
if ASV_USE_IMPL == "modin":
partitions = df._query_compiler._modin_frame._partitions.flatten()
mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls
if len(partitions) and hasattr(mgr_cls, "wait_partitions"):
mgr_cls.wait_partitions(partitions)
return
# compatibility with old Modin versions
all(
map(
lambda partition: partition.drain_call_queue() or True,
partitions,
)
)
if ASV_USE_ENGINE == "ray":
from ray import wait
all(map(lambda partition: wait([partition._data]), partitions))
elif ASV_USE_ENGINE == "dask":
from dask.distributed import wait
all(map(lambda partition: wait(partition._data), partitions))
elif ASV_USE_ENGINE == "python":
pass
elif ASV_USE_IMPL == "pandas":
pass
def get_shape_id(shape: tuple) -> str:
"""
Join shape numbers into a string with `_` delimiters.
Parameters
----------
shape : tuple
Same as pandas.Dataframe.shape.
Returns
-------
str
"""
return "_".join([str(element) for element in shape])
def prepare_io_data(test_filename: str, data_type: str, shapes: list):
"""
Prepare data for IO tests with caching.
Parameters
----------
test_filename : str
Unique file identifier that is used to distinguish data
for different tests.
data_type : {"int", "str_int", "true_false_int"}
Type of data generation.
shapes : list
Data shapes to prepare.
Returns
-------
test_filenames : dict
Dictionary that maps dataset shape to the file on disk.
"""
test_filenames = {}
for shape in shapes:
shape_id = get_shape_id(shape)
test_filenames[shape_id] = f"{test_filename}_{shape_id}_{data_type}.csv"
df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl="pandas")
df.to_csv(test_filenames[shape_id], index=False)
return test_filenames
def prepare_io_data_parquet(test_filename: str, data_type: str, shapes: list):
"""
Prepare data for IO tests with caching.
Parameters
----------
test_filename : str
Unique file identifier that is used to distinguish data
for different tests.
data_type : "str_int"
Type of data generation.
shapes : list
Data shapes to prepare.
Returns
-------
test_filenames : dict
Dictionary that maps dataset shape to the file on disk.
"""
test_filenames = {}
for shape in shapes:
shape_id = get_shape_id(shape)
test_filenames[shape_id] = f"{test_filename}_{shape_id}_{data_type}.parquet"
df = generate_dataframe(data_type, *shape, RAND_LOW, RAND_HIGH, impl="pandas")
df.to_parquet(test_filenames[shape_id], index=False)
return test_filenames
def setup(*args, **kwargs): # noqa: GL08
# This function just needs to be imported into each benchmark file to
# set up the random seed before each function. ASV run it automatically.
# https://asv.readthedocs.io/en/latest/writing_benchmarks.html
np.random.seed(42)
================================================
FILE: asv_bench/benchmarks/utils/compatibility.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Compatibility layer for parameters used by ASV."""
import os
import modin.pandas as pd
try:
from modin.config import NPartitions
NPARTITIONS = NPartitions.get()
except ImportError:
NPARTITIONS = pd.DEFAULT_NPARTITIONS
try:
from modin.config import AsvImplementation, Engine, StorageFormat, TestDatasetSize
ASV_USE_IMPL = AsvImplementation.get()
ASV_DATASET_SIZE = TestDatasetSize.get() or "Small"
ASV_USE_ENGINE = Engine.get()
ASV_USE_STORAGE_FORMAT = StorageFormat.get()
except ImportError:
# The same benchmarking code can be run for different versions of Modin, so in
# case of an error importing important variables, we'll just use predefined values
ASV_USE_IMPL = os.environ.get("MODIN_ASV_USE_IMPL", "modin")
ASV_DATASET_SIZE = os.environ.get("MODIN_TEST_DATASET_SIZE", "Small")
ASV_USE_ENGINE = os.environ.get("MODIN_ENGINE", "Ray")
ASV_USE_STORAGE_FORMAT = os.environ.get("MODIN_STORAGE_FORMAT", "Pandas")
ASV_USE_IMPL = ASV_USE_IMPL.lower()
ASV_DATASET_SIZE = ASV_DATASET_SIZE.lower()
ASV_USE_ENGINE = ASV_USE_ENGINE.lower()
ASV_USE_STORAGE_FORMAT = ASV_USE_STORAGE_FORMAT.lower()
assert ASV_USE_IMPL in ("modin", "pandas")
assert ASV_DATASET_SIZE in ("big", "small")
assert ASV_USE_ENGINE in ("ray", "dask", "python", "unidist")
assert ASV_USE_STORAGE_FORMAT in ("pandas")
================================================
FILE: asv_bench/benchmarks/utils/data_shapes.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
"""Define data shapes."""
import json
import os
from .compatibility import ASV_DATASET_SIZE
RAND_LOW = 0
RAND_HIGH = 100
BINARY_OP_DATA_SIZE = {
"big": [
[[5000, 5000], [5000, 5000]],
# the case extremely inefficient
# [[20, 500_000], [10, 1_000_000]],
[[500_000, 20], [1_000_000, 10]],
],
"small": [[[250, 250], [250, 250]], [[10_000, 20], [25_000, 10]]],
}
UNARY_OP_DATA_SIZE = {
"big": [
[5000, 5000],
# the case extremely inefficient
# [10, 1_000_000],
[1_000_000, 10],
],
"small": [[250, 250], [10_000, 10]],
}
SERIES_DATA_SIZE = {
"big": [[100_000, 1]],
"small": [[10_000, 1]],
}
BINARY_OP_SERIES_DATA_SIZE = {
"big": [
[[500_000, 1], [1_000_000, 1]],
[[500_000, 1], [500_000, 1]],
],
"small": [[[5_000, 1], [10_000, 1]]],
}
DEFAULT_GROUPBY_NGROUPS = {
"big": [100, "huge_amount_groups"],
"small": [5],
}
GROUPBY_NGROUPS = DEFAULT_GROUPBY_NGROUPS[ASV_DATASET_SIZE]
_DEFAULT_CONFIG_T = [
(
UNARY_OP_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeGroupByMultiColumn",
"TimeGroupByDefaultAggregations",
"TimeGroupByDictionaryAggregation",
"TimeSetItem",
"TimeInsert",
"TimeArithmetic",
"TimeSortValues",
"TimeDrop",
"TimeHead",
"TimeTail",
"TimeExplode",
"TimeFillna",
"TimeFillnaDataFrame",
"TimeValueCountsFrame",
"TimeValueCountsSeries",
"TimeIndexing",
"TimeMultiIndexing",
"TimeResetIndex",
"TimeAstype",
"TimeDescribe",
"TimeProperties",
"TimeReindex",
"TimeReindexMethod",
"TimeFillnaMethodDataframe",
"TimeDropDuplicatesDataframe",
"TimeStack",
"TimeUnstack",
"TimeRepr",
"TimeMaskBool",
"TimeIsnull",
"TimeDropna",
"TimeEquals",
# IO benchmarks
"TimeReadCsvSkiprows",
"TimeReadCsvTrueFalseValues",
"TimeReadCsvNamesDtype",
"TimeReadParquet",
# Scalability benchmarks
"TimeFromPandas",
"TimeToPandas",
"TimeToNumPy",
],
),
(
BINARY_OP_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeJoin",
"TimeMerge",
"TimeMergeDefault",
"TimeConcat",
"TimeAppend",
"TimeBinaryOp",
"TimeLevelAlign",
],
),
(
SERIES_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeFillnaSeries",
"TimeGroups",
"TimeIndexingNumericSeries",
"TimeFillnaMethodSeries",
"TimeDatetimeAccessor",
"TimeSetCategories",
"TimeRemoveCategories",
"TimeDropDuplicatesSeries",
],
),
(
BINARY_OP_SERIES_DATA_SIZE[ASV_DATASET_SIZE],
[
# Pandas storage format benchmarks
"TimeBinaryOpSeries",
],
),
]
DEFAULT_CONFIG = {}
DEFAULT_CONFIG["MergeCategoricals"] = (
[[10_000, 2]] if ASV_DATASET_SIZE == "big" else [[1_000, 2]]
)
DEFAULT_CONFIG["TimeJoinStringIndex"] = (
[[100_000, 64]] if ASV_DATASET_SIZE == "big" else [[1_000, 4]]
)
DEFAULT_CONFIG["TimeReplace"] = (
[[10_000, 2]] if ASV_DATASET_SIZE == "big" else [[1_000, 2]]
)
for config in (_DEFAULT_CONFIG_T,):
for _shape, _names in config:
DEFAULT_CONFIG.update({_name: _shape for _name in _names})
# Correct shapes in the case when the operation ended with a timeout error
if ASV_DATASET_SIZE == "big":
DEFAULT_CONFIG["TimeMergeDefault"] = [
[[1000, 1000], [1000, 1000]],
[[500_000, 20], [1_000_000, 10]],
]
DEFAULT_CONFIG["TimeLevelAlign"] = [
[[2500, 2500], [2500, 2500]],
[[250_000, 20], [500_000, 10]],
]
DEFAULT_CONFIG["TimeStack"] = [
[1500, 1500],
[100_000, 10],
]
DEFAULT_CONFIG["TimeUnstack"] = DEFAULT_CONFIG["TimeStack"]
CONFIG_FROM_FILE = None
def get_benchmark_shapes(bench_id: str):
"""
Get custom benchmark shapes from a json file stored in MODIN_ASV_DATASIZE_CONFIG.
If `bench_id` benchmark is not found in the file, then the default value will
be used.
Parameters
----------
bench_id : str
Unique benchmark identifier that is used to get shapes.
Returns
-------
list
Benchmark shapes.
"""
global CONFIG_FROM_FILE
if not CONFIG_FROM_FILE:
try:
from modin.config import AsvDataSizeConfig
filename = AsvDataSizeConfig.get()
except ImportError:
filename = os.environ.get("MODIN_ASV_DATASIZE_CONFIG", None)
if filename:
# should be json
with open(filename) as _f:
CONFIG_FROM_FILE = json.load(_f)
if CONFIG_FROM_FILE and bench_id in CONFIG_FROM_FILE:
return CONFIG_FROM_FILE[bench_id]
return DEFAULT_CONFIG[bench_id]
================================================
FILE: asv_bench/test/__init__.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
================================================
FILE: asv_bench/test/test_utils.py
================================================
# Licensed to Modin Development Team under one or more contributor license agreements.
# See the NOTICE file distributed with this work for additional information regarding
# copyright ownership. The Modin Development Team licenses this file to you under the
# Apache License, Version 2.0 (the "License"); you may not use this file except in
# compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under
# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific language
# governing permissions and limitations under the License.
from unittest.mock import Mock, mock_open, patch
import numpy as np
import pytest
from benchmarks.utils import data_shapes, execute, get_benchmark_shapes
import modin.pandas as pd
from modin.config import AsvDataSizeConfig
@pytest.mark.parametrize(
"asv_config_content, result",
[
(
'{"TimeJoin": [[[10, 10], [15, 15]], [[11, 11], [13, 13]]], \
"TimeGroupBy": [[11, 11], [13, 13]]}',
[
[
# binary shapes
[[10, 10], [15, 15]],
[[11, 11], [13, 13]],
],
[
# unary shapes
[11, 11],
[13, 13],
],
],
),
],
)
@patch.object(data_shapes, "CONFIG_FROM_FILE", new=None)
def test_get_benchmark_shapes(asv_config_content, result):
AsvDataSizeConfig.put("mock_filename")
with patch("builtins.open", mock_open(read_data=asv_config_content)):
assert result[0] == get_benchmark_shapes("TimeJoin")
assert result[1] == get_benchmark_shapes("TimeGroupBy")
@pytest.mark.parametrize(
"asv_config_content, result",
[
(
'{"TimeJoin": [[[10, 10], [15, 15]]]',
[[100, 100]],
),
],
)
@patch.object(data_shapes, "CONFIG_FROM_FILE", new=None)
def test_get_benchmark_shapes_default(asv_config_content, result):
AsvDataSizeConfig.put(None)
with patch.object(data_shapes, "DEFAULT_CONFIG", new={"TimeJoin": result}):
assert result == get_benchmark_shapes("TimeJoin")
def test_execute():
df = pd.DataFrame(np.random.rand(100, 64))
partitions = df._query_compiler._modin_frame._partitions.flatten()
mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls
with patch.object(mgr_cls, "wait_partitions", new=Mock()):
execute(df)
mgr_cls.wait_partitions.assert_called_once()
assert (mgr_cls.wait_partitions.call_args[0] == partitions).all()
================================================
FILE: ci/teamcity/Dockerfile.teamcity-ci
================================================
# Create images from this container like this (in modin repo root):
#
# git rev-parse HEAD > ci/teamcity/git-rev
#
# tar cf ci/teamcity/modin.tar .
#
# docker build --build-arg ENVIRONMENT=environment-dev.yml -t modin-project/teamcity-ci:${BUILD_NUMBER} -f ci/teamcity/Dockerfile.teamcity-ci ci/teamcity
FROM rayproject/ray:latest
ARG ENVIRONMENT=environment-dev.yml
ADD modin.tar /modin
ADD git-rev /modin/git-rev
WORKDIR /modin
RUN sudo chown -R ray /modin
# Make RUN commands use `bash --login`:
SHELL ["/bin/bash", "--login", "-c"]
# Initialize conda in bash config files:
RUN conda init bash
ENV PATH /home/ray/anaconda3/envs/modin/bin:$PATH
RUN conda config --set channel_priority strict
RUN conda update python -y
RUN conda env create -f ${ENVIRONMENT}
RUN conda install curl PyGithub
# Activate the environment, and make sure it's activated:
# The following line also removed conda initialization from
# ~/.bashrc so conda starts complaining that it should be
# initialized for bash. But it is necessary to do it because
# activation is not always executed when "docker exec" is used
# and then conda initialization overwrites PATH with its base
# environment where python doesn't have any packages installed.
RUN echo "conda activate modin" > ~/.bashrc
RUN echo "Make sure environment is activated"
RUN conda list -n modin
================================================
FILE: ci/teamcity/build-docker.py
================================================
import os
import sys
def execute_command(cmd):
status = os.system(cmd)
ec = os.WEXITSTATUS(status)
if ec != 0:
raise SystemExit('Command "{}" failed'.format(cmd))
if sys.platform.startswith("linux"):
execute_command("git rev-parse HEAD > git-rev")
execute_command(
"(cd ../.. && git archive -o ci/teamcity/modin.tar $(cat ci/teamcity/git-rev))"
)
base_image = "ray-project/deploy"
requirements = "requirements-dev.txt"
execute_command(
"docker build -f Dockerfile.modin-base --build-arg BASE_IMAGE={} -t modin-project/modin-base .".format(
base_image
)
)
else:
raise SystemExit(
"TeamCity CI in Docker containers is supported only on Linux at the moment."
)
execute_command(
"docker build -f Dockerfile.teamcity-ci --build-arg REQUIREMENTS={} -t modin-project/teamcity-ci .".format(
requirements
)
)
if sys.platform.startswith("linux"):
execute_command("rm ./modin.tar ./git-rev")
================================================
FILE: ci/teamcity/comment_on_pr.py
================================================
"""
Post the comment like the following to the PR:
```
:robot: TeamCity test results bot :robot:
TeamCity {} test results bot![](\"../../../img/pandas_multicore.png\")
![](\"../../../img/modin_multicore.png\")
\n",
"