Repository: eyaltrabelsi/pandas-log Branch: master Commit: 5ea73d91856c Files: 37 Total size: 158.7 KB Directory structure: gitextract_a1_r2osh/ ├── .github/ │ ├── ISSUE_TEMPLATE/ │ │ ├── bug_report.md │ │ ├── documentation_fix.md │ │ ├── new_examples.md │ │ └── new_proposed_feature.md │ └── pull_request_template.md ├── .gitignore ├── AUTHORS.rst ├── CONTRIBUTING.rst ├── HISTORY.rst ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.rst ├── TODO.rst ├── docs/ │ ├── Makefile │ ├── conf.py │ ├── contributing.rst │ ├── index.rst │ ├── installation.rst │ ├── modules.rst │ ├── pandas_log.rst │ ├── readme.rst │ └── usage.rst ├── examples/ │ ├── README.rst │ ├── __init__.py │ ├── pandas_log_intro.ipynb │ └── pokemon.csv ├── pandas_log/ │ ├── __init__.py │ ├── aop_utils.py │ ├── pandas_execution_stats.py │ ├── pandas_log.py │ ├── patched_logs_functions.py │ └── settings.py ├── requirements_dev.txt ├── setup.cfg ├── setup.py └── tox.ini ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/ISSUE_TEMPLATE/bug_report.md ================================================ --- name: Bug Report about: Please use this issue template if you are filing a bug report. --- # Brief Description # System Information - Operating system: macOS/Linux/Windows - OS details (optional): - Python version (required): # Minimally Reproducible Code # Error Messages ``` ``` ================================================ FILE: .github/ISSUE_TEMPLATE/documentation_fix.md ================================================ --- name: Propose a Documentation Fix about: Use this issue tracker template if you'd like to propose a fix to the documentation. --- # Brief Description of Fix Currently, the docs... I would like to propose a change, such that now the docs... # Relevant Context - [Link to documentation page](hhttps://pandas-log.readthedocs.io) ================================================ FILE: .github/ISSUE_TEMPLATE/new_examples.md ================================================ --- name: Add/Modify Notebooks about: Use this specific template if you'd like to contribute a notebook to the examples gallery or modify an existing one. --- # Brief Description I'd like to write a notebook that... (optional but encouraged) This notebook would likely cover the following pandas-log functionality: - - - # Dataset ================================================ FILE: .github/ISSUE_TEMPLATE/new_proposed_feature.md ================================================ --- name: Propose New Feature about: If you'd like to propose a new feature, please use this template. --- # Brief Description I would like to propose... ================================================ FILE: .github/pull_request_template.md ================================================ # PR Description Please describe the changes proposed in the pull request: - - - **This PR resolves #(put issue number here, and remove parentheses).** # PR Checklist Please ensure that you have done the following: 1. [ ] PR in from a fork off your branch. Do not PR from ``:master, but rather from ``:. 2. [ ] If you're not on the contributors list, add yourself to `AUTHORS.rst`. ## Quick Check To do a very quick check that everything is correct, follow these steps below: - [ ] Run the command `make format` from pandsa-log's top-level directory. This will automatically run: - black formatting - fix imports with isort ================================================ FILE: .gitignore ================================================ .idea explore_pandas_log.* pandas_log/__pycache__ .ipynb_checkpoints .cache .eggs *.egg-info ================================================ FILE: AUTHORS.rst ================================================ ======= Credits ======= Development Lead ---------------- * Eyal Trabelsi Contributors ------------ * Charles Davis ================================================ FILE: CONTRIBUTING.rst ================================================ ============ Contributing ============ Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given. The following sections detail a variety of ways to contribute, as well as how to get started. Types of Contributions ======================= Write Documentation -------------------- ``pandas-log`` could always use more documentation, whether as part of the official ``pandas-log`` docs, in docstrings, or the examples gallery. During sprints, we require newcomers to the project to first contribute a documentation fix before contributing a code fix. Doing so has numerous benefits: 1. You become familiar with the project by first reading through the docs. 2. Your documentation contribution will be a pain point that you have full context on. 3. Your contribution will be impactful because documentation is the project's front-facing interface. 4. Your first contribution will be simpler, because you won't have to wrestle with build systems. 5. You can choose between getting set up locally first (recommended), or instead directly making edits on the GitHub web UI (also not a problem). 6. Every newcomer is equal in our eyes, and it's the most egalitarian way to get started (regardless of experience). Remote contributors outside of sprints and prior contributors who are joining us at the sprints need not adhere to this rule, as a good prior assumption is that you are a motivated user of the library already. If you have made a prior pull request to the library, we would like to encourage you to mentor newcomers in lieu of coding contributions. Documentation can come in many forms. For example, you might want to contribute: - Fixes for a typographical, grammatical, or spelling error. - Changes for a docstring that was unclear. - Clarifications for installation/setup instructions that are unclear. - Corrections to a sentence/phrase/word choice that didn't make sense. - New example/tutorial notebooks using the library. - Edits to existing tutorial notebooks with better code style. In particular, contributing new tutorial notebooks and improving the clarity of existing ones are great ways to get familiar with the library and find pain points that you can propose as fixes or enhancements to the library. Report Bugs ------------ Report bugs at https://github.com/eyaltrabelsi/pandas-log/issues. If you are reporting a bug, please include: * Your operating system name and version. * Any details about your local setup that might be helpful in troubleshooting. * Detailed steps to reproduce the bug. Fix Bugs --------- Look through the GitHub issues for bugs. Anything tagged with ``bug`` and ``available to hack on`` is open to whoever wants to implement it. Do be sure to claim the issue for yourself by indicating, "I would like to work on this issue." If you would like to discuss it further before going forward, you are more than welcome to discuss on the GitHub issue tracker. Submit Feedback ----------------- The best way to send feedback is to file an issue at https://github.com/eyaltrabelsi/pandas-log/issues. If you are proposing a feature: * Explain in detail how it would work. * Keep the scope as narrow as possible, to make it easier to implement. * Remember that this is a volunteer-driven project, and that contributions are welcome :) Get Started! ==================== Ready to contribute? Here's how to set up ``pandas_log`` for local development. 1. Fork the `pandas_log` repo on GitHub: https://github.com/eyaltrabelsi/pandas-log. 2. Clone your fork locally:: $ git clone git@github.com:your_name_here/pandas_log.git 3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:: $ mkvirtualenv pandas_log $ cd pandas_log/ $ python setup.py develop 4. Create a branch for local development:: $ git checkout -b name-of-your-bugfix-or-feature Now you can make your changes locally. 5. When you're done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:: $ flake8 pandas_log tests $ python setup.py test or pytest $ tox To get flake8 and tox, just pip install them into your virtualenv. 6. Commit your changes and push your branch to GitHub:: $ git add . $ git commit -m "Your detailed description of your changes." $ git push origin name-of-your-bugfix-or-feature 7. Submit a pull request through the GitHub website. Pull Request Guidelines ---------------------- Before you submit a pull request, check that it meets these guidelines: 1. The pull request should include tests. 2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst. Deploying --------- A reminder for the maintainers on how to deploy. Make sure all your changes are committed (including an entry in HISTORY.rst). Then run:: $ bump2version patch # possible: major / minor / patch $ git push $ git push --tags Travis will then deploy to PyPI if tests pass. ================================================ FILE: HISTORY.rst ================================================ ======= History ======= 0.0.0 (2019-09-18) ------------------ * First release on PyPI. ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2019, Eyal Trabelsi Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: MANIFEST.in ================================================ include CONTRIBUTING.rst include LICENSE include README.rst recursive-include tests * recursive-exclude * __pycache__ recursive-exclude * *.py[co] recursive-include docs *.rst conf.py Makefile make.bat *.jpg *.png *.gif ================================================ FILE: Makefile ================================================ cat.PHONY: clean clean-test clean-pyc clean-build docs help .DEFAULT_GOAL := help define BROWSER_PYSCRIPT import os, webbrowser, sys try: from urllib import pathname2url except: from urllib.request import pathname2url webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1]))) endef export BROWSER_PYSCRIPT define PRINT_HELP_PYSCRIPT import re, sys for line in sys.stdin: make if match: target, help = match.groups() print("%-20s %s" % (target, help)) endef export PRINT_HELP_PYSCRIPT BROWSER := python -c "$$BROWSER_PYSCRIPT" help: @python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST) clean: find . -name '*.pyc' -exec rm -f {} + find . -name '*.pyo' -exec rm -f {} + find . -name '*~' -exec rm -f {} + find . -name '__pycache__' -exec rm -fr {} + rm -fr .tox/ rm -f .coverage rm -fr htmlcov/ rm -fr .pytest_cache rm -fr build/ rm -fr dist/ rm -fr .eggs/ find . -name '*.egg-info' -exec rm -fr {} + find . -name '*.egg' -exec rm -f {} + format: isort -rc pandas_log -y -up -tc black -l 79 pandas_log test: ## run tests on every Python version with tox tox docs: ## generate Sphinx HTML documentation, including API docs rm -f docs/pandas_log.rst rm -f docs/modules.rst sphinx-apidoc -o docs/ pandas_log $(MAKE) -C docs clean $(MAKE) -C docs html $(BROWSER) docs/_build/html/index.html release: dist ## package and upload a release twine upload dist/* dist: clean ## builds source and wheel package python setup.py sdist python setup.py bdist_wheel ls -l dist ================================================ FILE: README.rst ================================================ ========== pandas-log ========== .. image:: https://img.shields.io/pypi/v/pandas_log.svg :target: https://pypi.python.org/pypi/pandas_log .. image:: https://img.shields.io/travis/eyaltrabelsi/pandas-log.svg :target: https://travis-ci.org/eyaltrabelsi/pandas-log .. image:: https://readthedocs.org/projects/pandas-log/badge/?version=latest :target: https://pandas-log.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status .. image:: https://pyup.io/repos/github/eyaltrabelsi/pandas-log/shield.svg :target: https://pyup.io/repos/github/eyaltrabelsi/pandas-log/ :alt: Updates The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common functions, such as ``.query``, ``.apply``, ``.merge``, ``.group_by`` and more. Why pandas-log? --------------- ``Pandas-log`` is a Python implementation of the R package ``tidylog``, and provides a feedback about basic pandas operations. The pandas has been invaluable for the data science ecosystem and usually consists of a series of steps that involve transforming raw data into an understandable/usable format. These series of steps need to be run in a certain sequence and if the result is unexpected it's hard to understand what happened. ``Pandas-log`` log metadata on each operation which will allow to pinpoint the issues. Lets look at an example, first we need to load ``pandas-log`` after ``pandas`` and create a dataframe: .. code-block:: python import pandas import pandas_log with pandas_log.enable(): df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'], "toy": [np.nan, 'Batmobile', 'Bullwhip'], "born": [pd.NaT, pd.Timestamp("1940-04-25"), pd.NaT]}) ``pandas-log`` will give you feedback, for instance when filtering a data frame or adding a new variable: .. code-block:: python df.assign(toy=lambda x: x.toy.map(str.lower)) .query("name != 'Batman'") ``pandas-log`` can be especially helpful in longer pipes: .. code-block:: python df.assign(toy=lambda x: x.toy.map(str.lower)) .query("name != 'Batman'") .dropna()\ .assign(lower_name=lambda x: x.name.map(str.lower)) .reset_index() For medium article `go here `_ For a full walkthrough `go here `_ Installation ------------ ``pandas-log`` is currently installable from PyPI: .. code-block:: bash pip install pandas-log Contributing ------------ Follow `contribution docs `_ for a full description of the process of contributing to ``pandas-log``. ================================================ FILE: TODO.rst ================================================ - Add statistics from tidylog from it. - add formats of warnings. - todo check if copy is first - add logs for series as well. ================================================ FILE: docs/Makefile ================================================ # Minimal makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = python -msphinx SPHINXPROJ = pandas_log SOURCEDIR = . BUILDDIR = _build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) ================================================ FILE: docs/conf.py ================================================ #!/usr/bin/env python # -*- coding: utf-8 -*- # # pandas_log documentation build configuration file, created by # sphinx-quickstart on Fri Jun 9 13:47:02 2017. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. # If extensions (or modules to document with autodoc) are in another # directory, add these directories to sys.path here. If the directory is # relative to the documentation root, use os.path.abspath to make it # absolute, like shown here. # import os import sys sys.path.insert(0, os.path.abspath('..')) # import pandas_log # -- General configuration --------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. # # needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] source_suffix = '.rst' # The master toctree document. master_doc = 'index' # General information about the project. project = u'pandas-log' copyright = u"2019, Eyal Trabelsi" author = u"Eyal Trabelsi" # The version info for the project you're documenting, acts as replacement # for |version| and |release|, also used in various other places throughout # the built documents. # # The short X.Y version. version = '0.1.1' # The full version, including alpha/beta/rc tags. release = '0.1.1' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = None # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = False # -- Options for HTML output ------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = 'alabaster' # Theme options are theme-specific and customize the look and feel of a # theme further. For a list of options available for each theme, see the # documentation. # # html_theme_options = {} # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # -- Options for HTMLHelp output --------------------------------------- # Output file base name for HTML help builder. htmlhelp_basename = 'pandas_logdoc' # -- Options for LaTeX output ------------------------------------------ latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # # 'preamble': '', # Latex figure (float) alignment # # 'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass # [howto, manual, or own class]). latex_documents = [ (master_doc, 'pandas_log.tex', u'pandas-log Documentation', u'Eyal Trabelsi', 'manual'), ] # -- Options for manual page output ------------------------------------ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ (master_doc, 'pandas_log', u'pandas-log Documentation', [author], 1) ] # -- Options for Texinfo output ---------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ (master_doc, 'pandas_log', u'pandas-log Documentation', author, 'pandas_log', 'One line description of project.', 'Miscellaneous'), ] ================================================ FILE: docs/contributing.rst ================================================ .. include:: ../CONTRIBUTING.rst ================================================ FILE: docs/index.rst ================================================ Welcome to pandas-log's documentation! ====================================== .. toctree:: :maxdepth: 2 :caption: Contents: readme installation usage modules contributing Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` ================================================ FILE: docs/installation.rst ================================================ .. highlight:: shell ============ Installation ============ Stable release -------------- To install pandas-log, run this command in your terminal: .. code-block:: console $ pip install pandas-log This is the preferred method to install pandas-log, as it will always install the most recent stable release. If you don't have `pip`_ installed, this `Python installation guide`_ can guide you through the process. .. _pip: https://pip.pypa.io .. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/ From sources ------------ The sources for pandas-log can be downloaded from the `Github repo`_. You can either clone the public repository: .. code-block:: console $ git clone git://github.com/eyaltrabelsi/pandas-log Or download the `tarball`_: .. code-block:: console $ curl -OJL https://github.com/eyaltrabelsi/pandas-log/tarball/master Once you have a copy of the source, you can install it with: .. code-block:: console $ python setup.py install .. _Github repo: https://github.com/eyaltrabelsi/pandas-log .. _tarball: https://github.com/eyaltrabelsi/pandas-log/tarball/master ================================================ FILE: docs/modules.rst ================================================ pandas_log ========== .. toctree:: :maxdepth: 4 pandas_log ================================================ FILE: docs/pandas_log.rst ================================================ pandas\_log package =================== Submodules ---------- pandas\_log.aop\_utils module ----------------------------- .. automodule:: pandas_log.aop_utils :members: :undoc-members: :show-inheritance: pandas\_log.dataframe\_logger module ------------------------------------ .. automodule:: pandas_log.dataframe_logger :members: :undoc-members: :show-inheritance: pandas\_log.pandas\_log module ------------------------------ .. automodule:: pandas_log.pandas_log :members: :undoc-members: :show-inheritance: pandas\_log.patched\_logs\_functions module ------------------------------------------- .. automodule:: pandas_log.patched_logs_functions :members: :undoc-members: :show-inheritance: pandas\_log.settings module --------------------------- .. automodule:: pandas_log.settings :members: :undoc-members: :show-inheritance: pandas\_log.timer module ------------------------ .. automodule:: pandas_log.timer :members: :undoc-members: :show-inheritance: Module contents --------------- .. automodule:: pandas_log :members: :undoc-members: :show-inheritance: ================================================ FILE: docs/readme.rst ================================================ =============== Why pandas-log? =============== ``Pandas-log`` is a Python implementation of the R package ``tidylog``, and provides a feedback about basic pandas operations. The pandas has been invaluable for the data science ecosystem and usually consists of a series of steps that involve transforming raw data into an understandable/usable format. These series of steps need to be run in a certain sequence and if the result is unexpected it's hard to understand what happened. ``Pandas-log`` log metadata on each operation which will allow to pinpoint the issues like: - Wrong predicate expressions. - Copying of our DataFrames. - Wrong joins/merge. - Performance Issues. - More For medium article `go here `_ For a full walkthrough `go here `_ ================================================ FILE: docs/usage.rst ================================================ Usage ===== For a cleaner use-case I would `go here `_ First we need to load some libraries including pandas-log ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: ipython3 import pandas as pd import numpy as np import pandas_log Let’s take a look at our dataset: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: ipython3 df = pd.read_csv("pokemon.csv") df.head(10) .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation legendary
0 1 Bulbasaur Grass Poison 318 45 49 49 65 65 45 1 False
1 2 Ivysaur Grass Poison 405 60 62 63 80 80 60 1 False
2 3 Venusaur Grass Poison 525 80 82 83 100 100 80 1 False
3 3 VenusaurMega Venusaur Grass Poison 625 80 100 123 122 120 80 1 False
4 4 Charmander Fire NaN 309 39 52 43 60 50 65 1 False
5 5 Charmeleon Fire NaN 405 58 64 58 80 65 80 1 False
6 6 Charizard Fire Flying 534 78 84 78 109 85 100 1 False
7 6 CharizardMega Charizard X Fire Dragon 634 78 130 111 130 85 100 1 False
8 6 CharizardMega Charizard Y Fire Flying 634 78 104 78 159 115 100 1 False
9 7 Squirtle Water NaN 314 44 48 65 50 64 43 1 False
Lets say we want to find out: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Who is the weakest non-legendary fire pokemon? ---------------------------------------------- The strategy will probably be something like: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. Filter out legendary pokemons using ``.query()`` . 2. Keep only fire pokemons using ``.query()`` . 3. Drop Legendary column using ``.drop()`` . 4. Keep the weakest pokemon among them using ``.nsmallest()`` . 5. Reset index using ``.reset_index()`` . .. code:: ipython3 res = (df.copy() .query("legendary==0") .query("type_1=='fire' or type_2=='fire'") .drop("legendary", axis=1) .nsmallest(1,"total") .reset_index(drop=True) ) res .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation
OH NOO!!! Our code does not work !! We got no records ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If only there was a way to track those issue ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Fortunetly thats what **pandas-log** is for! either as a global function or context manager. This the example with pandas_log’s ``context_manager``. .. code:: ipython3 with pandas_log.enable(): res = (df.query("legendary==0") .query("type_1=='fire' or type_2=='fire'") .drop("legendary", axis=1) .nsmallest(1,"total") ) res .. parsed-literal:: 1) query(expr="legendary==0", inplace=False): Metadata: * Removed 65 rows (8.125%), 735 rows remaining. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 199.4 kB. * Output Dataframe size is 188.5 kB. 2) query(expr="type_1=='fire' or type_2=='fire'", inplace=False): Metadata: * Removed 735 rows (100.0%), 0 rows remaining. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 188.5 kB. * Output Dataframe size is 0 Bytes. 3) drop(labels="legendary", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'): Metadata: * Removed the following columns (legendary) now only have the following columns (attack, sp_def, speed, hp, total, type_2, #, name, type_1, generation, defense, sp_atk). * No change in number of rows of input df. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 0 Bytes. * Output Dataframe size is 0 Bytes. 4) nsmallest(n=1, columns="total", keep='first'): Metadata: * Picked 1 smallest rows by columns (total). Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 0 Bytes. * Output Dataframe size is 0 Bytes. .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation
We can see clearly that in the second step (``.query()``) we filter all the rows!! and indeed we should of writen Fire as oppose to fire ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: ipython3 res = (df.copy() .query("type_1=='Fire' or type_2=='Fire'") .query("legendary==0") .drop("legendary", axis=1) .nsmallest(1,"total") .reset_index(drop=True) ) res .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation
0 218 Slugma Fire NaN 250 40 40 40 70 40 20 2
Whoala we got Slugma !!!!!!!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some more advance usage ----------------------- One can use verbose variable which allows lower level logs functionalities like whether the dataframe was copied as part of pipeline. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This can explain comparision issues. .. code:: ipython3 with pandas_log.enable(verbose=True): res = (df.query("legendary==0") .query("type_1=='Fire' or type_2=='Fire'") .drop("legendary", axis=1) .nsmallest(1,"total") .reset_index(drop=True) ) res .. parsed-literal:: 1) query(expr="legendary==0", inplace=False): Metadata: * Removed 65 rows (8.125%), 735 rows remaining. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 199.4 kB. * Output Dataframe size is 188.5 kB. 2) query(expr="type_1=='Fire' or type_2=='Fire'", inplace=False): Metadata: * Removed 679 rows (92.38095238095238%), 56 rows remaining. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 188.5 kB. * Output Dataframe size is 14.4 kB. 3) drop(labels="legendary", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'): Metadata: * Removed the following columns (legendary) now only have the following columns (attack, sp_def, speed, hp, total, type_2, #, name, type_1, generation, defense, sp_atk). * No change in number of rows of input df. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.4 kB. * Output Dataframe size is 14.3 kB. X) __getitem__(key="total"): Metadata: Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.3 kB. * Output Dataframe size is 896 Bytes. X) copy(deep=True): Metadata: * Using default strategy (some metric might not be relevant). Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.3 kB. * Output Dataframe size is 14.3 kB. X) reset_index(level=None, drop=False, inplace=False, col_level=0, col_fill=''): Metadata: Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.3 kB. * Output Dataframe size is 14.0 kB. X) __getitem__(key="total"): Metadata: Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.0 kB. * Output Dataframe size is 576 Bytes. 4) nsmallest(n=1, columns="total", keep='first'): Metadata: * Picked 1 smallest rows by columns (total). Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.3 kB. * Output Dataframe size is 236 Bytes. X) copy(deep=True): Metadata: * Using default strategy (some metric might not be relevant). Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 236 Bytes. * Output Dataframe size is 236 Bytes. X) reset_index(level=None, drop=False, inplace=False, col_level=0, col_fill=''): Metadata: Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 236 Bytes. * Output Dataframe size is 356 Bytes. .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation
0 218 Slugma Fire NaN 250 40 40 40 70 40 20 2
as we can see after both the drop and nsmallest functions the dataframe was being copied One can use silent variable which allows to suppress stdout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: ipython3 with pandas_log.enable(silent=True): res = (df.copy() .query("legendary==0") .query("type_1=='Fire' or type_2=='Fire'") .drop("legendary", axis=1) .nsmallest(1,"total") .reset_index(drop=True) ) res .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation
0 218 Slugma Fire NaN 250 40 40 40 70 40 20 2
One can use full_signature variable which allows to suppress the signature ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: ipython3 with pandas_log.enable(full_signature=False): res = (df.query("legendary==0") .query("type_1=='Fire' or type_2=='Fire'") .drop("legendary", axis=1) .nsmallest(1,"total") .reset_index(drop=True) ) res .. parsed-literal:: 1) query(expr="legendary==0", inplace=False): Metadata: * Removed 65 rows (8.125%), 735 rows remaining. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 199.4 kB. * Output Dataframe size is 188.5 kB. 2) query(expr="type_1=='Fire' or type_2=='Fire'"): Metadata: * Removed 679 rows (92.38095238095238%), 56 rows remaining. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 188.5 kB. * Output Dataframe size is 14.4 kB. 3) drop(labels="legendary"): Metadata: * Removed the following columns (legendary) now only have the following columns (attack, sp_def, speed, hp, total, type_2, #, name, type_1, generation, defense, sp_atk). * No change in number of rows of input df. Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.4 kB. * Output Dataframe size is 14.3 kB. 4) nsmallest(n=1, columns="total"): Metadata: * Picked 1 smallest rows by columns (total). Execution Stats: * Execution time: Step Took a moment seconds.. * Input Dataframe size is 14.3 kB. * Output Dataframe size is 236 Bytes. .. raw:: html
# name type_1 type_2 total hp attack defense sp_atk sp_def speed generation
0 218 Slugma Fire NaN 250 40 40 40 70 40 20 2
================================================ FILE: examples/README.rst ================================================ Examples ======== This folder contains jupyter notebooks demonstrating different ways to implement pandas-log in your workflow. Guidelines ~~~~~~~~~~ When contributing example notebooks please include a short explanation of where the data came from and what it contains ================================================ FILE: examples/__init__.py ================================================ ================================================ FILE: examples/pandas_log_intro.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Pandas-Log Usage Walkthrough" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why pandas-log?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pandas-log is a Python implementation of the R package tidylog, and provides a feedback about basic pandas operations.\n", "\n", "The pandas has been invaluable for the data science ecosystem and usually consists of a series of steps that involve transforming raw data into an understandable/usable format. These series of steps need to be run in a certain sequence and if the result is unexpected it's hard to understand what happened. Pandas-log log metadata on each operation which will allow to pinpoint the issues." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pandas-log Demo\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### First we need to load some libraries including pandas-log" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import pandas_log " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Let's take a look at our dataset:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgenerationlegendary
01BulbasaurGrassPoison3184549496565451False
12IvysaurGrassPoison4056062638080601False
23VenusaurGrassPoison525808283100100801False
33VenusaurMega VenusaurGrassPoison62580100123122120801False
44CharmanderFireNaN3093952436050651False
55CharmeleonFireNaN4055864588065801False
66CharizardFireFlying534788478109851001False
76CharizardMega Charizard XFireDragon63478130111130851001False
86CharizardMega Charizard YFireFlying63478104781591151001False
97SquirtleWaterNaN3144448655064431False
\n", "
" ], "text/plain": [ " # name type_1 type_2 total hp attack defense \\\n", "0 1 Bulbasaur Grass Poison 318 45 49 49 \n", "1 2 Ivysaur Grass Poison 405 60 62 63 \n", "2 3 Venusaur Grass Poison 525 80 82 83 \n", "3 3 VenusaurMega Venusaur Grass Poison 625 80 100 123 \n", "4 4 Charmander Fire NaN 309 39 52 43 \n", "5 5 Charmeleon Fire NaN 405 58 64 58 \n", "6 6 Charizard Fire Flying 534 78 84 78 \n", "7 6 CharizardMega Charizard X Fire Dragon 634 78 130 111 \n", "8 6 CharizardMega Charizard Y Fire Flying 634 78 104 78 \n", "9 7 Squirtle Water NaN 314 44 48 65 \n", "\n", " sp_atk sp_def speed generation legendary \n", "0 65 65 45 1 False \n", "1 80 80 60 1 False \n", "2 100 100 80 1 False \n", "3 122 120 80 1 False \n", "4 60 50 65 1 False \n", "5 80 65 80 1 False \n", "6 109 85 100 1 False \n", "7 130 85 100 1 False \n", "8 159 115 100 1 False \n", "9 50 64 43 1 False " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_csv(\"pokemon.csv\")\n", "df.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Lets say we want to find out:\n", "## Who is the weakest non-legendary fire pokemon?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### The strategy will probably be something like:\n", "1. Filter out legendary pokemons using `.query()` .\n", "1. Keep only fire pokemons using `.query()` .\n", "1. Drop Legendary column using `.drop()` .\n", "1. Keep the weakest pokemon among them using `.nsmallest()` .\n", "1. Reset index using `.reset_index()` ." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgeneration
\n", "
" ], "text/plain": [ "Empty DataFrame\n", "Columns: [#, name, type_1, type_2, total, hp, attack, defense, sp_atk, sp_def, speed, generation]\n", "Index: []" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = (df.copy()\n", " .query(\"legendary==0\")\n", " .query(\"type_1=='fire' or type_2=='fire'\")\n", " .drop(\"legendary\", axis=1)\n", " .nsmallest(1,\"total\")\n", " .reset_index(drop=True)\n", " )\n", "res " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### OH NOO!!! Our code does not work !! We got no records\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### If only there was a way to track those issue\n", "\n", "Fortunetly thats what **pandas-log** is for! either as a global function or context manager.\n", "This the example with pandas_log's `context_manager`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "1) \u001b[1mquery\u001b[0m(expr=\"legendary==0\", inplace=False):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Removed 65 rows (8.125%), 735 rows remaining.\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.002567 seconds.\n", "\t* Input Dataframe size is 199.4 kB.\n", "\t* Output Dataframe size is 188.5 kB.\n", "\t\n", "\n", "2) \u001b[1mquery\u001b[0m(expr=\"type_1=='fire' or type_2=='fire'\", inplace=False):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Removed 735 rows (100.0%), 0 rows remaining.\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.003322 seconds.\n", "\t* Input Dataframe size is 188.5 kB.\n", "\t* Output Dataframe size is 0 Bytes.\n", "\t\n", "\n", "3) \u001b[1mdrop\u001b[0m(labels=\"legendary\", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* No change in number of rows of input df.\n", "\t* Removed the following columns (legendary) now only have the following columns (type_2, total, hp, defense, sp_def, speed, generation, sp_atk, type_1, attack, name, #).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.00073 seconds.\n", "\t* Input Dataframe size is 0 Bytes.\n", "\t* Output Dataframe size is 0 Bytes.\n", "\t\n", "\n", "4) \u001b[1mnsmallest\u001b[0m(n=1, columns=\"total\", keep='first'):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Picked 1 smallest rows by columns (total).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.010866 seconds.\n", "\t* Input Dataframe size is 0 Bytes.\n", "\t* Output Dataframe size is 0 Bytes.\n", "\t\u001b[4mTips\u001b[0m:\n", "\t* Number of rows didn't change, if you are working on the entire dataset you can remove this operation.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgeneration
\n", "
" ], "text/plain": [ "Empty DataFrame\n", "Columns: [#, name, type_1, type_2, total, hp, attack, defense, sp_atk, sp_def, speed, generation]\n", "Index: []" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with pandas_log.enable():\n", " res = (df.query(\"legendary==0\")\n", " .query(\"type_1=='fire' or type_2=='fire'\")\n", " .drop(\"legendary\", axis=1)\n", " .nsmallest(1,\"total\")\n", " )\n", "res " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### We can see clearly that in the second step (`.query()`) we filter all the rows!! and indeed we should of writen Fire as oppose to fire" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgeneration
0218SlugmaFireNaN2504040407040202
\n", "
" ], "text/plain": [ " # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n", "0 218 Slugma Fire NaN 250 40 40 40 70 40 \n", "\n", " speed generation \n", "0 20 2 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = (df.copy()\n", " .query(\"type_1=='Fire' or type_2=='Fire'\")\n", " .query(\"legendary==0\") \n", " .drop(\"legendary\", axis=1) \n", " .nsmallest(1,\"total\")\n", " .reset_index(drop=True)\n", " )\n", "res " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Whoala we got Slugma !!!!!!!!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Some more advance usage\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### One can use verbose variable which allows lower level logs functionalities like whether the dataframe was copied as part of pipeline.\n", "This can explain comparision issues." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "1) \u001b[1mquery\u001b[0m(expr=\"legendary==0\", inplace=False):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Removed 65 rows (8.125%), 735 rows remaining.\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.003288 seconds.\n", "\t* Input Dataframe size is 199.4 kB.\n", "\t* Output Dataframe size is 188.5 kB.\n", "\t\n", "\n", "2) \u001b[1mquery\u001b[0m(expr=\"type_1=='Fire' or type_2=='Fire'\", inplace=False):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Removed 679 rows (92.38095238095238%), 56 rows remaining.\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.003339 seconds.\n", "\t* Input Dataframe size is 188.5 kB.\n", "\t* Output Dataframe size is 14.4 kB.\n", "\t\n", "\n", "3) \u001b[1mdrop\u001b[0m(labels=\"legendary\", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* No change in number of rows of input df.\n", "\t* Removed the following columns (legendary) now only have the following columns (hp, sp_def, generation, name, #, attack, type_2, sp_atk, type_1, defense, total, speed).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.000604 seconds.\n", "\t* Input Dataframe size is 14.4 kB.\n", "\t* Output Dataframe size is 14.3 kB.\n", "\t\u001b[4mTips\u001b[0m:\n", "\t* Number of rows didn't change, if you are working on the entire dataset you can remove this operation.\n", "\n", "X) \u001b[1m__getitem__\u001b[0m(key=\"total\"):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* After transformation we received Series\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 3.3e-05 seconds.\n", "\t* Input Dataframe size is 14.3 kB.\n", "\t* Output Dataframe size is 896 Bytes.\n", "\t\n", "\n", "X) \u001b[1mcopy\u001b[0m(deep=True):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Using default strategy (some metric might not be relevant).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.000318 seconds.\n", "\t* Input Dataframe size is 14.3 kB.\n", "\t* Output Dataframe size is 14.3 kB.\n", "\t\n", "\n", "X) \u001b[1mreset_index\u001b[0m(level=None, drop=False, inplace=False, col_level=0, col_fill=''):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Using default strategy (some metric might not be relevant).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.00373 seconds.\n", "\t* Input Dataframe size is 14.3 kB.\n", "\t* Output Dataframe size is 14.0 kB.\n", "\t\n", "\n", "X) \u001b[1m__getitem__\u001b[0m(key=\"total\"):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* After transformation we received Series\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 1.1e-05 seconds.\n", "\t* Input Dataframe size is 14.0 kB.\n", "\t* Output Dataframe size is 576 Bytes.\n", "\t\n", "\n", "4) \u001b[1mnsmallest\u001b[0m(n=1, columns=\"total\", keep='first'):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Picked 1 smallest rows by columns (total).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.012324 seconds.\n", "\t* Input Dataframe size is 14.3 kB.\n", "\t* Output Dataframe size is 236 Bytes.\n", "\t\n", "\n", "X) \u001b[1mcopy\u001b[0m(deep=True):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Using default strategy (some metric might not be relevant).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.000137 seconds.\n", "\t* Input Dataframe size is 236 Bytes.\n", "\t* Output Dataframe size is 236 Bytes.\n", "\t\n", "\n", "X) \u001b[1mreset_index\u001b[0m(level=None, drop=False, inplace=False, col_level=0, col_fill=''):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Using default strategy (some metric might not be relevant).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.002781 seconds.\n", "\t* Input Dataframe size is 236 Bytes.\n", "\t* Output Dataframe size is 356 Bytes.\n", "\t\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgeneration
0218SlugmaFireNaN2504040407040202
\n", "
" ], "text/plain": [ " # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n", "0 218 Slugma Fire NaN 250 40 40 40 70 40 \n", "\n", " speed generation \n", "0 20 2 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with pandas_log.enable(verbose=True):\n", " res = (df.query(\"legendary==0\")\n", " .query(\"type_1=='Fire' or type_2=='Fire'\")\n", " .drop(\"legendary\", axis=1)\n", " .nsmallest(1,\"total\")\n", " .reset_index(drop=True)\n", " )\n", "res " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "as we can see after both the drop and nsmallest functions the dataframe was being copied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### One can use silent variable which allows to suppress stdout" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgeneration
0218SlugmaFireNaN2504040407040202
\n", "
" ], "text/plain": [ " # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n", "0 218 Slugma Fire NaN 250 40 40 40 70 40 \n", "\n", " speed generation \n", "0 20 2 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with pandas_log.enable(silent=True):\n", " res = (df.copy()\n", " .query(\"legendary==0\")\n", " .query(\"type_1=='Fire' or type_2=='Fire'\")\n", " .drop(\"legendary\", axis=1)\n", " .nsmallest(1,\"total\")\n", " .reset_index(drop=True)\n", " )\n", "res " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### One can use full_signature variable which allows to suppress the signature" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "1) \u001b[1mquery\u001b[0m(expr=\"legendary==0\", inplace=False):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Removed 65 rows (8.125%), 735 rows remaining.\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.001866 seconds.\n", "\t* Input Dataframe size is 199.4 kB.\n", "\t* Output Dataframe size is 188.5 kB.\n", "\t\n", "\n", "2) \u001b[1mquery\u001b[0m(expr=\"type_1=='Fire' or type_2=='Fire'\"):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Removed 679 rows (92.38095238095238%), 56 rows remaining.\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.002914 seconds.\n", "\t* Input Dataframe size is 188.5 kB.\n", "\t* Output Dataframe size is 14.4 kB.\n", "\t\n", "\n", "3) \u001b[1mdrop\u001b[0m(labels=\"legendary\"):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* No change in number of rows of input df.\n", "\t* Removed the following columns (legendary) now only have the following columns (hp, sp_def, generation, name, #, attack, type_2, sp_atk, type_1, defense, total, speed).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.000561 seconds.\n", "\t* Input Dataframe size is 14.4 kB.\n", "\t* Output Dataframe size is 14.3 kB.\n", "\t\u001b[4mTips\u001b[0m:\n", "\t* Number of rows didn't change, if you are working on the entire dataset you can remove this operation.\n", "\n", "4) \u001b[1mnsmallest\u001b[0m(n=1, columns=\"total\"):\n", "\t\u001b[4mMetadata\u001b[0m:\n", "\t* Picked 1 smallest rows by columns (total).\n", "\t\u001b[4mExecution Stats\u001b[0m:\n", "\t* Execution time: Step Took 0.008674 seconds.\n", "\t* Input Dataframe size is 14.3 kB.\n", "\t* Output Dataframe size is 236 Bytes.\n", "\t\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
#nametype_1type_2totalhpattackdefensesp_atksp_defspeedgeneration
0218SlugmaFireNaN2504040407040202
\n", "
" ], "text/plain": [ " # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n", "0 218 Slugma Fire NaN 250 40 40 40 70 40 \n", "\n", " speed generation \n", "0 20 2 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with pandas_log.enable(full_signature=False):\n", " res = (df.query(\"legendary==0\")\n", " .query(\"type_1=='Fire' or type_2=='Fire'\")\n", " .drop(\"legendary\", axis=1)\n", " .nsmallest(1,\"total\")\n", " .reset_index(drop=True)\n", " )\n", "res " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: examples/pokemon.csv ================================================ #,name,type_1,type_2,total,hp,attack,defense,sp_atk,sp_def,speed,generation,legendary 1,Bulbasaur,Grass,Poison,318,45,49,49,65,65,45,1,False 2,Ivysaur,Grass,Poison,405,60,62,63,80,80,60,1,False 3,Venusaur,Grass,Poison,525,80,82,83,100,100,80,1,False 3,VenusaurMega Venusaur,Grass,Poison,625,80,100,123,122,120,80,1,False 4,Charmander,Fire,,309,39,52,43,60,50,65,1,False 5,Charmeleon,Fire,,405,58,64,58,80,65,80,1,False 6,Charizard,Fire,Flying,534,78,84,78,109,85,100,1,False 6,CharizardMega Charizard X,Fire,Dragon,634,78,130,111,130,85,100,1,False 6,CharizardMega Charizard Y,Fire,Flying,634,78,104,78,159,115,100,1,False 7,Squirtle,Water,,314,44,48,65,50,64,43,1,False 8,Wartortle,Water,,405,59,63,80,65,80,58,1,False 9,Blastoise,Water,,530,79,83,100,85,105,78,1,False 9,BlastoiseMega Blastoise,Water,,630,79,103,120,135,115,78,1,False 10,Caterpie,Bug,,195,45,30,35,20,20,45,1,False 11,Metapod,Bug,,205,50,20,55,25,25,30,1,False 12,Butterfree,Bug,Flying,395,60,45,50,90,80,70,1,False 13,Weedle,Bug,Poison,195,40,35,30,20,20,50,1,False 14,Kakuna,Bug,Poison,205,45,25,50,25,25,35,1,False 15,Beedrill,Bug,Poison,395,65,90,40,45,80,75,1,False 15,BeedrillMega Beedrill,Bug,Poison,495,65,150,40,15,80,145,1,False 16,Pidgey,Normal,Flying,251,40,45,40,35,35,56,1,False 17,Pidgeotto,Normal,Flying,349,63,60,55,50,50,71,1,False 18,Pidgeot,Normal,Flying,479,83,80,75,70,70,101,1,False 18,PidgeotMega Pidgeot,Normal,Flying,579,83,80,80,135,80,121,1,False 19,Rattata,Normal,,253,30,56,35,25,35,72,1,False 20,Raticate,Normal,,413,55,81,60,50,70,97,1,False 21,Spearow,Normal,Flying,262,40,60,30,31,31,70,1,False 22,Fearow,Normal,Flying,442,65,90,65,61,61,100,1,False 23,Ekans,Poison,,288,35,60,44,40,54,55,1,False 24,Arbok,Poison,,438,60,85,69,65,79,80,1,False 25,Pikachu,Electric,,320,35,55,40,50,50,90,1,False 26,Raichu,Electric,,485,60,90,55,90,80,110,1,False 27,Sandshrew,Ground,,300,50,75,85,20,30,40,1,False 28,Sandslash,Ground,,450,75,100,110,45,55,65,1,False 29,Nidoran♀,Poison,,275,55,47,52,40,40,41,1,False 30,Nidorina,Poison,,365,70,62,67,55,55,56,1,False 31,Nidoqueen,Poison,Ground,505,90,92,87,75,85,76,1,False 32,Nidoran♂,Poison,,273,46,57,40,40,40,50,1,False 33,Nidorino,Poison,,365,61,72,57,55,55,65,1,False 34,Nidoking,Poison,Ground,505,81,102,77,85,75,85,1,False 35,Clefairy,Fairy,,323,70,45,48,60,65,35,1,False 36,Clefable,Fairy,,483,95,70,73,95,90,60,1,False 37,Vulpix,Fire,,299,38,41,40,50,65,65,1,False 38,Ninetales,Fire,,505,73,76,75,81,100,100,1,False 39,Jigglypuff,Normal,Fairy,270,115,45,20,45,25,20,1,False 40,Wigglytuff,Normal,Fairy,435,140,70,45,85,50,45,1,False 41,Zubat,Poison,Flying,245,40,45,35,30,40,55,1,False 42,Golbat,Poison,Flying,455,75,80,70,65,75,90,1,False 43,Oddish,Grass,Poison,320,45,50,55,75,65,30,1,False 44,Gloom,Grass,Poison,395,60,65,70,85,75,40,1,False 45,Vileplume,Grass,Poison,490,75,80,85,110,90,50,1,False 46,Paras,Bug,Grass,285,35,70,55,45,55,25,1,False 47,Parasect,Bug,Grass,405,60,95,80,60,80,30,1,False 48,Venonat,Bug,Poison,305,60,55,50,40,55,45,1,False 49,Venomoth,Bug,Poison,450,70,65,60,90,75,90,1,False 50,Diglett,Ground,,265,10,55,25,35,45,95,1,False 51,Dugtrio,Ground,,405,35,80,50,50,70,120,1,False 52,Meowth,Normal,,290,40,45,35,40,40,90,1,False 53,Persian,Normal,,440,65,70,60,65,65,115,1,False 54,Psyduck,Water,,320,50,52,48,65,50,55,1,False 55,Golduck,Water,,500,80,82,78,95,80,85,1,False 56,Mankey,Fighting,,305,40,80,35,35,45,70,1,False 57,Primeape,Fighting,,455,65,105,60,60,70,95,1,False 58,Growlithe,Fire,,350,55,70,45,70,50,60,1,False 59,Arcanine,Fire,,555,90,110,80,100,80,95,1,False 60,Poliwag,Water,,300,40,50,40,40,40,90,1,False 61,Poliwhirl,Water,,385,65,65,65,50,50,90,1,False 62,Poliwrath,Water,Fighting,510,90,95,95,70,90,70,1,False 63,Abra,Psychic,,310,25,20,15,105,55,90,1,False 64,Kadabra,Psychic,,400,40,35,30,120,70,105,1,False 65,Alakazam,Psychic,,500,55,50,45,135,95,120,1,False 65,AlakazamMega Alakazam,Psychic,,590,55,50,65,175,95,150,1,False 66,Machop,Fighting,,305,70,80,50,35,35,35,1,False 67,Machoke,Fighting,,405,80,100,70,50,60,45,1,False 68,Machamp,Fighting,,505,90,130,80,65,85,55,1,False 69,Bellsprout,Grass,Poison,300,50,75,35,70,30,40,1,False 70,Weepinbell,Grass,Poison,390,65,90,50,85,45,55,1,False 71,Victreebel,Grass,Poison,490,80,105,65,100,70,70,1,False 72,Tentacool,Water,Poison,335,40,40,35,50,100,70,1,False 73,Tentacruel,Water,Poison,515,80,70,65,80,120,100,1,False 74,Geodude,Rock,Ground,300,40,80,100,30,30,20,1,False 75,Graveler,Rock,Ground,390,55,95,115,45,45,35,1,False 76,Golem,Rock,Ground,495,80,120,130,55,65,45,1,False 77,Ponyta,Fire,,410,50,85,55,65,65,90,1,False 78,Rapidash,Fire,,500,65,100,70,80,80,105,1,False 79,Slowpoke,Water,Psychic,315,90,65,65,40,40,15,1,False 80,Slowbro,Water,Psychic,490,95,75,110,100,80,30,1,False 80,SlowbroMega Slowbro,Water,Psychic,590,95,75,180,130,80,30,1,False 81,Magnemite,Electric,Steel,325,25,35,70,95,55,45,1,False 82,Magneton,Electric,Steel,465,50,60,95,120,70,70,1,False 83,Farfetch'd,Normal,Flying,352,52,65,55,58,62,60,1,False 84,Doduo,Normal,Flying,310,35,85,45,35,35,75,1,False 85,Dodrio,Normal,Flying,460,60,110,70,60,60,100,1,False 86,Seel,Water,,325,65,45,55,45,70,45,1,False 87,Dewgong,Water,Ice,475,90,70,80,70,95,70,1,False 88,Grimer,Poison,,325,80,80,50,40,50,25,1,False 89,Muk,Poison,,500,105,105,75,65,100,50,1,False 90,Shellder,Water,,305,30,65,100,45,25,40,1,False 91,Cloyster,Water,Ice,525,50,95,180,85,45,70,1,False 92,Gastly,Ghost,Poison,310,30,35,30,100,35,80,1,False 93,Haunter,Ghost,Poison,405,45,50,45,115,55,95,1,False 94,Gengar,Ghost,Poison,500,60,65,60,130,75,110,1,False 94,GengarMega Gengar,Ghost,Poison,600,60,65,80,170,95,130,1,False 95,Onix,Rock,Ground,385,35,45,160,30,45,70,1,False 96,Drowzee,Psychic,,328,60,48,45,43,90,42,1,False 97,Hypno,Psychic,,483,85,73,70,73,115,67,1,False 98,Krabby,Water,,325,30,105,90,25,25,50,1,False 99,Kingler,Water,,475,55,130,115,50,50,75,1,False 100,Voltorb,Electric,,330,40,30,50,55,55,100,1,False 101,Electrode,Electric,,480,60,50,70,80,80,140,1,False 102,Exeggcute,Grass,Psychic,325,60,40,80,60,45,40,1,False 103,Exeggutor,Grass,Psychic,520,95,95,85,125,65,55,1,False 104,Cubone,Ground,,320,50,50,95,40,50,35,1,False 105,Marowak,Ground,,425,60,80,110,50,80,45,1,False 106,Hitmonlee,Fighting,,455,50,120,53,35,110,87,1,False 107,Hitmonchan,Fighting,,455,50,105,79,35,110,76,1,False 108,Lickitung,Normal,,385,90,55,75,60,75,30,1,False 109,Koffing,Poison,,340,40,65,95,60,45,35,1,False 110,Weezing,Poison,,490,65,90,120,85,70,60,1,False 111,Rhyhorn,Ground,Rock,345,80,85,95,30,30,25,1,False 112,Rhydon,Ground,Rock,485,105,130,120,45,45,40,1,False 113,Chansey,Normal,,450,250,5,5,35,105,50,1,False 114,Tangela,Grass,,435,65,55,115,100,40,60,1,False 115,Kangaskhan,Normal,,490,105,95,80,40,80,90,1,False 115,KangaskhanMega Kangaskhan,Normal,,590,105,125,100,60,100,100,1,False 116,Horsea,Water,,295,30,40,70,70,25,60,1,False 117,Seadra,Water,,440,55,65,95,95,45,85,1,False 118,Goldeen,Water,,320,45,67,60,35,50,63,1,False 119,Seaking,Water,,450,80,92,65,65,80,68,1,False 120,Staryu,Water,,340,30,45,55,70,55,85,1,False 121,Starmie,Water,Psychic,520,60,75,85,100,85,115,1,False 122,Mr. Mime,Psychic,Fairy,460,40,45,65,100,120,90,1,False 123,Scyther,Bug,Flying,500,70,110,80,55,80,105,1,False 124,Jynx,Ice,Psychic,455,65,50,35,115,95,95,1,False 125,Electabuzz,Electric,,490,65,83,57,95,85,105,1,False 126,Magmar,Fire,,495,65,95,57,100,85,93,1,False 127,Pinsir,Bug,,500,65,125,100,55,70,85,1,False 127,PinsirMega Pinsir,Bug,Flying,600,65,155,120,65,90,105,1,False 128,Tauros,Normal,,490,75,100,95,40,70,110,1,False 129,Magikarp,Water,,200,20,10,55,15,20,80,1,False 130,Gyarados,Water,Flying,540,95,125,79,60,100,81,1,False 130,GyaradosMega Gyarados,Water,Dark,640,95,155,109,70,130,81,1,False 131,Lapras,Water,Ice,535,130,85,80,85,95,60,1,False 132,Ditto,Normal,,288,48,48,48,48,48,48,1,False 133,Eevee,Normal,,325,55,55,50,45,65,55,1,False 134,Vaporeon,Water,,525,130,65,60,110,95,65,1,False 135,Jolteon,Electric,,525,65,65,60,110,95,130,1,False 136,Flareon,Fire,,525,65,130,60,95,110,65,1,False 137,Porygon,Normal,,395,65,60,70,85,75,40,1,False 138,Omanyte,Rock,Water,355,35,40,100,90,55,35,1,False 139,Omastar,Rock,Water,495,70,60,125,115,70,55,1,False 140,Kabuto,Rock,Water,355,30,80,90,55,45,55,1,False 141,Kabutops,Rock,Water,495,60,115,105,65,70,80,1,False 142,Aerodactyl,Rock,Flying,515,80,105,65,60,75,130,1,False 142,AerodactylMega Aerodactyl,Rock,Flying,615,80,135,85,70,95,150,1,False 143,Snorlax,Normal,,540,160,110,65,65,110,30,1,False 144,Articuno,Ice,Flying,580,90,85,100,95,125,85,1,True 145,Zapdos,Electric,Flying,580,90,90,85,125,90,100,1,True 146,Moltres,Fire,Flying,580,90,100,90,125,85,90,1,True 147,Dratini,Dragon,,300,41,64,45,50,50,50,1,False 148,Dragonair,Dragon,,420,61,84,65,70,70,70,1,False 149,Dragonite,Dragon,Flying,600,91,134,95,100,100,80,1,False 150,Mewtwo,Psychic,,680,106,110,90,154,90,130,1,True 150,MewtwoMega Mewtwo X,Psychic,Fighting,780,106,190,100,154,100,130,1,True 150,MewtwoMega Mewtwo Y,Psychic,,780,106,150,70,194,120,140,1,True 151,Mew,Psychic,,600,100,100,100,100,100,100,1,False 152,Chikorita,Grass,,318,45,49,65,49,65,45,2,False 153,Bayleef,Grass,,405,60,62,80,63,80,60,2,False 154,Meganium,Grass,,525,80,82,100,83,100,80,2,False 155,Cyndaquil,Fire,,309,39,52,43,60,50,65,2,False 156,Quilava,Fire,,405,58,64,58,80,65,80,2,False 157,Typhlosion,Fire,,534,78,84,78,109,85,100,2,False 158,Totodile,Water,,314,50,65,64,44,48,43,2,False 159,Croconaw,Water,,405,65,80,80,59,63,58,2,False 160,Feraligatr,Water,,530,85,105,100,79,83,78,2,False 161,Sentret,Normal,,215,35,46,34,35,45,20,2,False 162,Furret,Normal,,415,85,76,64,45,55,90,2,False 163,Hoothoot,Normal,Flying,262,60,30,30,36,56,50,2,False 164,Noctowl,Normal,Flying,442,100,50,50,76,96,70,2,False 165,Ledyba,Bug,Flying,265,40,20,30,40,80,55,2,False 166,Ledian,Bug,Flying,390,55,35,50,55,110,85,2,False 167,Spinarak,Bug,Poison,250,40,60,40,40,40,30,2,False 168,Ariados,Bug,Poison,390,70,90,70,60,60,40,2,False 169,Crobat,Poison,Flying,535,85,90,80,70,80,130,2,False 170,Chinchou,Water,Electric,330,75,38,38,56,56,67,2,False 171,Lanturn,Water,Electric,460,125,58,58,76,76,67,2,False 172,Pichu,Electric,,205,20,40,15,35,35,60,2,False 173,Cleffa,Fairy,,218,50,25,28,45,55,15,2,False 174,Igglybuff,Normal,Fairy,210,90,30,15,40,20,15,2,False 175,Togepi,Fairy,,245,35,20,65,40,65,20,2,False 176,Togetic,Fairy,Flying,405,55,40,85,80,105,40,2,False 177,Natu,Psychic,Flying,320,40,50,45,70,45,70,2,False 178,Xatu,Psychic,Flying,470,65,75,70,95,70,95,2,False 179,Mareep,Electric,,280,55,40,40,65,45,35,2,False 180,Flaaffy,Electric,,365,70,55,55,80,60,45,2,False 181,Ampharos,Electric,,510,90,75,85,115,90,55,2,False 181,AmpharosMega Ampharos,Electric,Dragon,610,90,95,105,165,110,45,2,False 182,Bellossom,Grass,,490,75,80,95,90,100,50,2,False 183,Marill,Water,Fairy,250,70,20,50,20,50,40,2,False 184,Azumarill,Water,Fairy,420,100,50,80,60,80,50,2,False 185,Sudowoodo,Rock,,410,70,100,115,30,65,30,2,False 186,Politoed,Water,,500,90,75,75,90,100,70,2,False 187,Hoppip,Grass,Flying,250,35,35,40,35,55,50,2,False 188,Skiploom,Grass,Flying,340,55,45,50,45,65,80,2,False 189,Jumpluff,Grass,Flying,460,75,55,70,55,95,110,2,False 190,Aipom,Normal,,360,55,70,55,40,55,85,2,False 191,Sunkern,Grass,,180,30,30,30,30,30,30,2,False 192,Sunflora,Grass,,425,75,75,55,105,85,30,2,False 193,Yanma,Bug,Flying,390,65,65,45,75,45,95,2,False 194,Wooper,Water,Ground,210,55,45,45,25,25,15,2,False 195,Quagsire,Water,Ground,430,95,85,85,65,65,35,2,False 196,Espeon,Psychic,,525,65,65,60,130,95,110,2,False 197,Umbreon,Dark,,525,95,65,110,60,130,65,2,False 198,Murkrow,Dark,Flying,405,60,85,42,85,42,91,2,False 199,Slowking,Water,Psychic,490,95,75,80,100,110,30,2,False 200,Misdreavus,Ghost,,435,60,60,60,85,85,85,2,False 201,Unown,Psychic,,336,48,72,48,72,48,48,2,False 202,Wobbuffet,Psychic,,405,190,33,58,33,58,33,2,False 203,Girafarig,Normal,Psychic,455,70,80,65,90,65,85,2,False 204,Pineco,Bug,,290,50,65,90,35,35,15,2,False 205,Forretress,Bug,Steel,465,75,90,140,60,60,40,2,False 206,Dunsparce,Normal,,415,100,70,70,65,65,45,2,False 207,Gligar,Ground,Flying,430,65,75,105,35,65,85,2,False 208,Steelix,Steel,Ground,510,75,85,200,55,65,30,2,False 208,SteelixMega Steelix,Steel,Ground,610,75,125,230,55,95,30,2,False 209,Snubbull,Fairy,,300,60,80,50,40,40,30,2,False 210,Granbull,Fairy,,450,90,120,75,60,60,45,2,False 211,Qwilfish,Water,Poison,430,65,95,75,55,55,85,2,False 212,Scizor,Bug,Steel,500,70,130,100,55,80,65,2,False 212,ScizorMega Scizor,Bug,Steel,600,70,150,140,65,100,75,2,False 213,Shuckle,Bug,Rock,505,20,10,230,10,230,5,2,False 214,Heracross,Bug,Fighting,500,80,125,75,40,95,85,2,False 214,HeracrossMega Heracross,Bug,Fighting,600,80,185,115,40,105,75,2,False 215,Sneasel,Dark,Ice,430,55,95,55,35,75,115,2,False 216,Teddiursa,Normal,,330,60,80,50,50,50,40,2,False 217,Ursaring,Normal,,500,90,130,75,75,75,55,2,False 218,Slugma,Fire,,250,40,40,40,70,40,20,2,False 219,Magcargo,Fire,Rock,410,50,50,120,80,80,30,2,False 220,Swinub,Ice,Ground,250,50,50,40,30,30,50,2,False 221,Piloswine,Ice,Ground,450,100,100,80,60,60,50,2,False 222,Corsola,Water,Rock,380,55,55,85,65,85,35,2,False 223,Remoraid,Water,,300,35,65,35,65,35,65,2,False 224,Octillery,Water,,480,75,105,75,105,75,45,2,False 225,Delibird,Ice,Flying,330,45,55,45,65,45,75,2,False 226,Mantine,Water,Flying,465,65,40,70,80,140,70,2,False 227,Skarmory,Steel,Flying,465,65,80,140,40,70,70,2,False 228,Houndour,Dark,Fire,330,45,60,30,80,50,65,2,False 229,Houndoom,Dark,Fire,500,75,90,50,110,80,95,2,False 229,HoundoomMega Houndoom,Dark,Fire,600,75,90,90,140,90,115,2,False 230,Kingdra,Water,Dragon,540,75,95,95,95,95,85,2,False 231,Phanpy,Ground,,330,90,60,60,40,40,40,2,False 232,Donphan,Ground,,500,90,120,120,60,60,50,2,False 233,Porygon2,Normal,,515,85,80,90,105,95,60,2,False 234,Stantler,Normal,,465,73,95,62,85,65,85,2,False 235,Smeargle,Normal,,250,55,20,35,20,45,75,2,False 236,Tyrogue,Fighting,,210,35,35,35,35,35,35,2,False 237,Hitmontop,Fighting,,455,50,95,95,35,110,70,2,False 238,Smoochum,Ice,Psychic,305,45,30,15,85,65,65,2,False 239,Elekid,Electric,,360,45,63,37,65,55,95,2,False 240,Magby,Fire,,365,45,75,37,70,55,83,2,False 241,Miltank,Normal,,490,95,80,105,40,70,100,2,False 242,Blissey,Normal,,540,255,10,10,75,135,55,2,False 243,Raikou,Electric,,580,90,85,75,115,100,115,2,True 244,Entei,Fire,,580,115,115,85,90,75,100,2,True 245,Suicune,Water,,580,100,75,115,90,115,85,2,True 246,Larvitar,Rock,Ground,300,50,64,50,45,50,41,2,False 247,Pupitar,Rock,Ground,410,70,84,70,65,70,51,2,False 248,Tyranitar,Rock,Dark,600,100,134,110,95,100,61,2,False 248,TyranitarMega Tyranitar,Rock,Dark,700,100,164,150,95,120,71,2,False 249,Lugia,Psychic,Flying,680,106,90,130,90,154,110,2,True 250,Ho-oh,Fire,Flying,680,106,130,90,110,154,90,2,True 251,Celebi,Psychic,Grass,600,100,100,100,100,100,100,2,False 252,Treecko,Grass,,310,40,45,35,65,55,70,3,False 253,Grovyle,Grass,,405,50,65,45,85,65,95,3,False 254,Sceptile,Grass,,530,70,85,65,105,85,120,3,False 254,SceptileMega Sceptile,Grass,Dragon,630,70,110,75,145,85,145,3,False 255,Torchic,Fire,,310,45,60,40,70,50,45,3,False 256,Combusken,Fire,Fighting,405,60,85,60,85,60,55,3,False 257,Blaziken,Fire,Fighting,530,80,120,70,110,70,80,3,False 257,BlazikenMega Blaziken,Fire,Fighting,630,80,160,80,130,80,100,3,False 258,Mudkip,Water,,310,50,70,50,50,50,40,3,False 259,Marshtomp,Water,Ground,405,70,85,70,60,70,50,3,False 260,Swampert,Water,Ground,535,100,110,90,85,90,60,3,False 260,SwampertMega Swampert,Water,Ground,635,100,150,110,95,110,70,3,False 261,Poochyena,Dark,,220,35,55,35,30,30,35,3,False 262,Mightyena,Dark,,420,70,90,70,60,60,70,3,False 263,Zigzagoon,Normal,,240,38,30,41,30,41,60,3,False 264,Linoone,Normal,,420,78,70,61,50,61,100,3,False 265,Wurmple,Bug,,195,45,45,35,20,30,20,3,False 266,Silcoon,Bug,,205,50,35,55,25,25,15,3,False 267,Beautifly,Bug,Flying,395,60,70,50,100,50,65,3,False 268,Cascoon,Bug,,205,50,35,55,25,25,15,3,False 269,Dustox,Bug,Poison,385,60,50,70,50,90,65,3,False 270,Lotad,Water,Grass,220,40,30,30,40,50,30,3,False 271,Lombre,Water,Grass,340,60,50,50,60,70,50,3,False 272,Ludicolo,Water,Grass,480,80,70,70,90,100,70,3,False 273,Seedot,Grass,,220,40,40,50,30,30,30,3,False 274,Nuzleaf,Grass,Dark,340,70,70,40,60,40,60,3,False 275,Shiftry,Grass,Dark,480,90,100,60,90,60,80,3,False 276,Taillow,Normal,Flying,270,40,55,30,30,30,85,3,False 277,Swellow,Normal,Flying,430,60,85,60,50,50,125,3,False 278,Wingull,Water,Flying,270,40,30,30,55,30,85,3,False 279,Pelipper,Water,Flying,430,60,50,100,85,70,65,3,False 280,Ralts,Psychic,Fairy,198,28,25,25,45,35,40,3,False 281,Kirlia,Psychic,Fairy,278,38,35,35,65,55,50,3,False 282,Gardevoir,Psychic,Fairy,518,68,65,65,125,115,80,3,False 282,GardevoirMega Gardevoir,Psychic,Fairy,618,68,85,65,165,135,100,3,False 283,Surskit,Bug,Water,269,40,30,32,50,52,65,3,False 284,Masquerain,Bug,Flying,414,70,60,62,80,82,60,3,False 285,Shroomish,Grass,,295,60,40,60,40,60,35,3,False 286,Breloom,Grass,Fighting,460,60,130,80,60,60,70,3,False 287,Slakoth,Normal,,280,60,60,60,35,35,30,3,False 288,Vigoroth,Normal,,440,80,80,80,55,55,90,3,False 289,Slaking,Normal,,670,150,160,100,95,65,100,3,False 290,Nincada,Bug,Ground,266,31,45,90,30,30,40,3,False 291,Ninjask,Bug,Flying,456,61,90,45,50,50,160,3,False 292,Shedinja,Bug,Ghost,236,1,90,45,30,30,40,3,False 293,Whismur,Normal,,240,64,51,23,51,23,28,3,False 294,Loudred,Normal,,360,84,71,43,71,43,48,3,False 295,Exploud,Normal,,490,104,91,63,91,73,68,3,False 296,Makuhita,Fighting,,237,72,60,30,20,30,25,3,False 297,Hariyama,Fighting,,474,144,120,60,40,60,50,3,False 298,Azurill,Normal,Fairy,190,50,20,40,20,40,20,3,False 299,Nosepass,Rock,,375,30,45,135,45,90,30,3,False 300,Skitty,Normal,,260,50,45,45,35,35,50,3,False 301,Delcatty,Normal,,380,70,65,65,55,55,70,3,False 302,Sableye,Dark,Ghost,380,50,75,75,65,65,50,3,False 302,SableyeMega Sableye,Dark,Ghost,480,50,85,125,85,115,20,3,False 303,Mawile,Steel,Fairy,380,50,85,85,55,55,50,3,False 303,MawileMega Mawile,Steel,Fairy,480,50,105,125,55,95,50,3,False 304,Aron,Steel,Rock,330,50,70,100,40,40,30,3,False 305,Lairon,Steel,Rock,430,60,90,140,50,50,40,3,False 306,Aggron,Steel,Rock,530,70,110,180,60,60,50,3,False 306,AggronMega Aggron,Steel,,630,70,140,230,60,80,50,3,False 307,Meditite,Fighting,Psychic,280,30,40,55,40,55,60,3,False 308,Medicham,Fighting,Psychic,410,60,60,75,60,75,80,3,False 308,MedichamMega Medicham,Fighting,Psychic,510,60,100,85,80,85,100,3,False 309,Electrike,Electric,,295,40,45,40,65,40,65,3,False 310,Manectric,Electric,,475,70,75,60,105,60,105,3,False 310,ManectricMega Manectric,Electric,,575,70,75,80,135,80,135,3,False 311,Plusle,Electric,,405,60,50,40,85,75,95,3,False 312,Minun,Electric,,405,60,40,50,75,85,95,3,False 313,Volbeat,Bug,,400,65,73,55,47,75,85,3,False 314,Illumise,Bug,,400,65,47,55,73,75,85,3,False 315,Roselia,Grass,Poison,400,50,60,45,100,80,65,3,False 316,Gulpin,Poison,,302,70,43,53,43,53,40,3,False 317,Swalot,Poison,,467,100,73,83,73,83,55,3,False 318,Carvanha,Water,Dark,305,45,90,20,65,20,65,3,False 319,Sharpedo,Water,Dark,460,70,120,40,95,40,95,3,False 319,SharpedoMega Sharpedo,Water,Dark,560,70,140,70,110,65,105,3,False 320,Wailmer,Water,,400,130,70,35,70,35,60,3,False 321,Wailord,Water,,500,170,90,45,90,45,60,3,False 322,Numel,Fire,Ground,305,60,60,40,65,45,35,3,False 323,Camerupt,Fire,Ground,460,70,100,70,105,75,40,3,False 323,CameruptMega Camerupt,Fire,Ground,560,70,120,100,145,105,20,3,False 324,Torkoal,Fire,,470,70,85,140,85,70,20,3,False 325,Spoink,Psychic,,330,60,25,35,70,80,60,3,False 326,Grumpig,Psychic,,470,80,45,65,90,110,80,3,False 327,Spinda,Normal,,360,60,60,60,60,60,60,3,False 328,Trapinch,Ground,,290,45,100,45,45,45,10,3,False 329,Vibrava,Ground,Dragon,340,50,70,50,50,50,70,3,False 330,Flygon,Ground,Dragon,520,80,100,80,80,80,100,3,False 331,Cacnea,Grass,,335,50,85,40,85,40,35,3,False 332,Cacturne,Grass,Dark,475,70,115,60,115,60,55,3,False 333,Swablu,Normal,Flying,310,45,40,60,40,75,50,3,False 334,Altaria,Dragon,Flying,490,75,70,90,70,105,80,3,False 334,AltariaMega Altaria,Dragon,Fairy,590,75,110,110,110,105,80,3,False 335,Zangoose,Normal,,458,73,115,60,60,60,90,3,False 336,Seviper,Poison,,458,73,100,60,100,60,65,3,False 337,Lunatone,Rock,Psychic,440,70,55,65,95,85,70,3,False 338,Solrock,Rock,Psychic,440,70,95,85,55,65,70,3,False 339,Barboach,Water,Ground,288,50,48,43,46,41,60,3,False 340,Whiscash,Water,Ground,468,110,78,73,76,71,60,3,False 341,Corphish,Water,,308,43,80,65,50,35,35,3,False 342,Crawdaunt,Water,Dark,468,63,120,85,90,55,55,3,False 343,Baltoy,Ground,Psychic,300,40,40,55,40,70,55,3,False 344,Claydol,Ground,Psychic,500,60,70,105,70,120,75,3,False 345,Lileep,Rock,Grass,355,66,41,77,61,87,23,3,False 346,Cradily,Rock,Grass,495,86,81,97,81,107,43,3,False 347,Anorith,Rock,Bug,355,45,95,50,40,50,75,3,False 348,Armaldo,Rock,Bug,495,75,125,100,70,80,45,3,False 349,Feebas,Water,,200,20,15,20,10,55,80,3,False 350,Milotic,Water,,540,95,60,79,100,125,81,3,False 351,Castform,Normal,,420,70,70,70,70,70,70,3,False 352,Kecleon,Normal,,440,60,90,70,60,120,40,3,False 353,Shuppet,Ghost,,295,44,75,35,63,33,45,3,False 354,Banette,Ghost,,455,64,115,65,83,63,65,3,False 354,BanetteMega Banette,Ghost,,555,64,165,75,93,83,75,3,False 355,Duskull,Ghost,,295,20,40,90,30,90,25,3,False 356,Dusclops,Ghost,,455,40,70,130,60,130,25,3,False 357,Tropius,Grass,Flying,460,99,68,83,72,87,51,3,False 358,Chimecho,Psychic,,425,65,50,70,95,80,65,3,False 359,Absol,Dark,,465,65,130,60,75,60,75,3,False 359,AbsolMega Absol,Dark,,565,65,150,60,115,60,115,3,False 360,Wynaut,Psychic,,260,95,23,48,23,48,23,3,False 361,Snorunt,Ice,,300,50,50,50,50,50,50,3,False 362,Glalie,Ice,,480,80,80,80,80,80,80,3,False 362,GlalieMega Glalie,Ice,,580,80,120,80,120,80,100,3,False 363,Spheal,Ice,Water,290,70,40,50,55,50,25,3,False 364,Sealeo,Ice,Water,410,90,60,70,75,70,45,3,False 365,Walrein,Ice,Water,530,110,80,90,95,90,65,3,False 366,Clamperl,Water,,345,35,64,85,74,55,32,3,False 367,Huntail,Water,,485,55,104,105,94,75,52,3,False 368,Gorebyss,Water,,485,55,84,105,114,75,52,3,False 369,Relicanth,Water,Rock,485,100,90,130,45,65,55,3,False 370,Luvdisc,Water,,330,43,30,55,40,65,97,3,False 371,Bagon,Dragon,,300,45,75,60,40,30,50,3,False 372,Shelgon,Dragon,,420,65,95,100,60,50,50,3,False 373,Salamence,Dragon,Flying,600,95,135,80,110,80,100,3,False 373,SalamenceMega Salamence,Dragon,Flying,700,95,145,130,120,90,120,3,False 374,Beldum,Steel,Psychic,300,40,55,80,35,60,30,3,False 375,Metang,Steel,Psychic,420,60,75,100,55,80,50,3,False 376,Metagross,Steel,Psychic,600,80,135,130,95,90,70,3,False 376,MetagrossMega Metagross,Steel,Psychic,700,80,145,150,105,110,110,3,False 377,Regirock,Rock,,580,80,100,200,50,100,50,3,True 378,Regice,Ice,,580,80,50,100,100,200,50,3,True 379,Registeel,Steel,,580,80,75,150,75,150,50,3,True 380,Latias,Dragon,Psychic,600,80,80,90,110,130,110,3,True 380,LatiasMega Latias,Dragon,Psychic,700,80,100,120,140,150,110,3,True 381,Latios,Dragon,Psychic,600,80,90,80,130,110,110,3,True 381,LatiosMega Latios,Dragon,Psychic,700,80,130,100,160,120,110,3,True 382,Kyogre,Water,,670,100,100,90,150,140,90,3,True 382,KyogrePrimal Kyogre,Water,,770,100,150,90,180,160,90,3,True 383,Groudon,Ground,,670,100,150,140,100,90,90,3,True 383,GroudonPrimal Groudon,Ground,Fire,770,100,180,160,150,90,90,3,True 384,Rayquaza,Dragon,Flying,680,105,150,90,150,90,95,3,True 384,RayquazaMega Rayquaza,Dragon,Flying,780,105,180,100,180,100,115,3,True 385,Jirachi,Steel,Psychic,600,100,100,100,100,100,100,3,True 386,DeoxysNormal Forme,Psychic,,600,50,150,50,150,50,150,3,True 386,DeoxysAttack Forme,Psychic,,600,50,180,20,180,20,150,3,True 386,DeoxysDefense Forme,Psychic,,600,50,70,160,70,160,90,3,True 386,DeoxysSpeed Forme,Psychic,,600,50,95,90,95,90,180,3,True 387,Turtwig,Grass,,318,55,68,64,45,55,31,4,False 388,Grotle,Grass,,405,75,89,85,55,65,36,4,False 389,Torterra,Grass,Ground,525,95,109,105,75,85,56,4,False 390,Chimchar,Fire,,309,44,58,44,58,44,61,4,False 391,Monferno,Fire,Fighting,405,64,78,52,78,52,81,4,False 392,Infernape,Fire,Fighting,534,76,104,71,104,71,108,4,False 393,Piplup,Water,,314,53,51,53,61,56,40,4,False 394,Prinplup,Water,,405,64,66,68,81,76,50,4,False 395,Empoleon,Water,Steel,530,84,86,88,111,101,60,4,False 396,Starly,Normal,Flying,245,40,55,30,30,30,60,4,False 397,Staravia,Normal,Flying,340,55,75,50,40,40,80,4,False 398,Staraptor,Normal,Flying,485,85,120,70,50,60,100,4,False 399,Bidoof,Normal,,250,59,45,40,35,40,31,4,False 400,Bibarel,Normal,Water,410,79,85,60,55,60,71,4,False 401,Kricketot,Bug,,194,37,25,41,25,41,25,4,False 402,Kricketune,Bug,,384,77,85,51,55,51,65,4,False 403,Shinx,Electric,,263,45,65,34,40,34,45,4,False 404,Luxio,Electric,,363,60,85,49,60,49,60,4,False 405,Luxray,Electric,,523,80,120,79,95,79,70,4,False 406,Budew,Grass,Poison,280,40,30,35,50,70,55,4,False 407,Roserade,Grass,Poison,515,60,70,65,125,105,90,4,False 408,Cranidos,Rock,,350,67,125,40,30,30,58,4,False 409,Rampardos,Rock,,495,97,165,60,65,50,58,4,False 410,Shieldon,Rock,Steel,350,30,42,118,42,88,30,4,False 411,Bastiodon,Rock,Steel,495,60,52,168,47,138,30,4,False 412,Burmy,Bug,,224,40,29,45,29,45,36,4,False 413,WormadamPlant Cloak,Bug,Grass,424,60,59,85,79,105,36,4,False 413,WormadamSandy Cloak,Bug,Ground,424,60,79,105,59,85,36,4,False 413,WormadamTrash Cloak,Bug,Steel,424,60,69,95,69,95,36,4,False 414,Mothim,Bug,Flying,424,70,94,50,94,50,66,4,False 415,Combee,Bug,Flying,244,30,30,42,30,42,70,4,False 416,Vespiquen,Bug,Flying,474,70,80,102,80,102,40,4,False 417,Pachirisu,Electric,,405,60,45,70,45,90,95,4,False 418,Buizel,Water,,330,55,65,35,60,30,85,4,False 419,Floatzel,Water,,495,85,105,55,85,50,115,4,False 420,Cherubi,Grass,,275,45,35,45,62,53,35,4,False 421,Cherrim,Grass,,450,70,60,70,87,78,85,4,False 422,Shellos,Water,,325,76,48,48,57,62,34,4,False 423,Gastrodon,Water,Ground,475,111,83,68,92,82,39,4,False 424,Ambipom,Normal,,482,75,100,66,60,66,115,4,False 425,Drifloon,Ghost,Flying,348,90,50,34,60,44,70,4,False 426,Drifblim,Ghost,Flying,498,150,80,44,90,54,80,4,False 427,Buneary,Normal,,350,55,66,44,44,56,85,4,False 428,Lopunny,Normal,,480,65,76,84,54,96,105,4,False 428,LopunnyMega Lopunny,Normal,Fighting,580,65,136,94,54,96,135,4,False 429,Mismagius,Ghost,,495,60,60,60,105,105,105,4,False 430,Honchkrow,Dark,Flying,505,100,125,52,105,52,71,4,False 431,Glameow,Normal,,310,49,55,42,42,37,85,4,False 432,Purugly,Normal,,452,71,82,64,64,59,112,4,False 433,Chingling,Psychic,,285,45,30,50,65,50,45,4,False 434,Stunky,Poison,Dark,329,63,63,47,41,41,74,4,False 435,Skuntank,Poison,Dark,479,103,93,67,71,61,84,4,False 436,Bronzor,Steel,Psychic,300,57,24,86,24,86,23,4,False 437,Bronzong,Steel,Psychic,500,67,89,116,79,116,33,4,False 438,Bonsly,Rock,,290,50,80,95,10,45,10,4,False 439,Mime Jr.,Psychic,Fairy,310,20,25,45,70,90,60,4,False 440,Happiny,Normal,,220,100,5,5,15,65,30,4,False 441,Chatot,Normal,Flying,411,76,65,45,92,42,91,4,False 442,Spiritomb,Ghost,Dark,485,50,92,108,92,108,35,4,False 443,Gible,Dragon,Ground,300,58,70,45,40,45,42,4,False 444,Gabite,Dragon,Ground,410,68,90,65,50,55,82,4,False 445,Garchomp,Dragon,Ground,600,108,130,95,80,85,102,4,False 445,GarchompMega Garchomp,Dragon,Ground,700,108,170,115,120,95,92,4,False 446,Munchlax,Normal,,390,135,85,40,40,85,5,4,False 447,Riolu,Fighting,,285,40,70,40,35,40,60,4,False 448,Lucario,Fighting,Steel,525,70,110,70,115,70,90,4,False 448,LucarioMega Lucario,Fighting,Steel,625,70,145,88,140,70,112,4,False 449,Hippopotas,Ground,,330,68,72,78,38,42,32,4,False 450,Hippowdon,Ground,,525,108,112,118,68,72,47,4,False 451,Skorupi,Poison,Bug,330,40,50,90,30,55,65,4,False 452,Drapion,Poison,Dark,500,70,90,110,60,75,95,4,False 453,Croagunk,Poison,Fighting,300,48,61,40,61,40,50,4,False 454,Toxicroak,Poison,Fighting,490,83,106,65,86,65,85,4,False 455,Carnivine,Grass,,454,74,100,72,90,72,46,4,False 456,Finneon,Water,,330,49,49,56,49,61,66,4,False 457,Lumineon,Water,,460,69,69,76,69,86,91,4,False 458,Mantyke,Water,Flying,345,45,20,50,60,120,50,4,False 459,Snover,Grass,Ice,334,60,62,50,62,60,40,4,False 460,Abomasnow,Grass,Ice,494,90,92,75,92,85,60,4,False 460,AbomasnowMega Abomasnow,Grass,Ice,594,90,132,105,132,105,30,4,False 461,Weavile,Dark,Ice,510,70,120,65,45,85,125,4,False 462,Magnezone,Electric,Steel,535,70,70,115,130,90,60,4,False 463,Lickilicky,Normal,,515,110,85,95,80,95,50,4,False 464,Rhyperior,Ground,Rock,535,115,140,130,55,55,40,4,False 465,Tangrowth,Grass,,535,100,100,125,110,50,50,4,False 466,Electivire,Electric,,540,75,123,67,95,85,95,4,False 467,Magmortar,Fire,,540,75,95,67,125,95,83,4,False 468,Togekiss,Fairy,Flying,545,85,50,95,120,115,80,4,False 469,Yanmega,Bug,Flying,515,86,76,86,116,56,95,4,False 470,Leafeon,Grass,,525,65,110,130,60,65,95,4,False 471,Glaceon,Ice,,525,65,60,110,130,95,65,4,False 472,Gliscor,Ground,Flying,510,75,95,125,45,75,95,4,False 473,Mamoswine,Ice,Ground,530,110,130,80,70,60,80,4,False 474,Porygon-Z,Normal,,535,85,80,70,135,75,90,4,False 475,Gallade,Psychic,Fighting,518,68,125,65,65,115,80,4,False 475,GalladeMega Gallade,Psychic,Fighting,618,68,165,95,65,115,110,4,False 476,Probopass,Rock,Steel,525,60,55,145,75,150,40,4,False 477,Dusknoir,Ghost,,525,45,100,135,65,135,45,4,False 478,Froslass,Ice,Ghost,480,70,80,70,80,70,110,4,False 479,Rotom,Electric,Ghost,440,50,50,77,95,77,91,4,False 479,RotomHeat Rotom,Electric,Fire,520,50,65,107,105,107,86,4,False 479,RotomWash Rotom,Electric,Water,520,50,65,107,105,107,86,4,False 479,RotomFrost Rotom,Electric,Ice,520,50,65,107,105,107,86,4,False 479,RotomFan Rotom,Electric,Flying,520,50,65,107,105,107,86,4,False 479,RotomMow Rotom,Electric,Grass,520,50,65,107,105,107,86,4,False 480,Uxie,Psychic,,580,75,75,130,75,130,95,4,True 481,Mesprit,Psychic,,580,80,105,105,105,105,80,4,True 482,Azelf,Psychic,,580,75,125,70,125,70,115,4,True 483,Dialga,Steel,Dragon,680,100,120,120,150,100,90,4,True 484,Palkia,Water,Dragon,680,90,120,100,150,120,100,4,True 485,Heatran,Fire,Steel,600,91,90,106,130,106,77,4,True 486,Regigigas,Normal,,670,110,160,110,80,110,100,4,True 487,GiratinaAltered Forme,Ghost,Dragon,680,150,100,120,100,120,90,4,True 487,GiratinaOrigin Forme,Ghost,Dragon,680,150,120,100,120,100,90,4,True 488,Cresselia,Psychic,,600,120,70,120,75,130,85,4,False 489,Phione,Water,,480,80,80,80,80,80,80,4,False 490,Manaphy,Water,,600,100,100,100,100,100,100,4,False 491,Darkrai,Dark,,600,70,90,90,135,90,125,4,True 492,ShayminLand Forme,Grass,,600,100,100,100,100,100,100,4,True 492,ShayminSky Forme,Grass,Flying,600,100,103,75,120,75,127,4,True 493,Arceus,Normal,,720,120,120,120,120,120,120,4,True 494,Victini,Psychic,Fire,600,100,100,100,100,100,100,5,True 495,Snivy,Grass,,308,45,45,55,45,55,63,5,False 496,Servine,Grass,,413,60,60,75,60,75,83,5,False 497,Serperior,Grass,,528,75,75,95,75,95,113,5,False 498,Tepig,Fire,,308,65,63,45,45,45,45,5,False 499,Pignite,Fire,Fighting,418,90,93,55,70,55,55,5,False 500,Emboar,Fire,Fighting,528,110,123,65,100,65,65,5,False 501,Oshawott,Water,,308,55,55,45,63,45,45,5,False 502,Dewott,Water,,413,75,75,60,83,60,60,5,False 503,Samurott,Water,,528,95,100,85,108,70,70,5,False 504,Patrat,Normal,,255,45,55,39,35,39,42,5,False 505,Watchog,Normal,,420,60,85,69,60,69,77,5,False 506,Lillipup,Normal,,275,45,60,45,25,45,55,5,False 507,Herdier,Normal,,370,65,80,65,35,65,60,5,False 508,Stoutland,Normal,,500,85,110,90,45,90,80,5,False 509,Purrloin,Dark,,281,41,50,37,50,37,66,5,False 510,Liepard,Dark,,446,64,88,50,88,50,106,5,False 511,Pansage,Grass,,316,50,53,48,53,48,64,5,False 512,Simisage,Grass,,498,75,98,63,98,63,101,5,False 513,Pansear,Fire,,316,50,53,48,53,48,64,5,False 514,Simisear,Fire,,498,75,98,63,98,63,101,5,False 515,Panpour,Water,,316,50,53,48,53,48,64,5,False 516,Simipour,Water,,498,75,98,63,98,63,101,5,False 517,Munna,Psychic,,292,76,25,45,67,55,24,5,False 518,Musharna,Psychic,,487,116,55,85,107,95,29,5,False 519,Pidove,Normal,Flying,264,50,55,50,36,30,43,5,False 520,Tranquill,Normal,Flying,358,62,77,62,50,42,65,5,False 521,Unfezant,Normal,Flying,488,80,115,80,65,55,93,5,False 522,Blitzle,Electric,,295,45,60,32,50,32,76,5,False 523,Zebstrika,Electric,,497,75,100,63,80,63,116,5,False 524,Roggenrola,Rock,,280,55,75,85,25,25,15,5,False 525,Boldore,Rock,,390,70,105,105,50,40,20,5,False 526,Gigalith,Rock,,515,85,135,130,60,80,25,5,False 527,Woobat,Psychic,Flying,313,55,45,43,55,43,72,5,False 528,Swoobat,Psychic,Flying,425,67,57,55,77,55,114,5,False 529,Drilbur,Ground,,328,60,85,40,30,45,68,5,False 530,Excadrill,Ground,Steel,508,110,135,60,50,65,88,5,False 531,Audino,Normal,,445,103,60,86,60,86,50,5,False 531,AudinoMega Audino,Normal,Fairy,545,103,60,126,80,126,50,5,False 532,Timburr,Fighting,,305,75,80,55,25,35,35,5,False 533,Gurdurr,Fighting,,405,85,105,85,40,50,40,5,False 534,Conkeldurr,Fighting,,505,105,140,95,55,65,45,5,False 535,Tympole,Water,,294,50,50,40,50,40,64,5,False 536,Palpitoad,Water,Ground,384,75,65,55,65,55,69,5,False 537,Seismitoad,Water,Ground,509,105,95,75,85,75,74,5,False 538,Throh,Fighting,,465,120,100,85,30,85,45,5,False 539,Sawk,Fighting,,465,75,125,75,30,75,85,5,False 540,Sewaddle,Bug,Grass,310,45,53,70,40,60,42,5,False 541,Swadloon,Bug,Grass,380,55,63,90,50,80,42,5,False 542,Leavanny,Bug,Grass,500,75,103,80,70,80,92,5,False 543,Venipede,Bug,Poison,260,30,45,59,30,39,57,5,False 544,Whirlipede,Bug,Poison,360,40,55,99,40,79,47,5,False 545,Scolipede,Bug,Poison,485,60,100,89,55,69,112,5,False 546,Cottonee,Grass,Fairy,280,40,27,60,37,50,66,5,False 547,Whimsicott,Grass,Fairy,480,60,67,85,77,75,116,5,False 548,Petilil,Grass,,280,45,35,50,70,50,30,5,False 549,Lilligant,Grass,,480,70,60,75,110,75,90,5,False 550,Basculin,Water,,460,70,92,65,80,55,98,5,False 551,Sandile,Ground,Dark,292,50,72,35,35,35,65,5,False 552,Krokorok,Ground,Dark,351,60,82,45,45,45,74,5,False 553,Krookodile,Ground,Dark,519,95,117,80,65,70,92,5,False 554,Darumaka,Fire,,315,70,90,45,15,45,50,5,False 555,DarmanitanStandard Mode,Fire,,480,105,140,55,30,55,95,5,False 555,DarmanitanZen Mode,Fire,Psychic,540,105,30,105,140,105,55,5,False 556,Maractus,Grass,,461,75,86,67,106,67,60,5,False 557,Dwebble,Bug,Rock,325,50,65,85,35,35,55,5,False 558,Crustle,Bug,Rock,475,70,95,125,65,75,45,5,False 559,Scraggy,Dark,Fighting,348,50,75,70,35,70,48,5,False 560,Scrafty,Dark,Fighting,488,65,90,115,45,115,58,5,False 561,Sigilyph,Psychic,Flying,490,72,58,80,103,80,97,5,False 562,Yamask,Ghost,,303,38,30,85,55,65,30,5,False 563,Cofagrigus,Ghost,,483,58,50,145,95,105,30,5,False 564,Tirtouga,Water,Rock,355,54,78,103,53,45,22,5,False 565,Carracosta,Water,Rock,495,74,108,133,83,65,32,5,False 566,Archen,Rock,Flying,401,55,112,45,74,45,70,5,False 567,Archeops,Rock,Flying,567,75,140,65,112,65,110,5,False 568,Trubbish,Poison,,329,50,50,62,40,62,65,5,False 569,Garbodor,Poison,,474,80,95,82,60,82,75,5,False 570,Zorua,Dark,,330,40,65,40,80,40,65,5,False 571,Zoroark,Dark,,510,60,105,60,120,60,105,5,False 572,Minccino,Normal,,300,55,50,40,40,40,75,5,False 573,Cinccino,Normal,,470,75,95,60,65,60,115,5,False 574,Gothita,Psychic,,290,45,30,50,55,65,45,5,False 575,Gothorita,Psychic,,390,60,45,70,75,85,55,5,False 576,Gothitelle,Psychic,,490,70,55,95,95,110,65,5,False 577,Solosis,Psychic,,290,45,30,40,105,50,20,5,False 578,Duosion,Psychic,,370,65,40,50,125,60,30,5,False 579,Reuniclus,Psychic,,490,110,65,75,125,85,30,5,False 580,Ducklett,Water,Flying,305,62,44,50,44,50,55,5,False 581,Swanna,Water,Flying,473,75,87,63,87,63,98,5,False 582,Vanillite,Ice,,305,36,50,50,65,60,44,5,False 583,Vanillish,Ice,,395,51,65,65,80,75,59,5,False 584,Vanilluxe,Ice,,535,71,95,85,110,95,79,5,False 585,Deerling,Normal,Grass,335,60,60,50,40,50,75,5,False 586,Sawsbuck,Normal,Grass,475,80,100,70,60,70,95,5,False 587,Emolga,Electric,Flying,428,55,75,60,75,60,103,5,False 588,Karrablast,Bug,,315,50,75,45,40,45,60,5,False 589,Escavalier,Bug,Steel,495,70,135,105,60,105,20,5,False 590,Foongus,Grass,Poison,294,69,55,45,55,55,15,5,False 591,Amoonguss,Grass,Poison,464,114,85,70,85,80,30,5,False 592,Frillish,Water,Ghost,335,55,40,50,65,85,40,5,False 593,Jellicent,Water,Ghost,480,100,60,70,85,105,60,5,False 594,Alomomola,Water,,470,165,75,80,40,45,65,5,False 595,Joltik,Bug,Electric,319,50,47,50,57,50,65,5,False 596,Galvantula,Bug,Electric,472,70,77,60,97,60,108,5,False 597,Ferroseed,Grass,Steel,305,44,50,91,24,86,10,5,False 598,Ferrothorn,Grass,Steel,489,74,94,131,54,116,20,5,False 599,Klink,Steel,,300,40,55,70,45,60,30,5,False 600,Klang,Steel,,440,60,80,95,70,85,50,5,False 601,Klinklang,Steel,,520,60,100,115,70,85,90,5,False 602,Tynamo,Electric,,275,35,55,40,45,40,60,5,False 603,Eelektrik,Electric,,405,65,85,70,75,70,40,5,False 604,Eelektross,Electric,,515,85,115,80,105,80,50,5,False 605,Elgyem,Psychic,,335,55,55,55,85,55,30,5,False 606,Beheeyem,Psychic,,485,75,75,75,125,95,40,5,False 607,Litwick,Ghost,Fire,275,50,30,55,65,55,20,5,False 608,Lampent,Ghost,Fire,370,60,40,60,95,60,55,5,False 609,Chandelure,Ghost,Fire,520,60,55,90,145,90,80,5,False 610,Axew,Dragon,,320,46,87,60,30,40,57,5,False 611,Fraxure,Dragon,,410,66,117,70,40,50,67,5,False 612,Haxorus,Dragon,,540,76,147,90,60,70,97,5,False 613,Cubchoo,Ice,,305,55,70,40,60,40,40,5,False 614,Beartic,Ice,,485,95,110,80,70,80,50,5,False 615,Cryogonal,Ice,,485,70,50,30,95,135,105,5,False 616,Shelmet,Bug,,305,50,40,85,40,65,25,5,False 617,Accelgor,Bug,,495,80,70,40,100,60,145,5,False 618,Stunfisk,Ground,Electric,471,109,66,84,81,99,32,5,False 619,Mienfoo,Fighting,,350,45,85,50,55,50,65,5,False 620,Mienshao,Fighting,,510,65,125,60,95,60,105,5,False 621,Druddigon,Dragon,,485,77,120,90,60,90,48,5,False 622,Golett,Ground,Ghost,303,59,74,50,35,50,35,5,False 623,Golurk,Ground,Ghost,483,89,124,80,55,80,55,5,False 624,Pawniard,Dark,Steel,340,45,85,70,40,40,60,5,False 625,Bisharp,Dark,Steel,490,65,125,100,60,70,70,5,False 626,Bouffalant,Normal,,490,95,110,95,40,95,55,5,False 627,Rufflet,Normal,Flying,350,70,83,50,37,50,60,5,False 628,Braviary,Normal,Flying,510,100,123,75,57,75,80,5,False 629,Vullaby,Dark,Flying,370,70,55,75,45,65,60,5,False 630,Mandibuzz,Dark,Flying,510,110,65,105,55,95,80,5,False 631,Heatmor,Fire,,484,85,97,66,105,66,65,5,False 632,Durant,Bug,Steel,484,58,109,112,48,48,109,5,False 633,Deino,Dark,Dragon,300,52,65,50,45,50,38,5,False 634,Zweilous,Dark,Dragon,420,72,85,70,65,70,58,5,False 635,Hydreigon,Dark,Dragon,600,92,105,90,125,90,98,5,False 636,Larvesta,Bug,Fire,360,55,85,55,50,55,60,5,False 637,Volcarona,Bug,Fire,550,85,60,65,135,105,100,5,False 638,Cobalion,Steel,Fighting,580,91,90,129,90,72,108,5,True 639,Terrakion,Rock,Fighting,580,91,129,90,72,90,108,5,True 640,Virizion,Grass,Fighting,580,91,90,72,90,129,108,5,True 641,TornadusIncarnate Forme,Flying,,580,79,115,70,125,80,111,5,True 641,TornadusTherian Forme,Flying,,580,79,100,80,110,90,121,5,True 642,ThundurusIncarnate Forme,Electric,Flying,580,79,115,70,125,80,111,5,True 642,ThundurusTherian Forme,Electric,Flying,580,79,105,70,145,80,101,5,True 643,Reshiram,Dragon,Fire,680,100,120,100,150,120,90,5,True 644,Zekrom,Dragon,Electric,680,100,150,120,120,100,90,5,True 645,LandorusIncarnate Forme,Ground,Flying,600,89,125,90,115,80,101,5,True 645,LandorusTherian Forme,Ground,Flying,600,89,145,90,105,80,91,5,True 646,Kyurem,Dragon,Ice,660,125,130,90,130,90,95,5,True 646,KyuremBlack Kyurem,Dragon,Ice,700,125,170,100,120,90,95,5,True 646,KyuremWhite Kyurem,Dragon,Ice,700,125,120,90,170,100,95,5,True 647,KeldeoOrdinary Forme,Water,Fighting,580,91,72,90,129,90,108,5,False 647,KeldeoResolute Forme,Water,Fighting,580,91,72,90,129,90,108,5,False 648,MeloettaAria Forme,Normal,Psychic,600,100,77,77,128,128,90,5,False 648,MeloettaPirouette Forme,Normal,Fighting,600,100,128,90,77,77,128,5,False 649,Genesect,Bug,Steel,600,71,120,95,120,95,99,5,False 650,Chespin,Grass,,313,56,61,65,48,45,38,6,False 651,Quilladin,Grass,,405,61,78,95,56,58,57,6,False 652,Chesnaught,Grass,Fighting,530,88,107,122,74,75,64,6,False 653,Fennekin,Fire,,307,40,45,40,62,60,60,6,False 654,Braixen,Fire,,409,59,59,58,90,70,73,6,False 655,Delphox,Fire,Psychic,534,75,69,72,114,100,104,6,False 656,Froakie,Water,,314,41,56,40,62,44,71,6,False 657,Frogadier,Water,,405,54,63,52,83,56,97,6,False 658,Greninja,Water,Dark,530,72,95,67,103,71,122,6,False 659,Bunnelby,Normal,,237,38,36,38,32,36,57,6,False 660,Diggersby,Normal,Ground,423,85,56,77,50,77,78,6,False 661,Fletchling,Normal,Flying,278,45,50,43,40,38,62,6,False 662,Fletchinder,Fire,Flying,382,62,73,55,56,52,84,6,False 663,Talonflame,Fire,Flying,499,78,81,71,74,69,126,6,False 664,Scatterbug,Bug,,200,38,35,40,27,25,35,6,False 665,Spewpa,Bug,,213,45,22,60,27,30,29,6,False 666,Vivillon,Bug,Flying,411,80,52,50,90,50,89,6,False 667,Litleo,Fire,Normal,369,62,50,58,73,54,72,6,False 668,Pyroar,Fire,Normal,507,86,68,72,109,66,106,6,False 669,Flabébé,Fairy,,303,44,38,39,61,79,42,6,False 670,Floette,Fairy,,371,54,45,47,75,98,52,6,False 671,Florges,Fairy,,552,78,65,68,112,154,75,6,False 672,Skiddo,Grass,,350,66,65,48,62,57,52,6,False 673,Gogoat,Grass,,531,123,100,62,97,81,68,6,False 674,Pancham,Fighting,,348,67,82,62,46,48,43,6,False 675,Pangoro,Fighting,Dark,495,95,124,78,69,71,58,6,False 676,Furfrou,Normal,,472,75,80,60,65,90,102,6,False 677,Espurr,Psychic,,355,62,48,54,63,60,68,6,False 678,MeowsticMale,Psychic,,466,74,48,76,83,81,104,6,False 678,MeowsticFemale,Psychic,,466,74,48,76,83,81,104,6,False 679,Honedge,Steel,Ghost,325,45,80,100,35,37,28,6,False 680,Doublade,Steel,Ghost,448,59,110,150,45,49,35,6,False 681,AegislashBlade Forme,Steel,Ghost,520,60,150,50,150,50,60,6,False 681,AegislashShield Forme,Steel,Ghost,520,60,50,150,50,150,60,6,False 682,Spritzee,Fairy,,341,78,52,60,63,65,23,6,False 683,Aromatisse,Fairy,,462,101,72,72,99,89,29,6,False 684,Swirlix,Fairy,,341,62,48,66,59,57,49,6,False 685,Slurpuff,Fairy,,480,82,80,86,85,75,72,6,False 686,Inkay,Dark,Psychic,288,53,54,53,37,46,45,6,False 687,Malamar,Dark,Psychic,482,86,92,88,68,75,73,6,False 688,Binacle,Rock,Water,306,42,52,67,39,56,50,6,False 689,Barbaracle,Rock,Water,500,72,105,115,54,86,68,6,False 690,Skrelp,Poison,Water,320,50,60,60,60,60,30,6,False 691,Dragalge,Poison,Dragon,494,65,75,90,97,123,44,6,False 692,Clauncher,Water,,330,50,53,62,58,63,44,6,False 693,Clawitzer,Water,,500,71,73,88,120,89,59,6,False 694,Helioptile,Electric,Normal,289,44,38,33,61,43,70,6,False 695,Heliolisk,Electric,Normal,481,62,55,52,109,94,109,6,False 696,Tyrunt,Rock,Dragon,362,58,89,77,45,45,48,6,False 697,Tyrantrum,Rock,Dragon,521,82,121,119,69,59,71,6,False 698,Amaura,Rock,Ice,362,77,59,50,67,63,46,6,False 699,Aurorus,Rock,Ice,521,123,77,72,99,92,58,6,False 700,Sylveon,Fairy,,525,95,65,65,110,130,60,6,False 701,Hawlucha,Fighting,Flying,500,78,92,75,74,63,118,6,False 702,Dedenne,Electric,Fairy,431,67,58,57,81,67,101,6,False 703,Carbink,Rock,Fairy,500,50,50,150,50,150,50,6,False 704,Goomy,Dragon,,300,45,50,35,55,75,40,6,False 705,Sliggoo,Dragon,,452,68,75,53,83,113,60,6,False 706,Goodra,Dragon,,600,90,100,70,110,150,80,6,False 707,Klefki,Steel,Fairy,470,57,80,91,80,87,75,6,False 708,Phantump,Ghost,Grass,309,43,70,48,50,60,38,6,False 709,Trevenant,Ghost,Grass,474,85,110,76,65,82,56,6,False 710,PumpkabooAverage Size,Ghost,Grass,335,49,66,70,44,55,51,6,False 710,PumpkabooSmall Size,Ghost,Grass,335,44,66,70,44,55,56,6,False 710,PumpkabooLarge Size,Ghost,Grass,335,54,66,70,44,55,46,6,False 710,PumpkabooSuper Size,Ghost,Grass,335,59,66,70,44,55,41,6,False 711,GourgeistAverage Size,Ghost,Grass,494,65,90,122,58,75,84,6,False 711,GourgeistSmall Size,Ghost,Grass,494,55,85,122,58,75,99,6,False 711,GourgeistLarge Size,Ghost,Grass,494,75,95,122,58,75,69,6,False 711,GourgeistSuper Size,Ghost,Grass,494,85,100,122,58,75,54,6,False 712,Bergmite,Ice,,304,55,69,85,32,35,28,6,False 713,Avalugg,Ice,,514,95,117,184,44,46,28,6,False 714,Noibat,Flying,Dragon,245,40,30,35,45,40,55,6,False 715,Noivern,Flying,Dragon,535,85,70,80,97,80,123,6,False 716,Xerneas,Fairy,,680,126,131,95,131,98,99,6,True 717,Yveltal,Dark,Flying,680,126,131,95,131,98,99,6,True 718,Zygarde50% Forme,Dragon,Ground,600,108,100,121,81,95,95,6,True 719,Diancie,Rock,Fairy,600,50,100,150,100,150,50,6,True 719,DiancieMega Diancie,Rock,Fairy,700,50,160,110,160,110,110,6,True 720,HoopaHoopa Confined,Psychic,Ghost,600,80,110,60,150,130,70,6,True 720,HoopaHoopa Unbound,Psychic,Dark,680,80,160,60,170,130,80,6,True 721,Volcanion,Fire,Water,600,80,110,120,130,90,70,6,True ================================================ FILE: pandas_log/__init__.py ================================================ # -*- coding: utf-8 -*- """Top-level package for pandas-log.""" from .pandas_log import * __author__ = """Eyal Trabelsi""" __email__ = "eyaltrabelsi@gmail.com" __version__ = "0.0.0" ORIGINAL_METHOD_PREFIX = "original_" ================================================ FILE: pandas_log/aop_utils.py ================================================ import itertools from inspect import signature import pandas as pd from pandas_log import settings from pandas_log.settings import DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE def set_df_attr(df, attr_name, attr_value): """ Hacky way to set attributes in dataframe :param df: DataFrame :param attr_name: Attribute name :param attr_value: Attribute value :return: None """ df.__dict__[attr_name] = attr_value def append_df_attr(df, attr_name, attr_value): """ Hacky way to append a value to dataframe :param df: DataFrame :param attr_name: Attribute name :param attr_value: Attribute value :return: None """ df.__dict__[attr_name].append(attr_value) def get_df_attr(df, attr_name, default_val): """ Get Dataframe attribute if exists otherwise default value :return: Dataframe attribute """ return df.__dict__.get(attr_name, default_val) def get_pandas_func(cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX): """ Get original pandas method :param cls: pandas class :param func: pandas method name :param prefix: the prefix used to keep original method :return: Original pandas method """ _raise_on_bad_class(cls) return getattr(cls, f"{prefix}{func.__name__}") def get_signature_repr(cls, fn, args, full_signature=True): """ Get the signature for the original pandas method with actual values :param cls: the pandas class :param fn: The pandas method :param args: The arguments used when it was applied :return: string representation of the signature for the applied pandas method """ _raise_on_bad_class(cls) def _get_bold_text(text): return f"\033[1m{text}\033[0m" def _get_orig_func_params(): return [ param_value if full_signature else param_name for param_name, param_value in signature( get_pandas_func(cls, fn) ).parameters.items() if param_name not in ("kwargs", "self") ] def _get_param_value(param_with_default, arg_value): res = str(param_with_default) if arg_value is not None: param_name = res.split("=")[0] arg_value = ( f'"{arg_value}"' if isinstance(arg_value, str) else arg_value ) res = ( param_name if isinstance(arg_value, pd.DataFrame) or isinstance(arg_value, pd.Series) else f"{param_name}={arg_value}" ) return res zip_func = itertools.zip_longest if full_signature else zip orig_func_params = _get_orig_func_params() args_vals = ", ".join( _get_param_value(param_with_default, arg_value) for param_with_default, arg_value in zip_func(orig_func_params, args) ) return f"{_get_bold_text(fn.__name__)}({args_vals}):" def _raise_on_bad_class(cls): implemented_classes = (pd.DataFrame, pd.Series) if cls not in implemented_classes: raise TypeError("cls must be one of {}".format(implemented_classes)) def restore_pandas_func_copy( cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX ): """ Restore the original pandas method instead of overridden one :param cls: class containing the method :param func: pandas method name :param prefix: the prefix used to keep original method :return: None """ _raise_on_bad_class(cls) original_method = getattr(cls, func) setattr(cls, func.replace(prefix, ""), original_method) def keep_pandas_func_copy(cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX): """ Saved copy of the pandas method before it overridden :param cls: class containing the method :param func: pandas method name :param prefix: the prefix used to keep original method :return: None """ _raise_on_bad_class(cls) original_method = getattr(cls, func) setattr(cls, f"{prefix}{func}", original_method) def calc_step_number(method_name, input_df): # TODO step_number = get_df_attr(input_df, "execution_history", 0) if step_number: step_number = step_number[-1].execution_stats.step_number if method_name not in DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE: step_number += 1 return step_number if __name__ == "__main__": pass ================================================ FILE: pandas_log/pandas_execution_stats.py ================================================ import warnings from collections import namedtuple from functools import partial from time import time import pandas as pd from pandas_log import patched_logs_functions from pandas_log.aop_utils import ( append_df_attr, calc_step_number, get_df_attr, get_pandas_func, get_signature_repr, set_df_attr, ) from pandas_log.settings import ( DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE, PATCHED_LOG_METHOD_PREFIX, ) with warnings.catch_warnings(): warnings.simplefilter("ignore") import humanize def get_execution_stats(cls, fn, input_df, fn_args, fn_kwargs, calculate_memory): start = time() output_df = get_pandas_func(cls, fn)(input_df, *fn_args, **fn_kwargs) exec_time = time() - start exec_time_pretty = humanize.naturaldelta(exec_time) if exec_time_pretty == "a moment": exec_time_pretty = f"{round(exec_time,6)} seconds" step_number = calc_step_number(fn.__name__, input_df) input_memory_size = ( StepStats.calc_df_series_memory(input_df) if calculate_memory else None ) output_memory_size = ( StepStats.calc_df_series_memory(output_df) if calculate_memory else None ) ExecutionStats = namedtuple( "ExecutionStats", "exec_time step_number input_memory_size output_memory_size", ) execution_stats = ExecutionStats( exec_time_pretty, step_number, input_memory_size, output_memory_size ) return output_df, execution_stats class StepStats: def __init__( self, execution_stats, cls, fn, fn_args, fn_kwargs, full_signature, input_df, output_df, ): """ Constructor :param execution_stats: execution_stats of the pandas operation both in time and memory :param cls: The calling object's pandas class :param fn: The original pandas method :param fn_args: The original pandas method args :param fn_kwargs: The original pandas method kwargs :param full_signature: adding additional information to function signature :param input_df: dataframe before step calculation :param output_df: dataframe after step calculation """ self.execution_stats = execution_stats self.full_signature = full_signature self.cls = cls self.fn = fn self.fn_args = fn_args self.fn_kwargs = fn_kwargs self.input_df = input_df self.output_df = output_df @staticmethod def calc_df_series_memory(df_or_series): res = None if isinstance(df_or_series, pd.Series): mem = df_or_series.memory_usage(index=True, deep=True) res = humanize.naturalsize(mem) elif isinstance(df_or_series, pd.DataFrame): mem = df_or_series.memory_usage(index=True, deep=True) res = humanize.naturalsize(mem.sum()) return res def persist_execution_stats(self): prev_exec_history = get_df_attr(self.input_df, "execution_history", []) set_df_attr(self.output_df, "execution_history", prev_exec_history) append_df_attr(self.output_df, "execution_history", self) def log_stats_if_needed(self, silent, verbose, copy_ok): from pandas_log.pandas_log import ALREADY_ENABLED if silent or not ALREADY_ENABLED: return if verbose or self.fn.__name__ not in DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE: s = self.__repr__(verbose, copy_ok) if s: # If this method isn't patched and verbose is False, __repr__ will give an empty string, which # we don't want to print print(s) def get_logs_for_specifc_method(self, verbose, copy_ok): self.fn_kwargs["kwargs"] = self.fn_kwargs.copy() self.fn_kwargs["copy_ok"] = copy_ok try: log_method = getattr( patched_logs_functions, f"{PATCHED_LOG_METHOD_PREFIX}{self.fn.__name__}", ) except AttributeError: # Method is listed as a method to override, but no patched function exists if verbose: log_method = getattr(patched_logs_functions, "log_default") else: log_method = getattr(patched_logs_functions, "log_no_message") log_method = partial(log_method, self.output_df, self.input_df) logs, tips = log_method(*self.fn_args, **self.fn_kwargs) return logs, tips def _repr_html_(self): pass def __repr__(self, verbose, copy_ok): # Step title func_sig = get_signature_repr( self.cls, self.fn, self.fn_args, self.full_signature ) step_number = ( "X" if self.fn.__name__ in DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE else self.execution_stats.step_number ) step_title = f"{step_number}) {func_sig}" # Step Metadata stats logs, tips = self.get_logs_for_specifc_method(verbose, copy_ok) metadata_stats = f"\033[4mMetadata\033[0m:\n{logs}" if logs else "" metadata_tips = f"\033[4mTips\033[0m:\n{tips}" if tips else "" # Step Execution stats exec_time_humanize = ( f"* Execution time: Step Took {self.execution_stats.exec_time}." ) exec_stats_raw = [exec_time_humanize] if self.execution_stats.input_memory_size is not None: exec_stats_raw.append( f"* Input Dataframe size is {self.execution_stats.input_memory_size}." ) if self.execution_stats.output_memory_size is not None: exec_stats_raw.append( f"* Output Dataframe size is {self.execution_stats.output_memory_size}." ) exec_stats_raw_str = "\n\t".join(exec_stats_raw) execution_stats = f"\033[4mExecution Stats\033[0m:\n\t{exec_stats_raw_str}" all_logs = [metadata_stats, execution_stats, metadata_tips] all_logs_str = "\n\t".join([x for x in all_logs if x]) return f"\n{step_title}\n\t{all_logs_str}" if __name__ == "__main__": pass ================================================ FILE: pandas_log/pandas_log.py ================================================ # -*- coding: utf-8 -*- """Main module.""" import warnings from contextlib import contextmanager from functools import wraps import pandas as pd import pandas_flavor as pf from pandas_log import settings from pandas_log.aop_utils import ( keep_pandas_func_copy, restore_pandas_func_copy, ) from pandas_log.pandas_execution_stats import StepStats, get_execution_stats __all__ = ["auto_enable", "auto_disable", "enable"] ALREADY_ENABLED = False def auto_disable(): """ Restore original pandas method without the additional log functionality (statistics) Note: we keep the original methods using original_ prefix. :return: None """ global ALREADY_ENABLED if not ALREADY_ENABLED: return for cls in (pd.DataFrame, pd.Series): for func in dir(cls): if func.startswith(settings.ORIGINAL_METHOD_PREFIX): restore_pandas_func_copy(cls, func) ALREADY_ENABLED = False @contextmanager def enable( verbose=False, silent=False, full_signature=True, copy_ok=True, calculate_memory=False, ): """ Adds the additional logging functionality (statistics) to pandas methods only for the scope of this context manager. :param verbose: Whether some inner functions should be recorded as well. For example: when a dataframe being copied :param silent: Whether additional the statistics get printed :param full_signature: adding additional information to function signature :param copy_ok: whether the dataframe is allowed to be copied to calculate more informative metadata logs :return: None """ auto_enable(verbose, silent, full_signature, copy_ok, calculate_memory) yield auto_disable() def auto_enable( verbose=False, silent=False, full_signature=True, copy_ok=True, calculate_memory=False, ): """ Adds the additional logging functionality (statistics) to pandas methods. :param verbose: Whether some inner functions should be recorded as well. For example: when a dataframe being copied :param silent: Whether additional the statistics get printed :param full_signature: adding additional information to function signature :param copy_ok: whether the dataframe is allowed to be copied to calculate more informative metadata logs :return: None """ global ALREADY_ENABLED if ALREADY_ENABLED: return settings.DATAFRAME_METHODS_TO_OVERIDE.extend( settings.DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE ) # Suppressing warning of the fact we override pandas functions. with warnings.catch_warnings(): warnings.simplefilter("ignore") for cls, overrides in [ (pd.DataFrame, settings.DATAFRAME_METHODS_TO_OVERIDE), (pd.Series, settings.SERIES_METHODS_TO_OVERIDE), ]: for func in dir(cls): if func in overrides: keep_pandas_func_copy(cls, func) create_overide_pandas_func( cls, func, verbose, silent, full_signature, copy_ok, calculate_memory, ) ALREADY_ENABLED = True def create_overide_pandas_func( cls, func, verbose, silent, full_signature, copy_ok, calculate_memory ): """ Create overridden pandas method dynamically with additional logging using DataFrameLogger Note: if we extracting _overide_pandas_method outside we need to implement decorator like here https://stackoverflow.com/questions/10176226/how-do-i-pass-extra-arguments-to-a-python-decorator :param cls: pandas class for which the method should be overriden :param func: pandas method name to be overridden :param silent: Whether additional the statistics get printed :param full_signature: adding additional information to function signature :param copy_ok: whether the dataframe is allowed to be copied to calculate more informative metadata logs :return: the same function with additional logging capabilities """ def _run_method_and_calc_stats( fn, fn_args, fn_kwargs, input_df, full_signature, silent, verbose, copy_ok, calculate_memory, ): if copy_ok: # If we're ok to make copies, copy the input_df so that we can compare against the output of inplace methods try: # Will hit infinite recursion if we use the patched copy method so use the original original_input_df = getattr( input_df, settings.ORIGINAL_METHOD_PREFIX + "copy" )(deep=True) except AttributeError: original_input_df = input_df.copy(deep=True) output_df, execution_stats = get_execution_stats( cls, fn, input_df, fn_args, fn_kwargs, calculate_memory ) if output_df is None: # The operation was strictly in place so we just call the dataframe the output_df as well output_df = input_df if copy_ok: # If this isn't true and the method was strictly inplace, input_df and output_df will just # point to the same object input_df = original_input_df step_stats = StepStats( execution_stats, cls, fn, fn_args, fn_kwargs, full_signature, input_df, output_df, ) step_stats.log_stats_if_needed(silent, verbose, copy_ok) if isinstance(output_df, pd.DataFrame) or isinstance(output_df, pd.Series): step_stats.persist_execution_stats() return output_df def _overide_pandas_method(fn): if cls == pd.DataFrame: register_method_wrapper = pf.register_dataframe_method elif cls == pd.Series: register_method_wrapper = pf.register_series_method @register_method_wrapper @wraps(fn) def wrapped(*args, **fn_kwargs): input_df, fn_args = args[0], args[1:] output_df = _run_method_and_calc_stats( fn, fn_args, fn_kwargs, input_df, full_signature, silent, verbose, copy_ok, calculate_memory, ) return output_df return wrapped return exec(f"@_overide_pandas_method\ndef {func}(df, *args, **kwargs): pass") if __name__ == "__main__": pass ================================================ FILE: pandas_log/patched_logs_functions.py ================================================ import warnings import numpy as np import pandas as pd # Values messages ALTERED_VALUES_MSG = "\t* Changed {values_changed} values, {values_unchanged} values were not changed." # Rows messages REMOVED_NO_ROWS_MSG = "\t* No change in number of rows of input df." FILTERED_ROWS_MSG = "\t* Removed {rows_removed} rows ({rows_removed_pct}%), {rows_remaining} rows remaining." # Cols messages REMOVED_NO_COLS_MSG = "\t* Removed no columns." FILTERED_COLS_MSG = ( "\t* Removed the following columns ({cols_removed}) now only have the following columns" "({cols_remaining})." ) ASSIGN_EXISTING_MSG = "\t* The columns {existing_cols} were reassigned." ASSIGN_NEW_MSG = "\t* The columns {new_cols} were created." # Group by messages GROUPBY_MSG = "\t* Grouping by {} resulted in {} groups like {}." # N/A messages FILLNA_NO_NA_MSG = "\t* There are no nulls." FILLNA_WITHH_NA_MSG = "\t* Filled {} with {}." # Mege messages JOIN_ROWS_MSG = "\t* Number of rows changed, after join is {output_rows} rows." JOIN_NEW_COLS_MSG = "\t* Added {num_new_columns} columns ({new_columns})." JOIN_TYPE_MSG = ( "\t* Its a {how} join with the following cardinality:\n\t\t> rows only in left is {left_only}." "\n\t\t> rows only in right is {right_only}.\n\t\t> rows in both is {both}." ) # Pick messages SAMPLE_MSG = "\t* Picked random sample of {output_rows} rows." NLARGEST_MSG = "\t* Picked {n} largest rows by columns ({cols})." NSMALLEST_MSG = "\t* Picked {n} smallest rows by columns ({cols})." HEAD_MSG = "\t* Picked the first {} rows." TAIL_MSG = "\t* Picked the last {} rows." # TIPS ITERROWS_TIPS = "\t*iterrows is not recommended, and in the majority of cases will have better alternatives" FILLNA_NO_NA_TIP = ( "\t* There are no nulls in this dataframe, if you are working on the entire dataset you can " "remove this operation." ) SHOULD_REDUCED_ROW_TIP = ( "\t* Number of rows didn't change, if you are working on the entire dataset you can remove " "this operation." ) # Others SORT_VALUES_MSG = "\t* Sorting by columns {} in a {} order." SORT_INDEX_MSG = "\t* Sorting by index in a {'ascending' if ascending else 'descending'} order." DEFAULT_STRATEGY_USED_MSG = ( "\t* Using default strategy (some metric might not be relevant)." ) TRANSFORMED_TO_DF_MSG = "\t* After transformation we received Series" COPY_WARNING_MSG = ( "Some pandas logging may involve copying dataframes, which can be time-/memory-intensive. " "Consider passing copy_ok=False to the enable/auto_enable functions in pandas_log if issues arise." ) def _stringify_list(l): return [str(x) for x in l] def rows_removed(input_df, output_df): return len(input_df) - len(output_df) def rows_removed_pct(input_df, output_df): return 100 * (rows_removed(input_df, output_df)) / len(input_df) def rows_remaining(output_df): return len(output_df) def cols_removed(input_df, output_df): return ", ".join( _stringify_list(set(input_df.columns) - set(output_df.columns)) ) def cols_remaining(output_df): return ", ".join(_stringify_list(set(output_df.columns))) def is_same_cols(input_df, output_df): return len(input_df.columns) == len(output_df.columns) def columns_changed(df, cols): return set(df.columns).intersection(set(cols)) def columns_added(df, cols): return set(cols) - set(df.columns) def is_same_rows(input_df, output_df): return len(input_df) == len(output_df) def num_of_na(df): return df.isnull().values.sum() def str_new_columns(input_df, output_df): return ", ".join( _stringify_list(set(output_df.columns) - set(input_df.columns)) ) def num_new_columns(input_df, output_df): return len(set(output_df.columns) - set(input_df.columns)) def num_values_changed(input_obj, output_obj): if ( isinstance(input_obj, pd.Series) and isinstance(output_obj, pd.Series) and input_obj.dtype != output_obj.dtype ): # Comparing values for equality across dtypes wouldn't be well-defined so we just say they all changed values_changed = len(input_obj) else: values_changed = ( (output_obj != input_obj) & ~(output_obj.isnull() & input_obj.isnull()) ).sum() if isinstance(input_obj, pd.DataFrame): # We only summed once so values_changed will be a series, so we sum again values_changed = values_changed.sum() values_unchanged = ( input_obj.shape[0] * input_obj.shape[1] ) - values_changed elif isinstance(input_obj, pd.Series): values_unchanged = len(output_obj) - values_changed return values_changed, values_unchanged def get_filter_rows_logs(input_df, output_df): tips = "" if is_same_rows(input_df, output_df): logs = REMOVED_NO_ROWS_MSG tips = SHOULD_REDUCED_ROW_TIP else: logs = FILTERED_ROWS_MSG.format( rows_removed=rows_removed(input_df, output_df), rows_removed_pct=rows_removed_pct(input_df, output_df), rows_remaining=rows_remaining(output_df), ) return logs, tips def log_default(output_df, input_df, *args, **kwargs): logs = [DEFAULT_STRATEGY_USED_MSG] tips = "" if isinstance(output_df, pd.DataFrame): if not is_same_cols(input_df, output_df): logs.append( FILTERED_COLS_MSG.format( cols_removed=cols_removed(input_df, output_df), cols_remaining=cols_remaining(output_df), ) ) if not is_same_rows(input_df, output_df): logs.append( FILTERED_ROWS_MSG.format( rows_removed=rows_removed(input_df, output_df), rows_removed_pct=rows_removed_pct(input_df, output_df), rows_remaining=rows_remaining(output_df), ) ) elif isinstance(output_df, pd.Series): logs.append(TRANSFORMED_TO_DF_MSG) logs = "\n".join(logs) return logs, tips def log_no_message(output_df, input_df, *args, **kwargs): return "", "" def log_drop( output_df, input_df, labels=None, axis=0, index=None, columns=None, level=None, inplace=False, errors="raise", *args, **kwargs, ): logs, tips = get_filter_rows_logs(input_df, output_df) tips = "" if is_same_cols(input_df, output_df): logs += f"\n{REMOVED_NO_COLS_MSG}" else: msg = FILTERED_COLS_MSG.format( cols_removed=cols_removed(input_df, output_df), cols_remaining=cols_remaining(output_df), ) logs += f"\n{msg}" return logs, tips def log_dropna( output_df, input_df, axis=0, how="any", thresh=None, subset=None, inplace=False, **kwargs, ): logs, tips = get_filter_rows_logs(input_df, output_df) if is_same_cols(input_df, output_df): logs += f"\n{REMOVED_NO_COLS_MSG}" tips = SHOULD_REDUCED_ROW_TIP else: msg = FILTERED_COLS_MSG.format( cols_removed=cols_removed(input_df, output_df), cols_remaining=cols_remaining(output_df), ) logs += f"\n{msg}" return logs, tips def log_assign(output_df, input_df, **kwargs): logs = [] tips = "" cols = [key for key in kwargs.keys() if key not in ["kwargs", "copy_ok"]] changed_cols = columns_changed(input_df, cols) added_cols = columns_added(input_df, cols) if changed_cols: if kwargs["copy_ok"]: warnings.warn(COPY_WARNING_MSG) # If copying is ok, we can check how many values actually changed for col in changed_cols: values_changed, values_unchanged = num_values_changed( input_df[col], output_df[col] ) logs.append( "\t* {}: {}".format( col, ALTERED_VALUES_MSG.format( values_changed=values_changed, values_unchanged=values_unchanged, )[3:], ) ) # [3:] to strip the "\t* " from the altered values message else: # Otherwise just indicated which columns changed # (Doing the above calculation would always say zero values changed in this case, since if copying isn't # ok then input_df and output_df point to the same object) logs.append( ASSIGN_EXISTING_MSG.format( existing_cols=", ".join(changed_cols) ) ) if added_cols: logs.append( ASSIGN_NEW_MSG.format( new_cols=", ".join(columns_added(input_df, cols)) ) ) logs = "\n".join(logs) return logs, tips def log___setitem__(output_df, input_df, key, value, **kwargs): if isinstance(key, str): # Only setting one column so can just use the assign logger as is kwargs[key] = value elif isinstance(key, list): # This would be kind of complicated but since we don't actually use the values in kwargs we can just # add the new columns as keys each with the full set of assigned values (as a placeholder) for subkey in key: kwargs[subkey] = value return log_assign(output_df, input_df, **kwargs) def log_query(output_df, input_df, expr, inplace=False, *args, **kwargs): logs, tips = get_filter_rows_logs(input_df, output_df) return logs, tips def log_sort_index( output_df, input_df, axis=0, level=None, ascending=True, inplace=False, kind="quicksort", na_position="last", sort_remaining=True, by=None, **kwargs, ): logs = SORT_INDEX_MSG.format(ascending) tips = "" return logs, tips def log_sort_values( output_df, input_df, by, axis=0, ascending=True, inplace=False, kind="quicksort", na_position="last", **kwargs, ): logs = SORT_VALUES_MSG.format( by, "ascending" if ascending else "descending" ) tips = "" return logs, tips def log_tail(output_df, input_df, n=5, **kwargs): logs = TAIL_MSG.format(n) tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else "" return logs, tips def log_head(output_df, input_df, n=5, **kwargs): logs = HEAD_MSG.format(n) tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else "" return logs, tips def log_merge( output_df, input_df, right, how="inner", on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=False, suffixes=("_x", "_y"), copy=True, indicator=False, validate=None, **kwargs, ): logs = [] tips = "" merged = input_df.original_merge( right, "outer", on, left_on, right_on, left_index, right_index, sort, suffixes, copy, True, validate, ) logs.append( JOIN_TYPE_MSG.format(how=how, **merged._merge.value_counts().to_dict()) ) if is_same_rows(input_df, output_df): logs.append(REMOVED_NO_ROWS_MSG) else: logs.append(JOIN_ROWS_MSG.format(output_rows=len(output_df))) if not is_same_cols(input_df, output_df): logs.append( JOIN_NEW_COLS_MSG.format( num_new_columns=num_new_columns(input_df, output_df), new_columns=str_new_columns(input_df, output_df), ) ) logs = "\n".join(logs) return logs, tips def log_applymap(output_df, input_df, func, **kwargs): values_changed, values_unchanged = num_values_changed(input_df, output_df) log = ALTERED_VALUES_MSG.format( values_changed=values_changed, values_unchanged=values_unchanged ) return log, "" def log_join( output_df, input_df, other, on=None, how="left", lsuffix="", rsuffix="", sort=False, **kwargs, ): logs = [] tips = "" merged = input_df.original_merge( other, "outer", on, input_df.index, other.index, indicator=True ) logs.append( JOIN_TYPE_MSG.format(how=how, **merged._merge.value_counts().to_dict()) ) logs.append(get_filter_rows_logs(input_df, output_df)) if not is_same_cols(input_df, output_df): logs.append( JOIN_NEW_COLS_MSG.format( num_new_columns=num_new_columns(input_df, output_df), new_columns=str_new_columns(input_df, output_df), ) ) logs = "\n".join(logs) return logs, tips def log_fillna( output_df, input_df, value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs, ): tips = "" value = "empty string" if value == "" else value if num_of_na(input_df) == num_of_na(output_df): logs = FILLNA_NO_NA_MSG tips = FILLNA_NO_NA_TIP else: logs = FILLNA_WITHH_NA_MSG.format(num_of_na(input_df), value) return logs, tips def log_mask( output_df, input_df, cond, other=np.nan, inplace=False, axis=None, level=None, errors="raise", try_cast=False, *args, **kwargs, ): values_changed, values_unchanged = num_values_changed(input_df, output_df) logs = [] logs.append( ALTERED_VALUES_MSG.format( values_changed=values_changed, values_unchanged=values_unchanged ) ) if isinstance(cond, pd.Series) and isinstance(input_df, pd.Series): # Calculate the values for which the condition was true but the value didn't change only for this simplest case, # where we can just & the two series true_but_unchanged = (cond & (output_df == input_df)).sum() if true_but_unchanged > 0: logs.append( "\t* {} rows met the masking condition and already had the masking value.".format( true_but_unchanged ) ) logs = "\n".join(logs) tips = "" return logs, tips def log_where( output_df, input_df, cond, other=np.nan, inplace=False, axis=None, level=None, errors="raise", try_cast=False, *args, **kwargs, ): return log_mask( output_df, input_df, ~cond, # Important other=np.nan, inplace=False, axis=None, level=None, errors="raise", try_cast=False, *args, **kwargs, ) def log_sample( output_df, input_df, n=None, frac=None, replace=False, weights=None, random_state=None, axis=None, *args, **kwargs, ): logs = SAMPLE_MSG.format(output_rows=len(output_df)) tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else "" return logs, tips def log_nlargest(output_df, input_df, n, columns, keep="first", **kwargs): # todo maybe wrong logs = NLARGEST_MSG.format(n=n, cols=columns) tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else "" return logs, tips def log_nsmallest(output_df, input_df, n, columns, keep="first", **kwargs): logs = NSMALLEST_MSG.format(n=n, cols=columns) tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else "" return logs, tips def log_groupby( output_df, input_df, by=None, axis=0, level=None, as_index=True, sort=True, group_keys=True, squeeze=False, observed=False, **kwargs, ): group_by = str(by) groups = list(output_df.groups) groups_len = len(groups) groups_repr = ( ",".join(["\n\t\t" + str(x) for x in groups[:5]]) + ",\n\t and more" ) tips = "" logs = GROUPBY_MSG.format(group_by, groups_len, groups_repr) return logs, tips def log__iterrows(output_df, input_df): tips = ITERROWS_TIPS logs = "" return logs, tips def log___getitem__(output_df, input_df, key, *args, **kwargs): logs = [] tips = "" if isinstance(output_df, pd.Series): # Naive handle of __getitem__ which return series logs = TRANSFORMED_TO_DF_MSG return logs, tips if not is_same_cols(input_df, output_df): logs.append( FILTERED_COLS_MSG.format( cols_removed=cols_removed(input_df, output_df), cols_remaining=cols_remaining(output_df), ) ) if not is_same_rows(input_df, output_df): logs.append( FILTERED_ROWS_MSG.format( rows_removed=rows_removed(input_df, output_df), rows_removed_pct=rows_removed_pct(input_df, output_df), rows_remaining=rows_remaining(output_df), ) ) logs = "\n".join(logs) return logs, tips # TODO add tip on types+cardinality if __name__ == "__main__": pass ================================================ FILE: pandas_log/settings.py ================================================ ORIGINAL_METHOD_PREFIX = "original_" PATCHED_LOG_METHOD_PREFIX = "log_" DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE = [ "copy", "reset_index", "__getitem__", "__setitem__", ] DATAFRAME_METHODS_TO_OVERIDE = [ "query", "drop", "dropna", "assign", "sort_index", "sort_values", "head", "tail", "sample", "fillna", "merge", "join", "nlargest", "nsmallest", "apply", "iterrows", "applymap", "pipe", "rolling", "groupby", "rename", "agg", "aggregate", "stack", "unstack", "pivot", "pivot_table", "mask", "max", "mean", "median", "melt", "replace", "skew", "notna", "kurt", "expanding", "drop_duplicates", "bfill", "corr", "corrwith", "droplevel", "explode", "ffill", "filter", "first", "kurtosis", "align", "transform", "update", "squeeze", "shift", "rank", "nunique", "min", "mod", "mode", "std", ] SERIES_METHODS_TO_OVERIDE = ["mask", "where"] ================================================ FILE: requirements_dev.txt ================================================ pip>=19.2.3 bump2version>=0.5.11 wheel>=0.33.6 watchdog>=0.9.0 flake8>=3.7.8 tox>=3.14.0 coverage>=4.5.4 Sphinx>=2.2.0 twine>=2.2.0 pytest>=5.1.3 pytest-runner>=5.1 pandas>=0.25.1 pandas_flavor>=0.1.2 humanize>=0.5.0 ================================================ FILE: setup.cfg ================================================ [bumpversion] current_version = 0.1.7 commit = True tag = True [bumpversion:file:setup.py] search = version='{current_version}' replace = version='{new_version}' [bumpversion:file:pandas_log/__init__.py] search = __version__ = '{current_version}' replace = __version__ = '{new_version}' [bdist_wheel] universal = 1 [flake8] exclude = docs [aliases] # Define setup.py command aliases here test = pytest [tool:pytest] collect_ignore = ['setup.py'] ================================================ FILE: setup.py ================================================ #!/usr/bin/env python # -*- coding: utf-8 -*- """The setup script.""" from setuptools import setup, find_packages with open('README.rst') as readme_file: readme = readme_file.read() requirements = ["humanize>=0.5.0", "pandas>=0.25.1", "pandas_flavor>=0.1.2"] setup_requirements = ['pytest-runner', ] test_requirements = ['pytest>=3', ] setup( name='pandas-log', version='0.1.7', description="pandas-log provides feedback about basic pandas operations. It provides simple wrapper functions for " "the most common functions, such as apply, map, query and more.", author="Eyal Trabelsi", author_email='eyaltrabelsi@gmail.com', url='https://github.com/eyaltrabelsi/pandas-log', packages=find_packages(include=['pandas_log', 'pandas_log.*']), install_requires=requirements, python_requires=">=3.4", license="MIT license", long_description="pandas-log provides feedback about basic pandas operations. It provides simple wrapper functions " "for the most common functions, such as apply, map, query and more.", long_description_content_type="text/x-rst", setup_requires=setup_requirements, test_suite='tests', tests_require=test_requirements ) ================================================ FILE: tox.ini ================================================ [tox] envlist = py27, py35, py36, py37 flake8 [travis] python = 3.7: py37 3.6: py36 3.5: py35 2.7: py27 [testenv:flake8] basepython = python deps = flake8 commands = flake8 pandas_log [testenv] setenv = PYTHONPATH = {toxinidir} deps = -r{toxinidir}/requirements_dev.txt ; If you want to make tox run the tests with the same versions, create a ; requirements.txt with the pinned versions and uncomment the following line: ; -r{toxinidir}/requirements.txt commands = pip install -U pip pytest --basetemp={envtmpdir}