Repository: eyaltrabelsi/pandas-log
Branch: master
Commit: 5ea73d91856c
Files: 37
Total size: 158.7 KB
Directory structure:
gitextract_a1_r2osh/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ ├── documentation_fix.md
│ │ ├── new_examples.md
│ │ └── new_proposed_feature.md
│ └── pull_request_template.md
├── .gitignore
├── AUTHORS.rst
├── CONTRIBUTING.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── TODO.rst
├── docs/
│ ├── Makefile
│ ├── conf.py
│ ├── contributing.rst
│ ├── index.rst
│ ├── installation.rst
│ ├── modules.rst
│ ├── pandas_log.rst
│ ├── readme.rst
│ └── usage.rst
├── examples/
│ ├── README.rst
│ ├── __init__.py
│ ├── pandas_log_intro.ipynb
│ └── pokemon.csv
├── pandas_log/
│ ├── __init__.py
│ ├── aop_utils.py
│ ├── pandas_execution_stats.py
│ ├── pandas_log.py
│ ├── patched_logs_functions.py
│ └── settings.py
├── requirements_dev.txt
├── setup.cfg
├── setup.py
└── tox.ini
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug Report
about: Please use this issue template if you are filing a bug report.
---
# Brief Description
<!-- Please provide a brief description of your bug. Do NOT paste the stack trace here. -->
# System Information
<!-- System information helps us. To keep things simple, just let us know the OS and Python version first.
You can provide the optional information later. -->
- Operating system: macOS/Linux/Windows <!-- delete the appropriate ones -->
- OS details (optional): <!-- e.g. version, or Linux distro -->
- Python version (required):
# Minimally Reproducible Code
<!-- If you provide minimal code that reproduces the problem, this makes it easier for us to debug what's going on.
Minimal code should be trivially copy/pastable into a Python interpreter in its entirety. Be sure to include imports.
-->
# Error Messages
<!-- If you get an error message, please paste it between the backticks here. -->
```
```
================================================
FILE: .github/ISSUE_TEMPLATE/documentation_fix.md
================================================
---
name: Propose a Documentation Fix
about: Use this issue tracker template if you'd like to propose a fix to the documentation.
---
# Brief Description of Fix
<!-- Please describe the fix in terms of a "before" and "after". In other words, what's not so good about the current docs
page, and what you would like to see it become.
Example starter wording is provided. -->
Currently, the docs...
I would like to propose a change, such that now the docs...
# Relevant Context
<!-- Please put here, in bullet points, links to the relevant docs page. A few starting template points are available
to get you started. -->
- [Link to documentation page](hhttps://pandas-log.readthedocs.io)
================================================
FILE: .github/ISSUE_TEMPLATE/new_examples.md
================================================
---
name: Add/Modify Notebooks
about: Use this specific template if you'd like to contribute a notebook to the examples gallery or modify an existing one.
---
# Brief Description
<!-- Please describe briefly what you'd like to do in the notebook. -->
I'd like to write a notebook that...
(optional but encouraged) This notebook would likely cover the following pandas-log functionality:
-
-
-
<!-- It's ok if you don't eventually use those functions, by the way! -->
# Dataset
<!-- Please list here where you plan to get the dataset from. -->
================================================
FILE: .github/ISSUE_TEMPLATE/new_proposed_feature.md
================================================
---
name: Propose New Feature
about: If you'd like to propose a new feature, please use this template.
---
# Brief Description
<!-- Please provide a brief description of what you'd like to propose. -->
I would like to propose...
================================================
FILE: .github/pull_request_template.md
================================================
<!-- Thank you for your PR!
BEFORE YOU CONTINUE! Please add the appropriate three-letter abbreviation to your title.
The abbreviations can be:
- [DOC]: Documentation fixes.
- [ENH]: Code contributions and new features.
- [TST]: Test-related contributions.
- [INF]: Infrastructure-related contributions.
Also, do not forget to tag the relevant issue here as well.
Finally, as commits come in, don't forget to regularly rebase!
-->
# PR Description
Please describe the changes proposed in the pull request:
-
-
-
<!-- Doing so provides maintainers with context on what the PR is, and can help us more effectively review your PR. -->
<!-- Please also identify below which issue that has been raised that you are going to close. -->
**This PR resolves #(put issue number here, and remove parentheses).**
<!-- As you go down the PR template, please feel free to delete sections that are irrelevant. -->
# PR Checklist
<!-- This checklist exists for newcomers who are not yet familiar with our requirements. If you are experienced with
the project, please feel free to delete this section. -->
Please ensure that you have done the following:
1. [ ] PR in from a fork off your branch. Do not PR from `<your_username>`:master, but rather from `<your_username>`:<branch_name>.
<!-- Doing this helps us keep the commit history much cleaner than it would otherwise be. -->
2. [ ] If you're not on the contributors list, add yourself to `AUTHORS.rst`.
<!-- We'd like to acknowledge your contributions! -->
## Quick Check
To do a very quick check that everything is correct, follow these steps below:
- [ ] Run the command `make format` from pandsa-log's top-level directory. This will automatically run:
- black formatting
- fix imports with isort
================================================
FILE: .gitignore
================================================
.idea
explore_pandas_log.*
pandas_log/__pycache__
.ipynb_checkpoints
.cache
.eggs
*.egg-info
================================================
FILE: AUTHORS.rst
================================================
=======
Credits
=======
Development Lead
----------------
* Eyal Trabelsi <eyaltrabelsi@gmail.com>
Contributors
------------
* Charles Davis <charles.m.davis.iv@gmail.com>
================================================
FILE: CONTRIBUTING.rst
================================================
============
Contributing
============
Contributions are welcome, and they are greatly appreciated!
Every little bit helps, and credit will always be given.
The following sections detail a variety of ways to contribute,
as well as how to get started.
Types of Contributions
=======================
Write Documentation
--------------------
``pandas-log`` could always use more documentation,
whether as part of the official ``pandas-log`` docs, in docstrings, or the examples gallery.
During sprints, we require newcomers to the project to
first contribute a documentation fix before contributing a code fix.
Doing so has numerous benefits:
1. You become familiar with the project by first reading through the docs.
2. Your documentation contribution will be a pain point that you have full context on.
3. Your contribution will be impactful because documentation is the project's front-facing interface.
4. Your first contribution will be simpler, because you won't have to wrestle with build systems.
5. You can choose between getting set up locally first (recommended), or instead directly making edits on the GitHub web UI (also not a problem).
6. Every newcomer is equal in our eyes, and it's the most egalitarian way to get started (regardless of experience).
Remote contributors outside of sprints and prior contributors
who are joining us at the sprints need not adhere to this rule,
as a good prior assumption is that you are a motivated user of the library already.
If you have made a prior pull request to the library,
we would like to encourage you to mentor newcomers in lieu of coding contributions.
Documentation can come in many forms. For example, you might want to contribute:
- Fixes for a typographical, grammatical, or spelling error.
- Changes for a docstring that was unclear.
- Clarifications for installation/setup instructions that are unclear.
- Corrections to a sentence/phrase/word choice that didn't make sense.
- New example/tutorial notebooks using the library.
- Edits to existing tutorial notebooks with better code style.
In particular, contributing new tutorial notebooks and
improving the clarity of existing ones are great ways to
get familiar with the library and find pain points that
you can propose as fixes or enhancements to the library.
Report Bugs
------------
Report bugs at https://github.com/eyaltrabelsi/pandas-log/issues.
If you are reporting a bug, please include:
* Your operating system name and version.
* Any details about your local setup that might be helpful in troubleshooting.
* Detailed steps to reproduce the bug.
Fix Bugs
---------
Look through the GitHub issues for bugs.
Anything tagged with ``bug`` and ``available to hack on`` is open to
whoever wants to implement it.
Do be sure to claim the issue for yourself by indicating,
"I would like to work on this issue."
If you would like to discuss it further before going forward,
you are more than welcome to discuss on the GitHub issue tracker.
Submit Feedback
-----------------
The best way to send feedback is to file an issue at https://github.com/eyaltrabelsi/pandas-log/issues.
If you are proposing a feature:
* Explain in detail how it would work.
* Keep the scope as narrow as possible, to make it easier to implement.
* Remember that this is a volunteer-driven project, and that contributions
are welcome :)
Get Started!
====================
Ready to contribute? Here's how to set up ``pandas_log`` for local development.
1. Fork the `pandas_log` repo on GitHub: https://github.com/eyaltrabelsi/pandas-log.
2. Clone your fork locally::
$ git clone git@github.com:your_name_here/pandas_log.git
3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development::
$ mkvirtualenv pandas_log
$ cd pandas_log/
$ python setup.py develop
4. Create a branch for local development::
$ git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
5. When you're done making changes, check that your changes pass flake8 and the
tests, including testing other Python versions with tox::
$ flake8 pandas_log tests
$ python setup.py test or pytest
$ tox
To get flake8 and tox, just pip install them into your virtualenv.
6. Commit your changes and push your branch to GitHub::
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
7. Submit a pull request through the GitHub website.
Pull Request Guidelines
----------------------
Before you submit a pull request, check that it meets these guidelines:
1. The pull request should include tests.
2. If the pull request adds functionality, the docs should be updated. Put
your new functionality into a function with a docstring, and add the
feature to the list in README.rst.
Deploying
---------
A reminder for the maintainers on how to deploy.
Make sure all your changes are committed (including an entry in HISTORY.rst).
Then run::
$ bump2version patch # possible: major / minor / patch
$ git push
$ git push --tags
Travis will then deploy to PyPI if tests pass.
================================================
FILE: HISTORY.rst
================================================
=======
History
=======
0.0.0 (2019-09-18)
------------------
* First release on PyPI.
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2019, Eyal Trabelsi
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: MANIFEST.in
================================================
include CONTRIBUTING.rst
include LICENSE
include README.rst
recursive-include tests *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]
recursive-include docs *.rst conf.py Makefile make.bat *.jpg *.png *.gif
================================================
FILE: Makefile
================================================
cat.PHONY: clean clean-test clean-pyc clean-build docs help
.DEFAULT_GOAL := help
define BROWSER_PYSCRIPT
import os, webbrowser, sys
try:
from urllib import pathname2url
except:
from urllib.request import pathname2url
webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1])))
endef
export BROWSER_PYSCRIPT
define PRINT_HELP_PYSCRIPT
import re, sys
for line in sys.stdin:
make
if match:
target, help = match.groups()
print("%-20s %s" % (target, help))
endef
export PRINT_HELP_PYSCRIPT
BROWSER := python -c "$$BROWSER_PYSCRIPT"
help:
@python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
clean:
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f {} +
find . -name '__pycache__' -exec rm -fr {} +
rm -fr .tox/
rm -f .coverage
rm -fr htmlcov/
rm -fr .pytest_cache
rm -fr build/
rm -fr dist/
rm -fr .eggs/
find . -name '*.egg-info' -exec rm -fr {} +
find . -name '*.egg' -exec rm -f {} +
format:
isort -rc pandas_log -y -up -tc
black -l 79 pandas_log
test: ## run tests on every Python version with tox
tox
docs: ## generate Sphinx HTML documentation, including API docs
rm -f docs/pandas_log.rst
rm -f docs/modules.rst
sphinx-apidoc -o docs/ pandas_log
$(MAKE) -C docs clean
$(MAKE) -C docs html
$(BROWSER) docs/_build/html/index.html
release: dist ## package and upload a release
twine upload dist/*
dist: clean ## builds source and wheel package
python setup.py sdist
python setup.py bdist_wheel
ls -l dist
================================================
FILE: README.rst
================================================
==========
pandas-log
==========
.. image:: https://img.shields.io/pypi/v/pandas_log.svg
:target: https://pypi.python.org/pypi/pandas_log
.. image:: https://img.shields.io/travis/eyaltrabelsi/pandas-log.svg
:target: https://travis-ci.org/eyaltrabelsi/pandas-log
.. image:: https://readthedocs.org/projects/pandas-log/badge/?version=latest
:target: https://pandas-log.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://pyup.io/repos/github/eyaltrabelsi/pandas-log/shield.svg
:target: https://pyup.io/repos/github/eyaltrabelsi/pandas-log/
:alt: Updates
The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common functions, such as ``.query``, ``.apply``, ``.merge``, ``.group_by`` and more.
Why pandas-log?
---------------
``Pandas-log`` is a Python implementation of the R package ``tidylog``, and provides a feedback about basic pandas operations.
The pandas has been invaluable for the data science ecosystem and usually consists of a series of steps that involve transforming raw data into an understandable/usable format.
These series of steps need to be run in a certain sequence and if the result is unexpected it's hard to understand what happened. ``Pandas-log`` log metadata on each operation which will allow to pinpoint the issues.
Lets look at an example, first we need to load ``pandas-log`` after ``pandas`` and create a dataframe:
.. code-block:: python
import pandas
import pandas_log
with pandas_log.enable():
df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
"toy": [np.nan, 'Batmobile', 'Bullwhip'],
"born": [pd.NaT, pd.Timestamp("1940-04-25"), pd.NaT]})
``pandas-log`` will give you feedback, for instance when filtering a data frame or adding a new variable:
.. code-block:: python
df.assign(toy=lambda x: x.toy.map(str.lower))
.query("name != 'Batman'")
``pandas-log`` can be especially helpful in longer pipes:
.. code-block:: python
df.assign(toy=lambda x: x.toy.map(str.lower))
.query("name != 'Batman'")
.dropna()\
.assign(lower_name=lambda x: x.name.map(str.lower))
.reset_index()
For medium article `go here
<https://towardsdatascience.com/introducing-pandas-log-3240a5e57e21>`_
For a full walkthrough `go here
<https://github.com/eyaltrabelsi/pandas-log/blob/master/examples/pandas_log_intro.ipynb>`_
Installation
------------
``pandas-log`` is currently installable from PyPI:
.. code-block:: bash
pip install pandas-log
Contributing
------------
Follow `contribution docs
<https://pandas-log.readthedocs.io/en/latest/contributing.html>`_ for a full description of the process of contributing to ``pandas-log``.
================================================
FILE: TODO.rst
================================================
- Add statistics from tidylog from it.
- add formats of warnings.
- todo check if copy is first
- add logs for series as well.
================================================
FILE: docs/Makefile
================================================
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = python -msphinx
SPHINXPROJ = pandas_log
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
================================================
FILE: docs/conf.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# pandas_log documentation build configuration file, created by
# sphinx-quickstart on Fri Jun 9 13:47:02 2017.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another
# directory, add these directories to sys.path here. If the directory is
# relative to the documentation root, use os.path.abspath to make it
# absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
# import pandas_log
# -- General configuration ---------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'pandas-log'
copyright = u"2019, Eyal Trabelsi"
author = u"Eyal Trabelsi"
# The version info for the project you're documenting, acts as replacement
# for |version| and |release|, also used in various other places throughout
# the built documents.
#
# The short X.Y version.
version = '0.1.1'
# The full version, including alpha/beta/rc tags.
release = '0.1.1'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output -------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a
# theme further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# -- Options for HTMLHelp output ---------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'pandas_logdoc'
# -- Options for LaTeX output ------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto, manual, or own class]).
latex_documents = [
(master_doc, 'pandas_log.tex',
u'pandas-log Documentation',
u'Eyal Trabelsi', 'manual'),
]
# -- Options for manual page output ------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'pandas_log',
u'pandas-log Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'pandas_log',
u'pandas-log Documentation',
author,
'pandas_log',
'One line description of project.',
'Miscellaneous'),
]
================================================
FILE: docs/contributing.rst
================================================
.. include:: ../CONTRIBUTING.rst
================================================
FILE: docs/index.rst
================================================
Welcome to pandas-log's documentation!
======================================
.. toctree::
:maxdepth: 2
:caption: Contents:
readme
installation
usage
modules
contributing
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
================================================
FILE: docs/installation.rst
================================================
.. highlight:: shell
============
Installation
============
Stable release
--------------
To install pandas-log, run this command in your terminal:
.. code-block:: console
$ pip install pandas-log
This is the preferred method to install pandas-log, as it will always install the most recent stable release.
If you don't have `pip`_ installed, this `Python installation guide`_ can guide
you through the process.
.. _pip: https://pip.pypa.io
.. _Python installation guide: http://docs.python-guide.org/en/latest/starting/installation/
From sources
------------
The sources for pandas-log can be downloaded from the `Github repo`_.
You can either clone the public repository:
.. code-block:: console
$ git clone git://github.com/eyaltrabelsi/pandas-log
Or download the `tarball`_:
.. code-block:: console
$ curl -OJL https://github.com/eyaltrabelsi/pandas-log/tarball/master
Once you have a copy of the source, you can install it with:
.. code-block:: console
$ python setup.py install
.. _Github repo: https://github.com/eyaltrabelsi/pandas-log
.. _tarball: https://github.com/eyaltrabelsi/pandas-log/tarball/master
================================================
FILE: docs/modules.rst
================================================
pandas_log
==========
.. toctree::
:maxdepth: 4
pandas_log
================================================
FILE: docs/pandas_log.rst
================================================
pandas\_log package
===================
Submodules
----------
pandas\_log.aop\_utils module
-----------------------------
.. automodule:: pandas_log.aop_utils
:members:
:undoc-members:
:show-inheritance:
pandas\_log.dataframe\_logger module
------------------------------------
.. automodule:: pandas_log.dataframe_logger
:members:
:undoc-members:
:show-inheritance:
pandas\_log.pandas\_log module
------------------------------
.. automodule:: pandas_log.pandas_log
:members:
:undoc-members:
:show-inheritance:
pandas\_log.patched\_logs\_functions module
-------------------------------------------
.. automodule:: pandas_log.patched_logs_functions
:members:
:undoc-members:
:show-inheritance:
pandas\_log.settings module
---------------------------
.. automodule:: pandas_log.settings
:members:
:undoc-members:
:show-inheritance:
pandas\_log.timer module
------------------------
.. automodule:: pandas_log.timer
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: pandas_log
:members:
:undoc-members:
:show-inheritance:
================================================
FILE: docs/readme.rst
================================================
===============
Why pandas-log?
===============
``Pandas-log`` is a Python implementation of the R package ``tidylog``, and provides a feedback about basic pandas operations.
The pandas has been invaluable for the data science ecosystem and usually consists of a series of steps that involve transforming raw data into an understandable/usable format.
These series of steps need to be run in a certain sequence and if the result is unexpected it's hard to understand what happened.
``Pandas-log`` log metadata on each operation which will allow to pinpoint the issues like:
- Wrong predicate expressions.
- Copying of our DataFrames.
- Wrong joins/merge.
- Performance Issues.
- More
For medium article `go here
<https://towardsdatascience.com/introducing-pandas-log-3240a5e57e21>`_
For a full walkthrough `go here
<https://github.com/eyaltrabelsi/pandas-log/blob/master/examples/pandas_log_intro.ipynb>`_
================================================
FILE: docs/usage.rst
================================================
Usage
=====
For a cleaner use-case I would `go here
<https://towardsdatascience.com/introducing-pandas-log-3240a5e57e21>`_
First we need to load some libraries including pandas-log
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
import pandas as pd
import numpy as np
import pandas_log
Let’s take a look at our dataset:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
df = pd.read_csv("pokemon.csv")
df.head(10)
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
<th>legendary</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>1</td>
<td>Bulbasaur</td>
<td>Grass</td>
<td>Poison</td>
<td>318</td>
<td>45</td>
<td>49</td>
<td>49</td>
<td>65</td>
<td>65</td>
<td>45</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>1</th>
<td>2</td>
<td>Ivysaur</td>
<td>Grass</td>
<td>Poison</td>
<td>405</td>
<td>60</td>
<td>62</td>
<td>63</td>
<td>80</td>
<td>80</td>
<td>60</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>2</th>
<td>3</td>
<td>Venusaur</td>
<td>Grass</td>
<td>Poison</td>
<td>525</td>
<td>80</td>
<td>82</td>
<td>83</td>
<td>100</td>
<td>100</td>
<td>80</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>3</th>
<td>3</td>
<td>VenusaurMega Venusaur</td>
<td>Grass</td>
<td>Poison</td>
<td>625</td>
<td>80</td>
<td>100</td>
<td>123</td>
<td>122</td>
<td>120</td>
<td>80</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>4</th>
<td>4</td>
<td>Charmander</td>
<td>Fire</td>
<td>NaN</td>
<td>309</td>
<td>39</td>
<td>52</td>
<td>43</td>
<td>60</td>
<td>50</td>
<td>65</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>5</th>
<td>5</td>
<td>Charmeleon</td>
<td>Fire</td>
<td>NaN</td>
<td>405</td>
<td>58</td>
<td>64</td>
<td>58</td>
<td>80</td>
<td>65</td>
<td>80</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>6</th>
<td>6</td>
<td>Charizard</td>
<td>Fire</td>
<td>Flying</td>
<td>534</td>
<td>78</td>
<td>84</td>
<td>78</td>
<td>109</td>
<td>85</td>
<td>100</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>7</th>
<td>6</td>
<td>CharizardMega Charizard X</td>
<td>Fire</td>
<td>Dragon</td>
<td>634</td>
<td>78</td>
<td>130</td>
<td>111</td>
<td>130</td>
<td>85</td>
<td>100</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>8</th>
<td>6</td>
<td>CharizardMega Charizard Y</td>
<td>Fire</td>
<td>Flying</td>
<td>634</td>
<td>78</td>
<td>104</td>
<td>78</td>
<td>159</td>
<td>115</td>
<td>100</td>
<td>1</td>
<td>False</td>
</tr>
<tr>
<th>9</th>
<td>7</td>
<td>Squirtle</td>
<td>Water</td>
<td>NaN</td>
<td>314</td>
<td>44</td>
<td>48</td>
<td>65</td>
<td>50</td>
<td>64</td>
<td>43</td>
<td>1</td>
<td>False</td>
</tr>
</tbody>
</table>
</div>
Lets say we want to find out:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Who is the weakest non-legendary fire pokemon?
----------------------------------------------
The strategy will probably be something like:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Filter out legendary pokemons using ``.query()`` .
2. Keep only fire pokemons using ``.query()`` .
3. Drop Legendary column using ``.drop()`` .
4. Keep the weakest pokemon among them using ``.nsmallest()`` .
5. Reset index using ``.reset_index()`` .
.. code:: ipython3
res = (df.copy()
.query("legendary==0")
.query("type_1=='fire' or type_2=='fire'")
.drop("legendary", axis=1)
.nsmallest(1,"total")
.reset_index(drop=True)
)
res
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
OH NOO!!! Our code does not work !! We got no records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If only there was a way to track those issue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Fortunetly thats what **pandas-log** is for! either as a global function
or context manager. This the example with pandas_log’s
``context_manager``.
.. code:: ipython3
with pandas_log.enable():
res = (df.query("legendary==0")
.query("type_1=='fire' or type_2=='fire'")
.drop("legendary", axis=1)
.nsmallest(1,"total")
)
res
.. parsed-literal::
1) [1mquery[0m(expr="legendary==0", inplace=False):
[4mMetadata[0m:
* Removed 65 rows (8.125%), 735 rows remaining.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 199.4 kB.
* Output Dataframe size is 188.5 kB.
2) [1mquery[0m(expr="type_1=='fire' or type_2=='fire'", inplace=False):
[4mMetadata[0m:
* Removed 735 rows (100.0%), 0 rows remaining.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 188.5 kB.
* Output Dataframe size is 0 Bytes.
3) [1mdrop[0m(labels="legendary", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'):
[4mMetadata[0m:
* Removed the following columns (legendary) now only have the following columns (attack, sp_def, speed, hp, total, type_2, #, name, type_1, generation, defense, sp_atk).
* No change in number of rows of input df.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 0 Bytes.
* Output Dataframe size is 0 Bytes.
4) [1mnsmallest[0m(n=1, columns="total", keep='first'):
[4mMetadata[0m:
* Picked 1 smallest rows by columns (total).
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 0 Bytes.
* Output Dataframe size is 0 Bytes.
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
We can see clearly that in the second step (``.query()``) we filter all the rows!! and indeed we should of writen Fire as oppose to fire
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
res = (df.copy()
.query("type_1=='Fire' or type_2=='Fire'")
.query("legendary==0")
.drop("legendary", axis=1)
.nsmallest(1,"total")
.reset_index(drop=True)
)
res
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>218</td>
<td>Slugma</td>
<td>Fire</td>
<td>NaN</td>
<td>250</td>
<td>40</td>
<td>40</td>
<td>40</td>
<td>70</td>
<td>40</td>
<td>20</td>
<td>2</td>
</tr>
</tbody>
</table>
</div>
Whoala we got Slugma !!!!!!!!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Some more advance usage
-----------------------
One can use verbose variable which allows lower level logs functionalities like whether the dataframe was copied as part of pipeline.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This can explain comparision issues.
.. code:: ipython3
with pandas_log.enable(verbose=True):
res = (df.query("legendary==0")
.query("type_1=='Fire' or type_2=='Fire'")
.drop("legendary", axis=1)
.nsmallest(1,"total")
.reset_index(drop=True)
)
res
.. parsed-literal::
1) [1mquery[0m(expr="legendary==0", inplace=False):
[4mMetadata[0m:
* Removed 65 rows (8.125%), 735 rows remaining.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 199.4 kB.
* Output Dataframe size is 188.5 kB.
2) [1mquery[0m(expr="type_1=='Fire' or type_2=='Fire'", inplace=False):
[4mMetadata[0m:
* Removed 679 rows (92.38095238095238%), 56 rows remaining.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 188.5 kB.
* Output Dataframe size is 14.4 kB.
3) [1mdrop[0m(labels="legendary", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'):
[4mMetadata[0m:
* Removed the following columns (legendary) now only have the following columns (attack, sp_def, speed, hp, total, type_2, #, name, type_1, generation, defense, sp_atk).
* No change in number of rows of input df.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.4 kB.
* Output Dataframe size is 14.3 kB.
X) [1m__getitem__[0m(key="total"):
Metadata:
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.3 kB.
* Output Dataframe size is 896 Bytes.
X) [1mcopy[0m(deep=True):
[4mMetadata[0m:
* Using default strategy (some metric might not be relevant).
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.3 kB.
* Output Dataframe size is 14.3 kB.
X) [1mreset_index[0m(level=None, drop=False, inplace=False, col_level=0, col_fill=''):
Metadata:
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.3 kB.
* Output Dataframe size is 14.0 kB.
X) [1m__getitem__[0m(key="total"):
Metadata:
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.0 kB.
* Output Dataframe size is 576 Bytes.
4) [1mnsmallest[0m(n=1, columns="total", keep='first'):
[4mMetadata[0m:
* Picked 1 smallest rows by columns (total).
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.3 kB.
* Output Dataframe size is 236 Bytes.
X) [1mcopy[0m(deep=True):
[4mMetadata[0m:
* Using default strategy (some metric might not be relevant).
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 236 Bytes.
* Output Dataframe size is 236 Bytes.
X) [1mreset_index[0m(level=None, drop=False, inplace=False, col_level=0, col_fill=''):
Metadata:
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 236 Bytes.
* Output Dataframe size is 356 Bytes.
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>218</td>
<td>Slugma</td>
<td>Fire</td>
<td>NaN</td>
<td>250</td>
<td>40</td>
<td>40</td>
<td>40</td>
<td>70</td>
<td>40</td>
<td>20</td>
<td>2</td>
</tr>
</tbody>
</table>
</div>
as we can see after both the drop and nsmallest functions the dataframe
was being copied
One can use silent variable which allows to suppress stdout
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
with pandas_log.enable(silent=True):
res = (df.copy()
.query("legendary==0")
.query("type_1=='Fire' or type_2=='Fire'")
.drop("legendary", axis=1)
.nsmallest(1,"total")
.reset_index(drop=True)
)
res
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>218</td>
<td>Slugma</td>
<td>Fire</td>
<td>NaN</td>
<td>250</td>
<td>40</td>
<td>40</td>
<td>40</td>
<td>70</td>
<td>40</td>
<td>20</td>
<td>2</td>
</tr>
</tbody>
</table>
</div>
One can use full_signature variable which allows to suppress the signature
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: ipython3
with pandas_log.enable(full_signature=False):
res = (df.query("legendary==0")
.query("type_1=='Fire' or type_2=='Fire'")
.drop("legendary", axis=1)
.nsmallest(1,"total")
.reset_index(drop=True)
)
res
.. parsed-literal::
1) [1mquery[0m(expr="legendary==0", inplace=False):
[4mMetadata[0m:
* Removed 65 rows (8.125%), 735 rows remaining.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 199.4 kB.
* Output Dataframe size is 188.5 kB.
2) [1mquery[0m(expr="type_1=='Fire' or type_2=='Fire'"):
[4mMetadata[0m:
* Removed 679 rows (92.38095238095238%), 56 rows remaining.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 188.5 kB.
* Output Dataframe size is 14.4 kB.
3) [1mdrop[0m(labels="legendary"):
[4mMetadata[0m:
* Removed the following columns (legendary) now only have the following columns (attack, sp_def, speed, hp, total, type_2, #, name, type_1, generation, defense, sp_atk).
* No change in number of rows of input df.
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.4 kB.
* Output Dataframe size is 14.3 kB.
4) [1mnsmallest[0m(n=1, columns="total"):
[4mMetadata[0m:
* Picked 1 smallest rows by columns (total).
[4mExecution Stats[0m:
* Execution time: Step Took a moment seconds..
* Input Dataframe size is 14.3 kB.
* Output Dataframe size is 236 Bytes.
.. raw:: html
<div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>#</th>
<th>name</th>
<th>type_1</th>
<th>type_2</th>
<th>total</th>
<th>hp</th>
<th>attack</th>
<th>defense</th>
<th>sp_atk</th>
<th>sp_def</th>
<th>speed</th>
<th>generation</th>
</tr>
</thead>
<tbody>
<tr>
<th>0</th>
<td>218</td>
<td>Slugma</td>
<td>Fire</td>
<td>NaN</td>
<td>250</td>
<td>40</td>
<td>40</td>
<td>40</td>
<td>70</td>
<td>40</td>
<td>20</td>
<td>2</td>
</tr>
</tbody>
</table>
</div>
================================================
FILE: examples/README.rst
================================================
Examples
========
This folder contains jupyter notebooks demonstrating different ways to
implement pandas-log in your workflow.
Guidelines
~~~~~~~~~~
When contributing example notebooks please include a short explanation of
where the data came from and what it contains
================================================
FILE: examples/__init__.py
================================================
================================================
FILE: examples/pandas_log_intro.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pandas-Log Usage Walkthrough"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why pandas-log?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Pandas-log is a Python implementation of the R package tidylog, and provides a feedback about basic pandas operations.\n",
"\n",
"The pandas has been invaluable for the data science ecosystem and usually consists of a series of steps that involve transforming raw data into an understandable/usable format. These series of steps need to be run in a certain sequence and if the result is unexpected it's hard to understand what happened. Pandas-log log metadata on each operation which will allow to pinpoint the issues."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas-log Demo\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### First we need to load some libraries including pandas-log"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import pandas_log "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Let's take a look at our dataset:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" <th>legendary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>Bulbasaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>318</td>\n",
" <td>45</td>\n",
" <td>49</td>\n",
" <td>49</td>\n",
" <td>65</td>\n",
" <td>65</td>\n",
" <td>45</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>Ivysaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>405</td>\n",
" <td>60</td>\n",
" <td>62</td>\n",
" <td>63</td>\n",
" <td>80</td>\n",
" <td>80</td>\n",
" <td>60</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>Venusaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>525</td>\n",
" <td>80</td>\n",
" <td>82</td>\n",
" <td>83</td>\n",
" <td>100</td>\n",
" <td>100</td>\n",
" <td>80</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>3</td>\n",
" <td>VenusaurMega Venusaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>625</td>\n",
" <td>80</td>\n",
" <td>100</td>\n",
" <td>123</td>\n",
" <td>122</td>\n",
" <td>120</td>\n",
" <td>80</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>4</td>\n",
" <td>Charmander</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>309</td>\n",
" <td>39</td>\n",
" <td>52</td>\n",
" <td>43</td>\n",
" <td>60</td>\n",
" <td>50</td>\n",
" <td>65</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>5</td>\n",
" <td>Charmeleon</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>405</td>\n",
" <td>58</td>\n",
" <td>64</td>\n",
" <td>58</td>\n",
" <td>80</td>\n",
" <td>65</td>\n",
" <td>80</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>6</td>\n",
" <td>Charizard</td>\n",
" <td>Fire</td>\n",
" <td>Flying</td>\n",
" <td>534</td>\n",
" <td>78</td>\n",
" <td>84</td>\n",
" <td>78</td>\n",
" <td>109</td>\n",
" <td>85</td>\n",
" <td>100</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>6</td>\n",
" <td>CharizardMega Charizard X</td>\n",
" <td>Fire</td>\n",
" <td>Dragon</td>\n",
" <td>634</td>\n",
" <td>78</td>\n",
" <td>130</td>\n",
" <td>111</td>\n",
" <td>130</td>\n",
" <td>85</td>\n",
" <td>100</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>6</td>\n",
" <td>CharizardMega Charizard Y</td>\n",
" <td>Fire</td>\n",
" <td>Flying</td>\n",
" <td>634</td>\n",
" <td>78</td>\n",
" <td>104</td>\n",
" <td>78</td>\n",
" <td>159</td>\n",
" <td>115</td>\n",
" <td>100</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>7</td>\n",
" <td>Squirtle</td>\n",
" <td>Water</td>\n",
" <td>NaN</td>\n",
" <td>314</td>\n",
" <td>44</td>\n",
" <td>48</td>\n",
" <td>65</td>\n",
" <td>50</td>\n",
" <td>64</td>\n",
" <td>43</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" # name type_1 type_2 total hp attack defense \\\n",
"0 1 Bulbasaur Grass Poison 318 45 49 49 \n",
"1 2 Ivysaur Grass Poison 405 60 62 63 \n",
"2 3 Venusaur Grass Poison 525 80 82 83 \n",
"3 3 VenusaurMega Venusaur Grass Poison 625 80 100 123 \n",
"4 4 Charmander Fire NaN 309 39 52 43 \n",
"5 5 Charmeleon Fire NaN 405 58 64 58 \n",
"6 6 Charizard Fire Flying 534 78 84 78 \n",
"7 6 CharizardMega Charizard X Fire Dragon 634 78 130 111 \n",
"8 6 CharizardMega Charizard Y Fire Flying 634 78 104 78 \n",
"9 7 Squirtle Water NaN 314 44 48 65 \n",
"\n",
" sp_atk sp_def speed generation legendary \n",
"0 65 65 45 1 False \n",
"1 80 80 60 1 False \n",
"2 100 100 80 1 False \n",
"3 122 120 80 1 False \n",
"4 60 50 65 1 False \n",
"5 80 65 80 1 False \n",
"6 109 85 100 1 False \n",
"7 130 85 100 1 False \n",
"8 159 115 100 1 False \n",
"9 50 64 43 1 False "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_csv(\"pokemon.csv\")\n",
"df.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Lets say we want to find out:\n",
"## Who is the weakest non-legendary fire pokemon?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"fire_pokemons.jpg\" width=\"540\" height=\"340\" align=\"left\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### The strategy will probably be something like:\n",
"1. Filter out legendary pokemons using `.query()` .\n",
"1. Keep only fire pokemons using `.query()` .\n",
"1. Drop Legendary column using `.drop()` .\n",
"1. Keep the weakest pokemon among them using `.nsmallest()` .\n",
"1. Reset index using `.reset_index()` ."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Empty DataFrame\n",
"Columns: [#, name, type_1, type_2, total, hp, attack, defense, sp_atk, sp_def, speed, generation]\n",
"Index: []"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"res = (df.copy()\n",
" .query(\"legendary==0\")\n",
" .query(\"type_1=='fire' or type_2=='fire'\")\n",
" .drop(\"legendary\", axis=1)\n",
" .nsmallest(1,\"total\")\n",
" .reset_index(drop=True)\n",
" )\n",
"res "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### OH NOO!!! Our code does not work !! We got no records\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"shocked.gif\" width=\"490\" height=\"340\" align=\"left\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### If only there was a way to track those issue\n",
"\n",
"Fortunetly thats what **pandas-log** is for! either as a global function or context manager.\n",
"This the example with pandas_log's `context_manager`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"1) \u001b[1mquery\u001b[0m(expr=\"legendary==0\", inplace=False):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Removed 65 rows (8.125%), 735 rows remaining.\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.002567 seconds.\n",
"\t* Input Dataframe size is 199.4 kB.\n",
"\t* Output Dataframe size is 188.5 kB.\n",
"\t\n",
"\n",
"2) \u001b[1mquery\u001b[0m(expr=\"type_1=='fire' or type_2=='fire'\", inplace=False):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Removed 735 rows (100.0%), 0 rows remaining.\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.003322 seconds.\n",
"\t* Input Dataframe size is 188.5 kB.\n",
"\t* Output Dataframe size is 0 Bytes.\n",
"\t\n",
"\n",
"3) \u001b[1mdrop\u001b[0m(labels=\"legendary\", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* No change in number of rows of input df.\n",
"\t* Removed the following columns (legendary) now only have the following columns (type_2, total, hp, defense, sp_def, speed, generation, sp_atk, type_1, attack, name, #).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.00073 seconds.\n",
"\t* Input Dataframe size is 0 Bytes.\n",
"\t* Output Dataframe size is 0 Bytes.\n",
"\t\n",
"\n",
"4) \u001b[1mnsmallest\u001b[0m(n=1, columns=\"total\", keep='first'):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Picked 1 smallest rows by columns (total).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.010866 seconds.\n",
"\t* Input Dataframe size is 0 Bytes.\n",
"\t* Output Dataframe size is 0 Bytes.\n",
"\t\u001b[4mTips\u001b[0m:\n",
"\t* Number of rows didn't change, if you are working on the entire dataset you can remove this operation.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Empty DataFrame\n",
"Columns: [#, name, type_1, type_2, total, hp, attack, defense, sp_atk, sp_def, speed, generation]\n",
"Index: []"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with pandas_log.enable():\n",
" res = (df.query(\"legendary==0\")\n",
" .query(\"type_1=='fire' or type_2=='fire'\")\n",
" .drop(\"legendary\", axis=1)\n",
" .nsmallest(1,\"total\")\n",
" )\n",
"res "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### We can see clearly that in the second step (`.query()`) we filter all the rows!! and indeed we should of writen Fire as oppose to fire"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>218</td>\n",
" <td>Slugma</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>250</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>70</td>\n",
" <td>40</td>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n",
"0 218 Slugma Fire NaN 250 40 40 40 70 40 \n",
"\n",
" speed generation \n",
"0 20 2 "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"res = (df.copy()\n",
" .query(\"type_1=='Fire' or type_2=='Fire'\")\n",
" .query(\"legendary==0\") \n",
" .drop(\"legendary\", axis=1) \n",
" .nsmallest(1,\"total\")\n",
" .reset_index(drop=True)\n",
" )\n",
"res "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Whoala we got Slugma !!!!!!!!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"slugma.jpg\" width=\"250\" height=\"340\" align=\"left\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Some more advance usage\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### One can use verbose variable which allows lower level logs functionalities like whether the dataframe was copied as part of pipeline.\n",
"This can explain comparision issues."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"1) \u001b[1mquery\u001b[0m(expr=\"legendary==0\", inplace=False):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Removed 65 rows (8.125%), 735 rows remaining.\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.003288 seconds.\n",
"\t* Input Dataframe size is 199.4 kB.\n",
"\t* Output Dataframe size is 188.5 kB.\n",
"\t\n",
"\n",
"2) \u001b[1mquery\u001b[0m(expr=\"type_1=='Fire' or type_2=='Fire'\", inplace=False):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Removed 679 rows (92.38095238095238%), 56 rows remaining.\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.003339 seconds.\n",
"\t* Input Dataframe size is 188.5 kB.\n",
"\t* Output Dataframe size is 14.4 kB.\n",
"\t\n",
"\n",
"3) \u001b[1mdrop\u001b[0m(labels=\"legendary\", axis=0, index=None, columns=None, level=None, inplace=False, errors='raise'):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* No change in number of rows of input df.\n",
"\t* Removed the following columns (legendary) now only have the following columns (hp, sp_def, generation, name, #, attack, type_2, sp_atk, type_1, defense, total, speed).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.000604 seconds.\n",
"\t* Input Dataframe size is 14.4 kB.\n",
"\t* Output Dataframe size is 14.3 kB.\n",
"\t\u001b[4mTips\u001b[0m:\n",
"\t* Number of rows didn't change, if you are working on the entire dataset you can remove this operation.\n",
"\n",
"X) \u001b[1m__getitem__\u001b[0m(key=\"total\"):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* After transformation we received Series\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 3.3e-05 seconds.\n",
"\t* Input Dataframe size is 14.3 kB.\n",
"\t* Output Dataframe size is 896 Bytes.\n",
"\t\n",
"\n",
"X) \u001b[1mcopy\u001b[0m(deep=True):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Using default strategy (some metric might not be relevant).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.000318 seconds.\n",
"\t* Input Dataframe size is 14.3 kB.\n",
"\t* Output Dataframe size is 14.3 kB.\n",
"\t\n",
"\n",
"X) \u001b[1mreset_index\u001b[0m(level=None, drop=False, inplace=False, col_level=0, col_fill=''):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Using default strategy (some metric might not be relevant).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.00373 seconds.\n",
"\t* Input Dataframe size is 14.3 kB.\n",
"\t* Output Dataframe size is 14.0 kB.\n",
"\t\n",
"\n",
"X) \u001b[1m__getitem__\u001b[0m(key=\"total\"):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* After transformation we received Series\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 1.1e-05 seconds.\n",
"\t* Input Dataframe size is 14.0 kB.\n",
"\t* Output Dataframe size is 576 Bytes.\n",
"\t\n",
"\n",
"4) \u001b[1mnsmallest\u001b[0m(n=1, columns=\"total\", keep='first'):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Picked 1 smallest rows by columns (total).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.012324 seconds.\n",
"\t* Input Dataframe size is 14.3 kB.\n",
"\t* Output Dataframe size is 236 Bytes.\n",
"\t\n",
"\n",
"X) \u001b[1mcopy\u001b[0m(deep=True):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Using default strategy (some metric might not be relevant).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.000137 seconds.\n",
"\t* Input Dataframe size is 236 Bytes.\n",
"\t* Output Dataframe size is 236 Bytes.\n",
"\t\n",
"\n",
"X) \u001b[1mreset_index\u001b[0m(level=None, drop=False, inplace=False, col_level=0, col_fill=''):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Using default strategy (some metric might not be relevant).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.002781 seconds.\n",
"\t* Input Dataframe size is 236 Bytes.\n",
"\t* Output Dataframe size is 356 Bytes.\n",
"\t\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>218</td>\n",
" <td>Slugma</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>250</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>70</td>\n",
" <td>40</td>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n",
"0 218 Slugma Fire NaN 250 40 40 40 70 40 \n",
"\n",
" speed generation \n",
"0 20 2 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with pandas_log.enable(verbose=True):\n",
" res = (df.query(\"legendary==0\")\n",
" .query(\"type_1=='Fire' or type_2=='Fire'\")\n",
" .drop(\"legendary\", axis=1)\n",
" .nsmallest(1,\"total\")\n",
" .reset_index(drop=True)\n",
" )\n",
"res "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"as we can see after both the drop and nsmallest functions the dataframe was being copied"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### One can use silent variable which allows to suppress stdout"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>218</td>\n",
" <td>Slugma</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>250</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>70</td>\n",
" <td>40</td>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n",
"0 218 Slugma Fire NaN 250 40 40 40 70 40 \n",
"\n",
" speed generation \n",
"0 20 2 "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with pandas_log.enable(silent=True):\n",
" res = (df.copy()\n",
" .query(\"legendary==0\")\n",
" .query(\"type_1=='Fire' or type_2=='Fire'\")\n",
" .drop(\"legendary\", axis=1)\n",
" .nsmallest(1,\"total\")\n",
" .reset_index(drop=True)\n",
" )\n",
"res "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### One can use full_signature variable which allows to suppress the signature"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"1) \u001b[1mquery\u001b[0m(expr=\"legendary==0\", inplace=False):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Removed 65 rows (8.125%), 735 rows remaining.\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.001866 seconds.\n",
"\t* Input Dataframe size is 199.4 kB.\n",
"\t* Output Dataframe size is 188.5 kB.\n",
"\t\n",
"\n",
"2) \u001b[1mquery\u001b[0m(expr=\"type_1=='Fire' or type_2=='Fire'\"):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Removed 679 rows (92.38095238095238%), 56 rows remaining.\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.002914 seconds.\n",
"\t* Input Dataframe size is 188.5 kB.\n",
"\t* Output Dataframe size is 14.4 kB.\n",
"\t\n",
"\n",
"3) \u001b[1mdrop\u001b[0m(labels=\"legendary\"):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* No change in number of rows of input df.\n",
"\t* Removed the following columns (legendary) now only have the following columns (hp, sp_def, generation, name, #, attack, type_2, sp_atk, type_1, defense, total, speed).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.000561 seconds.\n",
"\t* Input Dataframe size is 14.4 kB.\n",
"\t* Output Dataframe size is 14.3 kB.\n",
"\t\u001b[4mTips\u001b[0m:\n",
"\t* Number of rows didn't change, if you are working on the entire dataset you can remove this operation.\n",
"\n",
"4) \u001b[1mnsmallest\u001b[0m(n=1, columns=\"total\"):\n",
"\t\u001b[4mMetadata\u001b[0m:\n",
"\t* Picked 1 smallest rows by columns (total).\n",
"\t\u001b[4mExecution Stats\u001b[0m:\n",
"\t* Execution time: Step Took 0.008674 seconds.\n",
"\t* Input Dataframe size is 14.3 kB.\n",
"\t* Output Dataframe size is 236 Bytes.\n",
"\t\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>name</th>\n",
" <th>type_1</th>\n",
" <th>type_2</th>\n",
" <th>total</th>\n",
" <th>hp</th>\n",
" <th>attack</th>\n",
" <th>defense</th>\n",
" <th>sp_atk</th>\n",
" <th>sp_def</th>\n",
" <th>speed</th>\n",
" <th>generation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>218</td>\n",
" <td>Slugma</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>250</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>40</td>\n",
" <td>70</td>\n",
" <td>40</td>\n",
" <td>20</td>\n",
" <td>2</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" # name type_1 type_2 total hp attack defense sp_atk sp_def \\\n",
"0 218 Slugma Fire NaN 250 40 40 40 70 40 \n",
"\n",
" speed generation \n",
"0 20 2 "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with pandas_log.enable(full_signature=False):\n",
" res = (df.query(\"legendary==0\")\n",
" .query(\"type_1=='Fire' or type_2=='Fire'\")\n",
" .drop(\"legendary\", axis=1)\n",
" .nsmallest(1,\"total\")\n",
" .reset_index(drop=True)\n",
" )\n",
"res "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: examples/pokemon.csv
================================================
#,name,type_1,type_2,total,hp,attack,defense,sp_atk,sp_def,speed,generation,legendary
1,Bulbasaur,Grass,Poison,318,45,49,49,65,65,45,1,False
2,Ivysaur,Grass,Poison,405,60,62,63,80,80,60,1,False
3,Venusaur,Grass,Poison,525,80,82,83,100,100,80,1,False
3,VenusaurMega Venusaur,Grass,Poison,625,80,100,123,122,120,80,1,False
4,Charmander,Fire,,309,39,52,43,60,50,65,1,False
5,Charmeleon,Fire,,405,58,64,58,80,65,80,1,False
6,Charizard,Fire,Flying,534,78,84,78,109,85,100,1,False
6,CharizardMega Charizard X,Fire,Dragon,634,78,130,111,130,85,100,1,False
6,CharizardMega Charizard Y,Fire,Flying,634,78,104,78,159,115,100,1,False
7,Squirtle,Water,,314,44,48,65,50,64,43,1,False
8,Wartortle,Water,,405,59,63,80,65,80,58,1,False
9,Blastoise,Water,,530,79,83,100,85,105,78,1,False
9,BlastoiseMega Blastoise,Water,,630,79,103,120,135,115,78,1,False
10,Caterpie,Bug,,195,45,30,35,20,20,45,1,False
11,Metapod,Bug,,205,50,20,55,25,25,30,1,False
12,Butterfree,Bug,Flying,395,60,45,50,90,80,70,1,False
13,Weedle,Bug,Poison,195,40,35,30,20,20,50,1,False
14,Kakuna,Bug,Poison,205,45,25,50,25,25,35,1,False
15,Beedrill,Bug,Poison,395,65,90,40,45,80,75,1,False
15,BeedrillMega Beedrill,Bug,Poison,495,65,150,40,15,80,145,1,False
16,Pidgey,Normal,Flying,251,40,45,40,35,35,56,1,False
17,Pidgeotto,Normal,Flying,349,63,60,55,50,50,71,1,False
18,Pidgeot,Normal,Flying,479,83,80,75,70,70,101,1,False
18,PidgeotMega Pidgeot,Normal,Flying,579,83,80,80,135,80,121,1,False
19,Rattata,Normal,,253,30,56,35,25,35,72,1,False
20,Raticate,Normal,,413,55,81,60,50,70,97,1,False
21,Spearow,Normal,Flying,262,40,60,30,31,31,70,1,False
22,Fearow,Normal,Flying,442,65,90,65,61,61,100,1,False
23,Ekans,Poison,,288,35,60,44,40,54,55,1,False
24,Arbok,Poison,,438,60,85,69,65,79,80,1,False
25,Pikachu,Electric,,320,35,55,40,50,50,90,1,False
26,Raichu,Electric,,485,60,90,55,90,80,110,1,False
27,Sandshrew,Ground,,300,50,75,85,20,30,40,1,False
28,Sandslash,Ground,,450,75,100,110,45,55,65,1,False
29,Nidoran♀,Poison,,275,55,47,52,40,40,41,1,False
30,Nidorina,Poison,,365,70,62,67,55,55,56,1,False
31,Nidoqueen,Poison,Ground,505,90,92,87,75,85,76,1,False
32,Nidoran♂,Poison,,273,46,57,40,40,40,50,1,False
33,Nidorino,Poison,,365,61,72,57,55,55,65,1,False
34,Nidoking,Poison,Ground,505,81,102,77,85,75,85,1,False
35,Clefairy,Fairy,,323,70,45,48,60,65,35,1,False
36,Clefable,Fairy,,483,95,70,73,95,90,60,1,False
37,Vulpix,Fire,,299,38,41,40,50,65,65,1,False
38,Ninetales,Fire,,505,73,76,75,81,100,100,1,False
39,Jigglypuff,Normal,Fairy,270,115,45,20,45,25,20,1,False
40,Wigglytuff,Normal,Fairy,435,140,70,45,85,50,45,1,False
41,Zubat,Poison,Flying,245,40,45,35,30,40,55,1,False
42,Golbat,Poison,Flying,455,75,80,70,65,75,90,1,False
43,Oddish,Grass,Poison,320,45,50,55,75,65,30,1,False
44,Gloom,Grass,Poison,395,60,65,70,85,75,40,1,False
45,Vileplume,Grass,Poison,490,75,80,85,110,90,50,1,False
46,Paras,Bug,Grass,285,35,70,55,45,55,25,1,False
47,Parasect,Bug,Grass,405,60,95,80,60,80,30,1,False
48,Venonat,Bug,Poison,305,60,55,50,40,55,45,1,False
49,Venomoth,Bug,Poison,450,70,65,60,90,75,90,1,False
50,Diglett,Ground,,265,10,55,25,35,45,95,1,False
51,Dugtrio,Ground,,405,35,80,50,50,70,120,1,False
52,Meowth,Normal,,290,40,45,35,40,40,90,1,False
53,Persian,Normal,,440,65,70,60,65,65,115,1,False
54,Psyduck,Water,,320,50,52,48,65,50,55,1,False
55,Golduck,Water,,500,80,82,78,95,80,85,1,False
56,Mankey,Fighting,,305,40,80,35,35,45,70,1,False
57,Primeape,Fighting,,455,65,105,60,60,70,95,1,False
58,Growlithe,Fire,,350,55,70,45,70,50,60,1,False
59,Arcanine,Fire,,555,90,110,80,100,80,95,1,False
60,Poliwag,Water,,300,40,50,40,40,40,90,1,False
61,Poliwhirl,Water,,385,65,65,65,50,50,90,1,False
62,Poliwrath,Water,Fighting,510,90,95,95,70,90,70,1,False
63,Abra,Psychic,,310,25,20,15,105,55,90,1,False
64,Kadabra,Psychic,,400,40,35,30,120,70,105,1,False
65,Alakazam,Psychic,,500,55,50,45,135,95,120,1,False
65,AlakazamMega Alakazam,Psychic,,590,55,50,65,175,95,150,1,False
66,Machop,Fighting,,305,70,80,50,35,35,35,1,False
67,Machoke,Fighting,,405,80,100,70,50,60,45,1,False
68,Machamp,Fighting,,505,90,130,80,65,85,55,1,False
69,Bellsprout,Grass,Poison,300,50,75,35,70,30,40,1,False
70,Weepinbell,Grass,Poison,390,65,90,50,85,45,55,1,False
71,Victreebel,Grass,Poison,490,80,105,65,100,70,70,1,False
72,Tentacool,Water,Poison,335,40,40,35,50,100,70,1,False
73,Tentacruel,Water,Poison,515,80,70,65,80,120,100,1,False
74,Geodude,Rock,Ground,300,40,80,100,30,30,20,1,False
75,Graveler,Rock,Ground,390,55,95,115,45,45,35,1,False
76,Golem,Rock,Ground,495,80,120,130,55,65,45,1,False
77,Ponyta,Fire,,410,50,85,55,65,65,90,1,False
78,Rapidash,Fire,,500,65,100,70,80,80,105,1,False
79,Slowpoke,Water,Psychic,315,90,65,65,40,40,15,1,False
80,Slowbro,Water,Psychic,490,95,75,110,100,80,30,1,False
80,SlowbroMega Slowbro,Water,Psychic,590,95,75,180,130,80,30,1,False
81,Magnemite,Electric,Steel,325,25,35,70,95,55,45,1,False
82,Magneton,Electric,Steel,465,50,60,95,120,70,70,1,False
83,Farfetch'd,Normal,Flying,352,52,65,55,58,62,60,1,False
84,Doduo,Normal,Flying,310,35,85,45,35,35,75,1,False
85,Dodrio,Normal,Flying,460,60,110,70,60,60,100,1,False
86,Seel,Water,,325,65,45,55,45,70,45,1,False
87,Dewgong,Water,Ice,475,90,70,80,70,95,70,1,False
88,Grimer,Poison,,325,80,80,50,40,50,25,1,False
89,Muk,Poison,,500,105,105,75,65,100,50,1,False
90,Shellder,Water,,305,30,65,100,45,25,40,1,False
91,Cloyster,Water,Ice,525,50,95,180,85,45,70,1,False
92,Gastly,Ghost,Poison,310,30,35,30,100,35,80,1,False
93,Haunter,Ghost,Poison,405,45,50,45,115,55,95,1,False
94,Gengar,Ghost,Poison,500,60,65,60,130,75,110,1,False
94,GengarMega Gengar,Ghost,Poison,600,60,65,80,170,95,130,1,False
95,Onix,Rock,Ground,385,35,45,160,30,45,70,1,False
96,Drowzee,Psychic,,328,60,48,45,43,90,42,1,False
97,Hypno,Psychic,,483,85,73,70,73,115,67,1,False
98,Krabby,Water,,325,30,105,90,25,25,50,1,False
99,Kingler,Water,,475,55,130,115,50,50,75,1,False
100,Voltorb,Electric,,330,40,30,50,55,55,100,1,False
101,Electrode,Electric,,480,60,50,70,80,80,140,1,False
102,Exeggcute,Grass,Psychic,325,60,40,80,60,45,40,1,False
103,Exeggutor,Grass,Psychic,520,95,95,85,125,65,55,1,False
104,Cubone,Ground,,320,50,50,95,40,50,35,1,False
105,Marowak,Ground,,425,60,80,110,50,80,45,1,False
106,Hitmonlee,Fighting,,455,50,120,53,35,110,87,1,False
107,Hitmonchan,Fighting,,455,50,105,79,35,110,76,1,False
108,Lickitung,Normal,,385,90,55,75,60,75,30,1,False
109,Koffing,Poison,,340,40,65,95,60,45,35,1,False
110,Weezing,Poison,,490,65,90,120,85,70,60,1,False
111,Rhyhorn,Ground,Rock,345,80,85,95,30,30,25,1,False
112,Rhydon,Ground,Rock,485,105,130,120,45,45,40,1,False
113,Chansey,Normal,,450,250,5,5,35,105,50,1,False
114,Tangela,Grass,,435,65,55,115,100,40,60,1,False
115,Kangaskhan,Normal,,490,105,95,80,40,80,90,1,False
115,KangaskhanMega Kangaskhan,Normal,,590,105,125,100,60,100,100,1,False
116,Horsea,Water,,295,30,40,70,70,25,60,1,False
117,Seadra,Water,,440,55,65,95,95,45,85,1,False
118,Goldeen,Water,,320,45,67,60,35,50,63,1,False
119,Seaking,Water,,450,80,92,65,65,80,68,1,False
120,Staryu,Water,,340,30,45,55,70,55,85,1,False
121,Starmie,Water,Psychic,520,60,75,85,100,85,115,1,False
122,Mr. Mime,Psychic,Fairy,460,40,45,65,100,120,90,1,False
123,Scyther,Bug,Flying,500,70,110,80,55,80,105,1,False
124,Jynx,Ice,Psychic,455,65,50,35,115,95,95,1,False
125,Electabuzz,Electric,,490,65,83,57,95,85,105,1,False
126,Magmar,Fire,,495,65,95,57,100,85,93,1,False
127,Pinsir,Bug,,500,65,125,100,55,70,85,1,False
127,PinsirMega Pinsir,Bug,Flying,600,65,155,120,65,90,105,1,False
128,Tauros,Normal,,490,75,100,95,40,70,110,1,False
129,Magikarp,Water,,200,20,10,55,15,20,80,1,False
130,Gyarados,Water,Flying,540,95,125,79,60,100,81,1,False
130,GyaradosMega Gyarados,Water,Dark,640,95,155,109,70,130,81,1,False
131,Lapras,Water,Ice,535,130,85,80,85,95,60,1,False
132,Ditto,Normal,,288,48,48,48,48,48,48,1,False
133,Eevee,Normal,,325,55,55,50,45,65,55,1,False
134,Vaporeon,Water,,525,130,65,60,110,95,65,1,False
135,Jolteon,Electric,,525,65,65,60,110,95,130,1,False
136,Flareon,Fire,,525,65,130,60,95,110,65,1,False
137,Porygon,Normal,,395,65,60,70,85,75,40,1,False
138,Omanyte,Rock,Water,355,35,40,100,90,55,35,1,False
139,Omastar,Rock,Water,495,70,60,125,115,70,55,1,False
140,Kabuto,Rock,Water,355,30,80,90,55,45,55,1,False
141,Kabutops,Rock,Water,495,60,115,105,65,70,80,1,False
142,Aerodactyl,Rock,Flying,515,80,105,65,60,75,130,1,False
142,AerodactylMega Aerodactyl,Rock,Flying,615,80,135,85,70,95,150,1,False
143,Snorlax,Normal,,540,160,110,65,65,110,30,1,False
144,Articuno,Ice,Flying,580,90,85,100,95,125,85,1,True
145,Zapdos,Electric,Flying,580,90,90,85,125,90,100,1,True
146,Moltres,Fire,Flying,580,90,100,90,125,85,90,1,True
147,Dratini,Dragon,,300,41,64,45,50,50,50,1,False
148,Dragonair,Dragon,,420,61,84,65,70,70,70,1,False
149,Dragonite,Dragon,Flying,600,91,134,95,100,100,80,1,False
150,Mewtwo,Psychic,,680,106,110,90,154,90,130,1,True
150,MewtwoMega Mewtwo X,Psychic,Fighting,780,106,190,100,154,100,130,1,True
150,MewtwoMega Mewtwo Y,Psychic,,780,106,150,70,194,120,140,1,True
151,Mew,Psychic,,600,100,100,100,100,100,100,1,False
152,Chikorita,Grass,,318,45,49,65,49,65,45,2,False
153,Bayleef,Grass,,405,60,62,80,63,80,60,2,False
154,Meganium,Grass,,525,80,82,100,83,100,80,2,False
155,Cyndaquil,Fire,,309,39,52,43,60,50,65,2,False
156,Quilava,Fire,,405,58,64,58,80,65,80,2,False
157,Typhlosion,Fire,,534,78,84,78,109,85,100,2,False
158,Totodile,Water,,314,50,65,64,44,48,43,2,False
159,Croconaw,Water,,405,65,80,80,59,63,58,2,False
160,Feraligatr,Water,,530,85,105,100,79,83,78,2,False
161,Sentret,Normal,,215,35,46,34,35,45,20,2,False
162,Furret,Normal,,415,85,76,64,45,55,90,2,False
163,Hoothoot,Normal,Flying,262,60,30,30,36,56,50,2,False
164,Noctowl,Normal,Flying,442,100,50,50,76,96,70,2,False
165,Ledyba,Bug,Flying,265,40,20,30,40,80,55,2,False
166,Ledian,Bug,Flying,390,55,35,50,55,110,85,2,False
167,Spinarak,Bug,Poison,250,40,60,40,40,40,30,2,False
168,Ariados,Bug,Poison,390,70,90,70,60,60,40,2,False
169,Crobat,Poison,Flying,535,85,90,80,70,80,130,2,False
170,Chinchou,Water,Electric,330,75,38,38,56,56,67,2,False
171,Lanturn,Water,Electric,460,125,58,58,76,76,67,2,False
172,Pichu,Electric,,205,20,40,15,35,35,60,2,False
173,Cleffa,Fairy,,218,50,25,28,45,55,15,2,False
174,Igglybuff,Normal,Fairy,210,90,30,15,40,20,15,2,False
175,Togepi,Fairy,,245,35,20,65,40,65,20,2,False
176,Togetic,Fairy,Flying,405,55,40,85,80,105,40,2,False
177,Natu,Psychic,Flying,320,40,50,45,70,45,70,2,False
178,Xatu,Psychic,Flying,470,65,75,70,95,70,95,2,False
179,Mareep,Electric,,280,55,40,40,65,45,35,2,False
180,Flaaffy,Electric,,365,70,55,55,80,60,45,2,False
181,Ampharos,Electric,,510,90,75,85,115,90,55,2,False
181,AmpharosMega Ampharos,Electric,Dragon,610,90,95,105,165,110,45,2,False
182,Bellossom,Grass,,490,75,80,95,90,100,50,2,False
183,Marill,Water,Fairy,250,70,20,50,20,50,40,2,False
184,Azumarill,Water,Fairy,420,100,50,80,60,80,50,2,False
185,Sudowoodo,Rock,,410,70,100,115,30,65,30,2,False
186,Politoed,Water,,500,90,75,75,90,100,70,2,False
187,Hoppip,Grass,Flying,250,35,35,40,35,55,50,2,False
188,Skiploom,Grass,Flying,340,55,45,50,45,65,80,2,False
189,Jumpluff,Grass,Flying,460,75,55,70,55,95,110,2,False
190,Aipom,Normal,,360,55,70,55,40,55,85,2,False
191,Sunkern,Grass,,180,30,30,30,30,30,30,2,False
192,Sunflora,Grass,,425,75,75,55,105,85,30,2,False
193,Yanma,Bug,Flying,390,65,65,45,75,45,95,2,False
194,Wooper,Water,Ground,210,55,45,45,25,25,15,2,False
195,Quagsire,Water,Ground,430,95,85,85,65,65,35,2,False
196,Espeon,Psychic,,525,65,65,60,130,95,110,2,False
197,Umbreon,Dark,,525,95,65,110,60,130,65,2,False
198,Murkrow,Dark,Flying,405,60,85,42,85,42,91,2,False
199,Slowking,Water,Psychic,490,95,75,80,100,110,30,2,False
200,Misdreavus,Ghost,,435,60,60,60,85,85,85,2,False
201,Unown,Psychic,,336,48,72,48,72,48,48,2,False
202,Wobbuffet,Psychic,,405,190,33,58,33,58,33,2,False
203,Girafarig,Normal,Psychic,455,70,80,65,90,65,85,2,False
204,Pineco,Bug,,290,50,65,90,35,35,15,2,False
205,Forretress,Bug,Steel,465,75,90,140,60,60,40,2,False
206,Dunsparce,Normal,,415,100,70,70,65,65,45,2,False
207,Gligar,Ground,Flying,430,65,75,105,35,65,85,2,False
208,Steelix,Steel,Ground,510,75,85,200,55,65,30,2,False
208,SteelixMega Steelix,Steel,Ground,610,75,125,230,55,95,30,2,False
209,Snubbull,Fairy,,300,60,80,50,40,40,30,2,False
210,Granbull,Fairy,,450,90,120,75,60,60,45,2,False
211,Qwilfish,Water,Poison,430,65,95,75,55,55,85,2,False
212,Scizor,Bug,Steel,500,70,130,100,55,80,65,2,False
212,ScizorMega Scizor,Bug,Steel,600,70,150,140,65,100,75,2,False
213,Shuckle,Bug,Rock,505,20,10,230,10,230,5,2,False
214,Heracross,Bug,Fighting,500,80,125,75,40,95,85,2,False
214,HeracrossMega Heracross,Bug,Fighting,600,80,185,115,40,105,75,2,False
215,Sneasel,Dark,Ice,430,55,95,55,35,75,115,2,False
216,Teddiursa,Normal,,330,60,80,50,50,50,40,2,False
217,Ursaring,Normal,,500,90,130,75,75,75,55,2,False
218,Slugma,Fire,,250,40,40,40,70,40,20,2,False
219,Magcargo,Fire,Rock,410,50,50,120,80,80,30,2,False
220,Swinub,Ice,Ground,250,50,50,40,30,30,50,2,False
221,Piloswine,Ice,Ground,450,100,100,80,60,60,50,2,False
222,Corsola,Water,Rock,380,55,55,85,65,85,35,2,False
223,Remoraid,Water,,300,35,65,35,65,35,65,2,False
224,Octillery,Water,,480,75,105,75,105,75,45,2,False
225,Delibird,Ice,Flying,330,45,55,45,65,45,75,2,False
226,Mantine,Water,Flying,465,65,40,70,80,140,70,2,False
227,Skarmory,Steel,Flying,465,65,80,140,40,70,70,2,False
228,Houndour,Dark,Fire,330,45,60,30,80,50,65,2,False
229,Houndoom,Dark,Fire,500,75,90,50,110,80,95,2,False
229,HoundoomMega Houndoom,Dark,Fire,600,75,90,90,140,90,115,2,False
230,Kingdra,Water,Dragon,540,75,95,95,95,95,85,2,False
231,Phanpy,Ground,,330,90,60,60,40,40,40,2,False
232,Donphan,Ground,,500,90,120,120,60,60,50,2,False
233,Porygon2,Normal,,515,85,80,90,105,95,60,2,False
234,Stantler,Normal,,465,73,95,62,85,65,85,2,False
235,Smeargle,Normal,,250,55,20,35,20,45,75,2,False
236,Tyrogue,Fighting,,210,35,35,35,35,35,35,2,False
237,Hitmontop,Fighting,,455,50,95,95,35,110,70,2,False
238,Smoochum,Ice,Psychic,305,45,30,15,85,65,65,2,False
239,Elekid,Electric,,360,45,63,37,65,55,95,2,False
240,Magby,Fire,,365,45,75,37,70,55,83,2,False
241,Miltank,Normal,,490,95,80,105,40,70,100,2,False
242,Blissey,Normal,,540,255,10,10,75,135,55,2,False
243,Raikou,Electric,,580,90,85,75,115,100,115,2,True
244,Entei,Fire,,580,115,115,85,90,75,100,2,True
245,Suicune,Water,,580,100,75,115,90,115,85,2,True
246,Larvitar,Rock,Ground,300,50,64,50,45,50,41,2,False
247,Pupitar,Rock,Ground,410,70,84,70,65,70,51,2,False
248,Tyranitar,Rock,Dark,600,100,134,110,95,100,61,2,False
248,TyranitarMega Tyranitar,Rock,Dark,700,100,164,150,95,120,71,2,False
249,Lugia,Psychic,Flying,680,106,90,130,90,154,110,2,True
250,Ho-oh,Fire,Flying,680,106,130,90,110,154,90,2,True
251,Celebi,Psychic,Grass,600,100,100,100,100,100,100,2,False
252,Treecko,Grass,,310,40,45,35,65,55,70,3,False
253,Grovyle,Grass,,405,50,65,45,85,65,95,3,False
254,Sceptile,Grass,,530,70,85,65,105,85,120,3,False
254,SceptileMega Sceptile,Grass,Dragon,630,70,110,75,145,85,145,3,False
255,Torchic,Fire,,310,45,60,40,70,50,45,3,False
256,Combusken,Fire,Fighting,405,60,85,60,85,60,55,3,False
257,Blaziken,Fire,Fighting,530,80,120,70,110,70,80,3,False
257,BlazikenMega Blaziken,Fire,Fighting,630,80,160,80,130,80,100,3,False
258,Mudkip,Water,,310,50,70,50,50,50,40,3,False
259,Marshtomp,Water,Ground,405,70,85,70,60,70,50,3,False
260,Swampert,Water,Ground,535,100,110,90,85,90,60,3,False
260,SwampertMega Swampert,Water,Ground,635,100,150,110,95,110,70,3,False
261,Poochyena,Dark,,220,35,55,35,30,30,35,3,False
262,Mightyena,Dark,,420,70,90,70,60,60,70,3,False
263,Zigzagoon,Normal,,240,38,30,41,30,41,60,3,False
264,Linoone,Normal,,420,78,70,61,50,61,100,3,False
265,Wurmple,Bug,,195,45,45,35,20,30,20,3,False
266,Silcoon,Bug,,205,50,35,55,25,25,15,3,False
267,Beautifly,Bug,Flying,395,60,70,50,100,50,65,3,False
268,Cascoon,Bug,,205,50,35,55,25,25,15,3,False
269,Dustox,Bug,Poison,385,60,50,70,50,90,65,3,False
270,Lotad,Water,Grass,220,40,30,30,40,50,30,3,False
271,Lombre,Water,Grass,340,60,50,50,60,70,50,3,False
272,Ludicolo,Water,Grass,480,80,70,70,90,100,70,3,False
273,Seedot,Grass,,220,40,40,50,30,30,30,3,False
274,Nuzleaf,Grass,Dark,340,70,70,40,60,40,60,3,False
275,Shiftry,Grass,Dark,480,90,100,60,90,60,80,3,False
276,Taillow,Normal,Flying,270,40,55,30,30,30,85,3,False
277,Swellow,Normal,Flying,430,60,85,60,50,50,125,3,False
278,Wingull,Water,Flying,270,40,30,30,55,30,85,3,False
279,Pelipper,Water,Flying,430,60,50,100,85,70,65,3,False
280,Ralts,Psychic,Fairy,198,28,25,25,45,35,40,3,False
281,Kirlia,Psychic,Fairy,278,38,35,35,65,55,50,3,False
282,Gardevoir,Psychic,Fairy,518,68,65,65,125,115,80,3,False
282,GardevoirMega Gardevoir,Psychic,Fairy,618,68,85,65,165,135,100,3,False
283,Surskit,Bug,Water,269,40,30,32,50,52,65,3,False
284,Masquerain,Bug,Flying,414,70,60,62,80,82,60,3,False
285,Shroomish,Grass,,295,60,40,60,40,60,35,3,False
286,Breloom,Grass,Fighting,460,60,130,80,60,60,70,3,False
287,Slakoth,Normal,,280,60,60,60,35,35,30,3,False
288,Vigoroth,Normal,,440,80,80,80,55,55,90,3,False
289,Slaking,Normal,,670,150,160,100,95,65,100,3,False
290,Nincada,Bug,Ground,266,31,45,90,30,30,40,3,False
291,Ninjask,Bug,Flying,456,61,90,45,50,50,160,3,False
292,Shedinja,Bug,Ghost,236,1,90,45,30,30,40,3,False
293,Whismur,Normal,,240,64,51,23,51,23,28,3,False
294,Loudred,Normal,,360,84,71,43,71,43,48,3,False
295,Exploud,Normal,,490,104,91,63,91,73,68,3,False
296,Makuhita,Fighting,,237,72,60,30,20,30,25,3,False
297,Hariyama,Fighting,,474,144,120,60,40,60,50,3,False
298,Azurill,Normal,Fairy,190,50,20,40,20,40,20,3,False
299,Nosepass,Rock,,375,30,45,135,45,90,30,3,False
300,Skitty,Normal,,260,50,45,45,35,35,50,3,False
301,Delcatty,Normal,,380,70,65,65,55,55,70,3,False
302,Sableye,Dark,Ghost,380,50,75,75,65,65,50,3,False
302,SableyeMega Sableye,Dark,Ghost,480,50,85,125,85,115,20,3,False
303,Mawile,Steel,Fairy,380,50,85,85,55,55,50,3,False
303,MawileMega Mawile,Steel,Fairy,480,50,105,125,55,95,50,3,False
304,Aron,Steel,Rock,330,50,70,100,40,40,30,3,False
305,Lairon,Steel,Rock,430,60,90,140,50,50,40,3,False
306,Aggron,Steel,Rock,530,70,110,180,60,60,50,3,False
306,AggronMega Aggron,Steel,,630,70,140,230,60,80,50,3,False
307,Meditite,Fighting,Psychic,280,30,40,55,40,55,60,3,False
308,Medicham,Fighting,Psychic,410,60,60,75,60,75,80,3,False
308,MedichamMega Medicham,Fighting,Psychic,510,60,100,85,80,85,100,3,False
309,Electrike,Electric,,295,40,45,40,65,40,65,3,False
310,Manectric,Electric,,475,70,75,60,105,60,105,3,False
310,ManectricMega Manectric,Electric,,575,70,75,80,135,80,135,3,False
311,Plusle,Electric,,405,60,50,40,85,75,95,3,False
312,Minun,Electric,,405,60,40,50,75,85,95,3,False
313,Volbeat,Bug,,400,65,73,55,47,75,85,3,False
314,Illumise,Bug,,400,65,47,55,73,75,85,3,False
315,Roselia,Grass,Poison,400,50,60,45,100,80,65,3,False
316,Gulpin,Poison,,302,70,43,53,43,53,40,3,False
317,Swalot,Poison,,467,100,73,83,73,83,55,3,False
318,Carvanha,Water,Dark,305,45,90,20,65,20,65,3,False
319,Sharpedo,Water,Dark,460,70,120,40,95,40,95,3,False
319,SharpedoMega Sharpedo,Water,Dark,560,70,140,70,110,65,105,3,False
320,Wailmer,Water,,400,130,70,35,70,35,60,3,False
321,Wailord,Water,,500,170,90,45,90,45,60,3,False
322,Numel,Fire,Ground,305,60,60,40,65,45,35,3,False
323,Camerupt,Fire,Ground,460,70,100,70,105,75,40,3,False
323,CameruptMega Camerupt,Fire,Ground,560,70,120,100,145,105,20,3,False
324,Torkoal,Fire,,470,70,85,140,85,70,20,3,False
325,Spoink,Psychic,,330,60,25,35,70,80,60,3,False
326,Grumpig,Psychic,,470,80,45,65,90,110,80,3,False
327,Spinda,Normal,,360,60,60,60,60,60,60,3,False
328,Trapinch,Ground,,290,45,100,45,45,45,10,3,False
329,Vibrava,Ground,Dragon,340,50,70,50,50,50,70,3,False
330,Flygon,Ground,Dragon,520,80,100,80,80,80,100,3,False
331,Cacnea,Grass,,335,50,85,40,85,40,35,3,False
332,Cacturne,Grass,Dark,475,70,115,60,115,60,55,3,False
333,Swablu,Normal,Flying,310,45,40,60,40,75,50,3,False
334,Altaria,Dragon,Flying,490,75,70,90,70,105,80,3,False
334,AltariaMega Altaria,Dragon,Fairy,590,75,110,110,110,105,80,3,False
335,Zangoose,Normal,,458,73,115,60,60,60,90,3,False
336,Seviper,Poison,,458,73,100,60,100,60,65,3,False
337,Lunatone,Rock,Psychic,440,70,55,65,95,85,70,3,False
338,Solrock,Rock,Psychic,440,70,95,85,55,65,70,3,False
339,Barboach,Water,Ground,288,50,48,43,46,41,60,3,False
340,Whiscash,Water,Ground,468,110,78,73,76,71,60,3,False
341,Corphish,Water,,308,43,80,65,50,35,35,3,False
342,Crawdaunt,Water,Dark,468,63,120,85,90,55,55,3,False
343,Baltoy,Ground,Psychic,300,40,40,55,40,70,55,3,False
344,Claydol,Ground,Psychic,500,60,70,105,70,120,75,3,False
345,Lileep,Rock,Grass,355,66,41,77,61,87,23,3,False
346,Cradily,Rock,Grass,495,86,81,97,81,107,43,3,False
347,Anorith,Rock,Bug,355,45,95,50,40,50,75,3,False
348,Armaldo,Rock,Bug,495,75,125,100,70,80,45,3,False
349,Feebas,Water,,200,20,15,20,10,55,80,3,False
350,Milotic,Water,,540,95,60,79,100,125,81,3,False
351,Castform,Normal,,420,70,70,70,70,70,70,3,False
352,Kecleon,Normal,,440,60,90,70,60,120,40,3,False
353,Shuppet,Ghost,,295,44,75,35,63,33,45,3,False
354,Banette,Ghost,,455,64,115,65,83,63,65,3,False
354,BanetteMega Banette,Ghost,,555,64,165,75,93,83,75,3,False
355,Duskull,Ghost,,295,20,40,90,30,90,25,3,False
356,Dusclops,Ghost,,455,40,70,130,60,130,25,3,False
357,Tropius,Grass,Flying,460,99,68,83,72,87,51,3,False
358,Chimecho,Psychic,,425,65,50,70,95,80,65,3,False
359,Absol,Dark,,465,65,130,60,75,60,75,3,False
359,AbsolMega Absol,Dark,,565,65,150,60,115,60,115,3,False
360,Wynaut,Psychic,,260,95,23,48,23,48,23,3,False
361,Snorunt,Ice,,300,50,50,50,50,50,50,3,False
362,Glalie,Ice,,480,80,80,80,80,80,80,3,False
362,GlalieMega Glalie,Ice,,580,80,120,80,120,80,100,3,False
363,Spheal,Ice,Water,290,70,40,50,55,50,25,3,False
364,Sealeo,Ice,Water,410,90,60,70,75,70,45,3,False
365,Walrein,Ice,Water,530,110,80,90,95,90,65,3,False
366,Clamperl,Water,,345,35,64,85,74,55,32,3,False
367,Huntail,Water,,485,55,104,105,94,75,52,3,False
368,Gorebyss,Water,,485,55,84,105,114,75,52,3,False
369,Relicanth,Water,Rock,485,100,90,130,45,65,55,3,False
370,Luvdisc,Water,,330,43,30,55,40,65,97,3,False
371,Bagon,Dragon,,300,45,75,60,40,30,50,3,False
372,Shelgon,Dragon,,420,65,95,100,60,50,50,3,False
373,Salamence,Dragon,Flying,600,95,135,80,110,80,100,3,False
373,SalamenceMega Salamence,Dragon,Flying,700,95,145,130,120,90,120,3,False
374,Beldum,Steel,Psychic,300,40,55,80,35,60,30,3,False
375,Metang,Steel,Psychic,420,60,75,100,55,80,50,3,False
376,Metagross,Steel,Psychic,600,80,135,130,95,90,70,3,False
376,MetagrossMega Metagross,Steel,Psychic,700,80,145,150,105,110,110,3,False
377,Regirock,Rock,,580,80,100,200,50,100,50,3,True
378,Regice,Ice,,580,80,50,100,100,200,50,3,True
379,Registeel,Steel,,580,80,75,150,75,150,50,3,True
380,Latias,Dragon,Psychic,600,80,80,90,110,130,110,3,True
380,LatiasMega Latias,Dragon,Psychic,700,80,100,120,140,150,110,3,True
381,Latios,Dragon,Psychic,600,80,90,80,130,110,110,3,True
381,LatiosMega Latios,Dragon,Psychic,700,80,130,100,160,120,110,3,True
382,Kyogre,Water,,670,100,100,90,150,140,90,3,True
382,KyogrePrimal Kyogre,Water,,770,100,150,90,180,160,90,3,True
383,Groudon,Ground,,670,100,150,140,100,90,90,3,True
383,GroudonPrimal Groudon,Ground,Fire,770,100,180,160,150,90,90,3,True
384,Rayquaza,Dragon,Flying,680,105,150,90,150,90,95,3,True
384,RayquazaMega Rayquaza,Dragon,Flying,780,105,180,100,180,100,115,3,True
385,Jirachi,Steel,Psychic,600,100,100,100,100,100,100,3,True
386,DeoxysNormal Forme,Psychic,,600,50,150,50,150,50,150,3,True
386,DeoxysAttack Forme,Psychic,,600,50,180,20,180,20,150,3,True
386,DeoxysDefense Forme,Psychic,,600,50,70,160,70,160,90,3,True
386,DeoxysSpeed Forme,Psychic,,600,50,95,90,95,90,180,3,True
387,Turtwig,Grass,,318,55,68,64,45,55,31,4,False
388,Grotle,Grass,,405,75,89,85,55,65,36,4,False
389,Torterra,Grass,Ground,525,95,109,105,75,85,56,4,False
390,Chimchar,Fire,,309,44,58,44,58,44,61,4,False
391,Monferno,Fire,Fighting,405,64,78,52,78,52,81,4,False
392,Infernape,Fire,Fighting,534,76,104,71,104,71,108,4,False
393,Piplup,Water,,314,53,51,53,61,56,40,4,False
394,Prinplup,Water,,405,64,66,68,81,76,50,4,False
395,Empoleon,Water,Steel,530,84,86,88,111,101,60,4,False
396,Starly,Normal,Flying,245,40,55,30,30,30,60,4,False
397,Staravia,Normal,Flying,340,55,75,50,40,40,80,4,False
398,Staraptor,Normal,Flying,485,85,120,70,50,60,100,4,False
399,Bidoof,Normal,,250,59,45,40,35,40,31,4,False
400,Bibarel,Normal,Water,410,79,85,60,55,60,71,4,False
401,Kricketot,Bug,,194,37,25,41,25,41,25,4,False
402,Kricketune,Bug,,384,77,85,51,55,51,65,4,False
403,Shinx,Electric,,263,45,65,34,40,34,45,4,False
404,Luxio,Electric,,363,60,85,49,60,49,60,4,False
405,Luxray,Electric,,523,80,120,79,95,79,70,4,False
406,Budew,Grass,Poison,280,40,30,35,50,70,55,4,False
407,Roserade,Grass,Poison,515,60,70,65,125,105,90,4,False
408,Cranidos,Rock,,350,67,125,40,30,30,58,4,False
409,Rampardos,Rock,,495,97,165,60,65,50,58,4,False
410,Shieldon,Rock,Steel,350,30,42,118,42,88,30,4,False
411,Bastiodon,Rock,Steel,495,60,52,168,47,138,30,4,False
412,Burmy,Bug,,224,40,29,45,29,45,36,4,False
413,WormadamPlant Cloak,Bug,Grass,424,60,59,85,79,105,36,4,False
413,WormadamSandy Cloak,Bug,Ground,424,60,79,105,59,85,36,4,False
413,WormadamTrash Cloak,Bug,Steel,424,60,69,95,69,95,36,4,False
414,Mothim,Bug,Flying,424,70,94,50,94,50,66,4,False
415,Combee,Bug,Flying,244,30,30,42,30,42,70,4,False
416,Vespiquen,Bug,Flying,474,70,80,102,80,102,40,4,False
417,Pachirisu,Electric,,405,60,45,70,45,90,95,4,False
418,Buizel,Water,,330,55,65,35,60,30,85,4,False
419,Floatzel,Water,,495,85,105,55,85,50,115,4,False
420,Cherubi,Grass,,275,45,35,45,62,53,35,4,False
421,Cherrim,Grass,,450,70,60,70,87,78,85,4,False
422,Shellos,Water,,325,76,48,48,57,62,34,4,False
423,Gastrodon,Water,Ground,475,111,83,68,92,82,39,4,False
424,Ambipom,Normal,,482,75,100,66,60,66,115,4,False
425,Drifloon,Ghost,Flying,348,90,50,34,60,44,70,4,False
426,Drifblim,Ghost,Flying,498,150,80,44,90,54,80,4,False
427,Buneary,Normal,,350,55,66,44,44,56,85,4,False
428,Lopunny,Normal,,480,65,76,84,54,96,105,4,False
428,LopunnyMega Lopunny,Normal,Fighting,580,65,136,94,54,96,135,4,False
429,Mismagius,Ghost,,495,60,60,60,105,105,105,4,False
430,Honchkrow,Dark,Flying,505,100,125,52,105,52,71,4,False
431,Glameow,Normal,,310,49,55,42,42,37,85,4,False
432,Purugly,Normal,,452,71,82,64,64,59,112,4,False
433,Chingling,Psychic,,285,45,30,50,65,50,45,4,False
434,Stunky,Poison,Dark,329,63,63,47,41,41,74,4,False
435,Skuntank,Poison,Dark,479,103,93,67,71,61,84,4,False
436,Bronzor,Steel,Psychic,300,57,24,86,24,86,23,4,False
437,Bronzong,Steel,Psychic,500,67,89,116,79,116,33,4,False
438,Bonsly,Rock,,290,50,80,95,10,45,10,4,False
439,Mime Jr.,Psychic,Fairy,310,20,25,45,70,90,60,4,False
440,Happiny,Normal,,220,100,5,5,15,65,30,4,False
441,Chatot,Normal,Flying,411,76,65,45,92,42,91,4,False
442,Spiritomb,Ghost,Dark,485,50,92,108,92,108,35,4,False
443,Gible,Dragon,Ground,300,58,70,45,40,45,42,4,False
444,Gabite,Dragon,Ground,410,68,90,65,50,55,82,4,False
445,Garchomp,Dragon,Ground,600,108,130,95,80,85,102,4,False
445,GarchompMega Garchomp,Dragon,Ground,700,108,170,115,120,95,92,4,False
446,Munchlax,Normal,,390,135,85,40,40,85,5,4,False
447,Riolu,Fighting,,285,40,70,40,35,40,60,4,False
448,Lucario,Fighting,Steel,525,70,110,70,115,70,90,4,False
448,LucarioMega Lucario,Fighting,Steel,625,70,145,88,140,70,112,4,False
449,Hippopotas,Ground,,330,68,72,78,38,42,32,4,False
450,Hippowdon,Ground,,525,108,112,118,68,72,47,4,False
451,Skorupi,Poison,Bug,330,40,50,90,30,55,65,4,False
452,Drapion,Poison,Dark,500,70,90,110,60,75,95,4,False
453,Croagunk,Poison,Fighting,300,48,61,40,61,40,50,4,False
454,Toxicroak,Poison,Fighting,490,83,106,65,86,65,85,4,False
455,Carnivine,Grass,,454,74,100,72,90,72,46,4,False
456,Finneon,Water,,330,49,49,56,49,61,66,4,False
457,Lumineon,Water,,460,69,69,76,69,86,91,4,False
458,Mantyke,Water,Flying,345,45,20,50,60,120,50,4,False
459,Snover,Grass,Ice,334,60,62,50,62,60,40,4,False
460,Abomasnow,Grass,Ice,494,90,92,75,92,85,60,4,False
460,AbomasnowMega Abomasnow,Grass,Ice,594,90,132,105,132,105,30,4,False
461,Weavile,Dark,Ice,510,70,120,65,45,85,125,4,False
462,Magnezone,Electric,Steel,535,70,70,115,130,90,60,4,False
463,Lickilicky,Normal,,515,110,85,95,80,95,50,4,False
464,Rhyperior,Ground,Rock,535,115,140,130,55,55,40,4,False
465,Tangrowth,Grass,,535,100,100,125,110,50,50,4,False
466,Electivire,Electric,,540,75,123,67,95,85,95,4,False
467,Magmortar,Fire,,540,75,95,67,125,95,83,4,False
468,Togekiss,Fairy,Flying,545,85,50,95,120,115,80,4,False
469,Yanmega,Bug,Flying,515,86,76,86,116,56,95,4,False
470,Leafeon,Grass,,525,65,110,130,60,65,95,4,False
471,Glaceon,Ice,,525,65,60,110,130,95,65,4,False
472,Gliscor,Ground,Flying,510,75,95,125,45,75,95,4,False
473,Mamoswine,Ice,Ground,530,110,130,80,70,60,80,4,False
474,Porygon-Z,Normal,,535,85,80,70,135,75,90,4,False
475,Gallade,Psychic,Fighting,518,68,125,65,65,115,80,4,False
475,GalladeMega Gallade,Psychic,Fighting,618,68,165,95,65,115,110,4,False
476,Probopass,Rock,Steel,525,60,55,145,75,150,40,4,False
477,Dusknoir,Ghost,,525,45,100,135,65,135,45,4,False
478,Froslass,Ice,Ghost,480,70,80,70,80,70,110,4,False
479,Rotom,Electric,Ghost,440,50,50,77,95,77,91,4,False
479,RotomHeat Rotom,Electric,Fire,520,50,65,107,105,107,86,4,False
479,RotomWash Rotom,Electric,Water,520,50,65,107,105,107,86,4,False
479,RotomFrost Rotom,Electric,Ice,520,50,65,107,105,107,86,4,False
479,RotomFan Rotom,Electric,Flying,520,50,65,107,105,107,86,4,False
479,RotomMow Rotom,Electric,Grass,520,50,65,107,105,107,86,4,False
480,Uxie,Psychic,,580,75,75,130,75,130,95,4,True
481,Mesprit,Psychic,,580,80,105,105,105,105,80,4,True
482,Azelf,Psychic,,580,75,125,70,125,70,115,4,True
483,Dialga,Steel,Dragon,680,100,120,120,150,100,90,4,True
484,Palkia,Water,Dragon,680,90,120,100,150,120,100,4,True
485,Heatran,Fire,Steel,600,91,90,106,130,106,77,4,True
486,Regigigas,Normal,,670,110,160,110,80,110,100,4,True
487,GiratinaAltered Forme,Ghost,Dragon,680,150,100,120,100,120,90,4,True
487,GiratinaOrigin Forme,Ghost,Dragon,680,150,120,100,120,100,90,4,True
488,Cresselia,Psychic,,600,120,70,120,75,130,85,4,False
489,Phione,Water,,480,80,80,80,80,80,80,4,False
490,Manaphy,Water,,600,100,100,100,100,100,100,4,False
491,Darkrai,Dark,,600,70,90,90,135,90,125,4,True
492,ShayminLand Forme,Grass,,600,100,100,100,100,100,100,4,True
492,ShayminSky Forme,Grass,Flying,600,100,103,75,120,75,127,4,True
493,Arceus,Normal,,720,120,120,120,120,120,120,4,True
494,Victini,Psychic,Fire,600,100,100,100,100,100,100,5,True
495,Snivy,Grass,,308,45,45,55,45,55,63,5,False
496,Servine,Grass,,413,60,60,75,60,75,83,5,False
497,Serperior,Grass,,528,75,75,95,75,95,113,5,False
498,Tepig,Fire,,308,65,63,45,45,45,45,5,False
499,Pignite,Fire,Fighting,418,90,93,55,70,55,55,5,False
500,Emboar,Fire,Fighting,528,110,123,65,100,65,65,5,False
501,Oshawott,Water,,308,55,55,45,63,45,45,5,False
502,Dewott,Water,,413,75,75,60,83,60,60,5,False
503,Samurott,Water,,528,95,100,85,108,70,70,5,False
504,Patrat,Normal,,255,45,55,39,35,39,42,5,False
505,Watchog,Normal,,420,60,85,69,60,69,77,5,False
506,Lillipup,Normal,,275,45,60,45,25,45,55,5,False
507,Herdier,Normal,,370,65,80,65,35,65,60,5,False
508,Stoutland,Normal,,500,85,110,90,45,90,80,5,False
509,Purrloin,Dark,,281,41,50,37,50,37,66,5,False
510,Liepard,Dark,,446,64,88,50,88,50,106,5,False
511,Pansage,Grass,,316,50,53,48,53,48,64,5,False
512,Simisage,Grass,,498,75,98,63,98,63,101,5,False
513,Pansear,Fire,,316,50,53,48,53,48,64,5,False
514,Simisear,Fire,,498,75,98,63,98,63,101,5,False
515,Panpour,Water,,316,50,53,48,53,48,64,5,False
516,Simipour,Water,,498,75,98,63,98,63,101,5,False
517,Munna,Psychic,,292,76,25,45,67,55,24,5,False
518,Musharna,Psychic,,487,116,55,85,107,95,29,5,False
519,Pidove,Normal,Flying,264,50,55,50,36,30,43,5,False
520,Tranquill,Normal,Flying,358,62,77,62,50,42,65,5,False
521,Unfezant,Normal,Flying,488,80,115,80,65,55,93,5,False
522,Blitzle,Electric,,295,45,60,32,50,32,76,5,False
523,Zebstrika,Electric,,497,75,100,63,80,63,116,5,False
524,Roggenrola,Rock,,280,55,75,85,25,25,15,5,False
525,Boldore,Rock,,390,70,105,105,50,40,20,5,False
526,Gigalith,Rock,,515,85,135,130,60,80,25,5,False
527,Woobat,Psychic,Flying,313,55,45,43,55,43,72,5,False
528,Swoobat,Psychic,Flying,425,67,57,55,77,55,114,5,False
529,Drilbur,Ground,,328,60,85,40,30,45,68,5,False
530,Excadrill,Ground,Steel,508,110,135,60,50,65,88,5,False
531,Audino,Normal,,445,103,60,86,60,86,50,5,False
531,AudinoMega Audino,Normal,Fairy,545,103,60,126,80,126,50,5,False
532,Timburr,Fighting,,305,75,80,55,25,35,35,5,False
533,Gurdurr,Fighting,,405,85,105,85,40,50,40,5,False
534,Conkeldurr,Fighting,,505,105,140,95,55,65,45,5,False
535,Tympole,Water,,294,50,50,40,50,40,64,5,False
536,Palpitoad,Water,Ground,384,75,65,55,65,55,69,5,False
537,Seismitoad,Water,Ground,509,105,95,75,85,75,74,5,False
538,Throh,Fighting,,465,120,100,85,30,85,45,5,False
539,Sawk,Fighting,,465,75,125,75,30,75,85,5,False
540,Sewaddle,Bug,Grass,310,45,53,70,40,60,42,5,False
541,Swadloon,Bug,Grass,380,55,63,90,50,80,42,5,False
542,Leavanny,Bug,Grass,500,75,103,80,70,80,92,5,False
543,Venipede,Bug,Poison,260,30,45,59,30,39,57,5,False
544,Whirlipede,Bug,Poison,360,40,55,99,40,79,47,5,False
545,Scolipede,Bug,Poison,485,60,100,89,55,69,112,5,False
546,Cottonee,Grass,Fairy,280,40,27,60,37,50,66,5,False
547,Whimsicott,Grass,Fairy,480,60,67,85,77,75,116,5,False
548,Petilil,Grass,,280,45,35,50,70,50,30,5,False
549,Lilligant,Grass,,480,70,60,75,110,75,90,5,False
550,Basculin,Water,,460,70,92,65,80,55,98,5,False
551,Sandile,Ground,Dark,292,50,72,35,35,35,65,5,False
552,Krokorok,Ground,Dark,351,60,82,45,45,45,74,5,False
553,Krookodile,Ground,Dark,519,95,117,80,65,70,92,5,False
554,Darumaka,Fire,,315,70,90,45,15,45,50,5,False
555,DarmanitanStandard Mode,Fire,,480,105,140,55,30,55,95,5,False
555,DarmanitanZen Mode,Fire,Psychic,540,105,30,105,140,105,55,5,False
556,Maractus,Grass,,461,75,86,67,106,67,60,5,False
557,Dwebble,Bug,Rock,325,50,65,85,35,35,55,5,False
558,Crustle,Bug,Rock,475,70,95,125,65,75,45,5,False
559,Scraggy,Dark,Fighting,348,50,75,70,35,70,48,5,False
560,Scrafty,Dark,Fighting,488,65,90,115,45,115,58,5,False
561,Sigilyph,Psychic,Flying,490,72,58,80,103,80,97,5,False
562,Yamask,Ghost,,303,38,30,85,55,65,30,5,False
563,Cofagrigus,Ghost,,483,58,50,145,95,105,30,5,False
564,Tirtouga,Water,Rock,355,54,78,103,53,45,22,5,False
565,Carracosta,Water,Rock,495,74,108,133,83,65,32,5,False
566,Archen,Rock,Flying,401,55,112,45,74,45,70,5,False
567,Archeops,Rock,Flying,567,75,140,65,112,65,110,5,False
568,Trubbish,Poison,,329,50,50,62,40,62,65,5,False
569,Garbodor,Poison,,474,80,95,82,60,82,75,5,False
570,Zorua,Dark,,330,40,65,40,80,40,65,5,False
571,Zoroark,Dark,,510,60,105,60,120,60,105,5,False
572,Minccino,Normal,,300,55,50,40,40,40,75,5,False
573,Cinccino,Normal,,470,75,95,60,65,60,115,5,False
574,Gothita,Psychic,,290,45,30,50,55,65,45,5,False
575,Gothorita,Psychic,,390,60,45,70,75,85,55,5,False
576,Gothitelle,Psychic,,490,70,55,95,95,110,65,5,False
577,Solosis,Psychic,,290,45,30,40,105,50,20,5,False
578,Duosion,Psychic,,370,65,40,50,125,60,30,5,False
579,Reuniclus,Psychic,,490,110,65,75,125,85,30,5,False
580,Ducklett,Water,Flying,305,62,44,50,44,50,55,5,False
581,Swanna,Water,Flying,473,75,87,63,87,63,98,5,False
582,Vanillite,Ice,,305,36,50,50,65,60,44,5,False
583,Vanillish,Ice,,395,51,65,65,80,75,59,5,False
584,Vanilluxe,Ice,,535,71,95,85,110,95,79,5,False
585,Deerling,Normal,Grass,335,60,60,50,40,50,75,5,False
586,Sawsbuck,Normal,Grass,475,80,100,70,60,70,95,5,False
587,Emolga,Electric,Flying,428,55,75,60,75,60,103,5,False
588,Karrablast,Bug,,315,50,75,45,40,45,60,5,False
589,Escavalier,Bug,Steel,495,70,135,105,60,105,20,5,False
590,Foongus,Grass,Poison,294,69,55,45,55,55,15,5,False
591,Amoonguss,Grass,Poison,464,114,85,70,85,80,30,5,False
592,Frillish,Water,Ghost,335,55,40,50,65,85,40,5,False
593,Jellicent,Water,Ghost,480,100,60,70,85,105,60,5,False
594,Alomomola,Water,,470,165,75,80,40,45,65,5,False
595,Joltik,Bug,Electric,319,50,47,50,57,50,65,5,False
596,Galvantula,Bug,Electric,472,70,77,60,97,60,108,5,False
597,Ferroseed,Grass,Steel,305,44,50,91,24,86,10,5,False
598,Ferrothorn,Grass,Steel,489,74,94,131,54,116,20,5,False
599,Klink,Steel,,300,40,55,70,45,60,30,5,False
600,Klang,Steel,,440,60,80,95,70,85,50,5,False
601,Klinklang,Steel,,520,60,100,115,70,85,90,5,False
602,Tynamo,Electric,,275,35,55,40,45,40,60,5,False
603,Eelektrik,Electric,,405,65,85,70,75,70,40,5,False
604,Eelektross,Electric,,515,85,115,80,105,80,50,5,False
605,Elgyem,Psychic,,335,55,55,55,85,55,30,5,False
606,Beheeyem,Psychic,,485,75,75,75,125,95,40,5,False
607,Litwick,Ghost,Fire,275,50,30,55,65,55,20,5,False
608,Lampent,Ghost,Fire,370,60,40,60,95,60,55,5,False
609,Chandelure,Ghost,Fire,520,60,55,90,145,90,80,5,False
610,Axew,Dragon,,320,46,87,60,30,40,57,5,False
611,Fraxure,Dragon,,410,66,117,70,40,50,67,5,False
612,Haxorus,Dragon,,540,76,147,90,60,70,97,5,False
613,Cubchoo,Ice,,305,55,70,40,60,40,40,5,False
614,Beartic,Ice,,485,95,110,80,70,80,50,5,False
615,Cryogonal,Ice,,485,70,50,30,95,135,105,5,False
616,Shelmet,Bug,,305,50,40,85,40,65,25,5,False
617,Accelgor,Bug,,495,80,70,40,100,60,145,5,False
618,Stunfisk,Ground,Electric,471,109,66,84,81,99,32,5,False
619,Mienfoo,Fighting,,350,45,85,50,55,50,65,5,False
620,Mienshao,Fighting,,510,65,125,60,95,60,105,5,False
621,Druddigon,Dragon,,485,77,120,90,60,90,48,5,False
622,Golett,Ground,Ghost,303,59,74,50,35,50,35,5,False
623,Golurk,Ground,Ghost,483,89,124,80,55,80,55,5,False
624,Pawniard,Dark,Steel,340,45,85,70,40,40,60,5,False
625,Bisharp,Dark,Steel,490,65,125,100,60,70,70,5,False
626,Bouffalant,Normal,,490,95,110,95,40,95,55,5,False
627,Rufflet,Normal,Flying,350,70,83,50,37,50,60,5,False
628,Braviary,Normal,Flying,510,100,123,75,57,75,80,5,False
629,Vullaby,Dark,Flying,370,70,55,75,45,65,60,5,False
630,Mandibuzz,Dark,Flying,510,110,65,105,55,95,80,5,False
631,Heatmor,Fire,,484,85,97,66,105,66,65,5,False
632,Durant,Bug,Steel,484,58,109,112,48,48,109,5,False
633,Deino,Dark,Dragon,300,52,65,50,45,50,38,5,False
634,Zweilous,Dark,Dragon,420,72,85,70,65,70,58,5,False
635,Hydreigon,Dark,Dragon,600,92,105,90,125,90,98,5,False
636,Larvesta,Bug,Fire,360,55,85,55,50,55,60,5,False
637,Volcarona,Bug,Fire,550,85,60,65,135,105,100,5,False
638,Cobalion,Steel,Fighting,580,91,90,129,90,72,108,5,True
639,Terrakion,Rock,Fighting,580,91,129,90,72,90,108,5,True
640,Virizion,Grass,Fighting,580,91,90,72,90,129,108,5,True
641,TornadusIncarnate Forme,Flying,,580,79,115,70,125,80,111,5,True
641,TornadusTherian Forme,Flying,,580,79,100,80,110,90,121,5,True
642,ThundurusIncarnate Forme,Electric,Flying,580,79,115,70,125,80,111,5,True
642,ThundurusTherian Forme,Electric,Flying,580,79,105,70,145,80,101,5,True
643,Reshiram,Dragon,Fire,680,100,120,100,150,120,90,5,True
644,Zekrom,Dragon,Electric,680,100,150,120,120,100,90,5,True
645,LandorusIncarnate Forme,Ground,Flying,600,89,125,90,115,80,101,5,True
645,LandorusTherian Forme,Ground,Flying,600,89,145,90,105,80,91,5,True
646,Kyurem,Dragon,Ice,660,125,130,90,130,90,95,5,True
646,KyuremBlack Kyurem,Dragon,Ice,700,125,170,100,120,90,95,5,True
646,KyuremWhite Kyurem,Dragon,Ice,700,125,120,90,170,100,95,5,True
647,KeldeoOrdinary Forme,Water,Fighting,580,91,72,90,129,90,108,5,False
647,KeldeoResolute Forme,Water,Fighting,580,91,72,90,129,90,108,5,False
648,MeloettaAria Forme,Normal,Psychic,600,100,77,77,128,128,90,5,False
648,MeloettaPirouette Forme,Normal,Fighting,600,100,128,90,77,77,128,5,False
649,Genesect,Bug,Steel,600,71,120,95,120,95,99,5,False
650,Chespin,Grass,,313,56,61,65,48,45,38,6,False
651,Quilladin,Grass,,405,61,78,95,56,58,57,6,False
652,Chesnaught,Grass,Fighting,530,88,107,122,74,75,64,6,False
653,Fennekin,Fire,,307,40,45,40,62,60,60,6,False
654,Braixen,Fire,,409,59,59,58,90,70,73,6,False
655,Delphox,Fire,Psychic,534,75,69,72,114,100,104,6,False
656,Froakie,Water,,314,41,56,40,62,44,71,6,False
657,Frogadier,Water,,405,54,63,52,83,56,97,6,False
658,Greninja,Water,Dark,530,72,95,67,103,71,122,6,False
659,Bunnelby,Normal,,237,38,36,38,32,36,57,6,False
660,Diggersby,Normal,Ground,423,85,56,77,50,77,78,6,False
661,Fletchling,Normal,Flying,278,45,50,43,40,38,62,6,False
662,Fletchinder,Fire,Flying,382,62,73,55,56,52,84,6,False
663,Talonflame,Fire,Flying,499,78,81,71,74,69,126,6,False
664,Scatterbug,Bug,,200,38,35,40,27,25,35,6,False
665,Spewpa,Bug,,213,45,22,60,27,30,29,6,False
666,Vivillon,Bug,Flying,411,80,52,50,90,50,89,6,False
667,Litleo,Fire,Normal,369,62,50,58,73,54,72,6,False
668,Pyroar,Fire,Normal,507,86,68,72,109,66,106,6,False
669,Flabébé,Fairy,,303,44,38,39,61,79,42,6,False
670,Floette,Fairy,,371,54,45,47,75,98,52,6,False
671,Florges,Fairy,,552,78,65,68,112,154,75,6,False
672,Skiddo,Grass,,350,66,65,48,62,57,52,6,False
673,Gogoat,Grass,,531,123,100,62,97,81,68,6,False
674,Pancham,Fighting,,348,67,82,62,46,48,43,6,False
675,Pangoro,Fighting,Dark,495,95,124,78,69,71,58,6,False
676,Furfrou,Normal,,472,75,80,60,65,90,102,6,False
677,Espurr,Psychic,,355,62,48,54,63,60,68,6,False
678,MeowsticMale,Psychic,,466,74,48,76,83,81,104,6,False
678,MeowsticFemale,Psychic,,466,74,48,76,83,81,104,6,False
679,Honedge,Steel,Ghost,325,45,80,100,35,37,28,6,False
680,Doublade,Steel,Ghost,448,59,110,150,45,49,35,6,False
681,AegislashBlade Forme,Steel,Ghost,520,60,150,50,150,50,60,6,False
681,AegislashShield Forme,Steel,Ghost,520,60,50,150,50,150,60,6,False
682,Spritzee,Fairy,,341,78,52,60,63,65,23,6,False
683,Aromatisse,Fairy,,462,101,72,72,99,89,29,6,False
684,Swirlix,Fairy,,341,62,48,66,59,57,49,6,False
685,Slurpuff,Fairy,,480,82,80,86,85,75,72,6,False
686,Inkay,Dark,Psychic,288,53,54,53,37,46,45,6,False
687,Malamar,Dark,Psychic,482,86,92,88,68,75,73,6,False
688,Binacle,Rock,Water,306,42,52,67,39,56,50,6,False
689,Barbaracle,Rock,Water,500,72,105,115,54,86,68,6,False
690,Skrelp,Poison,Water,320,50,60,60,60,60,30,6,False
691,Dragalge,Poison,Dragon,494,65,75,90,97,123,44,6,False
692,Clauncher,Water,,330,50,53,62,58,63,44,6,False
693,Clawitzer,Water,,500,71,73,88,120,89,59,6,False
694,Helioptile,Electric,Normal,289,44,38,33,61,43,70,6,False
695,Heliolisk,Electric,Normal,481,62,55,52,109,94,109,6,False
696,Tyrunt,Rock,Dragon,362,58,89,77,45,45,48,6,False
697,Tyrantrum,Rock,Dragon,521,82,121,119,69,59,71,6,False
698,Amaura,Rock,Ice,362,77,59,50,67,63,46,6,False
699,Aurorus,Rock,Ice,521,123,77,72,99,92,58,6,False
700,Sylveon,Fairy,,525,95,65,65,110,130,60,6,False
701,Hawlucha,Fighting,Flying,500,78,92,75,74,63,118,6,False
702,Dedenne,Electric,Fairy,431,67,58,57,81,67,101,6,False
703,Carbink,Rock,Fairy,500,50,50,150,50,150,50,6,False
704,Goomy,Dragon,,300,45,50,35,55,75,40,6,False
705,Sliggoo,Dragon,,452,68,75,53,83,113,60,6,False
706,Goodra,Dragon,,600,90,100,70,110,150,80,6,False
707,Klefki,Steel,Fairy,470,57,80,91,80,87,75,6,False
708,Phantump,Ghost,Grass,309,43,70,48,50,60,38,6,False
709,Trevenant,Ghost,Grass,474,85,110,76,65,82,56,6,False
710,PumpkabooAverage Size,Ghost,Grass,335,49,66,70,44,55,51,6,False
710,PumpkabooSmall Size,Ghost,Grass,335,44,66,70,44,55,56,6,False
710,PumpkabooLarge Size,Ghost,Grass,335,54,66,70,44,55,46,6,False
710,PumpkabooSuper Size,Ghost,Grass,335,59,66,70,44,55,41,6,False
711,GourgeistAverage Size,Ghost,Grass,494,65,90,122,58,75,84,6,False
711,GourgeistSmall Size,Ghost,Grass,494,55,85,122,58,75,99,6,False
711,GourgeistLarge Size,Ghost,Grass,494,75,95,122,58,75,69,6,False
711,GourgeistSuper Size,Ghost,Grass,494,85,100,122,58,75,54,6,False
712,Bergmite,Ice,,304,55,69,85,32,35,28,6,False
713,Avalugg,Ice,,514,95,117,184,44,46,28,6,False
714,Noibat,Flying,Dragon,245,40,30,35,45,40,55,6,False
715,Noivern,Flying,Dragon,535,85,70,80,97,80,123,6,False
716,Xerneas,Fairy,,680,126,131,95,131,98,99,6,True
717,Yveltal,Dark,Flying,680,126,131,95,131,98,99,6,True
718,Zygarde50% Forme,Dragon,Ground,600,108,100,121,81,95,95,6,True
719,Diancie,Rock,Fairy,600,50,100,150,100,150,50,6,True
719,DiancieMega Diancie,Rock,Fairy,700,50,160,110,160,110,110,6,True
720,HoopaHoopa Confined,Psychic,Ghost,600,80,110,60,150,130,70,6,True
720,HoopaHoopa Unbound,Psychic,Dark,680,80,160,60,170,130,80,6,True
721,Volcanion,Fire,Water,600,80,110,120,130,90,70,6,True
================================================
FILE: pandas_log/__init__.py
================================================
# -*- coding: utf-8 -*-
"""Top-level package for pandas-log."""
from .pandas_log import *
__author__ = """Eyal Trabelsi"""
__email__ = "eyaltrabelsi@gmail.com"
__version__ = "0.0.0"
ORIGINAL_METHOD_PREFIX = "original_"
================================================
FILE: pandas_log/aop_utils.py
================================================
import itertools
from inspect import signature
import pandas as pd
from pandas_log import settings
from pandas_log.settings import DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE
def set_df_attr(df, attr_name, attr_value):
""" Hacky way to set attributes in dataframe
:param df: DataFrame
:param attr_name: Attribute name
:param attr_value: Attribute value
:return: None
"""
df.__dict__[attr_name] = attr_value
def append_df_attr(df, attr_name, attr_value):
""" Hacky way to append a value to dataframe
:param df: DataFrame
:param attr_name: Attribute name
:param attr_value: Attribute value
:return: None
"""
df.__dict__[attr_name].append(attr_value)
def get_df_attr(df, attr_name, default_val):
""" Get Dataframe attribute if exists otherwise default value
:return: Dataframe attribute
"""
return df.__dict__.get(attr_name, default_val)
def get_pandas_func(cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX):
""" Get original pandas method
:param cls: pandas class
:param func: pandas method name
:param prefix: the prefix used to keep original method
:return: Original pandas method
"""
_raise_on_bad_class(cls)
return getattr(cls, f"{prefix}{func.__name__}")
def get_signature_repr(cls, fn, args, full_signature=True):
""" Get the signature for the original pandas method with actual values
:param cls: the pandas class
:param fn: The pandas method
:param args: The arguments used when it was applied
:return: string representation of the signature for the applied pandas method
"""
_raise_on_bad_class(cls)
def _get_bold_text(text):
return f"\033[1m{text}\033[0m"
def _get_orig_func_params():
return [
param_value if full_signature else param_name
for param_name, param_value in signature(
get_pandas_func(cls, fn)
).parameters.items()
if param_name not in ("kwargs", "self")
]
def _get_param_value(param_with_default, arg_value):
res = str(param_with_default)
if arg_value is not None:
param_name = res.split("=")[0]
arg_value = (
f'"{arg_value}"' if isinstance(arg_value, str) else arg_value
)
res = (
param_name
if isinstance(arg_value, pd.DataFrame)
or isinstance(arg_value, pd.Series)
else f"{param_name}={arg_value}"
)
return res
zip_func = itertools.zip_longest if full_signature else zip
orig_func_params = _get_orig_func_params()
args_vals = ", ".join(
_get_param_value(param_with_default, arg_value)
for param_with_default, arg_value in zip_func(orig_func_params, args)
)
return f"{_get_bold_text(fn.__name__)}({args_vals}):"
def _raise_on_bad_class(cls):
implemented_classes = (pd.DataFrame, pd.Series)
if cls not in implemented_classes:
raise TypeError("cls must be one of {}".format(implemented_classes))
def restore_pandas_func_copy(
cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX
):
""" Restore the original pandas method instead of overridden one
:param cls: class containing the method
:param func: pandas method name
:param prefix: the prefix used to keep original method
:return: None
"""
_raise_on_bad_class(cls)
original_method = getattr(cls, func)
setattr(cls, func.replace(prefix, ""), original_method)
def keep_pandas_func_copy(cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX):
""" Saved copy of the pandas method before it overridden
:param cls: class containing the method
:param func: pandas method name
:param prefix: the prefix used to keep original method
:return: None
"""
_raise_on_bad_class(cls)
original_method = getattr(cls, func)
setattr(cls, f"{prefix}{func}", original_method)
def calc_step_number(method_name, input_df):
# TODO
step_number = get_df_attr(input_df, "execution_history", 0)
if step_number:
step_number = step_number[-1].execution_stats.step_number
if method_name not in DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE:
step_number += 1
return step_number
if __name__ == "__main__":
pass
================================================
FILE: pandas_log/pandas_execution_stats.py
================================================
import warnings
from collections import namedtuple
from functools import partial
from time import time
import pandas as pd
from pandas_log import patched_logs_functions
from pandas_log.aop_utils import (
append_df_attr,
calc_step_number,
get_df_attr,
get_pandas_func,
get_signature_repr,
set_df_attr,
)
from pandas_log.settings import (
DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE,
PATCHED_LOG_METHOD_PREFIX,
)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
import humanize
def get_execution_stats(cls, fn, input_df, fn_args, fn_kwargs, calculate_memory):
start = time()
output_df = get_pandas_func(cls, fn)(input_df, *fn_args, **fn_kwargs)
exec_time = time() - start
exec_time_pretty = humanize.naturaldelta(exec_time)
if exec_time_pretty == "a moment":
exec_time_pretty = f"{round(exec_time,6)} seconds"
step_number = calc_step_number(fn.__name__, input_df)
input_memory_size = (
StepStats.calc_df_series_memory(input_df) if calculate_memory else None
)
output_memory_size = (
StepStats.calc_df_series_memory(output_df) if calculate_memory else None
)
ExecutionStats = namedtuple(
"ExecutionStats", "exec_time step_number input_memory_size output_memory_size",
)
execution_stats = ExecutionStats(
exec_time_pretty, step_number, input_memory_size, output_memory_size
)
return output_df, execution_stats
class StepStats:
def __init__(
self,
execution_stats,
cls,
fn,
fn_args,
fn_kwargs,
full_signature,
input_df,
output_df,
):
""" Constructor
:param execution_stats: execution_stats of the pandas operation both in time and memory
:param cls: The calling object's pandas class
:param fn: The original pandas method
:param fn_args: The original pandas method args
:param fn_kwargs: The original pandas method kwargs
:param full_signature: adding additional information to function signature
:param input_df: dataframe before step calculation
:param output_df: dataframe after step calculation
"""
self.execution_stats = execution_stats
self.full_signature = full_signature
self.cls = cls
self.fn = fn
self.fn_args = fn_args
self.fn_kwargs = fn_kwargs
self.input_df = input_df
self.output_df = output_df
@staticmethod
def calc_df_series_memory(df_or_series):
res = None
if isinstance(df_or_series, pd.Series):
mem = df_or_series.memory_usage(index=True, deep=True)
res = humanize.naturalsize(mem)
elif isinstance(df_or_series, pd.DataFrame):
mem = df_or_series.memory_usage(index=True, deep=True)
res = humanize.naturalsize(mem.sum())
return res
def persist_execution_stats(self):
prev_exec_history = get_df_attr(self.input_df, "execution_history", [])
set_df_attr(self.output_df, "execution_history", prev_exec_history)
append_df_attr(self.output_df, "execution_history", self)
def log_stats_if_needed(self, silent, verbose, copy_ok):
from pandas_log.pandas_log import ALREADY_ENABLED
if silent or not ALREADY_ENABLED:
return
if verbose or self.fn.__name__ not in DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE:
s = self.__repr__(verbose, copy_ok)
if s:
# If this method isn't patched and verbose is False, __repr__ will give an empty string, which
# we don't want to print
print(s)
def get_logs_for_specifc_method(self, verbose, copy_ok):
self.fn_kwargs["kwargs"] = self.fn_kwargs.copy()
self.fn_kwargs["copy_ok"] = copy_ok
try:
log_method = getattr(
patched_logs_functions,
f"{PATCHED_LOG_METHOD_PREFIX}{self.fn.__name__}",
)
except AttributeError:
# Method is listed as a method to override, but no patched function exists
if verbose:
log_method = getattr(patched_logs_functions, "log_default")
else:
log_method = getattr(patched_logs_functions, "log_no_message")
log_method = partial(log_method, self.output_df, self.input_df)
logs, tips = log_method(*self.fn_args, **self.fn_kwargs)
return logs, tips
def _repr_html_(self):
pass
def __repr__(self, verbose, copy_ok):
# Step title
func_sig = get_signature_repr(
self.cls, self.fn, self.fn_args, self.full_signature
)
step_number = (
"X"
if self.fn.__name__ in DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE
else self.execution_stats.step_number
)
step_title = f"{step_number}) {func_sig}"
# Step Metadata stats
logs, tips = self.get_logs_for_specifc_method(verbose, copy_ok)
metadata_stats = f"\033[4mMetadata\033[0m:\n{logs}" if logs else ""
metadata_tips = f"\033[4mTips\033[0m:\n{tips}" if tips else ""
# Step Execution stats
exec_time_humanize = (
f"* Execution time: Step Took {self.execution_stats.exec_time}."
)
exec_stats_raw = [exec_time_humanize]
if self.execution_stats.input_memory_size is not None:
exec_stats_raw.append(
f"* Input Dataframe size is {self.execution_stats.input_memory_size}."
)
if self.execution_stats.output_memory_size is not None:
exec_stats_raw.append(
f"* Output Dataframe size is {self.execution_stats.output_memory_size}."
)
exec_stats_raw_str = "\n\t".join(exec_stats_raw)
execution_stats = f"\033[4mExecution Stats\033[0m:\n\t{exec_stats_raw_str}"
all_logs = [metadata_stats, execution_stats, metadata_tips]
all_logs_str = "\n\t".join([x for x in all_logs if x])
return f"\n{step_title}\n\t{all_logs_str}"
if __name__ == "__main__":
pass
================================================
FILE: pandas_log/pandas_log.py
================================================
# -*- coding: utf-8 -*-
"""Main module."""
import warnings
from contextlib import contextmanager
from functools import wraps
import pandas as pd
import pandas_flavor as pf
from pandas_log import settings
from pandas_log.aop_utils import (
keep_pandas_func_copy,
restore_pandas_func_copy,
)
from pandas_log.pandas_execution_stats import StepStats, get_execution_stats
__all__ = ["auto_enable", "auto_disable", "enable"]
ALREADY_ENABLED = False
def auto_disable():
""" Restore original pandas method without the additional log functionality (statistics)
Note: we keep the original methods using original_ prefix.
:return: None
"""
global ALREADY_ENABLED
if not ALREADY_ENABLED:
return
for cls in (pd.DataFrame, pd.Series):
for func in dir(cls):
if func.startswith(settings.ORIGINAL_METHOD_PREFIX):
restore_pandas_func_copy(cls, func)
ALREADY_ENABLED = False
@contextmanager
def enable(
verbose=False,
silent=False,
full_signature=True,
copy_ok=True,
calculate_memory=False,
):
""" Adds the additional logging functionality (statistics) to pandas methods only for the scope of this
context manager.
:param verbose: Whether some inner functions should be recorded as well.
For example: when a dataframe being copied
:param silent: Whether additional the statistics get printed
:param full_signature: adding additional information to function signature
:param copy_ok: whether the dataframe is allowed to be copied to calculate more informative metadata logs
:return: None
"""
auto_enable(verbose, silent, full_signature, copy_ok, calculate_memory)
yield
auto_disable()
def auto_enable(
verbose=False,
silent=False,
full_signature=True,
copy_ok=True,
calculate_memory=False,
):
""" Adds the additional logging functionality (statistics) to pandas methods.
:param verbose: Whether some inner functions should be recorded as well.
For example: when a dataframe being copied
:param silent: Whether additional the statistics get printed
:param full_signature: adding additional information to function signature
:param copy_ok: whether the dataframe is allowed to be copied to calculate more informative metadata logs
:return: None
"""
global ALREADY_ENABLED
if ALREADY_ENABLED:
return
settings.DATAFRAME_METHODS_TO_OVERIDE.extend(
settings.DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE
)
# Suppressing warning of the fact we override pandas functions.
with warnings.catch_warnings():
warnings.simplefilter("ignore")
for cls, overrides in [
(pd.DataFrame, settings.DATAFRAME_METHODS_TO_OVERIDE),
(pd.Series, settings.SERIES_METHODS_TO_OVERIDE),
]:
for func in dir(cls):
if func in overrides:
keep_pandas_func_copy(cls, func)
create_overide_pandas_func(
cls,
func,
verbose,
silent,
full_signature,
copy_ok,
calculate_memory,
)
ALREADY_ENABLED = True
def create_overide_pandas_func(
cls, func, verbose, silent, full_signature, copy_ok, calculate_memory
):
""" Create overridden pandas method dynamically with
additional logging using DataFrameLogger
Note: if we extracting _overide_pandas_method outside we need to implement decorator like here
https://stackoverflow.com/questions/10176226/how-do-i-pass-extra-arguments-to-a-python-decorator
:param cls: pandas class for which the method should be overriden
:param func: pandas method name to be overridden
:param silent: Whether additional the statistics get printed
:param full_signature: adding additional information to function signature
:param copy_ok: whether the dataframe is allowed to be copied to calculate more informative metadata logs
:return: the same function with additional logging capabilities
"""
def _run_method_and_calc_stats(
fn,
fn_args,
fn_kwargs,
input_df,
full_signature,
silent,
verbose,
copy_ok,
calculate_memory,
):
if copy_ok:
# If we're ok to make copies, copy the input_df so that we can compare against the output of inplace methods
try:
# Will hit infinite recursion if we use the patched copy method so use the original
original_input_df = getattr(
input_df, settings.ORIGINAL_METHOD_PREFIX + "copy"
)(deep=True)
except AttributeError:
original_input_df = input_df.copy(deep=True)
output_df, execution_stats = get_execution_stats(
cls, fn, input_df, fn_args, fn_kwargs, calculate_memory
)
if output_df is None:
# The operation was strictly in place so we just call the dataframe the output_df as well
output_df = input_df
if copy_ok:
# If this isn't true and the method was strictly inplace, input_df and output_df will just
# point to the same object
input_df = original_input_df
step_stats = StepStats(
execution_stats,
cls,
fn,
fn_args,
fn_kwargs,
full_signature,
input_df,
output_df,
)
step_stats.log_stats_if_needed(silent, verbose, copy_ok)
if isinstance(output_df, pd.DataFrame) or isinstance(output_df, pd.Series):
step_stats.persist_execution_stats()
return output_df
def _overide_pandas_method(fn):
if cls == pd.DataFrame:
register_method_wrapper = pf.register_dataframe_method
elif cls == pd.Series:
register_method_wrapper = pf.register_series_method
@register_method_wrapper
@wraps(fn)
def wrapped(*args, **fn_kwargs):
input_df, fn_args = args[0], args[1:]
output_df = _run_method_and_calc_stats(
fn,
fn_args,
fn_kwargs,
input_df,
full_signature,
silent,
verbose,
copy_ok,
calculate_memory,
)
return output_df
return wrapped
return exec(f"@_overide_pandas_method\ndef {func}(df, *args, **kwargs): pass")
if __name__ == "__main__":
pass
================================================
FILE: pandas_log/patched_logs_functions.py
================================================
import warnings
import numpy as np
import pandas as pd
# Values messages
ALTERED_VALUES_MSG = "\t* Changed {values_changed} values, {values_unchanged} values were not changed."
# Rows messages
REMOVED_NO_ROWS_MSG = "\t* No change in number of rows of input df."
FILTERED_ROWS_MSG = "\t* Removed {rows_removed} rows ({rows_removed_pct}%), {rows_remaining} rows remaining."
# Cols messages
REMOVED_NO_COLS_MSG = "\t* Removed no columns."
FILTERED_COLS_MSG = (
"\t* Removed the following columns ({cols_removed}) now only have the following columns"
"({cols_remaining})."
)
ASSIGN_EXISTING_MSG = "\t* The columns {existing_cols} were reassigned."
ASSIGN_NEW_MSG = "\t* The columns {new_cols} were created."
# Group by messages
GROUPBY_MSG = "\t* Grouping by {} resulted in {} groups like {}."
# N/A messages
FILLNA_NO_NA_MSG = "\t* There are no nulls."
FILLNA_WITHH_NA_MSG = "\t* Filled {} with {}."
# Mege messages
JOIN_ROWS_MSG = "\t* Number of rows changed, after join is {output_rows} rows."
JOIN_NEW_COLS_MSG = "\t* Added {num_new_columns} columns ({new_columns})."
JOIN_TYPE_MSG = (
"\t* Its a {how} join with the following cardinality:\n\t\t> rows only in left is {left_only}."
"\n\t\t> rows only in right is {right_only}.\n\t\t> rows in both is {both}."
)
# Pick messages
SAMPLE_MSG = "\t* Picked random sample of {output_rows} rows."
NLARGEST_MSG = "\t* Picked {n} largest rows by columns ({cols})."
NSMALLEST_MSG = "\t* Picked {n} smallest rows by columns ({cols})."
HEAD_MSG = "\t* Picked the first {} rows."
TAIL_MSG = "\t* Picked the last {} rows."
# TIPS
ITERROWS_TIPS = "\t*iterrows is not recommended, and in the majority of cases will have better alternatives"
FILLNA_NO_NA_TIP = (
"\t* There are no nulls in this dataframe, if you are working on the entire dataset you can "
"remove this operation."
)
SHOULD_REDUCED_ROW_TIP = (
"\t* Number of rows didn't change, if you are working on the entire dataset you can remove "
"this operation."
)
# Others
SORT_VALUES_MSG = "\t* Sorting by columns {} in a {} order."
SORT_INDEX_MSG = "\t* Sorting by index in a {'ascending' if ascending else 'descending'} order."
DEFAULT_STRATEGY_USED_MSG = (
"\t* Using default strategy (some metric might not be relevant)."
)
TRANSFORMED_TO_DF_MSG = "\t* After transformation we received Series"
COPY_WARNING_MSG = (
"Some pandas logging may involve copying dataframes, which can be time-/memory-intensive. "
"Consider passing copy_ok=False to the enable/auto_enable functions in pandas_log if issues arise."
)
def _stringify_list(l):
return [str(x) for x in l]
def rows_removed(input_df, output_df):
return len(input_df) - len(output_df)
def rows_removed_pct(input_df, output_df):
return 100 * (rows_removed(input_df, output_df)) / len(input_df)
def rows_remaining(output_df):
return len(output_df)
def cols_removed(input_df, output_df):
return ", ".join(
_stringify_list(set(input_df.columns) - set(output_df.columns))
)
def cols_remaining(output_df):
return ", ".join(_stringify_list(set(output_df.columns)))
def is_same_cols(input_df, output_df):
return len(input_df.columns) == len(output_df.columns)
def columns_changed(df, cols):
return set(df.columns).intersection(set(cols))
def columns_added(df, cols):
return set(cols) - set(df.columns)
def is_same_rows(input_df, output_df):
return len(input_df) == len(output_df)
def num_of_na(df):
return df.isnull().values.sum()
def str_new_columns(input_df, output_df):
return ", ".join(
_stringify_list(set(output_df.columns) - set(input_df.columns))
)
def num_new_columns(input_df, output_df):
return len(set(output_df.columns) - set(input_df.columns))
def num_values_changed(input_obj, output_obj):
if (
isinstance(input_obj, pd.Series)
and isinstance(output_obj, pd.Series)
and input_obj.dtype != output_obj.dtype
):
# Comparing values for equality across dtypes wouldn't be well-defined so we just say they all changed
values_changed = len(input_obj)
else:
values_changed = (
(output_obj != input_obj)
& ~(output_obj.isnull() & input_obj.isnull())
).sum()
if isinstance(input_obj, pd.DataFrame):
# We only summed once so values_changed will be a series, so we sum again
values_changed = values_changed.sum()
values_unchanged = (
input_obj.shape[0] * input_obj.shape[1]
) - values_changed
elif isinstance(input_obj, pd.Series):
values_unchanged = len(output_obj) - values_changed
return values_changed, values_unchanged
def get_filter_rows_logs(input_df, output_df):
tips = ""
if is_same_rows(input_df, output_df):
logs = REMOVED_NO_ROWS_MSG
tips = SHOULD_REDUCED_ROW_TIP
else:
logs = FILTERED_ROWS_MSG.format(
rows_removed=rows_removed(input_df, output_df),
rows_removed_pct=rows_removed_pct(input_df, output_df),
rows_remaining=rows_remaining(output_df),
)
return logs, tips
def log_default(output_df, input_df, *args, **kwargs):
logs = [DEFAULT_STRATEGY_USED_MSG]
tips = ""
if isinstance(output_df, pd.DataFrame):
if not is_same_cols(input_df, output_df):
logs.append(
FILTERED_COLS_MSG.format(
cols_removed=cols_removed(input_df, output_df),
cols_remaining=cols_remaining(output_df),
)
)
if not is_same_rows(input_df, output_df):
logs.append(
FILTERED_ROWS_MSG.format(
rows_removed=rows_removed(input_df, output_df),
rows_removed_pct=rows_removed_pct(input_df, output_df),
rows_remaining=rows_remaining(output_df),
)
)
elif isinstance(output_df, pd.Series):
logs.append(TRANSFORMED_TO_DF_MSG)
logs = "\n".join(logs)
return logs, tips
def log_no_message(output_df, input_df, *args, **kwargs):
return "", ""
def log_drop(
output_df,
input_df,
labels=None,
axis=0,
index=None,
columns=None,
level=None,
inplace=False,
errors="raise",
*args,
**kwargs,
):
logs, tips = get_filter_rows_logs(input_df, output_df)
tips = ""
if is_same_cols(input_df, output_df):
logs += f"\n{REMOVED_NO_COLS_MSG}"
else:
msg = FILTERED_COLS_MSG.format(
cols_removed=cols_removed(input_df, output_df),
cols_remaining=cols_remaining(output_df),
)
logs += f"\n{msg}"
return logs, tips
def log_dropna(
output_df,
input_df,
axis=0,
how="any",
thresh=None,
subset=None,
inplace=False,
**kwargs,
):
logs, tips = get_filter_rows_logs(input_df, output_df)
if is_same_cols(input_df, output_df):
logs += f"\n{REMOVED_NO_COLS_MSG}"
tips = SHOULD_REDUCED_ROW_TIP
else:
msg = FILTERED_COLS_MSG.format(
cols_removed=cols_removed(input_df, output_df),
cols_remaining=cols_remaining(output_df),
)
logs += f"\n{msg}"
return logs, tips
def log_assign(output_df, input_df, **kwargs):
logs = []
tips = ""
cols = [key for key in kwargs.keys() if key not in ["kwargs", "copy_ok"]]
changed_cols = columns_changed(input_df, cols)
added_cols = columns_added(input_df, cols)
if changed_cols:
if kwargs["copy_ok"]:
warnings.warn(COPY_WARNING_MSG)
# If copying is ok, we can check how many values actually changed
for col in changed_cols:
values_changed, values_unchanged = num_values_changed(
input_df[col], output_df[col]
)
logs.append(
"\t* {}: {}".format(
col,
ALTERED_VALUES_MSG.format(
values_changed=values_changed,
values_unchanged=values_unchanged,
)[3:],
)
) # [3:] to strip the "\t* " from the altered values message
else:
# Otherwise just indicated which columns changed
# (Doing the above calculation would always say zero values changed in this case, since if copying isn't
# ok then input_df and output_df point to the same object)
logs.append(
ASSIGN_EXISTING_MSG.format(
existing_cols=", ".join(changed_cols)
)
)
if added_cols:
logs.append(
ASSIGN_NEW_MSG.format(
new_cols=", ".join(columns_added(input_df, cols))
)
)
logs = "\n".join(logs)
return logs, tips
def log___setitem__(output_df, input_df, key, value, **kwargs):
if isinstance(key, str):
# Only setting one column so can just use the assign logger as is
kwargs[key] = value
elif isinstance(key, list):
# This would be kind of complicated but since we don't actually use the values in kwargs we can just
# add the new columns as keys each with the full set of assigned values (as a placeholder)
for subkey in key:
kwargs[subkey] = value
return log_assign(output_df, input_df, **kwargs)
def log_query(output_df, input_df, expr, inplace=False, *args, **kwargs):
logs, tips = get_filter_rows_logs(input_df, output_df)
return logs, tips
def log_sort_index(
output_df,
input_df,
axis=0,
level=None,
ascending=True,
inplace=False,
kind="quicksort",
na_position="last",
sort_remaining=True,
by=None,
**kwargs,
):
logs = SORT_INDEX_MSG.format(ascending)
tips = ""
return logs, tips
def log_sort_values(
output_df,
input_df,
by,
axis=0,
ascending=True,
inplace=False,
kind="quicksort",
na_position="last",
**kwargs,
):
logs = SORT_VALUES_MSG.format(
by, "ascending" if ascending else "descending"
)
tips = ""
return logs, tips
def log_tail(output_df, input_df, n=5, **kwargs):
logs = TAIL_MSG.format(n)
tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else ""
return logs, tips
def log_head(output_df, input_df, n=5, **kwargs):
logs = HEAD_MSG.format(n)
tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else ""
return logs, tips
def log_merge(
output_df,
input_df,
right,
how="inner",
on=None,
left_on=None,
right_on=None,
left_index=False,
right_index=False,
sort=False,
suffixes=("_x", "_y"),
copy=True,
indicator=False,
validate=None,
**kwargs,
):
logs = []
tips = ""
merged = input_df.original_merge(
right,
"outer",
on,
left_on,
right_on,
left_index,
right_index,
sort,
suffixes,
copy,
True,
validate,
)
logs.append(
JOIN_TYPE_MSG.format(how=how, **merged._merge.value_counts().to_dict())
)
if is_same_rows(input_df, output_df):
logs.append(REMOVED_NO_ROWS_MSG)
else:
logs.append(JOIN_ROWS_MSG.format(output_rows=len(output_df)))
if not is_same_cols(input_df, output_df):
logs.append(
JOIN_NEW_COLS_MSG.format(
num_new_columns=num_new_columns(input_df, output_df),
new_columns=str_new_columns(input_df, output_df),
)
)
logs = "\n".join(logs)
return logs, tips
def log_applymap(output_df, input_df, func, **kwargs):
values_changed, values_unchanged = num_values_changed(input_df, output_df)
log = ALTERED_VALUES_MSG.format(
values_changed=values_changed, values_unchanged=values_unchanged
)
return log, ""
def log_join(
output_df,
input_df,
other,
on=None,
how="left",
lsuffix="",
rsuffix="",
sort=False,
**kwargs,
):
logs = []
tips = ""
merged = input_df.original_merge(
other, "outer", on, input_df.index, other.index, indicator=True
)
logs.append(
JOIN_TYPE_MSG.format(how=how, **merged._merge.value_counts().to_dict())
)
logs.append(get_filter_rows_logs(input_df, output_df))
if not is_same_cols(input_df, output_df):
logs.append(
JOIN_NEW_COLS_MSG.format(
num_new_columns=num_new_columns(input_df, output_df),
new_columns=str_new_columns(input_df, output_df),
)
)
logs = "\n".join(logs)
return logs, tips
def log_fillna(
output_df,
input_df,
value=None,
method=None,
axis=None,
inplace=False,
limit=None,
downcast=None,
**kwargs,
):
tips = ""
value = "empty string" if value == "" else value
if num_of_na(input_df) == num_of_na(output_df):
logs = FILLNA_NO_NA_MSG
tips = FILLNA_NO_NA_TIP
else:
logs = FILLNA_WITHH_NA_MSG.format(num_of_na(input_df), value)
return logs, tips
def log_mask(
output_df,
input_df,
cond,
other=np.nan,
inplace=False,
axis=None,
level=None,
errors="raise",
try_cast=False,
*args,
**kwargs,
):
values_changed, values_unchanged = num_values_changed(input_df, output_df)
logs = []
logs.append(
ALTERED_VALUES_MSG.format(
values_changed=values_changed, values_unchanged=values_unchanged
)
)
if isinstance(cond, pd.Series) and isinstance(input_df, pd.Series):
# Calculate the values for which the condition was true but the value didn't change only for this simplest case,
# where we can just & the two series
true_but_unchanged = (cond & (output_df == input_df)).sum()
if true_but_unchanged > 0:
logs.append(
"\t* {} rows met the masking condition and already had the masking value.".format(
true_but_unchanged
)
)
logs = "\n".join(logs)
tips = ""
return logs, tips
def log_where(
output_df,
input_df,
cond,
other=np.nan,
inplace=False,
axis=None,
level=None,
errors="raise",
try_cast=False,
*args,
**kwargs,
):
return log_mask(
output_df,
input_df,
~cond, # Important
other=np.nan,
inplace=False,
axis=None,
level=None,
errors="raise",
try_cast=False,
*args,
**kwargs,
)
def log_sample(
output_df,
input_df,
n=None,
frac=None,
replace=False,
weights=None,
random_state=None,
axis=None,
*args,
**kwargs,
):
logs = SAMPLE_MSG.format(output_rows=len(output_df))
tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else ""
return logs, tips
def log_nlargest(output_df, input_df, n, columns, keep="first", **kwargs):
# todo maybe wrong
logs = NLARGEST_MSG.format(n=n, cols=columns)
tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else ""
return logs, tips
def log_nsmallest(output_df, input_df, n, columns, keep="first", **kwargs):
logs = NSMALLEST_MSG.format(n=n, cols=columns)
tips = SHOULD_REDUCED_ROW_TIP if is_same_rows(input_df, output_df) else ""
return logs, tips
def log_groupby(
output_df,
input_df,
by=None,
axis=0,
level=None,
as_index=True,
sort=True,
group_keys=True,
squeeze=False,
observed=False,
**kwargs,
):
group_by = str(by)
groups = list(output_df.groups)
groups_len = len(groups)
groups_repr = (
",".join(["\n\t\t" + str(x) for x in groups[:5]]) + ",\n\t and more"
)
tips = ""
logs = GROUPBY_MSG.format(group_by, groups_len, groups_repr)
return logs, tips
def log__iterrows(output_df, input_df):
tips = ITERROWS_TIPS
logs = ""
return logs, tips
def log___getitem__(output_df, input_df, key, *args, **kwargs):
logs = []
tips = ""
if isinstance(output_df, pd.Series):
# Naive handle of __getitem__ which return series
logs = TRANSFORMED_TO_DF_MSG
return logs, tips
if not is_same_cols(input_df, output_df):
logs.append(
FILTERED_COLS_MSG.format(
cols_removed=cols_removed(input_df, output_df),
cols_remaining=cols_remaining(output_df),
)
)
if not is_same_rows(input_df, output_df):
logs.append(
FILTERED_ROWS_MSG.format(
rows_removed=rows_removed(input_df, output_df),
rows_removed_pct=rows_removed_pct(input_df, output_df),
rows_remaining=rows_remaining(output_df),
)
)
logs = "\n".join(logs)
return logs, tips
# TODO add tip on types+cardinality
if __name__ == "__main__":
pass
================================================
FILE: pandas_log/settings.py
================================================
ORIGINAL_METHOD_PREFIX = "original_"
PATCHED_LOG_METHOD_PREFIX = "log_"
DATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE = [
"copy",
"reset_index",
"__getitem__",
"__setitem__",
]
DATAFRAME_METHODS_TO_OVERIDE = [
"query",
"drop",
"dropna",
"assign",
"sort_index",
"sort_values",
"head",
"tail",
"sample",
"fillna",
"merge",
"join",
"nlargest",
"nsmallest",
"apply",
"iterrows",
"applymap",
"pipe",
"rolling",
"groupby",
"rename",
"agg",
"aggregate",
"stack",
"unstack",
"pivot",
"pivot_table",
"mask",
"max",
"mean",
"median",
"melt",
"replace",
"skew",
"notna",
"kurt",
"expanding",
"drop_duplicates",
"bfill",
"corr",
"corrwith",
"droplevel",
"explode",
"ffill",
"filter",
"first",
"kurtosis",
"align",
"transform",
"update",
"squeeze",
"shift",
"rank",
"nunique",
"min",
"mod",
"mode",
"std",
]
SERIES_METHODS_TO_OVERIDE = ["mask", "where"]
================================================
FILE: requirements_dev.txt
================================================
pip>=19.2.3
bump2version>=0.5.11
wheel>=0.33.6
watchdog>=0.9.0
flake8>=3.7.8
tox>=3.14.0
coverage>=4.5.4
Sphinx>=2.2.0
twine>=2.2.0
pytest>=5.1.3
pytest-runner>=5.1
pandas>=0.25.1
pandas_flavor>=0.1.2
humanize>=0.5.0
================================================
FILE: setup.cfg
================================================
[bumpversion]
current_version = 0.1.7
commit = True
tag = True
[bumpversion:file:setup.py]
search = version='{current_version}'
replace = version='{new_version}'
[bumpversion:file:pandas_log/__init__.py]
search = __version__ = '{current_version}'
replace = __version__ = '{new_version}'
[bdist_wheel]
universal = 1
[flake8]
exclude = docs
[aliases]
# Define setup.py command aliases here
test = pytest
[tool:pytest]
collect_ignore = ['setup.py']
================================================
FILE: setup.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""The setup script."""
from setuptools import setup, find_packages
with open('README.rst') as readme_file:
readme = readme_file.read()
requirements = ["humanize>=0.5.0", "pandas>=0.25.1", "pandas_flavor>=0.1.2"]
setup_requirements = ['pytest-runner', ]
test_requirements = ['pytest>=3', ]
setup(
name='pandas-log',
version='0.1.7',
description="pandas-log provides feedback about basic pandas operations. It provides simple wrapper functions for "
"the most common functions, such as apply, map, query and more.",
author="Eyal Trabelsi",
author_email='eyaltrabelsi@gmail.com',
url='https://github.com/eyaltrabelsi/pandas-log',
packages=find_packages(include=['pandas_log', 'pandas_log.*']),
install_requires=requirements,
python_requires=">=3.4",
license="MIT license",
long_description="pandas-log provides feedback about basic pandas operations. It provides simple wrapper functions "
"for the most common functions, such as apply, map, query and more.",
long_description_content_type="text/x-rst",
setup_requires=setup_requirements,
test_suite='tests',
tests_require=test_requirements
)
================================================
FILE: tox.ini
================================================
[tox]
envlist = py27, py35, py36, py37 flake8
[travis]
python =
3.7: py37
3.6: py36
3.5: py35
2.7: py27
[testenv:flake8]
basepython = python
deps = flake8
commands = flake8 pandas_log
[testenv]
setenv =
PYTHONPATH = {toxinidir}
deps =
-r{toxinidir}/requirements_dev.txt
; If you want to make tox run the tests with the same versions, create a
; requirements.txt with the pinned versions and uncomment the following line:
; -r{toxinidir}/requirements.txt
commands =
pip install -U pip
pytest --basetemp={envtmpdir}
gitextract_a1_r2osh/ ├── .github/ │ ├── ISSUE_TEMPLATE/ │ │ ├── bug_report.md │ │ ├── documentation_fix.md │ │ ├── new_examples.md │ │ └── new_proposed_feature.md │ └── pull_request_template.md ├── .gitignore ├── AUTHORS.rst ├── CONTRIBUTING.rst ├── HISTORY.rst ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.rst ├── TODO.rst ├── docs/ │ ├── Makefile │ ├── conf.py │ ├── contributing.rst │ ├── index.rst │ ├── installation.rst │ ├── modules.rst │ ├── pandas_log.rst │ ├── readme.rst │ └── usage.rst ├── examples/ │ ├── README.rst │ ├── __init__.py │ ├── pandas_log_intro.ipynb │ └── pokemon.csv ├── pandas_log/ │ ├── __init__.py │ ├── aop_utils.py │ ├── pandas_execution_stats.py │ ├── pandas_log.py │ ├── patched_logs_functions.py │ └── settings.py ├── requirements_dev.txt ├── setup.cfg ├── setup.py └── tox.ini
SYMBOL INDEX (60 symbols across 4 files)
FILE: pandas_log/aop_utils.py
function set_df_attr (line 10) | def set_df_attr(df, attr_name, attr_value):
function append_df_attr (line 22) | def append_df_attr(df, attr_name, attr_value):
function get_df_attr (line 34) | def get_df_attr(df, attr_name, default_val):
function get_pandas_func (line 43) | def get_pandas_func(cls, func, prefix=settings.ORIGINAL_METHOD_PREFIX):
function get_signature_repr (line 56) | def get_signature_repr(cls, fn, args, full_signature=True):
function _raise_on_bad_class (line 103) | def _raise_on_bad_class(cls):
function restore_pandas_func_copy (line 109) | def restore_pandas_func_copy(
function keep_pandas_func_copy (line 125) | def keep_pandas_func_copy(cls, func, prefix=settings.ORIGINAL_METHOD_PRE...
function calc_step_number (line 139) | def calc_step_number(method_name, input_df):
FILE: pandas_log/pandas_execution_stats.py
function get_execution_stats (line 27) | def get_execution_stats(cls, fn, input_df, fn_args, fn_kwargs, calculate...
class StepStats (line 52) | class StepStats:
method __init__ (line 53) | def __init__(
method calc_df_series_memory (line 85) | def calc_df_series_memory(df_or_series):
method persist_execution_stats (line 95) | def persist_execution_stats(self):
method log_stats_if_needed (line 100) | def log_stats_if_needed(self, silent, verbose, copy_ok):
method get_logs_for_specifc_method (line 114) | def get_logs_for_specifc_method(self, verbose, copy_ok):
method _repr_html_ (line 133) | def _repr_html_(self):
method __repr__ (line 136) | def __repr__(self, verbose, copy_ok):
FILE: pandas_log/pandas_log.py
function auto_disable (line 25) | def auto_disable():
function enable (line 42) | def enable(
function auto_enable (line 65) | def auto_enable(
function create_overide_pandas_func (line 111) | def create_overide_pandas_func(
FILE: pandas_log/patched_logs_functions.py
function _stringify_list (line 69) | def _stringify_list(l):
function rows_removed (line 73) | def rows_removed(input_df, output_df):
function rows_removed_pct (line 77) | def rows_removed_pct(input_df, output_df):
function rows_remaining (line 81) | def rows_remaining(output_df):
function cols_removed (line 85) | def cols_removed(input_df, output_df):
function cols_remaining (line 91) | def cols_remaining(output_df):
function is_same_cols (line 95) | def is_same_cols(input_df, output_df):
function columns_changed (line 99) | def columns_changed(df, cols):
function columns_added (line 103) | def columns_added(df, cols):
function is_same_rows (line 107) | def is_same_rows(input_df, output_df):
function num_of_na (line 111) | def num_of_na(df):
function str_new_columns (line 115) | def str_new_columns(input_df, output_df):
function num_new_columns (line 121) | def num_new_columns(input_df, output_df):
function num_values_changed (line 125) | def num_values_changed(input_obj, output_obj):
function get_filter_rows_logs (line 149) | def get_filter_rows_logs(input_df, output_df):
function log_default (line 163) | def log_default(output_df, input_df, *args, **kwargs):
function log_no_message (line 188) | def log_no_message(output_df, input_df, *args, **kwargs):
function log_drop (line 192) | def log_drop(
function log_dropna (line 218) | def log_dropna(
function log_assign (line 241) | def log_assign(output_df, input_df, **kwargs):
function log___setitem__ (line 283) | def log___setitem__(output_df, input_df, key, value, **kwargs):
function log_query (line 295) | def log_query(output_df, input_df, expr, inplace=False, *args, **kwargs):
function log_sort_index (line 300) | def log_sort_index(
function log_sort_values (line 318) | def log_sort_values(
function log_tail (line 336) | def log_tail(output_df, input_df, n=5, **kwargs):
function log_head (line 342) | def log_head(output_df, input_df, n=5, **kwargs):
function log_merge (line 348) | def log_merge(
function log_applymap (line 400) | def log_applymap(output_df, input_df, func, **kwargs):
function log_join (line 408) | def log_join(
function log_fillna (line 440) | def log_fillna(
function log_mask (line 461) | def log_mask(
function log_where (line 496) | def log_where(
function log_sample (line 524) | def log_sample(
function log_nlargest (line 541) | def log_nlargest(output_df, input_df, n, columns, keep="first", **kwargs):
function log_nsmallest (line 548) | def log_nsmallest(output_df, input_df, n, columns, keep="first", **kwargs):
function log_groupby (line 554) | def log_groupby(
function log__iterrows (line 578) | def log__iterrows(output_df, input_df):
function log___getitem__ (line 584) | def log___getitem__(output_df, input_df, key, *args, **kwargs):
Condensed preview — 37 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (174K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 947,
"preview": "---\nname: Bug Report\nabout: Please use this issue template if you are filing a bug report.\n---\n\n# Brief Description\n\n<!-"
},
{
"path": ".github/ISSUE_TEMPLATE/documentation_fix.md",
"chars": 693,
"preview": "---\nname: Propose a Documentation Fix\nabout: Use this issue tracker template if you'd like to propose a fix to the docum"
},
{
"path": ".github/ISSUE_TEMPLATE/new_examples.md",
"chars": 554,
"preview": "---\nname: Add/Modify Notebooks\nabout: Use this specific template if you'd like to contribute a notebook to the examples "
},
{
"path": ".github/ISSUE_TEMPLATE/new_proposed_feature.md",
"chars": 232,
"preview": "---\nname: Propose New Feature\nabout: If you'd like to propose a new feature, please use this template.\n---\n\n# Brief Desc"
},
{
"path": ".github/pull_request_template.md",
"chars": 1767,
"preview": "<!-- Thank you for your PR! \n\nBEFORE YOU CONTINUE! Please add the appropriate three-letter abbreviation to your title.\n\n"
},
{
"path": ".gitignore",
"chars": 93,
"preview": ".idea\nexplore_pandas_log.*\npandas_log/__pycache__\n.ipynb_checkpoints\n.cache\n.eggs\n*.egg-info\n"
},
{
"path": "AUTHORS.rst",
"chars": 176,
"preview": "=======\nCredits\n=======\n\nDevelopment Lead\n----------------\n\n* Eyal Trabelsi <eyaltrabelsi@gmail.com>\n\nContributors\n-----"
},
{
"path": "CONTRIBUTING.rst",
"chars": 5205,
"preview": "\n============\nContributing\n============\n\nContributions are welcome, and they are greatly appreciated!\nEvery little bit h"
},
{
"path": "HISTORY.rst",
"chars": 89,
"preview": "=======\nHistory\n=======\n\n0.0.0 (2019-09-18)\n------------------\n\n* First release on PyPI.\n"
},
{
"path": "LICENSE",
"chars": 1072,
"preview": "MIT License\n\nCopyright (c) 2019, Eyal Trabelsi\n\nPermission is hereby granted, free of charge, to any person obtaining a "
},
{
"path": "MANIFEST.in",
"chars": 222,
"preview": "include CONTRIBUTING.rst\ninclude LICENSE\ninclude README.rst\n\nrecursive-include tests *\nrecursive-exclude * __pycache__\nr"
},
{
"path": "Makefile",
"chars": 1522,
"preview": "cat.PHONY: clean clean-test clean-pyc clean-build docs help\n.DEFAULT_GOAL := help\n\ndefine BROWSER_PYSCRIPT\nimport os, we"
},
{
"path": "README.rst",
"chars": 2840,
"preview": "==========\npandas-log\n==========\n\n\n.. image:: https://img.shields.io/pypi/v/pandas_log.svg\n :target: https://pypi"
},
{
"path": "TODO.rst",
"chars": 126,
"preview": "- Add statistics from tidylog from it.\n- add formats of warnings.\n- todo check if copy is first\n- add logs for series as"
},
{
"path": "docs/Makefile",
"chars": 611,
"preview": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHI"
},
{
"path": "docs/conf.py",
"chars": 4823,
"preview": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# pandas_log documentation build configuration file, created by\n# sphinx"
},
{
"path": "docs/contributing.rst",
"chars": 33,
"preview": ".. include:: ../CONTRIBUTING.rst\n"
},
{
"path": "docs/index.rst",
"chars": 285,
"preview": "Welcome to pandas-log's documentation!\n======================================\n\n.. toctree::\n :maxdepth: 2\n :caption:"
},
{
"path": "docs/installation.rst",
"chars": 1154,
"preview": ".. highlight:: shell\n\n============\nInstallation\n============\n\n\nStable release\n--------------\n\nTo install pandas-log, run"
},
{
"path": "docs/modules.rst",
"chars": 67,
"preview": "pandas_log\n==========\n\n.. toctree::\n :maxdepth: 4\n\n pandas_log\n"
},
{
"path": "docs/pandas_log.rst",
"chars": 1164,
"preview": "pandas\\_log package\n===================\n\nSubmodules\n----------\n\npandas\\_log.aop\\_utils module\n--------------------------"
},
{
"path": "docs/readme.rst",
"chars": 931,
"preview": "===============\nWhy pandas-log?\n===============\n\n``Pandas-log`` is a Python implementation of the R package ``tidylog``,"
},
{
"path": "docs/usage.rst",
"chars": 20530,
"preview": "\nUsage\n=====\n\nFor a cleaner use-case I would `go here\n<https://towardsdatascience.com/introducing-pandas-log-3240a5e57e2"
},
{
"path": "examples/README.rst",
"chars": 273,
"preview": "\nExamples\n========\n\nThis folder contains jupyter notebooks demonstrating different ways to\nimplement pandas-log in your "
},
{
"path": "examples/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/pandas_log_intro.ipynb",
"chars": 34771,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Pandas-Log Usage Walkthrough\"\n "
},
{
"path": "examples/pokemon.csv",
"chars": 44020,
"preview": "#,name,type_1,type_2,total,hp,attack,defense,sp_atk,sp_def,speed,generation,legendary\n1,Bulbasaur,Grass,Poison,318,45,49"
},
{
"path": "pandas_log/__init__.py",
"chars": 222,
"preview": "# -*- coding: utf-8 -*-\n\n\"\"\"Top-level package for pandas-log.\"\"\"\nfrom .pandas_log import *\n\n__author__ = \"\"\"Eyal Trabels"
},
{
"path": "pandas_log/aop_utils.py",
"chars": 4439,
"preview": "import itertools\nfrom inspect import signature\n\nimport pandas as pd\n\nfrom pandas_log import settings\nfrom pandas_log.set"
},
{
"path": "pandas_log/pandas_execution_stats.py",
"chars": 6191,
"preview": "import warnings\nfrom collections import namedtuple\nfrom functools import partial\nfrom time import time\n\nimport pandas as"
},
{
"path": "pandas_log/pandas_log.py",
"chars": 6821,
"preview": "# -*- coding: utf-8 -*-\n\n\"\"\"Main module.\"\"\"\n\nimport warnings\nfrom contextlib import contextmanager\nfrom functools import"
},
{
"path": "pandas_log/patched_logs_functions.py",
"chars": 17043,
"preview": "import warnings\n\nimport numpy as np\nimport pandas as pd\n\n# Values messages\nALTERED_VALUES_MSG = \"\\t* Changed {values_cha"
},
{
"path": "pandas_log/settings.py",
"chars": 1083,
"preview": "ORIGINAL_METHOD_PREFIX = \"original_\"\nPATCHED_LOG_METHOD_PREFIX = \"log_\"\nDATAFRAME_ADDITIONAL_METHODS_TO_OVERIDE = [\n "
},
{
"path": "requirements_dev.txt",
"chars": 217,
"preview": "pip>=19.2.3\nbump2version>=0.5.11\nwheel>=0.33.6\nwatchdog>=0.9.0\nflake8>=3.7.8\ntox>=3.14.0\ncoverage>=4.5.4\nSphinx>=2.2.0\nt"
},
{
"path": "setup.cfg",
"chars": 453,
"preview": "[bumpversion]\ncurrent_version = 0.1.7\ncommit = True\ntag = True\n\n[bumpversion:file:setup.py]\nsearch = version='{current_v"
},
{
"path": "setup.py",
"chars": 1245,
"preview": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\n\"\"\"The setup script.\"\"\"\n\nfrom setuptools import setup, find_packages\n\nwit"
},
{
"path": "tox.ini",
"chars": 554,
"preview": "[tox]\nenvlist = py27, py35, py36, py37 flake8\n\n[travis]\npython =\n 3.7: py37\n 3.6: py36\n 3.5: py35\n 2.7: py27"
}
]
About this extraction
This page contains the full source code of the eyaltrabelsi/pandas-log GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 37 files (158.7 KB), approximately 55.3k tokens, and a symbol index with 60 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.