[ { "path": ".gitignore", "content": "*.egg-info\ndist/\nbuild/\ndocs/_build\ndocs/source/.doctrees/\n__pycache__/\n.cache\n*.sqlite3\n.tox/\n*.swp\n*.pyc\n.coverage*\n.pytest_cache\nnode_modules\nvenv/\n\n# IDE and Tooling files\n.idea/*\n*~\n\n# macOS\n.DS_Store\n" }, { "path": ".readthedocs.yml", "content": "version: 2\nbuild:\n image: latest\nsphinx:\n configuration: docs/source/conf.py\npython:\n version: 3.6\n install:\n - requirements: docs/readthedocs.txt\n" }, { "path": ".travis.yml", "content": "dist: xenial\nlanguage: python\ncache: pip\nsudo: required\nmatrix:\n fast_finish: true\n include:\n - { python: \"3.6\", env: DJANGO=2.0 }\n - { python: \"3.6\", env: DJANGO=2.1 }\n - { python: \"3.6\", env: DJANGO=2.2 }\n - { python: \"3.7\", env: DJANGO=2.0 }\n - { python: \"3.7\", env: DJANGO=2.1 }\n - { python: \"3.7\", env: DJANGO=2.2 }\n - { python: \"3.7\", env: DJANGO=master }\n - { python: \"3.6\", env: TOXENV=lint }\n - { python: \"3.6\", env: TOXENV=docs }\n allow_failures:\n - env: TOXENV=docs\n - env: DJANGO=master\nservices:\n - docker\nbefore_install:\n - sudo /etc/init.d/postgresql stop\ninstall:\n - pip install '.[dev]' tox-travis\nbefore_script:\n - docker-compose -f dev/docker-compose.yml up -d\n - sleep 5\nscript:\n - tox\nafter_success:\n - pip install codecov\n - codecov -e TOXENV,DJANGO\nnotifications:\n email: false\n" }, { "path": "CONTRIBUTING.rst", "content": "Contributing Guidelines\n=======================\n\nIssue tracker\n-------------\n您可以通过 `issue tracker `__ 提交改进建议、缺陷报告或功能需求，但 **必须** 遵守以下规范：\n\n* **请勿** 重复提交相似的主题或内容。\n* **请勿** 讨论任何与本项目无关的内容。\n* 我们非常欢迎您提交程序缺陷报告，但在此之前，请确保您已经完整阅读过相关文档，并已经做了一些必要的调查，确定错误并非您自身造成的。在您编写程序缺陷报告时，\n 请详细描述您所出现的问题和复现步骤，并附带详细的信息，以便我们能尽快定位问题。\n\n----\n\nYou can submit improvement suggestions, bug reports, or feature requests through the `issue tracker `_,\nbut you **MUST** adhere to the following specifications:\n\n* **Do not** submit similar topics or content repeatedly.\n* **Do not** discuss any content not related to this project.\n* We welcome you to submit a bug report, but before doing so, please make sure that you have read the documentation in its entirety and\n have done some necessary investigations to determine that the error is not yours. When you write a bug report, Please describe in detail\n the problem and recurring steps that you have with detailed information so that we can locate the problem as quickly as possible.\n\nCode guidelines\n---------------\n* 本项目采用 `语义化版本 2.0.0 `_\n* 本项目使用了 `flask8` `isort` `black` 等代码静态检查工具。提交的代码 **必须** 通过 `lint` 工具检查。某些特殊情况不符合规范的部分，需要按照检查工具要求的方式具体标记出来。\n* 公开的 API **必须** 使用 Type Hint 并编写 Docstrings，其他部分 **建议** 使用并在必要的地方为代码编写注释，增强代码的可读性。\n* **必须** 限定非 Development 的外部依赖的模块版本为某一个完全兼容的系列。\n\n相关文档:\n\n| `Google Python Style Guide `_\n| `PEP 8 Style Guide for Python Code `_\n\n----\n\n* This project uses a `Semantic Version 2.0.0 `_\n* This project uses a code static check tool such as `flask8` `isort` `black`. The submitted code **MUST** be checked by the `lint` tool.\n Some special cases that do not meet the specifications need to be specifically marked in the way required by the inspection tool.\n* The public API **MUST** use Type Hint and write Docstrings, other parts **SHOULD** use it and write comments to the code where necessary\n to enhance the readability of code.\n* External dependencies published with the project **MUST** at least define a fully compatible version family.\n\nRelated documents:\n\n| `Google Python Style Guide `_\n| `PEP 8 Style Guide for Python Code `_\n" }, { "path": "CONTRIBUTORS", "content": "virusdefender \nmonouno \n" }, { "path": "LICENSE", "content": "Copyright (c) 2019 Chaitin Tech Co., Ltd.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.\nIN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,\nDAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR\nOTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE\nOR OTHER DEALINGS IN THE SOFTWARE." }, { "path": "README.rst", "content": "django-pg-partitioning\n======================\n.. image:: https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square\n :target: https://raw.githubusercontent.com/chaitin/django-pg-partitioning/master/LICENSE\n.. image:: https://img.shields.io/badge/Django-2.x-green.svg?style=flat-square&logo=django\n :target: https://www.djangoproject.com/\n.. image:: https://img.shields.io/badge/PostgreSQL-11-lightgrey.svg?style=flat-square&logo=postgresql\n :target: https://www.postgresql.org/\n.. image:: https://readthedocs.org/projects/django-pg-partitioning/badge/?version=latest&style=flat-square\n :target: https://django-pg-partitioning.readthedocs.io\n.. image:: https://img.shields.io/pypi/v/django-pg-partitioning.svg?style=flat-square\n :target: https://pypi.org/project/django-pg-partitioning/\n.. image:: https://api.travis-ci.org/chaitin/django-pg-partitioning.svg?branch=master\n :target: https://travis-ci.org/chaitin/django-pg-partitioning\n.. image:: https://api.codacy.com/project/badge/Grade/c872699c1b254e90b540b053343d1e81\n :target: https://www.codacy.com/app/xingji2163/django-pg-partitioning?utm_source=github.com&utm_medium=referral&utm_content=chaitin/django-pg-partitioning&utm_campaign=Badge_Grade\n.. image:: https://codecov.io/gh/chaitin/django-pg-partitioning/branch/master/graph/badge.svg\n :target: https://codecov.io/gh/chaitin/django-pg-partitioning\n\n⚠️\n----\n\n目前已经有了更好用的 Django 插件（比如 django-postgres-extra）使得基于 Django 开发的项目能够方便地使用 PostgreSQL 数据库的高级功能，因此本项目不再维护。你依然可以 fork 本项目并进行二次开发，祝你生活愉快 :)\n\nThere are already better Django plugins (such as django-postgres-extra) that make it easy for Django-based projects to use the advanced features of PostgreSQL databases, so this project is no longer maintained. You can still fork this project and do secondary development, have a nice life :)\n\n----\n\n一个支持 PostgreSQL 11 原生表分区的 Django 扩展，使您可以在 Django 中创建分区表并管理它们。目前它支持两种分区类型：\n\n- 时间范围分区（Time Range Partitioning）：将时序数据分开存储到不同的时间范围分区表中，支持创建连续且不重叠的时间范围分区并进行归档管理。\n- 列表分区（List Partitioning）：根据分区字段的确定值将数据分开存储到不同的分区表中。\n\nA Django extension that supports PostgreSQL 11 native table partitioning, allowing you to create partitioned tables in Django\nand manage them. Currently it supports the following two partition types:\n\n- **Time Range Partitioning**: Separate time series data into different time range partition tables,\n support the creation of continuous and non-overlapping time range partitions and archival management.\n- **List Partitioning**: Store data separately into different partition tables based on the determined values of the partition key.\n\nDocumentation\n https://django-pg-partitioning.readthedocs.io\n\n.. image:: https://raw.githubusercontent.com/chaitin/django-pg-partitioning/master/docs/source/_static/carbon.png\n :align: center\n\nTODO\n----\n- Improve the details of the function.\n- Improve documentation and testing.\n- Optimization implementation.\n\nmaybe more...\n\nContributing\n------------\nIf you want to contribute to a project and make it better, you help is very welcome!\nPlease read through `Contributing Guidelines `__.\n\nLicense\n-------\nThis project is licensed under the MIT. Please see `LICENSE `_.\n\nProject Practice\n----------------\n.. image:: https://raw.githubusercontent.com/chaitin/django-pg-timepart/master/docs/source/_static/safeline.svg?sanitize=true\n :target: https://www.chaitin.cn/en/safeline\n" }, { "path": "codecov.yml", "content": "coverage:\n precision: 2\n round: down\n range: \"80...100\"\n\n status:\n project: yes\n patch: no\n changes: no\n\ncomment: off\n" }, { "path": "dev/docker-compose.yml", "content": "version: \"3\"\nservices:\n postgres:\n image: postgres:${POSTGRES_VERSION:-11.1-alpine}\n container_name: postgres\n restart: always\n volumes:\n - ./postgres_init.sh:/docker-entrypoint-initdb.d/postgres_init.sh\n environment:\n - POSTGRES_DB=test\n - POSTGRES_USER=test\n - POSTGRES_PASSWORD=test\n ports:\n - \"127.0.0.1:5432:5432\"\n" }, { "path": "dev/postgres_init.sh", "content": "#!/usr/bin/env bash\n\nmkdir /tmp/data1 /tmp/data2\npsql -U test -c \"CREATE TABLESPACE data1 LOCATION '/tmp/data1'\"\npsql -U test -c \"CREATE TABLESPACE data2 LOCATION '/tmp/data2'\"\n" }, { "path": "docs/Makefile", "content": "# Minimal makefile for Sphinx documentation\n\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHINXBUILD = sphinx-build\nSPHINXPROJ = django-partitioning\nSOURCEDIR = source\nBUILDDIR = _build\n\n\n.PHONY: all\nall:\n\t@$(SPHINXBUILD) \"$(SOURCEDIR)\" \"$(BUILDDIR)/\"\n\n\n.PHONY: help\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n\n.PHONY: Makefile\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option. $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)" }, { "path": "docs/readthedocs.txt", "content": "alabaster==0.7.12\nsphinx==1.8.3\nsphinxcontrib-napoleon==0.7\nPygments==2.3.1\npsycopg2-binary==2.7.6.1\nDjango>=2.0,<2.2\npython-dateutil~=2.7\n" }, { "path": "docs/source/_templates/sidebarlogo.html", "content": "

\n \n \n \n
\n

\n" }, { "path": "docs/source/api.rst", "content": "API Reference\n=============\n\n.. py:currentmodule:: pg_partitioning.manager\n\nTime Range Partitioning\n-----------------------\n\n.. autoclass:: TimeRangePartitionManager\n :members:\n\n.. autoclass:: PartitionConfig\n :members: period, interval, attach_tablespace, detach_tablespace, save\n\n.. autoclass:: PartitionLog\n :members: is_attached, detach_time, save, delete\n\nList Partitioning\n-----------------\n\n.. autoclass:: ListPartitionManager\n :members:\n\n.. py:currentmodule:: pg_partitioning.shortcuts\n\nShortcuts\n---------\n\n.. automodule:: pg_partitioning.shortcuts\n :members: truncate_table, set_tablespace, drop_table\n" }, { "path": "docs/source/conf.py", "content": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\nimport os\nimport sys\n\nimport django\nfrom django.conf import settings\n\nsys.path.insert(0, os.path.abspath('../..'))\nsys.path.insert(0, os.path.abspath('./'))\n\nsettings.configure(\n INSTALLED_APPS=(\n \"pg_partitioning\",\n ),\n)\ndjango.setup()\n\nextensions = [\n 'sphinx.ext.autodoc',\n 'sphinx.ext.intersphinx',\n 'sphinx.ext.viewcode',\n 'sphinx.ext.napoleon',\n 'alabaster',\n]\n\ntemplates_path = ['_templates']\n\nsource_suffix = '.rst'\n\nmaster_doc = 'index'\n\nproject = 'django-pg-partitioning'\ncopyright = '2019, Chaitin Tech'\nauthor = 'Boyce Li'\n\nlanguage = 'en'\n\nexclude_patterns = ['_build']\n\npygments_style = 'sphinx'\n\nhtml_theme = 'alabaster'\n\nhtml_theme_options = {\n 'github_user': 'chaitin',\n 'github_repo': 'django-pg-partitioning',\n 'github_type': 'star',\n 'github_banner': 'true',\n 'show_powered_by': 'false',\n 'code_font_size': '14px',\n}\n\nhtml_static_path = ['_static']\n\nhtml_sidebars = {\n '**': [\n 'sidebarlogo.html',\n 'navigation.html',\n 'searchbox.html',\n ]\n}\n\nhtmlhelp_basename = 'django-pg-partitioning-doc'\n\nman_pages = [\n (master_doc, 'django-pg-partitioning', 'django-pg-partitioning Documentation',\n [author], 1)\n]\n\ntexinfo_documents = [\n (master_doc, 'django-pg-partitioning', 'django-pg-partitioning Documentation',\n author, 'django-pg-partitioning', 'A Django extension that supports PostgreSQL 11 time ranges and list partitioning.',\n 'Miscellaneous'),\n]\n\nintersphinx_mapping = {\n 'https://docs.python.org/3/': None,\n 'https://pika.readthedocs.io/en/0.10.0/': None,\n}\n\nadd_module_names = False\n" }, { "path": "docs/source/decorators.rst", "content": "Decorators\n==========\n\nNow the decorator must be used on a non-abstract model class that has not yet built a table in the database.\nIf you must use the decorator on a model class that has previously performed a migrate operation, you need\nto back up the model's data, then drop the table, and then import the data after you have created a\npartitioned table.\n\n.. py:currentmodule:: pg_partitioning.decorators\n\n.. autodata:: TimeRangePartitioning\n :annotation:\n\n.. autodata:: ListPartitioning\n :annotation:\n\nPost-Decoration\n---------------\n\nYou can run ``makemigrations`` and ``migrate`` commands to create and apply new migrations.\nOnce the table has been created, it is not possible to turn a regular table into a partitioned table or vice versa.\n" }, { "path": "docs/source/design.rst", "content": "Design\n======\n\nA brief description of the architecture of django-pg-partitioning.\n\nPartitioned Table Support\n-------------------------\n\nCurrently Django does not support creating partitioned tables, so django-partitioning monkey patch ``create_model`` method in\n``DatabaseSchemaEditor`` to make it generate SQL statements that create partitioned tables.\n\nSome of the operations that Django applies to regular database tables may not be supported or even conflicted on the partitioned\ntable, eventually throwing a database exception. Therefore, it is recommended that you read the section on the partition table\nin the official database documentation and refer to the relevant implementation inside Django.\n\nConstraint Limitations\n----------------------\n\nIt is important to note that PostgreSQL table partitioning has some restrictions on field constraints.\nIn order for the extension to work, we turned off Django's automatically generated primary key constraint, but did not do other legality checks.\nFor example, if you mistakenly used a unique or foreign key constraint, it will throw an exception directly, which is what you are coding and\nit needs to be manually circumvented during use.\n\nTablespace\n----------\n\n``pg_partitioning`` will silently set the tablespace of all local partitioned indexes under one partition to be consistent with\nthe partition.\n\nPartition Information\n---------------------\n\n``pg_partitioning`` saves partition configuration and state information in ``PartitionConfig`` and ``PartitionLog``.\nThe problem with this is that once this information is inconsistent with the actual situation, ``pg_partitioning``\nwill not work properly, so you can only fix it manually.\n\nManagement\n----------\n\nYou can use ``Model.partitioning.create_partition`` and ``Model.partitioning.detach_partition`` to automatically create and\narchive partitions. In addition setting ``default_detach_tablespace`` and ``default_attach_tablespace``, you can also use the\n``set_tablespace`` method of the PartitionLog object to move the partition. See :doc:`api` for details.\n" }, { "path": "docs/source/index.rst", "content": ".. include:: ../../README.rst\n\n.. toctree::\n :hidden:\n :maxdepth: 2\n\n installation\n design\n decorators\n api\n signals\n Contributing \n Project License \n" }, { "path": "docs/source/installation.rst", "content": "Installation\n============\n\nPyPI\n----\n\n.. code-block:: bash\n\n $ pip install django-pg-partitioning\n\nOr you can install from GitHub\n\n.. code-block:: bash\n\n $ pip install git+https://github.com/chaitin/django-pg-partitioning.git@master\n\nIntegrate with Django\n---------------------\n\nAdd ``pg_partitioning`` to ``INSTALLED_APPS`` in settings.py.\n\nImportant - Please note 'pg_partitioning' should be loaded earlier than other apps that depend on it::\n\n INSTALLED_APPS = [\n 'pg_partitioning',\n ...\n ]\n\n PARTITION_TIMEZONE = \"Asia/Shanghai\"\n\nYou can specify the time zone referenced by the time range partitioned table via ``PARTITION_TIMEZONE``,\nand if it is not specified, ``TIME_ZONE`` value is used.\n\nPost-Installation\n-----------------\n\nIn your Django root execute the command below to create 'pg_partitioning' database tables::\n\n ./manage.py migrate pg_partitioning\n\n" }, { "path": "docs/source/signals.rst", "content": "Signals\n=======\n\nNote that these signals are only triggered when the save methods of ``PartitionConfig`` and ``PartitionLog``\nYou can hook to them for your own needs (for example to create corresponding table index).\n\n.. py:currentmodule:: pg_partitioning.signals\n\n.. autodata:: post_create_partition(sender, partition_log)\n :annotation:\n.. autodata:: post_attach_partition(sender, partition_log)\n :annotation:\n.. autodata:: post_detach_partition(sender, partition_log)\n :annotation:\n" }, { "path": "pg_partitioning/__init__.py", "content": "\"\"\"\nA Django extension that supports PostgreSQL 11 time ranges and list partitioning.\n\"\"\"\nREQUIRED_DJANGO_VERSION = [(2, 0), (3, 0)]\nDJANGO_VERSION_ERROR = \"django-pg-partitioning isn't available on the currently installed version of Django.\"\n\ntry:\n import django\nexcept ImportError:\n raise ImportError(DJANGO_VERSION_ERROR)\n\nif REQUIRED_DJANGO_VERSION[0] > tuple(django.VERSION[:2]) or tuple(django.VERSION[:2]) > REQUIRED_DJANGO_VERSION[1]:\n raise ImportError(DJANGO_VERSION_ERROR)\n\n\n__version__ = \"0.11\"\n\ndefault_app_config = \"pg_partitioning.apps.AppConfig\"\n" }, { "path": "pg_partitioning/apps.py", "content": "from django.apps import AppConfig as DefaultAppConfig\n\n\nclass AppConfig(DefaultAppConfig):\n name = \"pg_partitioning\"\n\n def ready(self):\n from .patch import schema # noqa\n" }, { "path": "pg_partitioning/constants.py", "content": "SQL_CREATE_TIME_RANGE_PARTITION = \"\"\"\\\nCREATE TABLE IF NOT EXISTS %(child)s PARTITION OF %(parent)s FOR VALUES FROM (%(date_start)s) TO (%(date_end)s)\"\"\"\nSQL_CREATE_LIST_PARTITION = \"\"\"\\\nCREATE TABLE IF NOT EXISTS %(child)s PARTITION OF %(parent)s FOR VALUES IN (%(value)s)\"\"\"\nSQL_SET_TABLE_TABLESPACE = \"\"\"\\\nALTER TABLE IF EXISTS %(name)s SET TABLESPACE %(tablespace)s\"\"\"\nSQL_APPEND_TABLESPACE = \" TABLESPACE %(tablespace)s\"\nSQL_ATTACH_TIME_RANGE_PARTITION = \"\"\"\\\nALTER TABLE IF EXISTS %(parent)s ATTACH PARTITION %(child)s FOR VALUES FROM (%(date_start)s) TO (%(date_end)s)\"\"\"\nSQL_ATTACH_LIST_PARTITION = \"\"\"\\\nALTER TABLE IF EXISTS %(parent)s ATTACH PARTITION %(child)s FOR VALUES IN (%(value)s)\"\"\"\nSQL_DETACH_PARTITION = \"ALTER TABLE IF EXISTS %(parent)s DETACH PARTITION %(child)s\"\nSQL_DROP_TABLE = \"DROP TABLE IF EXISTS %(name)s\"\nSQL_TRUNCATE_TABLE = \"TRUNCATE TABLE %(name)s\"\nSQL_DROP_INDEX = \"DROP INDEX IF EXISTS %(name)s\"\nSQL_CREATE_INDEX = \"CREATE INDEX IF NOT EXISTS %(name)s ON %(table_name)s USING %(method)s (%(column_name)s)\"\nSQL_SET_INDEX_TABLESPACE = \"ALTER INDEX %(name)s SET TABLESPACE %(tablespace)s\"\nSQL_GET_TABLE_INDEXES = \"SELECT indexname FROM pg_indexes WHERE tablename = %(table_name)s\"\n\nDT_FORMAT = \"%Y-%m-%d\"\n\n\nclass PartitioningType:\n Range = \"RANGE\"\n List = \"LIST\"\n\n\nclass PeriodType:\n Day = \"Day\"\n Week = \"Week\"\n Month = \"Month\"\n Year = \"Year\"\n" }, { "path": "pg_partitioning/decorators.py", "content": "import logging\nfrom typing import Type\n\nfrom django.db import models\n\nfrom pg_partitioning.manager import ListPartitionManager, TimeRangePartitionManager\n\nlogger = logging.getLogger(__name__)\n\n\nclass _PartitioningBase:\n def __init__(self, partition_key: str, **options):\n self.partition_key = partition_key\n self.options = options\n\n def __call__(self, model: Type[models.Model]):\n if model._meta.abstract:\n raise NotImplementedError(\"Decorative abstract model classes are not supported.\")\n\n\nclass TimeRangePartitioning(_PartitioningBase):\n \"\"\"Use this decorator to declare the database table corresponding to the model to be partitioned by time range.\n\n Parameters:\n partition_key(str): Partition field name of DateTimeField.\n options: Currently supports the following keyword parameters:\n\n - default_period(PeriodType): Default partition period.\n - default_interval(int): Default detach partition interval.\n - default_attach_tablespace(str): Default tablespace for attached tables.\n - default_detach_tablespace(str): Default tablespace for attached tables.\n\n Example:\n .. code-block:: python\n\n from django.db import models\n from django.utils import timezone\n\n from pg_partitioning.decorators import TimeRangePartitioning\n\n\n @TimeRangePartitioning(partition_key=\"timestamp\")\n class MyLog(models.Model):\n name = models.TextField(default=\"Hello World!\")\n timestamp = models.DateTimeField(default=timezone.now, primary_key=True)\n \"\"\"\n\n def __call__(self, model: Type[models.Model]):\n super().__call__(model)\n if model._meta.get_field(self.partition_key).get_internal_type() != models.DateTimeField().get_internal_type():\n raise ValueError(\"The partition_key must be DateTimeField type.\")\n model.partitioning = TimeRangePartitionManager(model, self.partition_key, self.options)\n return model\n\n\nclass ListPartitioning(_PartitioningBase):\n \"\"\"Use this decorator to declare the database table corresponding to the model to be partitioned by list.\n\n Parameters:\n partition_key(str): Partition key name, the type of the key must be one of boolean, text or integer.\n\n Example:\n .. code-block:: python\n\n from django.db import models\n from django.utils import timezone\n\n from pg_partitioning.decorators import ListPartitioning\n\n\n @ListPartitioning(partition_key=\"category\")\n class MyLog(models.Model):\n category = models.TextField(default=\"A\")\n timestamp = models.DateTimeField(default=timezone.now, primary_key=True)\n \"\"\"\n\n def __call__(self, model: Type[models.Model]):\n super().__call__(model)\n model.partitioning = ListPartitionManager(model, self.partition_key, self.options)\n return model\n" }, { "path": "pg_partitioning/manager.py", "content": "import datetime\nfrom collections import Iterable\nfrom typing import Optional, Type, Union\n\nimport pytz\nfrom dateutil.relativedelta import MO, relativedelta\nfrom django.conf import settings\nfrom django.db import IntegrityError, models\nfrom django.db.models import Q\nfrom django.utils import timezone\n\nfrom pg_partitioning.shortcuts import double_quote, execute_sql, generate_set_indexes_tablespace_sql, single_quote\n\nfrom .constants import (\n DT_FORMAT,\n SQL_APPEND_TABLESPACE,\n SQL_ATTACH_LIST_PARTITION,\n SQL_CREATE_LIST_PARTITION,\n SQL_DETACH_PARTITION,\n SQL_SET_TABLE_TABLESPACE,\n PartitioningType,\n PeriodType,\n)\nfrom .models import PartitionConfig, PartitionLog\n\n\nclass _PartitionManagerBase:\n type = None\n\n def __init__(self, model: Type[models.Model], partition_key: str, options: dict):\n self.model = model\n self.partition_key = partition_key\n self.options = options\n\n\nclass TimeRangePartitionManager(_PartitionManagerBase):\n \"\"\"Manage time-based partition APIs.\"\"\"\n\n type = PartitioningType.Range\n\n @property\n def config(self) -> PartitionConfig:\n \"\"\"Get the latest PartitionConfig instance of this model.\n In order to avoid the race condition, we used **select_for_update** when querying.\n\n Returns:\n PartitionConfig: The latest PartitionConfig instance of this model.\n \"\"\"\n try:\n return PartitionConfig.objects.select_for_update().get(model_label=self.model._meta.label_lower)\n except PartitionConfig.DoesNotExist:\n try:\n return PartitionConfig.objects.create(\n model_label=self.model._meta.label_lower,\n period=self.options.get(\"default_period\", PeriodType.Month),\n interval=self.options.get(\"default_interval\"),\n attach_tablespace=self.options.get(\"default_attach_tablespace\"),\n detach_tablespace=self.options.get(\"default_detach_tablespace\"),\n )\n except IntegrityError:\n return PartitionConfig.objects.select_for_update().get(model_label=self.model._meta.label_lower)\n\n @property\n def latest(self) -> Optional[PartitionLog]:\n \"\"\"Get the latest PartitionLog instance of this model.\n\n Returns:\n Optional[PartitionLog]: The latest PartitionLog instance of this model or none.\n \"\"\"\n return self.config.logs.order_by(\"-id\").first()\n\n @classmethod\n def _get_period_bound(cls, date_start, initial, addition_zeros=None, is_week=False, **kwargs):\n zeros = {\"hour\": 0, \"minute\": 0, \"second\": 0, \"microsecond\": 0}\n if addition_zeros:\n zeros.update(addition_zeros)\n\n def func(): # lazy evaluation\n if initial:\n start = date_start.replace(**zeros)\n if is_week:\n start -= relativedelta(days=start.weekday())\n else:\n start = date_start\n end = start + relativedelta(**kwargs, **zeros)\n return start, end\n\n return func\n\n def create_partition(self, max_days_to_next_partition: int = 1) -> None:\n \"\"\"The partition of the next cycle is created according to the configuration.\n After modifying the period field, the new period will take effect the next time.\n The start time of the new partition is the end time of the previous partition table,\n or the start time of the current archive period when no partition exists.\n\n For example:\n the current time is June 5, 2018, and the archiving period is one year,\n then the start time of the first partition is 00:00:00 on January 1, 2018.\n\n Parameters:\n max_days_to_next_partition(int):\n If numbers of days remained in current partition is greater than ``max_days_to_next_partition``, no new partitions will be created.\n \"\"\"\n while True:\n if max_days_to_next_partition > 0 and self.latest and timezone.now() < (self.latest.end - relativedelta(days=max_days_to_next_partition)):\n return\n\n partition_timezone = getattr(settings, \"PARTITION_TIMEZONE\", None)\n if partition_timezone:\n partition_timezone = pytz.timezone(partition_timezone)\n date_start = timezone.localtime(self.latest.end if self.latest else None, timezone=partition_timezone)\n initial = not bool(self.latest)\n date_start, date_end = {\n PeriodType.Day: self._get_period_bound(date_start, initial, days=+1),\n PeriodType.Week: self._get_period_bound(date_start, initial, is_week=True, days=+1, weekday=MO),\n PeriodType.Month: self._get_period_bound(date_start, initial, addition_zeros=dict(day=1), months=+1),\n PeriodType.Year: self._get_period_bound(date_start, initial, addition_zeros=dict(month=1, day=1), years=+1),\n }[self.config.period]()\n\n partition_table_name = \"_\".join((self.model._meta.db_table, date_start.strftime(DT_FORMAT), date_end.strftime(DT_FORMAT)))\n PartitionLog.objects.create(config=self.config, table_name=partition_table_name, start=date_start, end=date_end)\n\n if not max_days_to_next_partition > 0:\n return\n\n def attach_partition(self, partition_log: Optional[Iterable] = None, detach_time: Optional[datetime.datetime] = None) -> None:\n \"\"\"Attach partitions.\n\n Parameters:\n partition_log(Optional[Iterable]):\n All partitions are attached when you don't specify partitions to attach.\n detach_time(Optional[datetime.datetime]):\n When the partition specifies the archive time, it will **not** be automatically archived until that time.\n \"\"\"\n if not partition_log:\n partition_log = PartitionLog.objects.filter(config=self.config, is_attached=False)\n\n for log in partition_log:\n log.is_attached = True\n log.detach_time = detach_time\n log.save()\n\n def detach_partition(self, partition_log: Optional[Iterable] = None) -> None:\n \"\"\"Detach partitions.\n\n Parameters:\n partition_log(Optional[Iterable]):\n Specify a partition to archive. When you don't specify a partition to archive, all partitions that meet the configuration rule are archived.\n \"\"\"\n if not partition_log:\n if self.config.interval:\n # fmt: off\n period = {PeriodType.Day: {\"days\": 1},\n PeriodType.Week: {\"weeks\": 1},\n PeriodType.Month: {\"months\": 1},\n PeriodType.Year: {\"years\": 1}}[self.config.period]\n # fmt: on\n now = timezone.now()\n detach_timeline = now - self.config.interval * relativedelta(**period)\n partition_log = PartitionLog.objects.filter(config=self.config, end__lt=detach_timeline, is_attached=True)\n partition_log = partition_log.filter(Q(detach_time=None) | Q(detach_time__lt=now))\n else:\n return\n\n for log in partition_log:\n log.is_attached = False\n log.detach_time = None\n log.save()\n\n def delete_partition(self, partition_log: Iterable) -> None:\n \"\"\"Delete partitions.\n\n Parameters:\n partition_log(Iterable): The partitions to be deleted.\n \"\"\"\n for log in partition_log:\n if log.config == self.config:\n log.delete()\n\n\ndef _db_value(value: Union[str, int, bool, None]) -> str:\n if value is None:\n return \"null\"\n return single_quote(value) if isinstance(value, str) else str(value)\n\n\nclass ListPartitionManager(_PartitionManagerBase):\n \"\"\"Manage list-based partition APIs.\"\"\"\n\n type = PartitioningType.List\n\n def create_partition(self, partition_name: str, value: Union[str, int, bool, None], tablespace: str = None) -> None:\n \"\"\"Create partitions.\n\n Parameters:\n partition_name(str): Partition name.\n value(Union[str, int, bool, None]): Partition key value.\n tablespace(str): Partition tablespace name.\n \"\"\"\n\n create_partition_sql = SQL_CREATE_LIST_PARTITION % {\n \"parent\": double_quote(self.model._meta.db_table),\n \"child\": double_quote(partition_name),\n \"value\": _db_value(value),\n }\n if tablespace:\n create_partition_sql += SQL_APPEND_TABLESPACE % {\"tablespace\": tablespace}\n execute_sql(create_partition_sql)\n execute_sql(generate_set_indexes_tablespace_sql(partition_name, tablespace))\n\n def attach_partition(self, partition_name: str, value: Union[str, int, bool, None], tablespace: str = None) -> None:\n \"\"\"Attach partitions.\n\n Parameters:\n partition_name(str): Partition name.\n value(Union[str, int, bool, None]): Partition key value.\n tablespace(str): Partition tablespace name.\n \"\"\"\n\n sql_sequence = list()\n if tablespace:\n sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {\"name\": double_quote(partition_name), \"tablespace\": tablespace})\n sql_sequence.extend(generate_set_indexes_tablespace_sql(partition_name, tablespace))\n sql_sequence.append(\n SQL_ATTACH_LIST_PARTITION % {\"parent\": double_quote(self.model._meta.db_table), \"child\": double_quote(partition_name), \"value\": _db_value(value)}\n )\n execute_sql(sql_sequence)\n\n def detach_partition(self, partition_name: str, tablespace: str = None) -> None:\n \"\"\"Detach partitions.\n\n Parameters:\n partition_name(str): Partition name.\n tablespace(str): Partition tablespace name.\n \"\"\"\n sql_sequence = [SQL_DETACH_PARTITION % {\"parent\": double_quote(self.model._meta.db_table), \"child\": double_quote(partition_name)}]\n if tablespace:\n sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {\"name\": double_quote(partition_name), \"tablespace\": tablespace})\n sql_sequence.extend(generate_set_indexes_tablespace_sql(partition_name, tablespace))\n execute_sql(sql_sequence)\n" }, { "path": "pg_partitioning/migrations/0001_initial.py", "content": "# Generated by Django 2.1.7 on 2019-02-17 12:00\n\nimport django.db.models.deletion\nfrom django.db import migrations, models\n\n\nclass Migration(migrations.Migration):\n\n initial = True\n\n dependencies = [\n ]\n\n operations = [\n migrations.CreateModel(\n name='PartitionConfig',\n fields=[\n ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),\n ('model_label', models.TextField(unique=True)),\n ('period', models.TextField(default='Month')),\n ('interval', models.PositiveIntegerField(null=True)),\n ('attach_tablespace', models.TextField(null=True)),\n ('detach_tablespace', models.TextField(null=True)),\n ],\n ),\n migrations.CreateModel(\n name='PartitionLog',\n fields=[\n ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),\n ('table_name', models.TextField(unique=True)),\n ('start', models.DateTimeField()),\n ('end', models.DateTimeField()),\n ('is_attached', models.BooleanField(default=True)),\n ('detach_time', models.DateTimeField(null=True)),\n ('config', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='logs', to='pg_partitioning.PartitionConfig')),\n ],\n options={\n 'ordering': ('-id',),\n },\n ),\n ]\n" }, { "path": "pg_partitioning/migrations/__init__.py", "content": "" }, { "path": "pg_partitioning/models.py", "content": "from django.apps import apps\nfrom django.db import models, transaction\n\nfrom pg_partitioning.signals import post_attach_partition, post_create_partition, post_detach_partition\n\nfrom .constants import (\n SQL_APPEND_TABLESPACE,\n SQL_ATTACH_TIME_RANGE_PARTITION,\n SQL_CREATE_TIME_RANGE_PARTITION,\n SQL_DETACH_PARTITION,\n SQL_SET_TABLE_TABLESPACE,\n PeriodType,\n)\nfrom .shortcuts import double_quote, drop_table, execute_sql, generate_set_indexes_tablespace_sql, single_quote\n\n\nclass PartitionConfig(models.Model):\n \"\"\"You can get the configuration object of the partition table through ``Model.partitioning.config``,\n You can only edit the following fields via the object's ``save`` method:\n \"\"\"\n\n model_label = models.TextField(unique=True)\n period = models.TextField(default=PeriodType.Month)\n \"\"\"Partition period. you can only set options in the `PeriodType`.\n The default value is ``PeriodType.Month``. Changing this value will trigger the ``detach_partition`` method.\"\"\"\n interval = models.PositiveIntegerField(null=True)\n \"\"\"Detaching period. The ``detach_partition`` method defaults to detach partitions before the interval * period.\n The default is None, ie no partition will be detached. Changing this value will trigger the ``detach_partition`` method.\"\"\"\n attach_tablespace = models.TextField(null=True)\n \"\"\"The name of the tablespace specified when creating or attaching a partition. Modifying this field will only affect subsequent operations.\n A table migration may occur at this time.\"\"\"\n detach_tablespace = models.TextField(null=True)\n \"\"\"The name of the tablespace specified when detaching a partition. Modifying this field will only affect subsequent operations.\n A table migration may occur at this time.\"\"\"\n\n def save(self, force_insert=False, force_update=False, using=None, update_fields=None):\n \"\"\"This setting will take effect immediately when you modify the value of\n ``interval`` in the configuration.\n \"\"\"\n\n adding = self._state.adding\n model = apps.get_model(self.model_label)\n\n if not adding:\n prev = self.__class__.objects.get(pk=self.pk)\n\n with transaction.atomic():\n super().save(force_insert, force_update, using, update_fields)\n if adding:\n # Creating first partition.\n model.partitioning.create_partition(0)\n\n if not adding:\n # Period or interval changed.\n if prev.period != self.period or (prev.interval != self.interval):\n model.partitioning.detach_partition()\n\n\nclass PartitionLog(models.Model):\n \"\"\"You can only edit the following fields via the object's ``save`` method:\"\"\"\n\n config = models.ForeignKey(PartitionConfig, on_delete=models.CASCADE, related_name=\"logs\")\n table_name = models.TextField(unique=True)\n # range bound: [start, end)\n start = models.DateTimeField()\n end = models.DateTimeField()\n is_attached = models.BooleanField(default=True)\n \"\"\"Whether the partition is a attached partition. changing the value will trigger an attaching or detaching operation.\"\"\"\n detach_time = models.DateTimeField(null=True)\n \"\"\"When the value is not `None`, the partition will not be automatically detached before this time. The default is `None`.\"\"\"\n\n def save(self, force_insert=False, force_update=False, using=None, update_fields=None):\n \"\"\"This setting will take effect immediately when you modify the value of\n ``is_attached`` in the configuration.\n \"\"\"\n\n model = apps.get_model(self.config.model_label)\n if self._state.adding:\n create_partition_sql = SQL_CREATE_TIME_RANGE_PARTITION % {\n \"parent\": double_quote(model._meta.db_table),\n \"child\": double_quote(self.table_name),\n \"date_start\": single_quote(self.start.isoformat()),\n \"date_end\": single_quote(self.end.isoformat()),\n }\n\n if self.config.attach_tablespace:\n create_partition_sql += SQL_APPEND_TABLESPACE % {\"tablespace\": self.config.attach_tablespace}\n\n with transaction.atomic():\n super().save(force_insert, force_update, using, update_fields)\n execute_sql(create_partition_sql)\n execute_sql(generate_set_indexes_tablespace_sql(self.table_name, self.config.attach_tablespace))\n post_create_partition.send(sender=model, partition_log=self)\n else:\n with transaction.atomic():\n prev = self.__class__.objects.select_for_update().get(pk=self.pk)\n # Detach partition.\n if prev.is_attached and (not self.is_attached):\n sql_sequence = [SQL_DETACH_PARTITION % {\"parent\": double_quote(model._meta.db_table), \"child\": double_quote(self.table_name)}]\n if self.config.detach_tablespace:\n sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {\"name\": double_quote(self.table_name), \"tablespace\": self.config.detach_tablespace})\n sql_sequence.extend(generate_set_indexes_tablespace_sql(self.table_name, self.config.detach_tablespace))\n\n super().save(force_insert, force_update, using, update_fields)\n execute_sql(sql_sequence)\n post_detach_partition.send(sender=model, partition_log=self)\n # Attach partition.\n elif (not prev.is_attached) and self.is_attached:\n sql_sequence = list()\n if self.config.attach_tablespace:\n sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {\"name\": double_quote(self.table_name), \"tablespace\": self.config.attach_tablespace})\n sql_sequence.extend(generate_set_indexes_tablespace_sql(self.table_name, self.config.attach_tablespace))\n\n sql_sequence.append(\n SQL_ATTACH_TIME_RANGE_PARTITION\n % {\n \"parent\": double_quote(model._meta.db_table),\n \"child\": double_quote(self.table_name),\n \"date_start\": single_quote(self.start.isoformat()),\n \"date_end\": single_quote(self.end.isoformat()),\n }\n )\n\n super().save(force_insert, force_update, using, update_fields)\n execute_sql(sql_sequence)\n post_attach_partition.send(sender=model, partition_log=self)\n # State has not changed.\n else:\n super().save(force_insert, force_update, using, update_fields)\n\n @transaction.atomic\n def delete(self, using=None, keep_parents=False):\n \"\"\"When the instance is deleted, the partition corresponding to it will also be deleted.\"\"\"\n\n drop_table(self.table_name)\n super().delete(using, keep_parents)\n\n class Meta:\n ordering = (\"-id\",)\n" }, { "path": "pg_partitioning/patch/__init__.py", "content": "" }, { "path": "pg_partitioning/patch/schema.py", "content": "import logging\nfrom importlib import import_module\n\nfrom django.apps.config import MODELS_MODULE_NAME\nfrom django.db.backends.postgresql.schema import DatabaseSchemaEditor\n\nfrom pg_partitioning.manager import _PartitionManagerBase\n\nlogger = logging.getLogger(__name__)\n\n\ndefault_create_model_method = DatabaseSchemaEditor.create_model\ndefault_sql_create_table = DatabaseSchemaEditor.sql_create_table\n\n\ndef create_model(self, model):\n meta = model._meta\n try:\n cls_module = import_module(f\"{meta.app_label}.{MODELS_MODULE_NAME}\")\n except ModuleNotFoundError:\n cls_module = None\n cls = getattr(cls_module, meta.object_name, None)\n partitioning = getattr(cls, \"partitioning\", None)\n if isinstance(partitioning, _PartitionManagerBase):\n # XXX: Monkeypatch create_model.\n logger.debug(\"Partitioned model detected: %s\", meta.label)\n _type = partitioning.type\n key = partitioning.partition_key\n DatabaseSchemaEditor.sql_create_table = f\"CREATE TABLE %(table)s (%(definition)s) PARTITION BY {_type} ({key})\"\n if meta.pk.name != key:\n \"\"\"The partition key must be part of the primary key,\n and currently Django does not support setting a composite primary key,\n so its properties are turned off.\"\"\"\n meta.pk.primary_key = False\n logger.info(\"Note that PK constraints for %s has been temporarily closed.\", meta.label)\n else:\n DatabaseSchemaEditor.sql_create_table = default_sql_create_table\n default_create_model_method(self, model)\n meta.pk.primary_key = True\n\n\nDatabaseSchemaEditor.create_model = create_model\n" }, { "path": "pg_partitioning/shortcuts.py", "content": "import logging\nfrom typing import List, Optional, Tuple, Union\n\nfrom django.db import connection\n\nfrom pg_partitioning.constants import SQL_DROP_TABLE, SQL_GET_TABLE_INDEXES, SQL_SET_INDEX_TABLESPACE, SQL_SET_TABLE_TABLESPACE, SQL_TRUNCATE_TABLE\n\nlogger = logging.getLogger(__name__)\n\n\ndef single_quote(name: str) -> str:\n \"\"\"Represent a string constants in SQL.\"\"\"\n\n if name.startswith(\"'\") and name.endswith(\"'\"):\n return name\n return \"'%s'\" % name\n\n\ndef double_quote(name: str) -> str:\n \"\"\"Represent a identify in SQL.\"\"\"\n\n if name.startswith('\"') and name.endswith('\"'):\n return name\n return '\"%s\"' % name\n\n\ndef execute_sql(sql_sequence: Union[str, List[str], Tuple[str]], fetch: bool = False) -> Optional[List]:\n \"\"\"Execute SQL sequence and returning result.\"\"\"\n if not sql_sequence:\n if fetch:\n return []\n return\n\n sql_str = \"\"\n for statement in sql_sequence if isinstance(sql_sequence, (list, tuple)) else [sql_sequence]:\n sql_str += \";\\n\" + statement if sql_str else statement\n logger.debug(\"The sequence of SQL statements to be executed:\\n %s\", sql_str)\n with connection.cursor() as cursor:\n cursor.execute(sql_str)\n if fetch:\n return cursor.fetchall()\n\n\ndef generate_set_indexes_tablespace_sql(table_name: str, tablespace: str) -> List[str]:\n \"\"\"Generate set indexes tablespace SQL sequence.\n\n Parameters:\n table_name(str): Table name.\n tablespace(str): Partition tablespace.\n \"\"\"\n\n sql_sequence = []\n result = execute_sql(SQL_GET_TABLE_INDEXES % {\"table_name\": single_quote(table_name)}, fetch=True)\n for item in result:\n sql_sequence.append(SQL_SET_INDEX_TABLESPACE % {\"name\": double_quote(item[0]), \"tablespace\": tablespace})\n return sql_sequence\n\n\ndef set_tablespace(table_name: str, tablespace: str) -> None:\n \"\"\"Set the tablespace for a table and indexes.\n\n Parameters:\n table_name(str): Table name.\n tablespace(str): Tablespace name.\n \"\"\"\n\n sql_sequence = [SQL_SET_TABLE_TABLESPACE % {\"name\": double_quote(table_name), \"tablespace\": tablespace}]\n sql_sequence.extend(generate_set_indexes_tablespace_sql(table_name, tablespace))\n execute_sql(sql_sequence)\n\n\ndef truncate_table(table_name: str) -> None:\n \"\"\"Truncate table.\n\n Parameters:\n table_name(str): Table name.\n \"\"\"\n\n execute_sql(SQL_TRUNCATE_TABLE % {\"name\": double_quote(table_name)})\n\n\ndef drop_table(table_name: str) -> None:\n \"\"\"Drop table.\n\n Parameters:\n table_name(str): Table name.\n \"\"\"\n\n execute_sql(SQL_DROP_TABLE % {\"name\": double_quote(table_name)})\n" }, { "path": "pg_partitioning/signals.py", "content": "from django.dispatch import Signal\n\npost_create_partition = Signal(providing_args=[\"partition_log\"])\n\"\"\"Sent when a partition is created.\n\"\"\"\npost_attach_partition = Signal(providing_args=[\"partition_log\"])\n\"\"\"Sent when a partition is attached.\n\"\"\"\npost_detach_partition = Signal(providing_args=[\"partition_log\"])\n\"\"\"Sent when a partition is detached.\n\"\"\"\n" }, { "path": "run_test.py", "content": "import argparse\nimport sys\n\nimport dj_database_url\nimport django\nfrom django.core.management import call_command\n\n\ndef setup_django_environment():\n from django.conf import settings\n\n settings.configure(\n DEBUG_PROPAGATE_EXCEPTIONS=True,\n DATABASES={\n \"default\": dj_database_url.config(env=\"DATABASE_URL\",\n default=\"postgres://test:test@localhost/test\",\n conn_max_age=20)\n },\n SECRET_KEY=\"not very secret in tests\",\n USE_I18N=True,\n USE_L10N=True,\n USE_TZ=True,\n TIME_ZONE=\"Asia/Shanghai\",\n INSTALLED_APPS=(\n \"pg_partitioning\",\n \"tests\",\n ),\n LOGGING={\n \"version\": 1,\n \"disable_existing_loggers\": False,\n \"formatters\": {\n \"standard\": {\n \"format\": \"[%(asctime)s] %(message)s\",\n \"datefmt\": \"%Y-%m-%d %H:%M:%S\"\n }\n },\n \"handlers\": {\n \"console\": {\n \"level\": \"DEBUG\",\n \"class\": \"logging.StreamHandler\",\n \"formatter\": \"standard\"\n }\n },\n \"loggers\": {\n \"pg_partitioning.shortcuts\": {\n \"handlers\": [\"console\"],\n \"level\": \"DEBUG\",\n \"propagate\": False,\n },\n \"pg_partitioning.patch.schema\": {\n \"handlers\": [\"console\"],\n \"level\": \"DEBUG\",\n \"propagate\": False,\n },\n },\n }\n )\n\n django.setup()\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Run the pg_partitioning test suite.\")\n parser.add_argument(\"-c\", \"--coverage\", dest=\"use_coverage\", action=\"store_true\", help=\"Run coverage to collect code coverage and generate report.\")\n options = parser.parse_args()\n\n if options.use_coverage:\n try:\n from coverage import coverage\n except ImportError:\n options.use_coverage = False\n\n if options.use_coverage:\n cov = coverage()\n cov.start()\n\n setup_django_environment()\n call_command(\"test\", verbosity=2, interactive=False, stdout=sys.stdout)\n\n if options.use_coverage:\n print(\"\\nRunning Code Coverage...\\n\")\n cov.stop()\n cov.report()\n cov.xml_report()\n" }, { "path": "setup.cfg", "content": "[flake8]\nmax-complexity = 20\nmax-line-length = 157\n\ninline-quotes = \"\nmultiline-quotes = \"\"\"\n\n[isort]\nnot_skip = __init__.py\nline_length = 157\ninclude_trailing_comma = true\ncombine_as_imports = true\nmulti_line_output = 3\norder_by_type = true\n\n[bdist_wheel]\nuniversal=1\n\n[coverage:run]\ninclude = pg_partitioning/*,test/*\nomit =\n pg_partitioning/migrations/*\nbranch = True\n\n[coverage:report]\nexclude_lines =\n pragma: no cover\n raise NotImplementedError\nignore_errors = True\n" }, { "path": "setup.py", "content": "import os\n\nfrom setuptools import setup\n\n\ndef rel(*xs):\n return os.path.join(os.path.abspath(os.path.dirname(__file__)), *xs)\n\n\nwith open(rel(\"README.rst\")) as f:\n long_description = f.read()\n\n\nwith open(rel(\"pg_partitioning\", \"__init__.py\"), \"r\") as f:\n version_marker = \"__version__ = \"\n for line in f:\n if line.startswith(version_marker):\n _, version = line.split(version_marker)\n version = version.strip().strip('\"')\n break\n else:\n raise RuntimeError(\"Version marker not found.\")\n\n\ndependencies = [\n \"python-dateutil~=2.7\",\n]\n\nextra_dependencies = {\n \"django\": [\n \"Django>=2.0,<3.0\"\n ],\n}\n\nextra_dependencies[\"all\"] = list(set(sum(extra_dependencies.values(), [])))\nextra_dependencies[\"dev\"] = extra_dependencies[\"all\"] + [\n # Pinned due to https://bitbucket.org/ned/coveragepy/issues/578/incomplete-file-path-in-xml-report\n \"coverage>=4.0,<4.4\",\n\n # Docs\n \"alabaster==0.7.12\",\n \"sphinx==1.8.3\",\n \"sphinxcontrib-napoleon==0.7\",\n\n # Linting\n \"flake8~=3.6.0\",\n \"isort~=4.3.4\",\n \"black~=18.9b0\",\n \"flake8-bugbear~=18.8.0\",\n \"flake8-quotes~=1.0.0\",\n\n # Misc\n \"dj-database-url==0.5.0\",\n \"psycopg2-binary==2.7.6.1\",\n \"twine==1.12.1\",\n\n # Testing\n \"tox==3.9.0\",\n \"tox-venv==0.4.0\"\n]\n\n\nsetup(\n name=\"django-pg-partitioning\",\n version=version,\n author=\"Boyce Li\",\n author_email=\"monobiao@gmail.com\",\n description=\"A Django extension that supports PostgreSQL 11 time ranges and list partitioning.\",\n long_description=long_description,\n long_description_content_type=\"text/x-rst\",\n url=\"https://github.com/chaitin/django-pg-partitioning\",\n packages=[\"pg_partitioning\", \"pg_partitioning.migrations\", \"pg_partitioning.patch\"],\n include_package_data=True,\n install_requires=dependencies,\n extras_require=extra_dependencies,\n python_requires=\">=3.6\",\n classifiers=[\n \"Development Status :: 4 - Beta\",\n \"Programming Language :: Python :: 3.6\",\n \"Programming Language :: Python :: 3.7\",\n \"Framework :: Django :: 2.0\",\n \"Framework :: Django :: 2.1\",\n \"License :: OSI Approved :: MIT License\",\n \"Operating System :: OS Independent\",\n \"Topic :: Software Development :: Libraries :: Python Modules\"\n ],\n)\n" }, { "path": "tests/__init__.py", "content": "" }, { "path": "tests/models.py", "content": "from django.contrib.postgres.indexes import BrinIndex\nfrom django.db import models\nfrom django.utils import timezone\n\nfrom pg_partitioning.constants import PeriodType\nfrom pg_partitioning.decorators import ListPartitioning, TimeRangePartitioning\n\n\n@TimeRangePartitioning(partition_key=\"timestamp\", default_period=PeriodType.Month, default_attach_tablespace=\"data1\", default_detach_tablespace=\"data2\")\nclass TimeRangeTableA(models.Model):\n text = models.TextField()\n timestamp = models.DateTimeField(default=timezone.now)\n\n class Meta:\n db_tablespace = \"pg_default\"\n indexes = [BrinIndex(fields=[\"timestamp\"])]\n unique_together = (\"text\", \"timestamp\")\n ordering = [\"text\"]\n\n\n@TimeRangePartitioning(partition_key=\"timestamp\", default_period=PeriodType.Day, default_attach_tablespace=\"data2\", default_detach_tablespace=\"data1\")\nclass TimeRangeTableB(models.Model):\n text = models.TextField()\n timestamp = models.DateTimeField(default=timezone.now)\n\n class Meta:\n indexes = [BrinIndex(fields=[\"timestamp\"])]\n unique_together = (\"text\", \"timestamp\")\n ordering = [\"text\"]\n\n\n@ListPartitioning(partition_key=\"category\")\nclass ListTableText(models.Model):\n category = models.TextField(default=\"A\", null=True, blank=True)\n timestamp = models.DateTimeField(default=timezone.now)\n\n\n@ListPartitioning(partition_key=\"category\")\nclass ListTableInt(models.Model):\n category = models.IntegerField(default=0, null=True)\n timestamp = models.DateTimeField(default=timezone.now)\n\n\n@ListPartitioning(partition_key=\"category\")\nclass ListTableBool(models.Model):\n category = models.NullBooleanField(default=False, null=True)\n timestamp = models.DateTimeField(default=timezone.now)\n" }, { "path": "tests/tests.py", "content": "import datetime\nfrom unittest.mock import patch\n\nfrom dateutil.relativedelta import MO, relativedelta\nfrom django.db import connection\nfrom django.test import TestCase\nfrom django.utils import timezone\nfrom django.utils.crypto import get_random_string\n\nfrom pg_partitioning.constants import SQL_GET_TABLE_INDEXES, PeriodType\nfrom pg_partitioning.models import PartitionConfig, PartitionLog\nfrom pg_partitioning.shortcuts import single_quote\n\nfrom .models import ListTableBool, ListTableInt, ListTableText, TimeRangeTableA, TimeRangeTableB\n\n\ndef t(year=2018, month=8, day=25, hour=7, minute=15, second=15, millisecond=0):\n \"\"\"A point in time.\"\"\"\n return timezone.get_current_timezone().localize(datetime.datetime(year, month, day, hour, minute, second, millisecond))\n\n\ndef tz(time):\n return timezone.localtime(time)\n\n\nclass GeneralTestCase(TestCase):\n def assertTablespace(self, table_name, tablespace):\n with connection.cursor() as cursor:\n cursor.execute(f\"SELECT tablespace FROM pg_tables WHERE tablename = {single_quote(table_name)};\")\n rows = cursor.fetchall()\n self.assertEqual(tablespace, rows[0][0])\n cursor.execute(SQL_GET_TABLE_INDEXES % {\"table_name\": single_quote(table_name)})\n rows = cursor.fetchall()\n for row in rows:\n cursor.execute(f\"SELECT tablespace FROM pg_indexes WHERE indexname = {single_quote(row[0])};\")\n rows = cursor.fetchall()\n self.assertEqual(tablespace, rows[0][0])\n\n\nclass TimeRangePartitioningTestCase(GeneralTestCase):\n def assertTimeRangeEqual(self, model, time_start, time_end):\n self.assertListEqual([time_start, time_end], [tz(model.partitioning.latest.start), tz(model.partitioning.latest.end)])\n\n # Verify that the partition has been created by inserting data.\n model.objects.create(text=get_random_string(length=32), timestamp=time_start)\n model.objects.create(text=get_random_string(length=32), timestamp=time_end - relativedelta(microseconds=1))\n\n def _create_partition(self, period, start_date, delta):\n TimeRangeTableB.partitioning.options[\"default_period\"] = period\n\n for i in range(0, 3):\n if i == 0:\n TimeRangeTableB.partitioning.config # Create first partition by side effect.\n else:\n TimeRangeTableB.partitioning.create_partition(0)\n end_date = start_date + delta\n self.assertTimeRangeEqual(TimeRangeTableB, start_date, end_date)\n start_date = end_date\n\n @patch(\"django.utils.timezone.now\", new=t)\n def test_create_partition_week(self):\n self._create_partition(PeriodType.Week, t(2018, 8, 20, 0, 0, 0), relativedelta(days=1, weekday=MO))\n\n @patch(\"django.utils.timezone.now\", new=t)\n def test_create_partition_day(self):\n self._create_partition(PeriodType.Day, t(2018, 8, 25, 0, 0, 0), relativedelta(days=1))\n\n @patch(\"django.utils.timezone.now\", new=t)\n def test_create_partition_month(self):\n self._create_partition(PeriodType.Month, t(2018, 8, 1, 0, 0, 0), relativedelta(months=1))\n\n @patch(\"django.utils.timezone.now\", new=t)\n def test_create_partition_year(self):\n self._create_partition(PeriodType.Year, t(2018, 1, 1, 0, 0, 0), relativedelta(years=1))\n\n @classmethod\n def _update_config_period(cls, config: PartitionConfig, period: str):\n config.period = period\n config.save()\n\n def test_create_partition(self):\n with patch(\"django.utils.timezone.now\", new=t):\n config_a: PartitionConfig = TimeRangeTableA.partitioning.config\n\n self.assertEqual(1, config_a.logs.count())\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 1, 0, 0, 0), t(2018, 9, 1, 0, 0, 0))\n\n # Repeated calls will not produce wrong results (idempotence).\n for _ in range(3):\n TimeRangeTableA.partitioning.create_partition()\n\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 1, 0, 0, 0), t(2018, 9, 1, 0, 0, 0))\n\n # Perform a series of partition creation operations.\n self._update_config_period(config_a, PeriodType.Week)\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 9, 3, 0, 0, 0))\n\n self._update_config_period(config_a, PeriodType.Day)\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 3, 0, 0, 0), t(2018, 9, 4, 0, 0, 0))\n\n self._update_config_period(config_a, PeriodType.Month)\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 4, 0, 0, 0), t(2018, 10, 1, 0, 0, 0))\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 10, 1, 0, 0, 0), t(2018, 11, 1, 0, 0, 0))\n\n self._update_config_period(config_a, PeriodType.Year)\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 11, 1, 0, 0, 0), t(2019, 1, 1, 0, 0, 0))\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2019, 1, 1, 0, 0, 0), t(2020, 1, 1, 0, 0, 0))\n\n def test_max_days_to_next_partition(self):\n with patch(\"django.utils.timezone.now\", new=t):\n TimeRangeTableA.partitioning.create_partition()\n TimeRangeTableA.partitioning.create_partition(0)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 10, 1, 0, 0, 0))\n\n with patch(\"django.utils.timezone.now\", return_value=t(2018, 8, 2, 0, 0, 0)):\n TimeRangeTableA.partitioning.create_partition(59)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 10, 1, 0, 0, 0))\n TimeRangeTableA.partitioning.create_partition(60)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 10, 1, 0, 0, 0), t(2018, 11, 1, 0, 0, 0))\n\n with patch(\"django.utils.timezone.now\", return_value=t(2019, 3, 3, 0, 0, 0)):\n TimeRangeTableA.partitioning.create_partition(5)\n self.assertTimeRangeEqual(TimeRangeTableA, t(2019, 3, 1, 0, 0, 0), t(2019, 4, 1, 0, 0, 0))\n\n def test_attach_or_detach_partition(self):\n self.test_create_partition()\n\n config_a: PartitionConfig = TimeRangeTableA.partitioning.config\n\n self.assertEqual(0, config_a.logs.filter(is_attached=False).count())\n TimeRangeTableA.partitioning.detach_partition(config_a.logs.all())\n self.assertEqual(0, config_a.logs.filter(is_attached=True).count())\n TimeRangeTableA.partitioning.attach_partition(config_a.logs.all())\n self.assertEqual(0, config_a.logs.filter(is_attached=False).count())\n\n with patch(\"django.utils.timezone.now\", return_value=t(2018, 10, 15, 12, 1, 4)):\n config_a.period = PeriodType.Day\n config_a.interval = 15\n config_a.save()\n self.assertEqual(t(2018, 10, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by(\"start\").first().end))\n TimeRangeTableA.partitioning.attach_partition(config_a.logs.all())\n\n config_a.period = PeriodType.Week\n config_a.interval = 2\n config_a.save()\n self.assertEqual(t(2018, 11, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by(\"start\").first().end))\n TimeRangeTableA.partitioning.attach_partition(config_a.logs.all())\n\n log = config_a.logs.filter(is_attached=True).order_by(\"start\").first()\n self.assertEqual(t(2018, 9, 1, 0, 0, 0), log.end)\n\n log.detach_time = t(2018, 10, 15, 12, 1, 5)\n log.save()\n\n config_a.period = PeriodType.Month\n config_a.interval = 1\n config_a.save()\n\n self.assertEqual(t(2018, 9, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by(\"start\").first().end))\n log.refresh_from_db()\n self.assertEqual(True, log.is_attached)\n\n log.detach_time = None\n log.save()\n\n TimeRangeTableA.partitioning.detach_partition()\n self.assertEqual(t(2018, 10, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by(\"start\").first().end))\n log.refresh_from_db()\n self.assertEqual(False, log.is_attached)\n\n def test_delete_partition(self):\n for _ in range(4):\n TimeRangeTableA.partitioning.create_partition(0)\n\n config_a: PartitionConfig = TimeRangeTableA.partitioning.config\n\n TimeRangeTableA.partitioning.delete_partition(config_a.logs.all())\n self.assertEqual(0, config_a.logs.count())\n\n with patch(\"django.utils.timezone.now\", new=t):\n self._update_config_period(config_a, PeriodType.Day)\n TimeRangeTableA.partitioning.create_partition()\n\n self.assertEqual(2, config_a.logs.count())\n self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 26, 0, 0, 0), t(2018, 8, 27, 0, 0, 0))\n\n def test_attach_detach_tablespace(self):\n TimeRangeTableA.partitioning.create_partition()\n log: PartitionLog = TimeRangeTableA.partitioning.latest\n self.assertEqual(True, log.is_attached)\n self.assertTablespace(log.table_name, log.config.attach_tablespace)\n\n TimeRangeTableA.partitioning.detach_partition([log])\n log.refresh_from_db()\n self.assertEqual(False, log.is_attached)\n self.assertTablespace(log.table_name, log.config.detach_tablespace)\n\n TimeRangeTableA.partitioning.attach_partition()\n log.refresh_from_db()\n self.assertEqual(True, log.is_attached)\n self.assertTablespace(log.table_name, log.config.attach_tablespace)\n\n\nclass ListPartitioningTestCase(GeneralTestCase):\n @classmethod\n def assertCreated(cls, model, category):\n # Verify that the partition has been created by inserting data.\n model.objects.create(category=category)\n\n def test_create_partition(self):\n ListTableText.partitioning.create_partition(\"list_table_text_a\", \"A\", \"data1\")\n self.assertCreated(ListTableText, \"A\")\n\n ListTableText.partitioning.create_partition(\"list_table_text_b\", \"B\")\n self.assertCreated(ListTableText, \"B\")\n\n ListTableText.partitioning.create_partition(\"list_table_text_blank\", \"\")\n self.assertCreated(ListTableText, \"\")\n\n ListTableText.partitioning.create_partition(\"list_table_text_none\", None, \"data2\")\n self.assertCreated(ListTableText, None)\n\n ListTableInt.partitioning.create_partition(\"list_table_int_1\", 1, \"data1\")\n self.assertCreated(ListTableInt, 1)\n\n ListTableInt.partitioning.create_partition(\"list_table_int_2\", 2)\n self.assertCreated(ListTableInt, 2)\n\n ListTableInt.partitioning.create_partition(\"list_table_int_none\", None, \"data2\")\n self.assertCreated(ListTableInt, None)\n\n ListTableBool.partitioning.create_partition(\"list_table_bool_true\", True, \"data1\")\n self.assertCreated(ListTableBool, True)\n\n ListTableBool.partitioning.create_partition(\"list_table_bool_false\", False)\n self.assertCreated(ListTableBool, False)\n\n ListTableBool.partitioning.create_partition(\"list_table_bool_none\", None, \"data2\")\n self.assertCreated(ListTableBool, None)\n\n def assertTablespace(self, table_name, tablespace):\n with connection.cursor() as cursor:\n cursor.execute(f\"SELECT tablespace FROM pg_tables WHERE tablename = '{table_name}';\")\n rows = cursor.fetchall()\n self.assertEqual(tablespace, rows[0][0])\n\n def test_attach_or_detach_partition(self):\n self.test_create_partition()\n ListTableText.partitioning.detach_partition(\"list_table_text_none\", \"data1\")\n self.assertTablespace(\"list_table_text_none\", \"data1\")\n ListTableText.partitioning.attach_partition(\"list_table_text_none\", None, \"data2\")\n self.assertTablespace(\"list_table_text_none\", \"data2\")\n" }, { "path": "tox.ini", "content": "[tox]\nenvlist =\n {py36,py37}-django{20,21,22}\n docs\n lint\n\n[travis:env]\nDJANGO =\n 2.0: django20\n 2.1: django21\n 2.2: django22\n\n[testenv]\nsetenv =\n PYTHONDONTWRITEBYTECODE=1\n PYTHONWARNINGS=once\nenvdir = {toxworkdir}/venvs/{envname}\nextras =\n dev\ndeps =\n lint: isort\n black\n flake8\n flake8-bugbear\n flake8-quotes\n django20: Django>=2.0,<2.1\n django21: Django>=2.1,<2.2\n django22: Django>=2.2a1,<3.0\n\ncommands =\n python3 ./run_test.py -c {posargs}\n\n[testenv:docs]\nbasepython = python3.6\nwhitelist_externals = make\nchangedir = docs\ncommands =\n make html\n\n[testenv:lint]\nbasepython = python3.6\ncommands =\n isort -rc pg_partitioning tests\n black pg_partitioning tests -l 157 --exclude pg_partitioning/migrations/*\n flake8 {toxinidir}/pg_partitioning {toxinidir}/tests {toxinidir}/*.py --exclude */migrations\n" } ]