Repository: chaitin/django-pg-timepart Branch: master Commit: 129f5ddd11c6 Files: 39 Total size: 64.0 KB Directory structure: gitextract_of6gshgr/ ├── .gitignore ├── .readthedocs.yml ├── .travis.yml ├── CONTRIBUTING.rst ├── CONTRIBUTORS ├── LICENSE ├── README.rst ├── codecov.yml ├── dev/ │ ├── docker-compose.yml │ └── postgres_init.sh ├── docs/ │ ├── Makefile │ ├── readthedocs.txt │ └── source/ │ ├── _templates/ │ │ └── sidebarlogo.html │ ├── api.rst │ ├── conf.py │ ├── decorators.rst │ ├── design.rst │ ├── index.rst │ ├── installation.rst │ └── signals.rst ├── pg_partitioning/ │ ├── __init__.py │ ├── apps.py │ ├── constants.py │ ├── decorators.py │ ├── manager.py │ ├── migrations/ │ │ ├── 0001_initial.py │ │ └── __init__.py │ ├── models.py │ ├── patch/ │ │ ├── __init__.py │ │ └── schema.py │ ├── shortcuts.py │ └── signals.py ├── run_test.py ├── setup.cfg ├── setup.py ├── tests/ │ ├── __init__.py │ ├── models.py │ └── tests.py └── tox.ini ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ *.egg-info dist/ build/ docs/_build docs/source/.doctrees/ __pycache__/ .cache *.sqlite3 .tox/ *.swp *.pyc .coverage* .pytest_cache node_modules venv/ # IDE and Tooling files .idea/* *~ # macOS .DS_Store ================================================ FILE: .readthedocs.yml ================================================ version: 2 build: image: latest sphinx: configuration: docs/source/conf.py python: version: 3.6 install: - requirements: docs/readthedocs.txt ================================================ FILE: .travis.yml ================================================ dist: xenial language: python cache: pip sudo: required matrix: fast_finish: true include: - { python: "3.6", env: DJANGO=2.0 } - { python: "3.6", env: DJANGO=2.1 } - { python: "3.6", env: DJANGO=2.2 } - { python: "3.7", env: DJANGO=2.0 } - { python: "3.7", env: DJANGO=2.1 } - { python: "3.7", env: DJANGO=2.2 } - { python: "3.7", env: DJANGO=master } - { python: "3.6", env: TOXENV=lint } - { python: "3.6", env: TOXENV=docs } allow_failures: - env: TOXENV=docs - env: DJANGO=master services: - docker before_install: - sudo /etc/init.d/postgresql stop install: - pip install '.[dev]' tox-travis before_script: - docker-compose -f dev/docker-compose.yml up -d - sleep 5 script: - tox after_success: - pip install codecov - codecov -e TOXENV,DJANGO notifications: email: false ================================================ FILE: CONTRIBUTING.rst ================================================ Contributing Guidelines ======================= Issue tracker ------------- 您可以通过 `issue tracker `__ 提交改进建议、缺陷报告或功能需求，但 **必须** 遵守以下规范： * **请勿** 重复提交相似的主题或内容。 * **请勿** 讨论任何与本项目无关的内容。 * 我们非常欢迎您提交程序缺陷报告，但在此之前，请确保您已经完整阅读过相关文档，并已经做了一些必要的调查，确定错误并非您自身造成的。在您编写程序缺陷报告时， 请详细描述您所出现的问题和复现步骤，并附带详细的信息，以便我们能尽快定位问题。 ---- You can submit improvement suggestions, bug reports, or feature requests through the `issue tracker `_, but you **MUST** adhere to the following specifications: * **Do not** submit similar topics or content repeatedly. * **Do not** discuss any content not related to this project. * We welcome you to submit a bug report, but before doing so, please make sure that you have read the documentation in its entirety and have done some necessary investigations to determine that the error is not yours. When you write a bug report, Please describe in detail the problem and recurring steps that you have with detailed information so that we can locate the problem as quickly as possible. Code guidelines --------------- * 本项目采用 `语义化版本 2.0.0 `_ * 本项目使用了 `flask8` `isort` `black` 等代码静态检查工具。提交的代码 **必须** 通过 `lint` 工具检查。某些特殊情况不符合规范的部分，需要按照检查工具要求的方式具体标记出来。 * 公开的 API **必须** 使用 Type Hint 并编写 Docstrings，其他部分 **建议** 使用并在必要的地方为代码编写注释，增强代码的可读性。 * **必须** 限定非 Development 的外部依赖的模块版本为某一个完全兼容的系列。 相关文档: | `Google Python Style Guide `_ | `PEP 8 Style Guide for Python Code `_ ---- * This project uses a `Semantic Version 2.0.0 `_ * This project uses a code static check tool such as `flask8` `isort` `black`. The submitted code **MUST** be checked by the `lint` tool. Some special cases that do not meet the specifications need to be specifically marked in the way required by the inspection tool. * The public API **MUST** use Type Hint and write Docstrings, other parts **SHOULD** use it and write comments to the code where necessary to enhance the readability of code. * External dependencies published with the project **MUST** at least define a fully compatible version family. Related documents: | `Google Python Style Guide `_ | `PEP 8 Style Guide for Python Code `_ ================================================ FILE: CONTRIBUTORS ================================================ virusdefender monouno ================================================ FILE: LICENSE ================================================ Copyright (c) 2019 Chaitin Tech Co., Ltd. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.rst ================================================ django-pg-partitioning ====================== .. image:: https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square :target: https://raw.githubusercontent.com/chaitin/django-pg-partitioning/master/LICENSE .. image:: https://img.shields.io/badge/Django-2.x-green.svg?style=flat-square&logo=django :target: https://www.djangoproject.com/ .. image:: https://img.shields.io/badge/PostgreSQL-11-lightgrey.svg?style=flat-square&logo=postgresql :target: https://www.postgresql.org/ .. image:: https://readthedocs.org/projects/django-pg-partitioning/badge/?version=latest&style=flat-square :target: https://django-pg-partitioning.readthedocs.io .. image:: https://img.shields.io/pypi/v/django-pg-partitioning.svg?style=flat-square :target: https://pypi.org/project/django-pg-partitioning/ .. image:: https://api.travis-ci.org/chaitin/django-pg-partitioning.svg?branch=master :target: https://travis-ci.org/chaitin/django-pg-partitioning .. image:: https://api.codacy.com/project/badge/Grade/c872699c1b254e90b540b053343d1e81 :target: https://www.codacy.com/app/xingji2163/django-pg-partitioning?utm_source=github.com&utm_medium=referral&utm_content=chaitin/django-pg-partitioning&utm_campaign=Badge_Grade .. image:: https://codecov.io/gh/chaitin/django-pg-partitioning/branch/master/graph/badge.svg :target: https://codecov.io/gh/chaitin/django-pg-partitioning ⚠️ ---- 目前已经有了更好用的 Django 插件（比如 django-postgres-extra）使得基于 Django 开发的项目能够方便地使用 PostgreSQL 数据库的高级功能，因此本项目不再维护。你依然可以 fork 本项目并进行二次开发，祝你生活愉快 :) There are already better Django plugins (such as django-postgres-extra) that make it easy for Django-based projects to use the advanced features of PostgreSQL databases, so this project is no longer maintained. You can still fork this project and do secondary development, have a nice life :) ---- 一个支持 PostgreSQL 11 原生表分区的 Django 扩展，使您可以在 Django 中创建分区表并管理它们。目前它支持两种分区类型： - 时间范围分区（Time Range Partitioning）：将时序数据分开存储到不同的时间范围分区表中，支持创建连续且不重叠的时间范围分区并进行归档管理。 - 列表分区（List Partitioning）：根据分区字段的确定值将数据分开存储到不同的分区表中。 A Django extension that supports PostgreSQL 11 native table partitioning, allowing you to create partitioned tables in Django and manage them. Currently it supports the following two partition types: - **Time Range Partitioning**: Separate time series data into different time range partition tables, support the creation of continuous and non-overlapping time range partitions and archival management. - **List Partitioning**: Store data separately into different partition tables based on the determined values of the partition key. Documentation https://django-pg-partitioning.readthedocs.io .. image:: https://raw.githubusercontent.com/chaitin/django-pg-partitioning/master/docs/source/_static/carbon.png :align: center TODO ---- - Improve the details of the function. - Improve documentation and testing. - Optimization implementation. maybe more... Contributing ------------ If you want to contribute to a project and make it better, you help is very welcome! Please read through `Contributing Guidelines `__. License ------- This project is licensed under the MIT. Please see `LICENSE `_. Project Practice ---------------- .. image:: https://raw.githubusercontent.com/chaitin/django-pg-timepart/master/docs/source/_static/safeline.svg?sanitize=true :target: https://www.chaitin.cn/en/safeline ================================================ FILE: codecov.yml ================================================ coverage: precision: 2 round: down range: "80...100" status: project: yes patch: no changes: no comment: off ================================================ FILE: dev/docker-compose.yml ================================================ version: "3" services: postgres: image: postgres:${POSTGRES_VERSION:-11.1-alpine} container_name: postgres restart: always volumes: - ./postgres_init.sh:/docker-entrypoint-initdb.d/postgres_init.sh environment: - POSTGRES_DB=test - POSTGRES_USER=test - POSTGRES_PASSWORD=test ports: - "127.0.0.1:5432:5432" ================================================ FILE: dev/postgres_init.sh ================================================ #!/usr/bin/env bash mkdir /tmp/data1 /tmp/data2 psql -U test -c "CREATE TABLESPACE data1 LOCATION '/tmp/data1'" psql -U test -c "CREATE TABLESPACE data2 LOCATION '/tmp/data2'" ================================================ FILE: docs/Makefile ================================================ # Minimal makefile for Sphinx documentation # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build SPHINXPROJ = django-partitioning SOURCEDIR = source BUILDDIR = _build .PHONY: all all: @$(SPHINXBUILD) "$(SOURCEDIR)" "$(BUILDDIR)/" .PHONY: help help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) ================================================ FILE: docs/readthedocs.txt ================================================ alabaster==0.7.12 sphinx==1.8.3 sphinxcontrib-napoleon==0.7 Pygments==2.3.1 psycopg2-binary==2.7.6.1 Django>=2.0,<2.2 python-dateutil~=2.7 ================================================ FILE: docs/source/_templates/sidebarlogo.html ================================================ ================================================ FILE: docs/source/api.rst ================================================ API Reference ============= .. py:currentmodule:: pg_partitioning.manager Time Range Partitioning ----------------------- .. autoclass:: TimeRangePartitionManager :members: .. autoclass:: PartitionConfig :members: period, interval, attach_tablespace, detach_tablespace, save .. autoclass:: PartitionLog :members: is_attached, detach_time, save, delete List Partitioning ----------------- .. autoclass:: ListPartitionManager :members: .. py:currentmodule:: pg_partitioning.shortcuts Shortcuts --------- .. automodule:: pg_partitioning.shortcuts :members: truncate_table, set_tablespace, drop_table ================================================ FILE: docs/source/conf.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- import os import sys import django from django.conf import settings sys.path.insert(0, os.path.abspath('../..')) sys.path.insert(0, os.path.abspath('./')) settings.configure( INSTALLED_APPS=( "pg_partitioning", ), ) django.setup() extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', 'sphinx.ext.napoleon', 'alabaster', ] templates_path = ['_templates'] source_suffix = '.rst' master_doc = 'index' project = 'django-pg-partitioning' copyright = '2019, Chaitin Tech' author = 'Boyce Li' language = 'en' exclude_patterns = ['_build'] pygments_style = 'sphinx' html_theme = 'alabaster' html_theme_options = { 'github_user': 'chaitin', 'github_repo': 'django-pg-partitioning', 'github_type': 'star', 'github_banner': 'true', 'show_powered_by': 'false', 'code_font_size': '14px', } html_static_path = ['_static'] html_sidebars = { '**': [ 'sidebarlogo.html', 'navigation.html', 'searchbox.html', ] } htmlhelp_basename = 'django-pg-partitioning-doc' man_pages = [ (master_doc, 'django-pg-partitioning', 'django-pg-partitioning Documentation', [author], 1) ] texinfo_documents = [ (master_doc, 'django-pg-partitioning', 'django-pg-partitioning Documentation', author, 'django-pg-partitioning', 'A Django extension that supports PostgreSQL 11 time ranges and list partitioning.', 'Miscellaneous'), ] intersphinx_mapping = { 'https://docs.python.org/3/': None, 'https://pika.readthedocs.io/en/0.10.0/': None, } add_module_names = False ================================================ FILE: docs/source/decorators.rst ================================================ Decorators ========== Now the decorator must be used on a non-abstract model class that has not yet built a table in the database. If you must use the decorator on a model class that has previously performed a migrate operation, you need to back up the model's data, then drop the table, and then import the data after you have created a partitioned table. .. py:currentmodule:: pg_partitioning.decorators .. autodata:: TimeRangePartitioning :annotation: .. autodata:: ListPartitioning :annotation: Post-Decoration --------------- You can run ``makemigrations`` and ``migrate`` commands to create and apply new migrations. Once the table has been created, it is not possible to turn a regular table into a partitioned table or vice versa. ================================================ FILE: docs/source/design.rst ================================================ Design ====== A brief description of the architecture of django-pg-partitioning. Partitioned Table Support ------------------------- Currently Django does not support creating partitioned tables, so django-partitioning monkey patch ``create_model`` method in ``DatabaseSchemaEditor`` to make it generate SQL statements that create partitioned tables. Some of the operations that Django applies to regular database tables may not be supported or even conflicted on the partitioned table, eventually throwing a database exception. Therefore, it is recommended that you read the section on the partition table in the official database documentation and refer to the relevant implementation inside Django. Constraint Limitations ---------------------- It is important to note that PostgreSQL table partitioning has some restrictions on field constraints. In order for the extension to work, we turned off Django's automatically generated primary key constraint, but did not do other legality checks. For example, if you mistakenly used a unique or foreign key constraint, it will throw an exception directly, which is what you are coding and it needs to be manually circumvented during use. Tablespace ---------- ``pg_partitioning`` will silently set the tablespace of all local partitioned indexes under one partition to be consistent with the partition. Partition Information --------------------- ``pg_partitioning`` saves partition configuration and state information in ``PartitionConfig`` and ``PartitionLog``. The problem with this is that once this information is inconsistent with the actual situation, ``pg_partitioning`` will not work properly, so you can only fix it manually. Management ---------- You can use ``Model.partitioning.create_partition`` and ``Model.partitioning.detach_partition`` to automatically create and archive partitions. In addition setting ``default_detach_tablespace`` and ``default_attach_tablespace``, you can also use the ``set_tablespace`` method of the PartitionLog object to move the partition. See :doc:`api` for details. ================================================ FILE: docs/source/index.rst ================================================ .. include:: ../../README.rst .. toctree:: :hidden: :maxdepth: 2 installation design decorators api signals Contributing Project License ================================================ FILE: docs/source/installation.rst ================================================ Installation ============ PyPI ---- .. code-block:: bash $ pip install django-pg-partitioning Or you can install from GitHub .. code-block:: bash $ pip install git+https://github.com/chaitin/django-pg-partitioning.git@master Integrate with Django --------------------- Add ``pg_partitioning`` to ``INSTALLED_APPS`` in settings.py. Important - Please note 'pg_partitioning' should be loaded earlier than other apps that depend on it:: INSTALLED_APPS = [ 'pg_partitioning', ... ] PARTITION_TIMEZONE = "Asia/Shanghai" You can specify the time zone referenced by the time range partitioned table via ``PARTITION_TIMEZONE``, and if it is not specified, ``TIME_ZONE`` value is used. Post-Installation ----------------- In your Django root execute the command below to create 'pg_partitioning' database tables:: ./manage.py migrate pg_partitioning ================================================ FILE: docs/source/signals.rst ================================================ Signals ======= Note that these signals are only triggered when the save methods of ``PartitionConfig`` and ``PartitionLog`` You can hook to them for your own needs (for example to create corresponding table index). .. py:currentmodule:: pg_partitioning.signals .. autodata:: post_create_partition(sender, partition_log) :annotation: .. autodata:: post_attach_partition(sender, partition_log) :annotation: .. autodata:: post_detach_partition(sender, partition_log) :annotation: ================================================ FILE: pg_partitioning/__init__.py ================================================ """ A Django extension that supports PostgreSQL 11 time ranges and list partitioning. """ REQUIRED_DJANGO_VERSION = [(2, 0), (3, 0)] DJANGO_VERSION_ERROR = "django-pg-partitioning isn't available on the currently installed version of Django." try: import django except ImportError: raise ImportError(DJANGO_VERSION_ERROR) if REQUIRED_DJANGO_VERSION[0] > tuple(django.VERSION[:2]) or tuple(django.VERSION[:2]) > REQUIRED_DJANGO_VERSION[1]: raise ImportError(DJANGO_VERSION_ERROR) __version__ = "0.11" default_app_config = "pg_partitioning.apps.AppConfig" ================================================ FILE: pg_partitioning/apps.py ================================================ from django.apps import AppConfig as DefaultAppConfig class AppConfig(DefaultAppConfig): name = "pg_partitioning" def ready(self): from .patch import schema # noqa ================================================ FILE: pg_partitioning/constants.py ================================================ SQL_CREATE_TIME_RANGE_PARTITION = """\ CREATE TABLE IF NOT EXISTS %(child)s PARTITION OF %(parent)s FOR VALUES FROM (%(date_start)s) TO (%(date_end)s)""" SQL_CREATE_LIST_PARTITION = """\ CREATE TABLE IF NOT EXISTS %(child)s PARTITION OF %(parent)s FOR VALUES IN (%(value)s)""" SQL_SET_TABLE_TABLESPACE = """\ ALTER TABLE IF EXISTS %(name)s SET TABLESPACE %(tablespace)s""" SQL_APPEND_TABLESPACE = " TABLESPACE %(tablespace)s" SQL_ATTACH_TIME_RANGE_PARTITION = """\ ALTER TABLE IF EXISTS %(parent)s ATTACH PARTITION %(child)s FOR VALUES FROM (%(date_start)s) TO (%(date_end)s)""" SQL_ATTACH_LIST_PARTITION = """\ ALTER TABLE IF EXISTS %(parent)s ATTACH PARTITION %(child)s FOR VALUES IN (%(value)s)""" SQL_DETACH_PARTITION = "ALTER TABLE IF EXISTS %(parent)s DETACH PARTITION %(child)s" SQL_DROP_TABLE = "DROP TABLE IF EXISTS %(name)s" SQL_TRUNCATE_TABLE = "TRUNCATE TABLE %(name)s" SQL_DROP_INDEX = "DROP INDEX IF EXISTS %(name)s" SQL_CREATE_INDEX = "CREATE INDEX IF NOT EXISTS %(name)s ON %(table_name)s USING %(method)s (%(column_name)s)" SQL_SET_INDEX_TABLESPACE = "ALTER INDEX %(name)s SET TABLESPACE %(tablespace)s" SQL_GET_TABLE_INDEXES = "SELECT indexname FROM pg_indexes WHERE tablename = %(table_name)s" DT_FORMAT = "%Y-%m-%d" class PartitioningType: Range = "RANGE" List = "LIST" class PeriodType: Day = "Day" Week = "Week" Month = "Month" Year = "Year" ================================================ FILE: pg_partitioning/decorators.py ================================================ import logging from typing import Type from django.db import models from pg_partitioning.manager import ListPartitionManager, TimeRangePartitionManager logger = logging.getLogger(__name__) class _PartitioningBase: def __init__(self, partition_key: str, **options): self.partition_key = partition_key self.options = options def __call__(self, model: Type[models.Model]): if model._meta.abstract: raise NotImplementedError("Decorative abstract model classes are not supported.") class TimeRangePartitioning(_PartitioningBase): """Use this decorator to declare the database table corresponding to the model to be partitioned by time range. Parameters: partition_key(str): Partition field name of DateTimeField. options: Currently supports the following keyword parameters: - default_period(PeriodType): Default partition period. - default_interval(int): Default detach partition interval. - default_attach_tablespace(str): Default tablespace for attached tables. - default_detach_tablespace(str): Default tablespace for attached tables. Example: .. code-block:: python from django.db import models from django.utils import timezone from pg_partitioning.decorators import TimeRangePartitioning @TimeRangePartitioning(partition_key="timestamp") class MyLog(models.Model): name = models.TextField(default="Hello World!") timestamp = models.DateTimeField(default=timezone.now, primary_key=True) """ def __call__(self, model: Type[models.Model]): super().__call__(model) if model._meta.get_field(self.partition_key).get_internal_type() != models.DateTimeField().get_internal_type(): raise ValueError("The partition_key must be DateTimeField type.") model.partitioning = TimeRangePartitionManager(model, self.partition_key, self.options) return model class ListPartitioning(_PartitioningBase): """Use this decorator to declare the database table corresponding to the model to be partitioned by list. Parameters: partition_key(str): Partition key name, the type of the key must be one of boolean, text or integer. Example: .. code-block:: python from django.db import models from django.utils import timezone from pg_partitioning.decorators import ListPartitioning @ListPartitioning(partition_key="category") class MyLog(models.Model): category = models.TextField(default="A") timestamp = models.DateTimeField(default=timezone.now, primary_key=True) """ def __call__(self, model: Type[models.Model]): super().__call__(model) model.partitioning = ListPartitionManager(model, self.partition_key, self.options) return model ================================================ FILE: pg_partitioning/manager.py ================================================ import datetime from collections import Iterable from typing import Optional, Type, Union import pytz from dateutil.relativedelta import MO, relativedelta from django.conf import settings from django.db import IntegrityError, models from django.db.models import Q from django.utils import timezone from pg_partitioning.shortcuts import double_quote, execute_sql, generate_set_indexes_tablespace_sql, single_quote from .constants import ( DT_FORMAT, SQL_APPEND_TABLESPACE, SQL_ATTACH_LIST_PARTITION, SQL_CREATE_LIST_PARTITION, SQL_DETACH_PARTITION, SQL_SET_TABLE_TABLESPACE, PartitioningType, PeriodType, ) from .models import PartitionConfig, PartitionLog class _PartitionManagerBase: type = None def __init__(self, model: Type[models.Model], partition_key: str, options: dict): self.model = model self.partition_key = partition_key self.options = options class TimeRangePartitionManager(_PartitionManagerBase): """Manage time-based partition APIs.""" type = PartitioningType.Range @property def config(self) -> PartitionConfig: """Get the latest PartitionConfig instance of this model. In order to avoid the race condition, we used **select_for_update** when querying. Returns: PartitionConfig: The latest PartitionConfig instance of this model. """ try: return PartitionConfig.objects.select_for_update().get(model_label=self.model._meta.label_lower) except PartitionConfig.DoesNotExist: try: return PartitionConfig.objects.create( model_label=self.model._meta.label_lower, period=self.options.get("default_period", PeriodType.Month), interval=self.options.get("default_interval"), attach_tablespace=self.options.get("default_attach_tablespace"), detach_tablespace=self.options.get("default_detach_tablespace"), ) except IntegrityError: return PartitionConfig.objects.select_for_update().get(model_label=self.model._meta.label_lower) @property def latest(self) -> Optional[PartitionLog]: """Get the latest PartitionLog instance of this model. Returns: Optional[PartitionLog]: The latest PartitionLog instance of this model or none. """ return self.config.logs.order_by("-id").first() @classmethod def _get_period_bound(cls, date_start, initial, addition_zeros=None, is_week=False, **kwargs): zeros = {"hour": 0, "minute": 0, "second": 0, "microsecond": 0} if addition_zeros: zeros.update(addition_zeros) def func(): # lazy evaluation if initial: start = date_start.replace(**zeros) if is_week: start -= relativedelta(days=start.weekday()) else: start = date_start end = start + relativedelta(**kwargs, **zeros) return start, end return func def create_partition(self, max_days_to_next_partition: int = 1) -> None: """The partition of the next cycle is created according to the configuration. After modifying the period field, the new period will take effect the next time. The start time of the new partition is the end time of the previous partition table, or the start time of the current archive period when no partition exists. For example: the current time is June 5, 2018, and the archiving period is one year, then the start time of the first partition is 00:00:00 on January 1, 2018. Parameters: max_days_to_next_partition(int): If numbers of days remained in current partition is greater than ``max_days_to_next_partition``, no new partitions will be created. """ while True: if max_days_to_next_partition > 0 and self.latest and timezone.now() < (self.latest.end - relativedelta(days=max_days_to_next_partition)): return partition_timezone = getattr(settings, "PARTITION_TIMEZONE", None) if partition_timezone: partition_timezone = pytz.timezone(partition_timezone) date_start = timezone.localtime(self.latest.end if self.latest else None, timezone=partition_timezone) initial = not bool(self.latest) date_start, date_end = { PeriodType.Day: self._get_period_bound(date_start, initial, days=+1), PeriodType.Week: self._get_period_bound(date_start, initial, is_week=True, days=+1, weekday=MO), PeriodType.Month: self._get_period_bound(date_start, initial, addition_zeros=dict(day=1), months=+1), PeriodType.Year: self._get_period_bound(date_start, initial, addition_zeros=dict(month=1, day=1), years=+1), }[self.config.period]() partition_table_name = "_".join((self.model._meta.db_table, date_start.strftime(DT_FORMAT), date_end.strftime(DT_FORMAT))) PartitionLog.objects.create(config=self.config, table_name=partition_table_name, start=date_start, end=date_end) if not max_days_to_next_partition > 0: return def attach_partition(self, partition_log: Optional[Iterable] = None, detach_time: Optional[datetime.datetime] = None) -> None: """Attach partitions. Parameters: partition_log(Optional[Iterable]): All partitions are attached when you don't specify partitions to attach. detach_time(Optional[datetime.datetime]): When the partition specifies the archive time, it will **not** be automatically archived until that time. """ if not partition_log: partition_log = PartitionLog.objects.filter(config=self.config, is_attached=False) for log in partition_log: log.is_attached = True log.detach_time = detach_time log.save() def detach_partition(self, partition_log: Optional[Iterable] = None) -> None: """Detach partitions. Parameters: partition_log(Optional[Iterable]): Specify a partition to archive. When you don't specify a partition to archive, all partitions that meet the configuration rule are archived. """ if not partition_log: if self.config.interval: # fmt: off period = {PeriodType.Day: {"days": 1}, PeriodType.Week: {"weeks": 1}, PeriodType.Month: {"months": 1}, PeriodType.Year: {"years": 1}}[self.config.period] # fmt: on now = timezone.now() detach_timeline = now - self.config.interval * relativedelta(**period) partition_log = PartitionLog.objects.filter(config=self.config, end__lt=detach_timeline, is_attached=True) partition_log = partition_log.filter(Q(detach_time=None) | Q(detach_time__lt=now)) else: return for log in partition_log: log.is_attached = False log.detach_time = None log.save() def delete_partition(self, partition_log: Iterable) -> None: """Delete partitions. Parameters: partition_log(Iterable): The partitions to be deleted. """ for log in partition_log: if log.config == self.config: log.delete() def _db_value(value: Union[str, int, bool, None]) -> str: if value is None: return "null" return single_quote(value) if isinstance(value, str) else str(value) class ListPartitionManager(_PartitionManagerBase): """Manage list-based partition APIs.""" type = PartitioningType.List def create_partition(self, partition_name: str, value: Union[str, int, bool, None], tablespace: str = None) -> None: """Create partitions. Parameters: partition_name(str): Partition name. value(Union[str, int, bool, None]): Partition key value. tablespace(str): Partition tablespace name. """ create_partition_sql = SQL_CREATE_LIST_PARTITION % { "parent": double_quote(self.model._meta.db_table), "child": double_quote(partition_name), "value": _db_value(value), } if tablespace: create_partition_sql += SQL_APPEND_TABLESPACE % {"tablespace": tablespace} execute_sql(create_partition_sql) execute_sql(generate_set_indexes_tablespace_sql(partition_name, tablespace)) def attach_partition(self, partition_name: str, value: Union[str, int, bool, None], tablespace: str = None) -> None: """Attach partitions. Parameters: partition_name(str): Partition name. value(Union[str, int, bool, None]): Partition key value. tablespace(str): Partition tablespace name. """ sql_sequence = list() if tablespace: sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(partition_name), "tablespace": tablespace}) sql_sequence.extend(generate_set_indexes_tablespace_sql(partition_name, tablespace)) sql_sequence.append( SQL_ATTACH_LIST_PARTITION % {"parent": double_quote(self.model._meta.db_table), "child": double_quote(partition_name), "value": _db_value(value)} ) execute_sql(sql_sequence) def detach_partition(self, partition_name: str, tablespace: str = None) -> None: """Detach partitions. Parameters: partition_name(str): Partition name. tablespace(str): Partition tablespace name. """ sql_sequence = [SQL_DETACH_PARTITION % {"parent": double_quote(self.model._meta.db_table), "child": double_quote(partition_name)}] if tablespace: sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(partition_name), "tablespace": tablespace}) sql_sequence.extend(generate_set_indexes_tablespace_sql(partition_name, tablespace)) execute_sql(sql_sequence) ================================================ FILE: pg_partitioning/migrations/0001_initial.py ================================================ # Generated by Django 2.1.7 on 2019-02-17 12:00 import django.db.models.deletion from django.db import migrations, models class Migration(migrations.Migration): initial = True dependencies = [ ] operations = [ migrations.CreateModel( name='PartitionConfig', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('model_label', models.TextField(unique=True)), ('period', models.TextField(default='Month')), ('interval', models.PositiveIntegerField(null=True)), ('attach_tablespace', models.TextField(null=True)), ('detach_tablespace', models.TextField(null=True)), ], ), migrations.CreateModel( name='PartitionLog', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('table_name', models.TextField(unique=True)), ('start', models.DateTimeField()), ('end', models.DateTimeField()), ('is_attached', models.BooleanField(default=True)), ('detach_time', models.DateTimeField(null=True)), ('config', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='logs', to='pg_partitioning.PartitionConfig')), ], options={ 'ordering': ('-id',), }, ), ] ================================================ FILE: pg_partitioning/migrations/__init__.py ================================================ ================================================ FILE: pg_partitioning/models.py ================================================ from django.apps import apps from django.db import models, transaction from pg_partitioning.signals import post_attach_partition, post_create_partition, post_detach_partition from .constants import ( SQL_APPEND_TABLESPACE, SQL_ATTACH_TIME_RANGE_PARTITION, SQL_CREATE_TIME_RANGE_PARTITION, SQL_DETACH_PARTITION, SQL_SET_TABLE_TABLESPACE, PeriodType, ) from .shortcuts import double_quote, drop_table, execute_sql, generate_set_indexes_tablespace_sql, single_quote class PartitionConfig(models.Model): """You can get the configuration object of the partition table through ``Model.partitioning.config``, You can only edit the following fields via the object's ``save`` method: """ model_label = models.TextField(unique=True) period = models.TextField(default=PeriodType.Month) """Partition period. you can only set options in the `PeriodType`. The default value is ``PeriodType.Month``. Changing this value will trigger the ``detach_partition`` method.""" interval = models.PositiveIntegerField(null=True) """Detaching period. The ``detach_partition`` method defaults to detach partitions before the interval * period. The default is None, ie no partition will be detached. Changing this value will trigger the ``detach_partition`` method.""" attach_tablespace = models.TextField(null=True) """The name of the tablespace specified when creating or attaching a partition. Modifying this field will only affect subsequent operations. A table migration may occur at this time.""" detach_tablespace = models.TextField(null=True) """The name of the tablespace specified when detaching a partition. Modifying this field will only affect subsequent operations. A table migration may occur at this time.""" def save(self, force_insert=False, force_update=False, using=None, update_fields=None): """This setting will take effect immediately when you modify the value of ``interval`` in the configuration. """ adding = self._state.adding model = apps.get_model(self.model_label) if not adding: prev = self.__class__.objects.get(pk=self.pk) with transaction.atomic(): super().save(force_insert, force_update, using, update_fields) if adding: # Creating first partition. model.partitioning.create_partition(0) if not adding: # Period or interval changed. if prev.period != self.period or (prev.interval != self.interval): model.partitioning.detach_partition() class PartitionLog(models.Model): """You can only edit the following fields via the object's ``save`` method:""" config = models.ForeignKey(PartitionConfig, on_delete=models.CASCADE, related_name="logs") table_name = models.TextField(unique=True) # range bound: [start, end) start = models.DateTimeField() end = models.DateTimeField() is_attached = models.BooleanField(default=True) """Whether the partition is a attached partition. changing the value will trigger an attaching or detaching operation.""" detach_time = models.DateTimeField(null=True) """When the value is not `None`, the partition will not be automatically detached before this time. The default is `None`.""" def save(self, force_insert=False, force_update=False, using=None, update_fields=None): """This setting will take effect immediately when you modify the value of ``is_attached`` in the configuration. """ model = apps.get_model(self.config.model_label) if self._state.adding: create_partition_sql = SQL_CREATE_TIME_RANGE_PARTITION % { "parent": double_quote(model._meta.db_table), "child": double_quote(self.table_name), "date_start": single_quote(self.start.isoformat()), "date_end": single_quote(self.end.isoformat()), } if self.config.attach_tablespace: create_partition_sql += SQL_APPEND_TABLESPACE % {"tablespace": self.config.attach_tablespace} with transaction.atomic(): super().save(force_insert, force_update, using, update_fields) execute_sql(create_partition_sql) execute_sql(generate_set_indexes_tablespace_sql(self.table_name, self.config.attach_tablespace)) post_create_partition.send(sender=model, partition_log=self) else: with transaction.atomic(): prev = self.__class__.objects.select_for_update().get(pk=self.pk) # Detach partition. if prev.is_attached and (not self.is_attached): sql_sequence = [SQL_DETACH_PARTITION % {"parent": double_quote(model._meta.db_table), "child": double_quote(self.table_name)}] if self.config.detach_tablespace: sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(self.table_name), "tablespace": self.config.detach_tablespace}) sql_sequence.extend(generate_set_indexes_tablespace_sql(self.table_name, self.config.detach_tablespace)) super().save(force_insert, force_update, using, update_fields) execute_sql(sql_sequence) post_detach_partition.send(sender=model, partition_log=self) # Attach partition. elif (not prev.is_attached) and self.is_attached: sql_sequence = list() if self.config.attach_tablespace: sql_sequence.append(SQL_SET_TABLE_TABLESPACE % {"name": double_quote(self.table_name), "tablespace": self.config.attach_tablespace}) sql_sequence.extend(generate_set_indexes_tablespace_sql(self.table_name, self.config.attach_tablespace)) sql_sequence.append( SQL_ATTACH_TIME_RANGE_PARTITION % { "parent": double_quote(model._meta.db_table), "child": double_quote(self.table_name), "date_start": single_quote(self.start.isoformat()), "date_end": single_quote(self.end.isoformat()), } ) super().save(force_insert, force_update, using, update_fields) execute_sql(sql_sequence) post_attach_partition.send(sender=model, partition_log=self) # State has not changed. else: super().save(force_insert, force_update, using, update_fields) @transaction.atomic def delete(self, using=None, keep_parents=False): """When the instance is deleted, the partition corresponding to it will also be deleted.""" drop_table(self.table_name) super().delete(using, keep_parents) class Meta: ordering = ("-id",) ================================================ FILE: pg_partitioning/patch/__init__.py ================================================ ================================================ FILE: pg_partitioning/patch/schema.py ================================================ import logging from importlib import import_module from django.apps.config import MODELS_MODULE_NAME from django.db.backends.postgresql.schema import DatabaseSchemaEditor from pg_partitioning.manager import _PartitionManagerBase logger = logging.getLogger(__name__) default_create_model_method = DatabaseSchemaEditor.create_model default_sql_create_table = DatabaseSchemaEditor.sql_create_table def create_model(self, model): meta = model._meta try: cls_module = import_module(f"{meta.app_label}.{MODELS_MODULE_NAME}") except ModuleNotFoundError: cls_module = None cls = getattr(cls_module, meta.object_name, None) partitioning = getattr(cls, "partitioning", None) if isinstance(partitioning, _PartitionManagerBase): # XXX: Monkeypatch create_model. logger.debug("Partitioned model detected: %s", meta.label) _type = partitioning.type key = partitioning.partition_key DatabaseSchemaEditor.sql_create_table = f"CREATE TABLE %(table)s (%(definition)s) PARTITION BY {_type} ({key})" if meta.pk.name != key: """The partition key must be part of the primary key, and currently Django does not support setting a composite primary key, so its properties are turned off.""" meta.pk.primary_key = False logger.info("Note that PK constraints for %s has been temporarily closed.", meta.label) else: DatabaseSchemaEditor.sql_create_table = default_sql_create_table default_create_model_method(self, model) meta.pk.primary_key = True DatabaseSchemaEditor.create_model = create_model ================================================ FILE: pg_partitioning/shortcuts.py ================================================ import logging from typing import List, Optional, Tuple, Union from django.db import connection from pg_partitioning.constants import SQL_DROP_TABLE, SQL_GET_TABLE_INDEXES, SQL_SET_INDEX_TABLESPACE, SQL_SET_TABLE_TABLESPACE, SQL_TRUNCATE_TABLE logger = logging.getLogger(__name__) def single_quote(name: str) -> str: """Represent a string constants in SQL.""" if name.startswith("'") and name.endswith("'"): return name return "'%s'" % name def double_quote(name: str) -> str: """Represent a identify in SQL.""" if name.startswith('"') and name.endswith('"'): return name return '"%s"' % name def execute_sql(sql_sequence: Union[str, List[str], Tuple[str]], fetch: bool = False) -> Optional[List]: """Execute SQL sequence and returning result.""" if not sql_sequence: if fetch: return [] return sql_str = "" for statement in sql_sequence if isinstance(sql_sequence, (list, tuple)) else [sql_sequence]: sql_str += ";\n" + statement if sql_str else statement logger.debug("The sequence of SQL statements to be executed:\n %s", sql_str) with connection.cursor() as cursor: cursor.execute(sql_str) if fetch: return cursor.fetchall() def generate_set_indexes_tablespace_sql(table_name: str, tablespace: str) -> List[str]: """Generate set indexes tablespace SQL sequence. Parameters: table_name(str): Table name. tablespace(str): Partition tablespace. """ sql_sequence = [] result = execute_sql(SQL_GET_TABLE_INDEXES % {"table_name": single_quote(table_name)}, fetch=True) for item in result: sql_sequence.append(SQL_SET_INDEX_TABLESPACE % {"name": double_quote(item[0]), "tablespace": tablespace}) return sql_sequence def set_tablespace(table_name: str, tablespace: str) -> None: """Set the tablespace for a table and indexes. Parameters: table_name(str): Table name. tablespace(str): Tablespace name. """ sql_sequence = [SQL_SET_TABLE_TABLESPACE % {"name": double_quote(table_name), "tablespace": tablespace}] sql_sequence.extend(generate_set_indexes_tablespace_sql(table_name, tablespace)) execute_sql(sql_sequence) def truncate_table(table_name: str) -> None: """Truncate table. Parameters: table_name(str): Table name. """ execute_sql(SQL_TRUNCATE_TABLE % {"name": double_quote(table_name)}) def drop_table(table_name: str) -> None: """Drop table. Parameters: table_name(str): Table name. """ execute_sql(SQL_DROP_TABLE % {"name": double_quote(table_name)}) ================================================ FILE: pg_partitioning/signals.py ================================================ from django.dispatch import Signal post_create_partition = Signal(providing_args=["partition_log"]) """Sent when a partition is created. """ post_attach_partition = Signal(providing_args=["partition_log"]) """Sent when a partition is attached. """ post_detach_partition = Signal(providing_args=["partition_log"]) """Sent when a partition is detached. """ ================================================ FILE: run_test.py ================================================ import argparse import sys import dj_database_url import django from django.core.management import call_command def setup_django_environment(): from django.conf import settings settings.configure( DEBUG_PROPAGATE_EXCEPTIONS=True, DATABASES={ "default": dj_database_url.config(env="DATABASE_URL", default="postgres://test:test@localhost/test", conn_max_age=20) }, SECRET_KEY="not very secret in tests", USE_I18N=True, USE_L10N=True, USE_TZ=True, TIME_ZONE="Asia/Shanghai", INSTALLED_APPS=( "pg_partitioning", "tests", ), LOGGING={ "version": 1, "disable_existing_loggers": False, "formatters": { "standard": { "format": "[%(asctime)s] %(message)s", "datefmt": "%Y-%m-%d %H:%M:%S" } }, "handlers": { "console": { "level": "DEBUG", "class": "logging.StreamHandler", "formatter": "standard" } }, "loggers": { "pg_partitioning.shortcuts": { "handlers": ["console"], "level": "DEBUG", "propagate": False, }, "pg_partitioning.patch.schema": { "handlers": ["console"], "level": "DEBUG", "propagate": False, }, }, } ) django.setup() if __name__ == "__main__": parser = argparse.ArgumentParser(description="Run the pg_partitioning test suite.") parser.add_argument("-c", "--coverage", dest="use_coverage", action="store_true", help="Run coverage to collect code coverage and generate report.") options = parser.parse_args() if options.use_coverage: try: from coverage import coverage except ImportError: options.use_coverage = False if options.use_coverage: cov = coverage() cov.start() setup_django_environment() call_command("test", verbosity=2, interactive=False, stdout=sys.stdout) if options.use_coverage: print("\nRunning Code Coverage...\n") cov.stop() cov.report() cov.xml_report() ================================================ FILE: setup.cfg ================================================ [flake8] max-complexity = 20 max-line-length = 157 inline-quotes = " multiline-quotes = """ [isort] not_skip = __init__.py line_length = 157 include_trailing_comma = true combine_as_imports = true multi_line_output = 3 order_by_type = true [bdist_wheel] universal=1 [coverage:run] include = pg_partitioning/*,test/* omit = pg_partitioning/migrations/* branch = True [coverage:report] exclude_lines = pragma: no cover raise NotImplementedError ignore_errors = True ================================================ FILE: setup.py ================================================ import os from setuptools import setup def rel(*xs): return os.path.join(os.path.abspath(os.path.dirname(__file__)), *xs) with open(rel("README.rst")) as f: long_description = f.read() with open(rel("pg_partitioning", "__init__.py"), "r") as f: version_marker = "__version__ = " for line in f: if line.startswith(version_marker): _, version = line.split(version_marker) version = version.strip().strip('"') break else: raise RuntimeError("Version marker not found.") dependencies = [ "python-dateutil~=2.7", ] extra_dependencies = { "django": [ "Django>=2.0,<3.0" ], } extra_dependencies["all"] = list(set(sum(extra_dependencies.values(), []))) extra_dependencies["dev"] = extra_dependencies["all"] + [ # Pinned due to https://bitbucket.org/ned/coveragepy/issues/578/incomplete-file-path-in-xml-report "coverage>=4.0,<4.4", # Docs "alabaster==0.7.12", "sphinx==1.8.3", "sphinxcontrib-napoleon==0.7", # Linting "flake8~=3.6.0", "isort~=4.3.4", "black~=18.9b0", "flake8-bugbear~=18.8.0", "flake8-quotes~=1.0.0", # Misc "dj-database-url==0.5.0", "psycopg2-binary==2.7.6.1", "twine==1.12.1", # Testing "tox==3.9.0", "tox-venv==0.4.0" ] setup( name="django-pg-partitioning", version=version, author="Boyce Li", author_email="monobiao@gmail.com", description="A Django extension that supports PostgreSQL 11 time ranges and list partitioning.", long_description=long_description, long_description_content_type="text/x-rst", url="https://github.com/chaitin/django-pg-partitioning", packages=["pg_partitioning", "pg_partitioning.migrations", "pg_partitioning.patch"], include_package_data=True, install_requires=dependencies, extras_require=extra_dependencies, python_requires=">=3.6", classifiers=[ "Development Status :: 4 - Beta", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Framework :: Django :: 2.0", "Framework :: Django :: 2.1", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Topic :: Software Development :: Libraries :: Python Modules" ], ) ================================================ FILE: tests/__init__.py ================================================ ================================================ FILE: tests/models.py ================================================ from django.contrib.postgres.indexes import BrinIndex from django.db import models from django.utils import timezone from pg_partitioning.constants import PeriodType from pg_partitioning.decorators import ListPartitioning, TimeRangePartitioning @TimeRangePartitioning(partition_key="timestamp", default_period=PeriodType.Month, default_attach_tablespace="data1", default_detach_tablespace="data2") class TimeRangeTableA(models.Model): text = models.TextField() timestamp = models.DateTimeField(default=timezone.now) class Meta: db_tablespace = "pg_default" indexes = [BrinIndex(fields=["timestamp"])] unique_together = ("text", "timestamp") ordering = ["text"] @TimeRangePartitioning(partition_key="timestamp", default_period=PeriodType.Day, default_attach_tablespace="data2", default_detach_tablespace="data1") class TimeRangeTableB(models.Model): text = models.TextField() timestamp = models.DateTimeField(default=timezone.now) class Meta: indexes = [BrinIndex(fields=["timestamp"])] unique_together = ("text", "timestamp") ordering = ["text"] @ListPartitioning(partition_key="category") class ListTableText(models.Model): category = models.TextField(default="A", null=True, blank=True) timestamp = models.DateTimeField(default=timezone.now) @ListPartitioning(partition_key="category") class ListTableInt(models.Model): category = models.IntegerField(default=0, null=True) timestamp = models.DateTimeField(default=timezone.now) @ListPartitioning(partition_key="category") class ListTableBool(models.Model): category = models.NullBooleanField(default=False, null=True) timestamp = models.DateTimeField(default=timezone.now) ================================================ FILE: tests/tests.py ================================================ import datetime from unittest.mock import patch from dateutil.relativedelta import MO, relativedelta from django.db import connection from django.test import TestCase from django.utils import timezone from django.utils.crypto import get_random_string from pg_partitioning.constants import SQL_GET_TABLE_INDEXES, PeriodType from pg_partitioning.models import PartitionConfig, PartitionLog from pg_partitioning.shortcuts import single_quote from .models import ListTableBool, ListTableInt, ListTableText, TimeRangeTableA, TimeRangeTableB def t(year=2018, month=8, day=25, hour=7, minute=15, second=15, millisecond=0): """A point in time.""" return timezone.get_current_timezone().localize(datetime.datetime(year, month, day, hour, minute, second, millisecond)) def tz(time): return timezone.localtime(time) class GeneralTestCase(TestCase): def assertTablespace(self, table_name, tablespace): with connection.cursor() as cursor: cursor.execute(f"SELECT tablespace FROM pg_tables WHERE tablename = {single_quote(table_name)};") rows = cursor.fetchall() self.assertEqual(tablespace, rows[0][0]) cursor.execute(SQL_GET_TABLE_INDEXES % {"table_name": single_quote(table_name)}) rows = cursor.fetchall() for row in rows: cursor.execute(f"SELECT tablespace FROM pg_indexes WHERE indexname = {single_quote(row[0])};") rows = cursor.fetchall() self.assertEqual(tablespace, rows[0][0]) class TimeRangePartitioningTestCase(GeneralTestCase): def assertTimeRangeEqual(self, model, time_start, time_end): self.assertListEqual([time_start, time_end], [tz(model.partitioning.latest.start), tz(model.partitioning.latest.end)]) # Verify that the partition has been created by inserting data. model.objects.create(text=get_random_string(length=32), timestamp=time_start) model.objects.create(text=get_random_string(length=32), timestamp=time_end - relativedelta(microseconds=1)) def _create_partition(self, period, start_date, delta): TimeRangeTableB.partitioning.options["default_period"] = period for i in range(0, 3): if i == 0: TimeRangeTableB.partitioning.config # Create first partition by side effect. else: TimeRangeTableB.partitioning.create_partition(0) end_date = start_date + delta self.assertTimeRangeEqual(TimeRangeTableB, start_date, end_date) start_date = end_date @patch("django.utils.timezone.now", new=t) def test_create_partition_week(self): self._create_partition(PeriodType.Week, t(2018, 8, 20, 0, 0, 0), relativedelta(days=1, weekday=MO)) @patch("django.utils.timezone.now", new=t) def test_create_partition_day(self): self._create_partition(PeriodType.Day, t(2018, 8, 25, 0, 0, 0), relativedelta(days=1)) @patch("django.utils.timezone.now", new=t) def test_create_partition_month(self): self._create_partition(PeriodType.Month, t(2018, 8, 1, 0, 0, 0), relativedelta(months=1)) @patch("django.utils.timezone.now", new=t) def test_create_partition_year(self): self._create_partition(PeriodType.Year, t(2018, 1, 1, 0, 0, 0), relativedelta(years=1)) @classmethod def _update_config_period(cls, config: PartitionConfig, period: str): config.period = period config.save() def test_create_partition(self): with patch("django.utils.timezone.now", new=t): config_a: PartitionConfig = TimeRangeTableA.partitioning.config self.assertEqual(1, config_a.logs.count()) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 1, 0, 0, 0), t(2018, 9, 1, 0, 0, 0)) # Repeated calls will not produce wrong results (idempotence). for _ in range(3): TimeRangeTableA.partitioning.create_partition() self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 1, 0, 0, 0), t(2018, 9, 1, 0, 0, 0)) # Perform a series of partition creation operations. self._update_config_period(config_a, PeriodType.Week) TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 9, 3, 0, 0, 0)) self._update_config_period(config_a, PeriodType.Day) TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 3, 0, 0, 0), t(2018, 9, 4, 0, 0, 0)) self._update_config_period(config_a, PeriodType.Month) TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 4, 0, 0, 0), t(2018, 10, 1, 0, 0, 0)) TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 10, 1, 0, 0, 0), t(2018, 11, 1, 0, 0, 0)) self._update_config_period(config_a, PeriodType.Year) TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 11, 1, 0, 0, 0), t(2019, 1, 1, 0, 0, 0)) TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2019, 1, 1, 0, 0, 0), t(2020, 1, 1, 0, 0, 0)) def test_max_days_to_next_partition(self): with patch("django.utils.timezone.now", new=t): TimeRangeTableA.partitioning.create_partition() TimeRangeTableA.partitioning.create_partition(0) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 10, 1, 0, 0, 0)) with patch("django.utils.timezone.now", return_value=t(2018, 8, 2, 0, 0, 0)): TimeRangeTableA.partitioning.create_partition(59) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 9, 1, 0, 0, 0), t(2018, 10, 1, 0, 0, 0)) TimeRangeTableA.partitioning.create_partition(60) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 10, 1, 0, 0, 0), t(2018, 11, 1, 0, 0, 0)) with patch("django.utils.timezone.now", return_value=t(2019, 3, 3, 0, 0, 0)): TimeRangeTableA.partitioning.create_partition(5) self.assertTimeRangeEqual(TimeRangeTableA, t(2019, 3, 1, 0, 0, 0), t(2019, 4, 1, 0, 0, 0)) def test_attach_or_detach_partition(self): self.test_create_partition() config_a: PartitionConfig = TimeRangeTableA.partitioning.config self.assertEqual(0, config_a.logs.filter(is_attached=False).count()) TimeRangeTableA.partitioning.detach_partition(config_a.logs.all()) self.assertEqual(0, config_a.logs.filter(is_attached=True).count()) TimeRangeTableA.partitioning.attach_partition(config_a.logs.all()) self.assertEqual(0, config_a.logs.filter(is_attached=False).count()) with patch("django.utils.timezone.now", return_value=t(2018, 10, 15, 12, 1, 4)): config_a.period = PeriodType.Day config_a.interval = 15 config_a.save() self.assertEqual(t(2018, 10, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) TimeRangeTableA.partitioning.attach_partition(config_a.logs.all()) config_a.period = PeriodType.Week config_a.interval = 2 config_a.save() self.assertEqual(t(2018, 11, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) TimeRangeTableA.partitioning.attach_partition(config_a.logs.all()) log = config_a.logs.filter(is_attached=True).order_by("start").first() self.assertEqual(t(2018, 9, 1, 0, 0, 0), log.end) log.detach_time = t(2018, 10, 15, 12, 1, 5) log.save() config_a.period = PeriodType.Month config_a.interval = 1 config_a.save() self.assertEqual(t(2018, 9, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) log.refresh_from_db() self.assertEqual(True, log.is_attached) log.detach_time = None log.save() TimeRangeTableA.partitioning.detach_partition() self.assertEqual(t(2018, 10, 1, 0, 0, 0), tz(config_a.logs.filter(is_attached=True).order_by("start").first().end)) log.refresh_from_db() self.assertEqual(False, log.is_attached) def test_delete_partition(self): for _ in range(4): TimeRangeTableA.partitioning.create_partition(0) config_a: PartitionConfig = TimeRangeTableA.partitioning.config TimeRangeTableA.partitioning.delete_partition(config_a.logs.all()) self.assertEqual(0, config_a.logs.count()) with patch("django.utils.timezone.now", new=t): self._update_config_period(config_a, PeriodType.Day) TimeRangeTableA.partitioning.create_partition() self.assertEqual(2, config_a.logs.count()) self.assertTimeRangeEqual(TimeRangeTableA, t(2018, 8, 26, 0, 0, 0), t(2018, 8, 27, 0, 0, 0)) def test_attach_detach_tablespace(self): TimeRangeTableA.partitioning.create_partition() log: PartitionLog = TimeRangeTableA.partitioning.latest self.assertEqual(True, log.is_attached) self.assertTablespace(log.table_name, log.config.attach_tablespace) TimeRangeTableA.partitioning.detach_partition([log]) log.refresh_from_db() self.assertEqual(False, log.is_attached) self.assertTablespace(log.table_name, log.config.detach_tablespace) TimeRangeTableA.partitioning.attach_partition() log.refresh_from_db() self.assertEqual(True, log.is_attached) self.assertTablespace(log.table_name, log.config.attach_tablespace) class ListPartitioningTestCase(GeneralTestCase): @classmethod def assertCreated(cls, model, category): # Verify that the partition has been created by inserting data. model.objects.create(category=category) def test_create_partition(self): ListTableText.partitioning.create_partition("list_table_text_a", "A", "data1") self.assertCreated(ListTableText, "A") ListTableText.partitioning.create_partition("list_table_text_b", "B") self.assertCreated(ListTableText, "B") ListTableText.partitioning.create_partition("list_table_text_blank", "") self.assertCreated(ListTableText, "") ListTableText.partitioning.create_partition("list_table_text_none", None, "data2") self.assertCreated(ListTableText, None) ListTableInt.partitioning.create_partition("list_table_int_1", 1, "data1") self.assertCreated(ListTableInt, 1) ListTableInt.partitioning.create_partition("list_table_int_2", 2) self.assertCreated(ListTableInt, 2) ListTableInt.partitioning.create_partition("list_table_int_none", None, "data2") self.assertCreated(ListTableInt, None) ListTableBool.partitioning.create_partition("list_table_bool_true", True, "data1") self.assertCreated(ListTableBool, True) ListTableBool.partitioning.create_partition("list_table_bool_false", False) self.assertCreated(ListTableBool, False) ListTableBool.partitioning.create_partition("list_table_bool_none", None, "data2") self.assertCreated(ListTableBool, None) def assertTablespace(self, table_name, tablespace): with connection.cursor() as cursor: cursor.execute(f"SELECT tablespace FROM pg_tables WHERE tablename = '{table_name}';") rows = cursor.fetchall() self.assertEqual(tablespace, rows[0][0]) def test_attach_or_detach_partition(self): self.test_create_partition() ListTableText.partitioning.detach_partition("list_table_text_none", "data1") self.assertTablespace("list_table_text_none", "data1") ListTableText.partitioning.attach_partition("list_table_text_none", None, "data2") self.assertTablespace("list_table_text_none", "data2") ================================================ FILE: tox.ini ================================================ [tox] envlist = {py36,py37}-django{20,21,22} docs lint [travis:env] DJANGO = 2.0: django20 2.1: django21 2.2: django22 [testenv] setenv = PYTHONDONTWRITEBYTECODE=1 PYTHONWARNINGS=once envdir = {toxworkdir}/venvs/{envname} extras = dev deps = lint: isort black flake8 flake8-bugbear flake8-quotes django20: Django>=2.0,<2.1 django21: Django>=2.1,<2.2 django22: Django>=2.2a1,<3.0 commands = python3 ./run_test.py -c {posargs} [testenv:docs] basepython = python3.6 whitelist_externals = make changedir = docs commands = make html [testenv:lint] basepython = python3.6 commands = isort -rc pg_partitioning tests black pg_partitioning tests -l 157 --exclude pg_partitioning/migrations/* flake8 {toxinidir}/pg_partitioning {toxinidir}/tests {toxinidir}/*.py --exclude */migrations