[ { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*,cover\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n.env\n.idea\n\n# vim temporary files\n.*.swp\nexample\ntests/functional/output*py\n" }, { "path": ".travis.yml", "content": "language: python\npython:\n - \"3.3\"\n - \"3.4\"\n - \"3.5\"\n - \"3.6\"\nservices:\n - docker\nbefore_install:\n - pip install -U pip setuptools\ninstall:\n - pip install -Ur requirements.txt\n - pip install .\n - python setup.py develop\nscript:\n - mypy --ignore-missing-imports py_backwards\n - py.test -vvvv --capture=sys --enable-functional\n" }, { "path": "Dockerfile", "content": "FROM python:3.6\nMAINTAINER Vladimir Iakovlev \n\nCOPY . /src/\nRUN pip install /src\n\nWORKDIR /data/\n\nENTRYPOINT [\"py-backwards\"]\n" }, { "path": "README.md", "content": "# Py-backwards [![Build Status](https://travis-ci.org/nvbn/py-backwards.svg?branch=master)](https://travis-ci.org/nvbn/py-backwards)\n\nPython to python compiler that allows you to use some Python 3.6 features in older versions, you can try it in [the online demo](https://py-backwards.herokuapp.com/).\n\nRequires Python 3.3+ to run, can compile down to 2.7.\n\n## Supported features\n\nTarget 3.5:\n* [formatted string literals](https://docs.python.org/3/whatsnew/3.6.html#pep-498-formatted-string-literals) like `f'hi {x}'`\n* [variables annotations](https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep526) like `x: int = 10` and `x: int`\n* [underscores in numeric literals](https://docs.python.org/3/whatsnew/3.6.html#pep-515-underscores-in-numeric-literals) like `1_000_000` (works automatically)\n\nTarget 3.4:\n* [starred unpacking](https://docs.python.org/3/whatsnew/3.5.html#pep-448-additional-unpacking-generalizations) like `[*range(1, 5), *range(10, 15)]` and `print(*[1, 2], 3, *[4, 5])`\n* [dict unpacking](https://docs.python.org/3/whatsnew/3.5.html#pep-448-additional-unpacking-generalizations) like `{1: 2, **{3: 4}}`\n\nTarget 3.3:\n* import [pathlib2](https://pypi.python.org/pypi/pathlib2/) instead of pathlib\n\nTarget 3.2:\n* [yield from](https://docs.python.org/3/whatsnew/3.3.html#pep-380)\n* [return from generator](https://docs.python.org/3/whatsnew/3.3.html#pep-380)\n\nTarget 2.7:\n* [functions annotations](https://www.python.org/dev/peps/pep-3107/) like `def fn(a: int) -> str`\n* [imports from `__future__`](https://docs.python.org/3/howto/pyporting.html#prevent-compatibility-regressions)\n* [super without arguments](https://www.python.org/dev/peps/pep-3135/)\n* classes without base like `class A: pass`\n* imports from [six moves](https://pythonhosted.org/six/#module-six.moves)\n* metaclass\n* string/unicode literals (works automatically)\n* `str` to `unicode`\n* define encoding (not transformer)\n* `dbm => anydbm` and `dbm.ndbm => dbm`\n\nFor example, if you have some python 3.6 code, like:\n\n```python\ndef returning_range(x: int):\n yield from range(x)\n return x\n\n\ndef x_printer(x):\n val: int\n val = yield from returning_range(x)\n print(f'val {val}')\n\n\ndef formatter(x: int) -> dict:\n items: list = [*x_printer(x), x]\n print(*items, *items)\n return {'items': items}\n\n\nresult = {'x': 10, **formatter(10)}\nprint(result)\n\n\nclass NumberManager:\n def ten(self):\n return 10\n\n @classmethod\n def eleven(cls):\n return 11\n\n\nclass ImportantNumberManager(NumberManager):\n def ten(self):\n return super().ten()\n\n @classmethod\n def eleven(cls):\n return super().eleven()\n\n\nprint(ImportantNumberManager().ten())\nprint(ImportantNumberManager.eleven())\n```\n\nYou can compile it for python 2.7 with:\n\n```bash\n➜ py-backwards -i input.py -o output.py -t 2.7\n```\n\nGot some [ugly code](https://gist.github.com/nvbn/51b1536dc05bddc09439f848461cef6a) and ensure that it works:\n\n```bash\n➜ python3.6 input.py\nval 10\n0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10\n{'x': 10, 'items': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}\n10\n11\n➜ python2 output.py \nval 10\n0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10\n{'x': 10, 'items': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}\n10\n11\n```\n\n## Usage\n\nInstallation:\n\n```bash\npip install py-backwards\n```\n\nCompile code:\n\n```bash\npy-backwards -i src -o compiled -t 2.7\n```\n\n### Testing compiled code\n\nFor testing compiled code with each supported python version you can use [tox](https://tox.readthedocs.io/en/latest/)\nand [tox-py-backwards](https://github.com/nvbn/tox-py-backwards). You need to install them:\n\n```bash\npip install tox tox-py-backwards\n```\n\nFill `tox.ini` (`py_backwards = true` in `testenv` section enables py-backwards), like:\n\n```ini\n[tox]\nenvlist = py27,py33,py34,py35,py36\n\n[testenv]\ndeps = pytest\ncommands = py.test\npy_backwards = true\n```\n\nAnd run tests with:\n\n```bash\ntox\n```\n\n### Distributing compiled code\n\nFor distributing packages compiled with py-backwards you can use [py-backwards-packager](https://github.com/nvbn/py-backwards-packager).\nInstall it with:\n\n```python\npip install py-backwards-packager\n```\n\nAnd change `setup` import in `setup.py` to:\n \n```python\ntry:\n from py_backwards_packager import setup\nexcept ImportError:\n from setuptools import setup\n```\n\nBy default all targets enabled, but you can limit them with:\n \n```python\nsetup(...,\n py_backwards_targets=['2.7', '3.3'])\n```\n\nAfter that your code will be automatically compiled on `bdist` and `bdist_wheel`.\n\n### Running on systems without Python 3.3+\n\nYou can use docker for running py-backwards on systems without Python 3.3+, for example\nfor testing on travis-ci with Python 2.7:\n\n```bash\ndocker run -v $(pwd):/data/ nvbn/py-backwards -i example -o out -t 2.7\n```\n\n## Development\n\nSetup:\n\n```bash\npip install .\npython setup.py develop\npip install -r requirements.txt\n```\n\nRun tests:\n\n```bash\n py.test -vvvv --capture=sys --enable-functional\n```\n\nRun tests on systems without docker:\n\n```bash\n py.test -vvvv\n```\n\n## Writing code transformers\n\nFirst of all, you need to inherit from `BaseTransformer`, `BaseNodeTransformer` (if you want to use\n[NodeTransfromer](https://docs.python.org/3/library/ast.html#ast.NodeTransformer) interface),\nor `BaseImportRewrite` (if you want just to change import).\n\nIf you use `BaseTransformer`, override class method `def transform(cls, tree: ast.AST) -> TransformationResult`, like:\n\n```python\nfrom ..types import TransformationResult\nfrom .base import BaseTransformer\n\n\nclass MyTransformer(BaseTransformer):\n @classmethod\n def transform(cls, tree: ast.AST) -> TransformationResult:\n return TransformationResult(tree=tree,\n tree_changed=True,\n dependencies=[])\n```\n\nIf you use `BaseNodeTransformer`, override `visit_*` methods, for simplification this class\nhave a whole tree in `self._tree`, you should also set `self._tree_changed = True` if the tree\nwas changed:\n\n```python\nfrom .base import BaseNodeTransformer\n\n\nclass MyTransformer(BaseNodeTransformer):\n dependencies = [] # additional dependencies\n\n def visit_FunctionDef(self, node: ast.FunctionDef) -> ast.FunctionDef:\n self._tree_changed = True # Mark that transformer changed tree\n return self.generic_visit(node)\n```\n\nIf you use `BaseImportRewrite`, just override `rewrites`, like:\n\n```python\nfrom .base import BaseImportRewrite\n\n\nclass MyTransformer(BaseImportRewrite):\n dependencies = ['pathlib2']\n\n rewrites = [('pathlib', 'pathlib2')]\n```\n\nAfter that you need to add your transformer to `transformers.__init__.transformers`.\n\nIt's hard to write code in AST, because of that we have [snippets](https://github.com/nvbn/py-backwards/blob/master/py_backwards/utils/snippet.py#L102):\n\n```python\nfrom ..utils.snippet import snippet, let, extend\n\n\n@snippet\ndef my_snippet(class_name, class_body):\n class class_name: # will be replaced with `class_name`\n extend(class_body) # body of the class will be extended with `class_body`\n \n def fn(self):\n let(x) # x will be replaced everywhere with unique name, like `_py_backwards_x_1`\n x = 10\n return x\n```\n\nAnd you can easily get content of snippet with:\n\n```python\nmy_snippet.get_body(class_name='MyClass',\n class_body=[ast.Expr(...), ...])\n```\n\nAlso please look at [tree utils](https://github.com/nvbn/py-backwards/blob/master/py_backwards/utils/tree.py),\nit contains such useful functions like `find`, `get_parent` and etc.\n\n## Related projects\n\n* [py-backwards-astunparse](https://github.com/nvbn/py-backwards-astunparse)\n* [tox-py-backwards](https://github.com/nvbn/tox-py-backwards)\n* [py-backwards-packager](https://github.com/nvbn/py-backwards-packager)\n* [pytest-docker-pexpect](https://github.com/nvbn/pytest-docker-pexpect)\n\n## License MIT\n" }, { "path": "py_backwards/__init__.py", "content": "" }, { "path": "py_backwards/compiler.py", "content": "from copy import deepcopy\nfrom time import time\nfrom traceback import format_exc\nfrom typing import List, Tuple, Optional\nfrom typed_ast import ast3 as ast\nfrom astunparse import unparse, dump\nfrom autopep8 import fix_code\nfrom .files import get_input_output_paths, InputOutput\nfrom .transformers import transformers\nfrom .types import CompilationTarget, CompilationResult\nfrom .exceptions import CompilationError, TransformationError\nfrom .utils.helpers import debug\nfrom . import const\n\n\ndef _transform(path: str, code: str, target: CompilationTarget) -> Tuple[str, List[str]]:\n \"\"\"Applies all transformation for passed target.\"\"\"\n debug(lambda: 'Compiling \"{}\"'.format(path))\n dependencies = [] # type: List[str]\n tree = ast.parse(code, path)\n debug(lambda: 'Initial ast:\\n{}'.format(dump(tree)))\n\n for transformer in transformers:\n if transformer.target < target:\n debug(lambda: 'Skip transformer \"{}\"'.format(transformer.__name__))\n continue\n\n debug(lambda: 'Use transformer \"{}\"'.format(transformer.__name__))\n\n working_tree = deepcopy(tree)\n try:\n result = transformer.transform(working_tree)\n except:\n raise TransformationError(path, transformer,\n dump(tree), format_exc())\n\n if not result.tree_changed:\n debug(lambda: 'Tree not changed')\n continue\n\n tree = working_tree\n debug(lambda: 'Tree changed:\\n{}'.format(dump(tree)))\n dependencies.extend(result.dependencies)\n\n try:\n code = unparse(tree)\n debug(lambda: 'Code changed:\\n{}'.format(code))\n except:\n raise TransformationError(path, transformer,\n dump(tree), format_exc())\n\n return fix_code(code), dependencies\n\n\ndef _compile_file(paths: InputOutput, target: CompilationTarget) -> List[str]:\n \"\"\"Compiles a single file.\"\"\"\n with paths.input.open() as f:\n code = f.read()\n\n try:\n transformed, dependencies = _transform(paths.input.as_posix(),\n code, target)\n except SyntaxError as e:\n raise CompilationError(paths.input.as_posix(),\n code, e.lineno, e.offset)\n\n try:\n paths.output.parent.mkdir(parents=True)\n except FileExistsError:\n pass\n\n if target == const.TARGETS['2.7']:\n transformed = '# -*- coding: utf-8 -*-\\n{}'.format(transformed)\n\n with paths.output.open('w') as f:\n f.write(transformed)\n\n return dependencies\n\n\ndef compile_files(input_: str, output: str, target: CompilationTarget,\n root: Optional[str] = None) -> CompilationResult:\n \"\"\"Compiles all files from input_ to output.\"\"\"\n dependencies = set()\n start = time()\n count = 0\n for paths in get_input_output_paths(input_, output, root):\n count += 1\n dependencies.update(_compile_file(paths, target))\n return CompilationResult(count, time() - start, target,\n sorted(dependencies))\n" }, { "path": "py_backwards/conf.py", "content": "from argparse import Namespace\n\n\nclass Settings:\n def __init__(self) -> None:\n self.debug = False\n\n\nsettings = Settings()\n\n\ndef init_settings(args: Namespace) -> None:\n if args.debug:\n settings.debug = True\n" }, { "path": "py_backwards/const.py", "content": "from collections import OrderedDict\n\nTARGETS = OrderedDict([('2.7', (2, 7)),\n ('3.0', (3, 0)),\n ('3.1', (3, 1)),\n ('3.2', (3, 2)),\n ('3.3', (3, 3)),\n ('3.4', (3, 4)),\n ('3.5', (3, 5)),\n ('3.6', (3, 6))])\n\nSYNTAX_ERROR_OFFSET = 5\n\nTARGET_ALL = (9999, 9999)\n" }, { "path": "py_backwards/exceptions.py", "content": "from typing import Type, TYPE_CHECKING\n\nif TYPE_CHECKING:\n from .transformers.base import BaseTransformer\n\n\nclass CompilationError(Exception):\n \"\"\"Raises when compilation failed because fo syntax error.\"\"\"\n\n def __init__(self, filename: str, code: str,\n lineno: int, offset: int) -> None:\n self.filename = filename\n self.code = code\n self.lineno = lineno\n self.offset = offset\n\n\nclass TransformationError(Exception):\n \"\"\"Raises when transformation failed.\"\"\"\n\n def __init__(self, filename: str,\n transformer: 'Type[BaseTransformer]',\n ast: str,\n traceback: str) -> None:\n self.filename = filename\n self.transformer = transformer\n self.ast = ast\n self.traceback = traceback\n\n\nclass InvalidInputOutput(Exception):\n \"\"\"Raises when input is a directory, but output is a file.\"\"\"\n\n\nclass InputDoesntExists(Exception):\n \"\"\"Raises when input doesn't exists.\"\"\"\n\n\nclass NodeNotFound(Exception):\n \"\"\"Raises when node not found.\"\"\"\n" }, { "path": "py_backwards/files.py", "content": "from typing import Iterable, Optional\n\ntry:\n from pathlib import Path\nexcept ImportError:\n from pathlib2 import Path # type: ignore\n\nfrom .types import InputOutput\nfrom .exceptions import InvalidInputOutput, InputDoesntExists\n\n\ndef get_input_output_paths(input_: str, output: str,\n root: Optional[str]) -> Iterable[InputOutput]:\n \"\"\"Get input/output paths pairs.\"\"\"\n if output.endswith('.py') and not input_.endswith('.py'):\n raise InvalidInputOutput\n\n if not Path(input_).exists():\n raise InputDoesntExists\n\n if input_.endswith('.py'):\n if output.endswith('.py'):\n yield InputOutput(Path(input_), Path(output))\n else:\n input_path = Path(input_)\n if root is None:\n output_path = Path(output).joinpath(input_path.name)\n else:\n output_path = Path(output).joinpath(input_path.relative_to(root))\n yield InputOutput(input_path, output_path)\n else:\n output_path = Path(output)\n input_path = Path(input_)\n root_path = input_path if root is None else Path(root)\n for child_input in input_path.glob('**/*.py'):\n child_output = output_path.joinpath(\n child_input.relative_to(root_path))\n yield InputOutput(child_input, child_output)\n" }, { "path": "py_backwards/main.py", "content": "from colorama import init\n\ninit()\n\nfrom argparse import ArgumentParser\nimport sys\nfrom .compiler import compile_files\nfrom .conf import init_settings\nfrom . import const, messages, exceptions\n\n\ndef main() -> int:\n parser = ArgumentParser(\n 'py-backwards',\n description='Python to python compiler that allows you to use some '\n 'Python 3.6 features in older versions.')\n parser.add_argument('-i', '--input', type=str, nargs='+', required=True,\n help='input file or folder')\n parser.add_argument('-o', '--output', type=str, required=True,\n help='output file or folder')\n parser.add_argument('-t', '--target', type=str,\n required=True, choices=const.TARGETS.keys(),\n help='target python version')\n parser.add_argument('-r', '--root', type=str, required=False,\n help='sources root')\n parser.add_argument('-d', '--debug', action='store_true', required=False,\n help='enable debug output')\n args = parser.parse_args()\n init_settings(args)\n\n try:\n for input_ in args.input:\n result = compile_files(input_, args.output,\n const.TARGETS[args.target],\n args.root)\n except exceptions.CompilationError as e:\n print(messages.syntax_error(e), file=sys.stderr)\n return 1\n except exceptions.TransformationError as e:\n print(messages.transformation_error(e), file=sys.stderr)\n return 1\n except exceptions.InputDoesntExists:\n print(messages.input_doesnt_exists(args.input), file=sys.stderr)\n return 1\n except exceptions.InvalidInputOutput:\n print(messages.invalid_output(args.input, args.output),\n file=sys.stderr)\n return 1\n except PermissionError:\n print(messages.permission_error(args.output), file=sys.stderr)\n return 1\n\n print(messages.compilation_result(result))\n return 0\n" }, { "path": "py_backwards/messages.py", "content": "from typing import Iterable\nfrom colorama import Fore, Style\nfrom .exceptions import CompilationError, TransformationError\nfrom .types import CompilationResult\nfrom . import const\n\n\ndef _format_line(line: str, n: int, padding: int) -> str:\n \"\"\"Format single line of code.\"\"\"\n return ' {dim}{n}{reset}: {line}'.format(dim=Style.DIM,\n n=str(n + 1).zfill(padding),\n line=line,\n reset=Style.RESET_ALL)\n\n\ndef _get_lines_with_highlighted_error(e: CompilationError) -> Iterable[str]:\n \"\"\"Format code with highlighted syntax error.\"\"\"\n error_line = e.lineno - 1\n lines = e.code.split('\\n')\n padding = len(str(len(lines)))\n\n from_line = error_line - const.SYNTAX_ERROR_OFFSET\n if from_line < 0:\n from_line = 0\n\n if from_line < error_line:\n for n in range(from_line, error_line):\n yield _format_line(lines[n], n, padding)\n\n yield ' {dim}{n}{reset}: {bright}{line}{reset}'.format(\n dim=Style.DIM,\n n=str(error_line + 1).zfill(padding),\n line=lines[error_line],\n reset=Style.RESET_ALL,\n bright=Style.BRIGHT)\n yield ' {padding}{bright}^{reset}'.format(\n padding=' ' * (padding + e.offset + 1),\n bright=Style.BRIGHT,\n reset=Style.RESET_ALL)\n\n to_line = error_line + const.SYNTAX_ERROR_OFFSET\n if to_line > len(lines):\n to_line = len(lines)\n for n in range(error_line + 1, to_line):\n yield _format_line(lines[n], n, padding)\n\n\ndef syntax_error(e: CompilationError) -> str:\n lines = _get_lines_with_highlighted_error(e)\n\n return ('{red}Syntax error in \"{e.filename}\", '\n 'line {e.lineno}, pos {e.offset}:{reset}\\n{lines}').format(\n red=Fore.RED,\n e=e,\n reset=Style.RESET_ALL,\n bright=Style.BRIGHT,\n lines='\\n'.join(lines))\n\n\ndef transformation_error(e: TransformationError) -> str:\n return ('{red}{bright}Transformation error in \"{e.filename}\", '\n 'transformer \"{e.transformer.__name__}\" '\n 'failed with:{reset}\\n{e.traceback}\\n'\n '{bright}AST:{reset}\\n{e.ast}').format(\n red=Fore.RED,\n e=e,\n reset=Style.RESET_ALL,\n bright=Style.BRIGHT)\n\n\ndef input_doesnt_exists(input_: str) -> str:\n return '{red}Input path \"{path}\" doesn\\'t exists{reset}'.format(\n red=Fore.RED, path=input_, reset=Style.RESET_ALL)\n\n\ndef invalid_output(input_: str, output: str) -> str:\n return ('{red}Invalid output, when input \"{input}\" is a directory,'\n 'output \"{output}\" should be a directory too{reset}').format(\n red=Fore.RED, input=input_, output=output, reset=Style.RESET_ALL)\n\n\ndef permission_error(output: str) -> str:\n return '{red}Permission denied to \"{output}\"{reset}'.format(\n red=Fore.RED, output=output, reset=Style.RESET_ALL)\n\n\ndef compilation_result(result: CompilationResult) -> str:\n if result.dependencies:\n dependencies = ('\\n Additional dependencies:\\n'\n '{bright} {dependencies}{reset}').format(\n dependencies='\\n '.join(dep for dep in result.dependencies),\n bright=Style.BRIGHT,\n reset=Style.RESET_ALL)\n else:\n dependencies = ''\n\n return ('{bright}Compilation succeed{reset}:\\n'\n ' target: {bright}{target}{reset}\\n'\n ' files: {bright}{files}{reset}\\n'\n ' took: {bright}{time:.2f}{reset} seconds{dependencies}').format(\n bright=Style.BRIGHT,\n reset=Style.RESET_ALL,\n target='{}.{}'.format(*result.target),\n files=result.files,\n time=result.time,\n dependencies=dependencies)\n\n\ndef warn(message: str) -> str:\n return '{bright}{red}WARN:{reset} {message}'.format(\n bright=Style.BRIGHT,\n red=Fore.RED,\n reset=Style.RESET_ALL,\n message=message)\n\n\ndef debug(message: str) -> str:\n return '{bright}{blue}DEBUG:{reset} {message}'.format(\n bright=Style.BRIGHT,\n blue=Fore.BLUE,\n reset=Style.RESET_ALL,\n message=message)\n" }, { "path": "py_backwards/transformers/__init__.py", "content": "from typing import List, Type\nfrom .dict_unpacking import DictUnpackingTransformer\nfrom .formatted_values import FormattedValuesTransformer\nfrom .functions_annotations import FunctionsAnnotationsTransformer\nfrom .starred_unpacking import StarredUnpackingTransformer\nfrom .variables_annotations import VariablesAnnotationsTransformer\nfrom .yield_from import YieldFromTransformer\nfrom .return_from_generator import ReturnFromGeneratorTransformer\nfrom .python2_future import Python2FutureTransformer\nfrom .super_without_arguments import SuperWithoutArgumentsTransformer\nfrom .class_without_bases import ClassWithoutBasesTransformer\nfrom .import_pathlib import ImportPathlibTransformer\nfrom .six_moves import SixMovesTransformer\nfrom .metaclass import MetaclassTransformer\nfrom .string_types import StringTypesTransformer\nfrom .import_dbm import ImportDbmTransformer\nfrom .base import BaseTransformer\n\ntransformers = [\n # 3.5\n VariablesAnnotationsTransformer,\n FormattedValuesTransformer,\n # 3.4\n DictUnpackingTransformer,\n StarredUnpackingTransformer,\n # 3.2\n YieldFromTransformer,\n ReturnFromGeneratorTransformer,\n # 2.7\n FunctionsAnnotationsTransformer,\n SuperWithoutArgumentsTransformer,\n ClassWithoutBasesTransformer,\n ImportPathlibTransformer,\n SixMovesTransformer,\n MetaclassTransformer,\n StringTypesTransformer,\n ImportDbmTransformer,\n Python2FutureTransformer, # always should be the last transformer\n] # type: List[Type[BaseTransformer]]\n" }, { "path": "py_backwards/transformers/base.py", "content": "from abc import ABCMeta, abstractmethod\nfrom typing import List, Tuple, Union, Optional, Iterable, Dict\nfrom typed_ast import ast3 as ast\nfrom ..types import CompilationTarget, TransformationResult\nfrom ..utils.snippet import snippet, extend\n\n\nclass BaseTransformer(metaclass=ABCMeta):\n target = None # type: CompilationTarget\n\n @classmethod\n @abstractmethod\n def transform(cls, tree: ast.AST) -> TransformationResult:\n ...\n\n\nclass BaseNodeTransformer(BaseTransformer, ast.NodeTransformer):\n dependencies = [] # type: List[str]\n\n def __init__(self, tree: ast.AST) -> None:\n super().__init__()\n self._tree = tree\n self._tree_changed = False\n\n @classmethod\n def transform(cls, tree: ast.AST) -> TransformationResult:\n inst = cls(tree)\n inst.visit(tree)\n return TransformationResult(tree, inst._tree_changed, cls.dependencies)\n\n\n@snippet\ndef import_rewrite(previous, current):\n try:\n extend(previous)\n except ImportError:\n extend(current)\n\n\nclass BaseImportRewrite(BaseNodeTransformer):\n rewrites = [] # type: List[Tuple[str, str]]\n wrapper = import_rewrite # type: snippet\n\n def _get_matched_rewrite(self, name: Optional[str]) -> Optional[Tuple[str, str]]:\n \"\"\"Returns rewrite for module name.\"\"\"\n if name is None:\n return None\n\n for from_, to in self.rewrites:\n if name == from_ or name.startswith(from_ + '.'):\n return from_, to\n\n return None\n\n def _replace_import(self, node: ast.Import, from_: str, to: str) -> ast.Try:\n \"\"\"Replace import with try/except with old and new import.\"\"\"\n self._tree_changed = True\n\n rewrote_name = node.names[0].name.replace(from_, to, 1)\n import_as = node.names[0].asname or node.names[0].name.split('.')[-1]\n\n rewrote = ast.Import(names=[\n ast.alias(name=rewrote_name,\n asname=import_as)])\n\n return self.wrapper.get_body(previous=node, # type: ignore\n current=rewrote)[0]\n\n def visit_Import(self, node: ast.Import) -> Union[ast.Import, ast.Try]:\n rewrite = self._get_matched_rewrite(node.names[0].name)\n if rewrite:\n return self._replace_import(node, *rewrite)\n\n return self.generic_visit(node)\n\n def _replace_import_from_module(self, node: ast.ImportFrom, from_: str, to: str) -> ast.Try:\n \"\"\"Replaces import from with try/except with old and new import module.\"\"\"\n self._tree_changed = True\n\n rewrote_module = node.module.replace(from_, to, 1)\n rewrote = ast.ImportFrom(module=rewrote_module,\n names=node.names,\n level=node.level)\n\n return self.wrapper.get_body(previous=node, # type: ignore\n current=rewrote)[0]\n\n def _get_names_to_replace(self, node: ast.ImportFrom) -> Iterable[Tuple[str, Tuple[str, str]]]:\n \"\"\"Finds names/aliases to replace.\"\"\"\n for alias in node.names:\n full_name = '{}.{}'.format(node.module, alias.name)\n if alias.name != '*':\n rewrite = self._get_matched_rewrite(full_name)\n if rewrite:\n yield (full_name, rewrite)\n\n def _get_replaced_import_from_part(self, node: ast.ImportFrom, alias: ast.alias,\n names_to_replace: Dict[str, Tuple[str, str]]) -> ast.ImportFrom:\n \"\"\"Returns import from statement with changed module or alias.\"\"\"\n full_name = '{}.{}'.format(node.module, alias.name)\n if full_name in names_to_replace:\n full_name = full_name.replace(names_to_replace[full_name][0],\n names_to_replace[full_name][1],\n 1)\n module_name = '.'.join(full_name.split('.')[:-1])\n name = full_name.split('.')[-1]\n return ast.ImportFrom(\n module=module_name,\n names=[ast.alias(name=name,\n asname=alias.asname or alias.name)],\n level=node.level)\n\n def _replace_import_from_names(self, node: ast.ImportFrom,\n names_to_replace: Dict[str, Tuple[str, str]]) -> ast.Try:\n \"\"\"Replaces import from with try/except with old and new \n import module and names.\n \n \"\"\"\n self._tree_changed = True\n\n rewrotes = [\n self._get_replaced_import_from_part(node, alias, names_to_replace)\n for alias in node.names]\n\n return self.wrapper.get_body(previous=node, # type: ignore\n current=rewrotes)[0]\n\n def visit_ImportFrom(self, node: ast.ImportFrom) -> Union[ast.ImportFrom, ast.Try, ast.AST]:\n rewrite = self._get_matched_rewrite(node.module)\n if rewrite:\n return self._replace_import_from_module(node, *rewrite)\n\n names_to_replace = dict(self._get_names_to_replace(node))\n if names_to_replace:\n return self._replace_import_from_names(node, names_to_replace)\n\n return self.generic_visit(node)\n" }, { "path": "py_backwards/transformers/class_without_bases.py", "content": "from typed_ast import ast3 as ast\nfrom .base import BaseNodeTransformer\n\n\nclass ClassWithoutBasesTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n class A:\n pass\n To:\n class A(object)\n \n \"\"\"\n target = (2, 7)\n\n def visit_ClassDef(self, node: ast.ClassDef) -> ast.ClassDef:\n if not node.bases:\n node.bases = [ast.Name(id='object')]\n self._tree_changed = True\n\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/dict_unpacking.py", "content": "from typing import Union, Iterable, Optional, List, Tuple\nfrom typed_ast import ast3 as ast\nfrom ..utils.tree import insert_at\nfrom ..utils.snippet import snippet\nfrom .base import BaseNodeTransformer\n\n\n@snippet\ndef merge_dicts():\n def _py_backwards_merge_dicts(dicts):\n result = {}\n for dict_ in dicts:\n result.update(dict_)\n return result\n\n\nSplitted = List[Union[List[Tuple[ast.expr, ast.expr]], ast.expr]]\nPair = Tuple[Optional[ast.expr], ast.expr]\n\n\nclass DictUnpackingTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n \n {1: 1, **dict_a}\n \n To:\n \n _py_backwards_merge_dicts([{1: 1}], dict_a})\n \n \"\"\"\n target = (3, 4)\n\n def _split_by_None(self, pairs: Iterable[Pair]) -> Splitted:\n \"\"\"Splits pairs to lists separated by dict unpacking statements.\"\"\"\n result = [[]] # type: Splitted\n for key, value in pairs:\n if key is None:\n result.append(value)\n result.append([])\n else:\n assert isinstance(result[-1], list)\n result[-1].append((key, value))\n\n return result\n\n def _prepare_splitted(self, splitted: Splitted) \\\n -> Iterable[Union[ast.Call, ast.Dict]]:\n \"\"\"Wraps splitted in Call or Dict.\"\"\"\n for group in splitted:\n if not isinstance(group, list):\n yield ast.Call(\n func=ast.Name(id='dict'),\n args=[group],\n keywords=[])\n elif group:\n yield ast.Dict(keys=[key for key, _ in group],\n values=[value for _, value in group])\n\n def _merge_dicts(self, xs: Iterable[Union[ast.Call, ast.Dict]]) \\\n -> ast.Call:\n \"\"\"Creates call of function for merging dicts.\"\"\"\n return ast.Call(\n func=ast.Name(id='_py_backwards_merge_dicts'),\n args=[ast.List(elts=list(xs))],\n keywords=[])\n\n def visit_Module(self, node: ast.Module) -> ast.Module:\n insert_at(0, node, merge_dicts.get_body()) # type: ignore\n return self.generic_visit(node) # type: ignore\n\n def visit_Dict(self, node: ast.Dict) -> Union[ast.Dict, ast.Call]:\n if None not in node.keys:\n return self.generic_visit(node) # type: ignore\n\n self._tree_changed = True\n pairs = zip(node.keys, node.values)\n splitted = self._split_by_None(pairs)\n prepared = self._prepare_splitted(splitted)\n return self._merge_dicts(prepared)\n" }, { "path": "py_backwards/transformers/formatted_values.py", "content": "from typed_ast import ast3 as ast\nfrom ..const import TARGET_ALL\nfrom .base import BaseNodeTransformer\n\n\nclass FormattedValuesTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n f\"hello {x}\"\n To\n ''.join(['hello ', '{}'.format(x)])\n \n \"\"\"\n target = TARGET_ALL\n\n def visit_FormattedValue(self, node: ast.FormattedValue) -> ast.Call:\n self._tree_changed = True\n\n if node.format_spec:\n template = ''.join(['{:', node.format_spec.s, '}']) # type: ignore\n else:\n template = '{}'\n\n format_call = ast.Call(func=ast.Attribute(value=ast.Str(s=template),\n attr='format'),\n args=[node.value],\n keywords=[])\n return self.generic_visit(format_call) # type: ignore\n\n def visit_JoinedStr(self, node: ast.JoinedStr) -> ast.Call:\n self._tree_changed = True\n\n join_call = ast.Call(func=ast.Attribute(value=ast.Str(s=''),\n attr='join'),\n args=[ast.List(elts=node.values)],\n keywords=[])\n return self.generic_visit(join_call) # type: ignore\n" }, { "path": "py_backwards/transformers/functions_annotations.py", "content": "from typed_ast import ast3 as ast\nfrom .base import BaseNodeTransformer\n\n\nclass FunctionsAnnotationsTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n def fn(x: int) -> int:\n pass\n To:\n def fn(x):\n pass\n \n \"\"\"\n target = (2, 7)\n\n def visit_arg(self, node: ast.arg) -> ast.arg:\n self._tree_changed = True\n node.annotation = None\n return self.generic_visit(node) # type: ignore\n\n def visit_FunctionDef(self, node: ast.FunctionDef):\n self._tree_changed = True\n node.returns = None\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/import_dbm.py", "content": "from typing import Union\nfrom typed_ast import ast3 as ast\nfrom ..utils.snippet import snippet, extend\nfrom .base import BaseImportRewrite\n\n\n@snippet\ndef import_rewrite(previous, current):\n if __import__('six').PY2:\n extend(current)\n else:\n extend(previous)\n\n\nclass ImportDbmTransformer(BaseImportRewrite):\n \"\"\"Replaces:\n \n dbm => anydbm\n dbm.ndbm => dbm\n\n \"\"\"\n target = (2, 7)\n rewrites = [('dbm.ndbm', 'dbm'),\n ('dbm', 'anydbm')]\n wrapper = import_rewrite\n dependencies = ['six']\n\n def visit_Import(self, node: ast.Import) -> Union[ast.Import, ast.Try]:\n if node.names[0].name == 'dbm' and node.names[0].asname == 'ndbm':\n return node\n\n return super().visit_Import(node)\n\n def visit_ImportFrom(self, node: ast.ImportFrom) -> Union[ast.ImportFrom, ast.Try, ast.AST]:\n names = [name.name for name in node.names]\n if node.module == 'dbm' and names == ['ndbm']:\n import_ = ast.Import(names=[ast.alias(name='dbm',\n asname='ndbm')])\n return self.wrapper.get_body(previous=node, current=import_)[0] # type: ignore\n\n return super().visit_ImportFrom(node)\n" }, { "path": "py_backwards/transformers/import_pathlib.py", "content": "from .base import BaseImportRewrite\n\n\nclass ImportPathlibTransformer(BaseImportRewrite):\n \"\"\"Replaces pathlib with backported pathlib2.\"\"\"\n target = (3, 3)\n rewrites = [('pathlib', 'pathlib2')]\n dependencies = ['pathlib2']\n" }, { "path": "py_backwards/transformers/metaclass.py", "content": "from typed_ast import ast3 as ast\nfrom ..utils.snippet import snippet\nfrom ..utils.tree import insert_at\nfrom .base import BaseNodeTransformer\n\n\n@snippet\ndef six_import():\n from six import with_metaclass as _py_backwards_six_withmetaclass\n\n\n@snippet\ndef class_bases(metaclass, bases):\n _py_backwards_six_withmetaclass(metaclass, *bases)\n\n\nclass MetaclassTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n class A(metaclass=B):\n pass\n To:\n class A(_py_backwards_six_with_metaclass(B))\n \n \"\"\"\n target = (2, 7)\n dependencies = ['six']\n\n def visit_Module(self, node: ast.Module) -> ast.Module:\n insert_at(0, node, six_import.get_body())\n return self.generic_visit(node) # type: ignore\n\n def visit_ClassDef(self, node: ast.ClassDef) -> ast.ClassDef:\n if node.keywords:\n metaclass = node.keywords[0].value\n node.bases = class_bases.get_body(metaclass=metaclass, # type: ignore\n bases=ast.List(elts=node.bases))\n node.keywords = []\n self._tree_changed = True\n\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/python2_future.py", "content": "from typed_ast import ast3 as ast\nfrom ..utils.snippet import snippet\nfrom .base import BaseNodeTransformer\n\n\n@snippet\ndef imports(future):\n from future import absolute_import\n from future import division\n from future import print_function\n from future import unicode_literals\n\n\nclass Python2FutureTransformer(BaseNodeTransformer):\n \"\"\"Prepends module with:\n from __future__ import absolute_import\n from __future__ import division\n from __future__ import print_function\n from __future__ import unicode_literals\n \n \"\"\"\n target = (2, 7)\n\n def visit_Module(self, node: ast.Module) -> ast.Module:\n self._tree_changed = True\n node.body = imports.get_body(future='__future__') + node.body # type: ignore\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/return_from_generator.py", "content": "from typing import List, Tuple, Any\nfrom typed_ast import ast3 as ast\nfrom ..utils.snippet import snippet, let\nfrom .base import BaseNodeTransformer\n\n\n@snippet\ndef return_from_generator(return_value):\n let(exc)\n exc = StopIteration()\n exc.value = return_value\n raise exc\n\n\nclass ReturnFromGeneratorTransformer(BaseNodeTransformer):\n \"\"\"Compiles return in generators like:\n def fn():\n yield 1\n return 5\n To:\n def fn():\n yield 1\n exc = StopIteration()\n exc.value = 5\n raise exc\n \"\"\"\n target = (3, 2)\n\n def _find_generator_returns(self, node: ast.FunctionDef) \\\n -> List[Tuple[ast.stmt, ast.Return]]:\n \"\"\"Using bfs find all `return` statements in function.\"\"\"\n to_check = [(node, x) for x in node.body] # type: ignore\n returns = []\n has_yield = False\n while to_check:\n parent, current = to_check.pop()\n\n if isinstance(current, ast.FunctionDef):\n continue\n elif hasattr(current, 'value'):\n to_check.append((current, current.value)) # type: ignore\n elif hasattr(current, 'body') and isinstance(current.body, list): # type: ignore\n to_check.extend([(parent, x) for x in current.body]) # type: ignore\n\n if isinstance(current, ast.Yield) or isinstance(current, ast.YieldFrom):\n has_yield = True\n\n if isinstance(current, ast.Return) and current.value is not None:\n returns.append((parent, current))\n\n if has_yield:\n return returns # type: ignore\n else:\n return []\n\n def _replace_return(self, parent: Any, return_: ast.Return) -> None:\n \"\"\"Replace return with exception raising.\"\"\"\n index = parent.body.index(return_)\n parent.body.pop(index)\n\n for line in return_from_generator.get_body(return_value=return_.value)[::-1]:\n parent.body.insert(index, line)\n\n def visit_FunctionDef(self, node: ast.FunctionDef) -> ast.FunctionDef:\n generator_returns = self._find_generator_returns(node)\n\n if generator_returns:\n self._tree_changed = True\n\n for parent, return_ in generator_returns:\n self._replace_return(parent, return_)\n\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/six_moves.py", "content": "# type: ignore\nfrom ..utils.helpers import eager\nfrom .base import BaseImportRewrite\n\n\n# Special class for handling six moves:\nclass MovedAttribute:\n def __init__(self, name, old_mod, new_mod, old_attr=None, new_attr=None):\n self.name = name\n if new_mod is None:\n new_mod = name\n self.new_mod = new_mod\n if new_attr is None:\n if old_attr is None:\n new_attr = name\n else:\n new_attr = old_attr\n self.new_attr = new_attr\n\n\nclass MovedModule:\n def __init__(self, name, old, new=None):\n self.name = name\n if new is None:\n new = name\n self.new = new\n\n\n# Code copied from six:\n\n_moved_attributes = [\n MovedAttribute(\"cStringIO\", \"cStringIO\", \"io\", \"StringIO\"),\n MovedAttribute(\"filter\", \"itertools\", \"builtins\", \"ifilter\", \"filter\"),\n MovedAttribute(\"filterfalse\", \"itertools\", \"itertools\", \"ifilterfalse\", \"filterfalse\"),\n MovedAttribute(\"input\", \"__builtin__\", \"builtins\", \"raw_input\", \"input\"),\n MovedAttribute(\"intern\", \"__builtin__\", \"sys\"),\n MovedAttribute(\"map\", \"itertools\", \"builtins\", \"imap\", \"map\"),\n MovedAttribute(\"getcwd\", \"os\", \"os\", \"getcwdu\", \"getcwd\"),\n MovedAttribute(\"getcwdb\", \"os\", \"os\", \"getcwd\", \"getcwdb\"),\n MovedAttribute(\"getstatusoutput\", \"commands\", \"subprocess\"),\n MovedAttribute(\"getoutput\", \"commands\", \"subprocess\"),\n MovedAttribute(\"range\", \"__builtin__\", \"builtins\", \"xrange\", \"range\"),\n MovedAttribute(\"reload_module\", \"__builtin__\", \"imp\", \"reload\"),\n MovedAttribute(\"reload_module\", \"__builtin__\", \"importlib\", \"reload\"),\n MovedAttribute(\"reduce\", \"__builtin__\", \"functools\"),\n MovedAttribute(\"shlex_quote\", \"pipes\", \"shlex\", \"quote\"),\n MovedAttribute(\"StringIO\", \"StringIO\", \"io\"),\n MovedAttribute(\"UserDict\", \"UserDict\", \"collections\"),\n MovedAttribute(\"UserList\", \"UserList\", \"collections\"),\n MovedAttribute(\"UserString\", \"UserString\", \"collections\"),\n MovedAttribute(\"xrange\", \"__builtin__\", \"builtins\", \"xrange\", \"range\"),\n MovedAttribute(\"zip\", \"itertools\", \"builtins\", \"izip\", \"zip\"),\n MovedAttribute(\"zip_longest\", \"itertools\", \"itertools\", \"izip_longest\", \"zip_longest\"),\n MovedModule(\"builtins\", \"__builtin__\"),\n MovedModule(\"configparser\", \"ConfigParser\"),\n MovedModule(\"copyreg\", \"copy_reg\"),\n MovedModule(\"dbm_gnu\", \"gdbm\", \"dbm.gnu\"),\n MovedModule(\"_dummy_thread\", \"dummy_thread\", \"_dummy_thread\"),\n MovedModule(\"http_cookiejar\", \"cookielib\", \"http.cookiejar\"),\n MovedModule(\"http_cookies\", \"Cookie\", \"http.cookies\"),\n MovedModule(\"html_entities\", \"htmlentitydefs\", \"html.entities\"),\n MovedModule(\"html_parser\", \"HTMLParser\", \"html.parser\"),\n MovedModule(\"http_client\", \"httplib\", \"http.client\"),\n MovedModule(\"email_mime_base\", \"email.MIMEBase\", \"email.mime.base\"),\n MovedModule(\"email_mime_image\", \"email.MIMEImage\", \"email.mime.image\"),\n MovedModule(\"email_mime_multipart\", \"email.MIMEMultipart\", \"email.mime.multipart\"),\n MovedModule(\"email_mime_nonmultipart\", \"email.MIMENonMultipart\", \"email.mime.nonmultipart\"),\n MovedModule(\"email_mime_text\", \"email.MIMEText\", \"email.mime.text\"),\n MovedModule(\"BaseHTTPServer\", \"BaseHTTPServer\", \"http.server\"),\n MovedModule(\"CGIHTTPServer\", \"CGIHTTPServer\", \"http.server\"),\n MovedModule(\"SimpleHTTPServer\", \"SimpleHTTPServer\", \"http.server\"),\n MovedModule(\"cPickle\", \"cPickle\", \"pickle\"),\n MovedModule(\"queue\", \"Queue\"),\n MovedModule(\"reprlib\", \"repr\"),\n MovedModule(\"socketserver\", \"SocketServer\"),\n MovedModule(\"_thread\", \"thread\", \"_thread\"),\n MovedModule(\"tkinter\", \"Tkinter\"),\n MovedModule(\"tkinter_dialog\", \"Dialog\", \"tkinter.dialog\"),\n MovedModule(\"tkinter_filedialog\", \"FileDialog\", \"tkinter.filedialog\"),\n MovedModule(\"tkinter_scrolledtext\", \"ScrolledText\", \"tkinter.scrolledtext\"),\n MovedModule(\"tkinter_simpledialog\", \"SimpleDialog\", \"tkinter.simpledialog\"),\n MovedModule(\"tkinter_tix\", \"Tix\", \"tkinter.tix\"),\n MovedModule(\"tkinter_ttk\", \"ttk\", \"tkinter.ttk\"),\n MovedModule(\"tkinter_constants\", \"Tkconstants\", \"tkinter.constants\"),\n MovedModule(\"tkinter_dnd\", \"Tkdnd\", \"tkinter.dnd\"),\n MovedModule(\"tkinter_colorchooser\", \"tkColorChooser\",\n \"tkinter.colorchooser\"),\n MovedModule(\"tkinter_commondialog\", \"tkCommonDialog\",\n \"tkinter.commondialog\"),\n MovedModule(\"tkinter_tkfiledialog\", \"tkFileDialog\", \"tkinter.filedialog\"),\n MovedModule(\"tkinter_font\", \"tkFont\", \"tkinter.font\"),\n MovedModule(\"tkinter_messagebox\", \"tkMessageBox\", \"tkinter.messagebox\"),\n MovedModule(\"tkinter_tksimpledialog\", \"tkSimpleDialog\",\n \"tkinter.simpledialog\"),\n MovedModule(\"urllib_parse\", __name__ + \".moves.urllib_parse\", \"urllib.parse\"),\n MovedModule(\"urllib_error\", __name__ + \".moves.urllib_error\", \"urllib.error\"),\n MovedModule(\"urllib\", __name__ + \".moves.urllib\", __name__ + \".moves.urllib\"),\n MovedModule(\"urllib_robotparser\", \"robotparser\", \"urllib.robotparser\"),\n MovedModule(\"xmlrpc_client\", \"xmlrpclib\", \"xmlrpc.client\"),\n MovedModule(\"xmlrpc_server\", \"SimpleXMLRPCServer\", \"xmlrpc.server\"),\n]\n\n_moved_attributes += [\n MovedModule(\"winreg\", \"_winreg\"),\n]\n\n_urllib_parse_moved_attributes = [\n MovedAttribute(\"ParseResult\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"SplitResult\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"parse_qs\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"parse_qsl\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"urldefrag\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"urljoin\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"urlparse\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"urlsplit\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"urlunparse\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"urlunsplit\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"quote\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"quote_plus\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"unquote\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"unquote_plus\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"unquote_to_bytes\", \"urllib\", \"urllib.parse\", \"unquote\", \"unquote_to_bytes\"),\n MovedAttribute(\"urlencode\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"splitquery\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"splittag\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"splituser\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"splitvalue\", \"urllib\", \"urllib.parse\"),\n MovedAttribute(\"uses_fragment\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"uses_netloc\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"uses_params\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"uses_query\", \"urlparse\", \"urllib.parse\"),\n MovedAttribute(\"uses_relative\", \"urlparse\", \"urllib.parse\"),\n]\n\n_urllib_error_moved_attributes = [\n MovedAttribute(\"URLError\", \"urllib2\", \"urllib.error\"),\n MovedAttribute(\"HTTPError\", \"urllib2\", \"urllib.error\"),\n MovedAttribute(\"ContentTooShortError\", \"urllib\", \"urllib.error\"),\n]\n\n_urllib_request_moved_attributes = [\n MovedAttribute(\"urlopen\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"install_opener\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"build_opener\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"pathname2url\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"url2pathname\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"getproxies\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"Request\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"OpenerDirector\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPDefaultErrorHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPRedirectHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPCookieProcessor\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"ProxyHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"BaseHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPPasswordMgr\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPPasswordMgrWithDefaultRealm\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"AbstractBasicAuthHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPBasicAuthHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"ProxyBasicAuthHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"AbstractDigestAuthHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPDigestAuthHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"ProxyDigestAuthHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPSHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"FileHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"FTPHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"CacheFTPHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"UnknownHandler\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"HTTPErrorProcessor\", \"urllib2\", \"urllib.request\"),\n MovedAttribute(\"urlretrieve\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"urlcleanup\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"URLopener\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"FancyURLopener\", \"urllib\", \"urllib.request\"),\n MovedAttribute(\"proxy_bypass\", \"urllib\", \"urllib.request\"),\n]\n\n_urllib_response_moved_attributes = [\n MovedAttribute(\"addbase\", \"urllib\", \"urllib.response\"),\n MovedAttribute(\"addclosehook\", \"urllib\", \"urllib.response\"),\n MovedAttribute(\"addinfo\", \"urllib\", \"urllib.response\"),\n MovedAttribute(\"addinfourl\", \"urllib\", \"urllib.response\"),\n]\n\n_urllib_robotparser_moved_attributes = [\n MovedAttribute(\"RobotFileParser\", \"robotparser\", \"urllib.robotparser\"),\n]\n\n# end of code from six\n\nprefixed_moves = [('', _moved_attributes),\n ('.urllib.parse', _urllib_parse_moved_attributes),\n ('.urllib.error', _urllib_error_moved_attributes),\n ('.urllib.request', _urllib_request_moved_attributes),\n ('.urllib.response', _urllib_response_moved_attributes),\n ('.urllib.robotparser', _urllib_robotparser_moved_attributes)]\n\n\n@eager\ndef _get_rewrites():\n for prefix, moves in prefixed_moves:\n for move in moves:\n if isinstance(move, MovedAttribute):\n path = '{}.{}'.format(move.new_mod, move.new_attr)\n yield (path, 'six.moves{}.{}'.format(prefix, move.name))\n elif isinstance(move, MovedModule):\n yield (move.new, 'six.moves{}.{}'.format(prefix, move.name))\n\n\nclass SixMovesTransformer(BaseImportRewrite):\n \"\"\"Replaces moved modules with ones from `six.moves`.\"\"\"\n target = (2, 7)\n rewrites = _get_rewrites()\n dependencies = ['six']\n" }, { "path": "py_backwards/transformers/starred_unpacking.py", "content": "from typing import Union, Iterable, List\nfrom typed_ast import ast3 as ast\nfrom .base import BaseNodeTransformer\n\nSplitted = Union[List[ast.expr], ast.Starred]\nListEntry = Union[ast.Call, ast.List]\n\n\nclass StarredUnpackingTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n [2, *range(10), 1]\n print(*range(1), *range(3))\n To:\n [2] + list(range(10)) + [1]\n print(*(list(range(1)) + list(range(3))))\n \n \"\"\"\n target = (3, 4)\n\n def _has_starred(self, xs: List[ast.expr]) -> bool:\n for x in xs:\n if isinstance(x, ast.Starred):\n return True\n\n return False\n\n def _split_by_starred(self, xs: Iterable[ast.expr]) -> List[Splitted]:\n \"\"\"Split `xs` to separate list by Starred.\"\"\"\n lists = [[]] # type: List[Splitted]\n for x in xs:\n if isinstance(x, ast.Starred):\n lists.append(x)\n lists.append([])\n else:\n assert isinstance(lists[-1], list)\n lists[-1].append(x)\n return lists\n\n def _prepare_lists(self, xs: List[Splitted]) -> Iterable[ListEntry]:\n \"\"\"Wrap starred in list call and list elts to just List.\"\"\"\n for x in xs:\n if isinstance(x, ast.Starred):\n yield ast.Call(\n func=ast.Name(id='list'),\n args=[x.value],\n keywords=[])\n elif x:\n yield ast.List(elts=x)\n\n def _merge_lists(self, xs: List[ListEntry]) -> Union[ast.BinOp, ListEntry]:\n \"\"\"Merge lists by summing them.\"\"\"\n if len(xs) == 1:\n return xs[0]\n\n result = ast.BinOp(left=xs[0], right=xs[1], op=ast.Add())\n for x in xs[2:]:\n result = ast.BinOp(left=result, right=x, op=ast.Add())\n return result\n\n def _to_sum_of_lists(self, xs: List[ast.expr]) -> Union[ast.BinOp, ListEntry]:\n \"\"\"Convert list of arguments / list to sum of lists.\"\"\"\n splitted = self._split_by_starred(xs)\n prepared = list(self._prepare_lists(splitted))\n return self._merge_lists(prepared)\n\n def visit_List(self, node: ast.List) -> ast.List:\n if not self._has_starred(node.elts):\n return self.generic_visit(node) # type: ignore\n\n self._tree_changed = True\n\n return self.generic_visit(self._to_sum_of_lists(node.elts)) # type: ignore\n\n def visit_Call(self, node: ast.Call) -> ast.Call:\n if not self._has_starred(node.args):\n return self.generic_visit(self.generic_visit(node)) # type: ignore\n\n self._tree_changed = True\n\n args = self._to_sum_of_lists(node.args)\n node.args = [ast.Starred(value=args)]\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/string_types.py", "content": "from typed_ast import ast3 as ast\nfrom ..utils.tree import find\nfrom ..types import TransformationResult\nfrom .base import BaseTransformer\n\n\nclass StringTypesTransformer(BaseTransformer):\n \"\"\"Replaces `str` with `unicode`. \n\n \"\"\"\n target = (2, 7)\n\n @classmethod\n def transform(cls, tree: ast.AST) -> TransformationResult:\n tree_changed = False\n\n for node in find(tree, ast.Name):\n if node.id == 'str':\n node.id = 'unicode'\n tree_changed = True\n\n return TransformationResult(tree, tree_changed, [])\n" }, { "path": "py_backwards/transformers/super_without_arguments.py", "content": "from typed_ast import ast3 as ast\nfrom ..utils.tree import get_closest_parent_of\nfrom ..utils.helpers import warn\nfrom ..exceptions import NodeNotFound\nfrom .base import BaseNodeTransformer\n\n\nclass SuperWithoutArgumentsTransformer(BaseNodeTransformer):\n \"\"\"Compiles:\n super()\n To:\n super(Cls, self)\n super(Cls, cls)\n \n \"\"\"\n target = (2, 7)\n\n def _replace_super_args(self, node: ast.Call) -> None:\n try:\n func = get_closest_parent_of(self._tree, node, ast.FunctionDef)\n except NodeNotFound:\n warn('super() outside of function')\n return\n\n try:\n cls = get_closest_parent_of(self._tree, node, ast.ClassDef)\n except NodeNotFound:\n warn('super() outside of class')\n return\n\n node.args = [ast.Name(id=cls.name), ast.Name(id=func.args.args[0].arg)]\n\n def visit_Call(self, node: ast.Call) -> ast.Call:\n if isinstance(node.func, ast.Name) and node.func.id == 'super' and not len(node.args):\n self._replace_super_args(node)\n self._tree_changed = True\n\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/transformers/variables_annotations.py", "content": "from typed_ast import ast3 as ast\nfrom ..utils.tree import find, get_node_position, insert_at\nfrom ..utils.helpers import warn\nfrom ..types import TransformationResult\nfrom ..exceptions import NodeNotFound\nfrom .base import BaseTransformer\n\n\nclass VariablesAnnotationsTransformer(BaseTransformer):\n \"\"\"Compiles:\n a: int = 10\n b: int\n To:\n a = 10\n\n \"\"\"\n target = (3, 5)\n\n @classmethod\n def transform(cls, tree: ast.AST) -> TransformationResult:\n tree_changed = False\n\n for node in find(tree, ast.AnnAssign):\n try:\n position = get_node_position(tree, node)\n except NodeNotFound:\n warn('Assignment outside of body')\n continue\n\n tree_changed = True\n position.holder.pop(position.index) # type: ignore\n\n if node.value is not None:\n insert_at(position.index, position.parent,\n ast.Assign(targets=[node.target], # type: ignore\n value=node.value,\n type_comment=node.annotation),\n position.attribute)\n\n return TransformationResult(tree, tree_changed, [])\n" }, { "path": "py_backwards/transformers/yield_from.py", "content": "from typing import Optional, List, Type, Union\nfrom typed_ast import ast3 as ast\nfrom ..utils.tree import insert_at\nfrom ..utils.snippet import snippet, let, extend\nfrom ..utils.helpers import VariablesGenerator\nfrom .base import BaseNodeTransformer\n\nNode = Union[ast.Try, ast.If, ast.While, ast.For, ast.FunctionDef, ast.Module]\nHolder = Union[ast.Expr, ast.Assign]\n\n\n@snippet\ndef result_assignment(exc, target):\n if hasattr(exc, 'value'):\n target = exc.value\n\n\n@snippet\ndef yield_from(generator, exc, assignment):\n let(iterable)\n iterable = iter(generator)\n while True:\n try:\n yield next(iterable)\n except StopIteration as exc:\n extend(assignment)\n break\n\n\nclass YieldFromTransformer(BaseNodeTransformer):\n \"\"\"Compiles yield from to special while statement.\"\"\"\n target = (3, 2)\n\n def _get_yield_from_index(self, node: ast.AST,\n type_: Type[Holder]) -> Optional[int]:\n if hasattr(node, 'body') and isinstance(node.body, list): # type: ignore\n for n, child in enumerate(node.body): # type: ignore\n if isinstance(child, type_) and isinstance(child.value, ast.YieldFrom):\n return n\n\n return None\n\n def _emulate_yield_from(self, target: Optional[ast.AST],\n node: ast.YieldFrom) -> List[ast.AST]:\n exc = VariablesGenerator.generate('exc')\n if target is not None:\n assignment = result_assignment.get_body(exc=exc, target=target)\n else:\n assignment = []\n\n return yield_from.get_body(generator=node.value,\n assignment=assignment,\n exc=exc)\n\n def _handle_assignments(self, node: Node) -> Node:\n while True:\n index = self._get_yield_from_index(node, ast.Assign)\n if index is None:\n return node\n\n assign = node.body.pop(index)\n yield_from_ast = self._emulate_yield_from(assign.targets[0], # type: ignore\n assign.value) # type: ignore\n insert_at(index, node, yield_from_ast)\n self._tree_changed = True\n\n def _handle_expressions(self, node: Node) -> Node:\n while True:\n index = self._get_yield_from_index(node, ast.Expr)\n if index is None:\n return node\n\n exp = node.body.pop(index)\n yield_from_ast = self._emulate_yield_from(None, exp.value) # type: ignore\n insert_at(index, node, yield_from_ast)\n self._tree_changed = True\n\n def visit(self, node: ast.AST) -> ast.AST:\n node = self._handle_assignments(node) # type: ignore\n node = self._handle_expressions(node) # type: ignore\n return self.generic_visit(node) # type: ignore\n" }, { "path": "py_backwards/types.py", "content": "from typing import NamedTuple, Tuple, List\nfrom typed_ast import ast3 as ast\n\ntry:\n from pathlib import Path\nexcept ImportError:\n from pathlib2 import Path # type: ignore\n\n# Target python version\nCompilationTarget = Tuple[int, int]\n\n# Information about compilation\nCompilationResult = NamedTuple('CompilationResult',\n [('files', int),\n ('time', float),\n ('target', CompilationTarget),\n ('dependencies', List[str])])\n\n# Input/output pair\nInputOutput = NamedTuple('InputOutput', [('input', Path),\n ('output', Path)])\n\n# Result of transformers transformation\nTransformationResult = NamedTuple('TransformationResult',\n [('tree', ast.AST),\n ('tree_changed', bool),\n ('dependencies', List[str])])\n\n# Node position in tree:\nNodePosition = NamedTuple('NodePosition',\n [('parent', ast.AST),\n ('attribute', str),\n ('holder', List[ast.AST]),\n ('index', int)])\n" }, { "path": "py_backwards/utils/__init__.py", "content": "" }, { "path": "py_backwards/utils/helpers.py", "content": "from inspect import getsource\nimport re\nimport sys\nfrom typing import Any, Callable, Iterable, List, TypeVar\nfrom functools import wraps\nfrom ..conf import settings\nfrom .. import messages\n\nT = TypeVar('T')\n\n\ndef eager(fn: Callable[..., Iterable[T]]) -> Callable[..., List[T]]:\n @wraps(fn)\n def wrapped(*args: Any, **kwargs: Any) -> List[T]:\n return list(fn(*args, **kwargs))\n\n return wrapped\n\n\nclass VariablesGenerator:\n _counter = 0\n\n @classmethod\n def generate(cls, variable: str) -> str:\n \"\"\"Generates unique name for variable.\"\"\"\n try:\n return '_py_backwards_{}_{}'.format(variable, cls._counter)\n finally:\n cls._counter += 1\n\n\ndef get_source(fn: Callable[..., Any]) -> str:\n \"\"\"Returns source code of the function.\"\"\"\n source_lines = getsource(fn).split('\\n')\n padding = len(re.findall(r'^(\\s*)', source_lines[0])[0])\n return '\\n'.join(line[padding:] for line in source_lines)\n\n\ndef warn(message: str) -> None:\n print(messages.warn(message), file=sys.stderr)\n\n\ndef debug(get_message: Callable[[], str]) -> None:\n if settings.debug:\n print(messages.debug(get_message()), file=sys.stderr)\n" }, { "path": "py_backwards/utils/snippet.py", "content": "from typing import Callable, Any, List, Dict, Iterable, Union, TypeVar\nfrom typed_ast import ast3 as ast\nfrom .tree import find, get_node_position, replace_at\nfrom .helpers import eager, VariablesGenerator, get_source\n\nVariable = Union[ast.AST, List[ast.AST], str]\n\n\n@eager\ndef find_variables(tree: ast.AST) -> Iterable[str]:\n \"\"\"Finds variables and remove `let` calls.\"\"\"\n for node in find(tree, ast.Call):\n if isinstance(node.func, ast.Name) and node.func.id == 'let':\n position = get_node_position(tree, node)\n position.holder.pop(position.index) # type: ignore\n yield node.args[0].id # type: ignore\n\n\nT = TypeVar('T', bound=ast.AST)\n\n\nclass VariablesReplacer(ast.NodeTransformer):\n \"\"\"Replaces declared variables with unique names.\"\"\"\n\n def __init__(self, variables: Dict[str, Variable]) -> None:\n self._variables = variables\n\n def _replace_field_or_node(self, node: T, field: str, all_types=False) -> T:\n value = getattr(node, field, None)\n if value in self._variables:\n if isinstance(self._variables[value], str):\n setattr(node, field, self._variables[value])\n elif all_types or isinstance(self._variables[value], type(node)):\n node = self._variables[value] # type: ignore\n\n return node\n\n def visit_Name(self, node: ast.Name) -> ast.Name:\n node = self._replace_field_or_node(node, 'id', True)\n return self.generic_visit(node) # type: ignore\n\n def visit_FunctionDef(self, node: ast.FunctionDef) -> ast.FunctionDef:\n node = self._replace_field_or_node(node, 'name')\n return self.generic_visit(node) # type: ignore\n\n def visit_Attribute(self, node: ast.Attribute) -> ast.Attribute:\n node = self._replace_field_or_node(node, 'name')\n return self.generic_visit(node) # type: ignore\n\n def visit_keyword(self, node: ast.keyword) -> ast.keyword:\n node = self._replace_field_or_node(node, 'arg')\n return self.generic_visit(node) # type: ignore\n\n def visit_ClassDef(self, node: ast.ClassDef) -> ast.ClassDef:\n node = self._replace_field_or_node(node, 'name')\n return self.generic_visit(node) # type: ignore\n\n def visit_arg(self, node: ast.arg) -> ast.arg:\n node = self._replace_field_or_node(node, 'arg')\n return self.generic_visit(node) # type: ignore\n\n def _replace_module(self, module: str) -> str:\n def _replace(name):\n if name in self._variables:\n if isinstance(self._variables[name], str):\n return self._variables[name]\n\n return name\n\n return '.'.join(_replace(part) for part in module.split('.'))\n\n def visit_ImportFrom(self, node: ast.ImportFrom) -> ast.ImportFrom:\n node.module = self._replace_module(node.module)\n return self.generic_visit(node) # type: ignore\n\n def visit_alias(self, node: ast.alias) -> ast.alias:\n node.name = self._replace_module(node.name)\n node = self._replace_field_or_node(node, 'asname')\n return self.generic_visit(node) # type: ignore\n\n def visit_ExceptHandler(self, node: ast.ExceptHandler) -> ast.ExceptHandler:\n node = self._replace_field_or_node(node, 'name')\n return self.generic_visit(node) # type: ignore\n\n @classmethod\n def replace(cls, tree: T, variables: Dict[str, Variable]) -> T:\n \"\"\"Replaces all variables with unique names.\"\"\"\n inst = cls(variables)\n inst.visit(tree)\n return tree\n\n\ndef extend_tree(tree: ast.AST, variables: Dict[str, Variable]) -> None:\n for node in find(tree, ast.Call):\n if isinstance(node.func, ast.Name) and node.func.id == 'extend':\n position = get_node_position(tree, node)\n replace_at(position.index, position.parent, # type: ignore\n variables[node.args[0].id], # type: ignore\n position.attribute) # type: ignore\n\n\n# Public api:\n\nclass snippet:\n \"\"\"Snippet of code.\"\"\"\n\n def __init__(self, fn: Callable[..., None]) -> None:\n self._fn = fn\n\n def _get_variables(self, tree: ast.AST,\n snippet_kwargs: Dict[str, Variable]) -> Dict[str, Variable]:\n names = find_variables(tree)\n variables = {name: VariablesGenerator.generate(name)\n for name in names}\n\n for key, val in snippet_kwargs.items():\n if isinstance(val, ast.Name):\n variables[key] = val.id\n else:\n variables[key] = val # type: ignore\n\n return variables # type: ignore\n\n def get_body(self, **snippet_kwargs: Variable) -> List[ast.AST]:\n \"\"\"Get AST of snippet body with replaced variables.\"\"\"\n source = get_source(self._fn)\n tree = ast.parse(source)\n variables = self._get_variables(tree, snippet_kwargs)\n extend_tree(tree, variables)\n VariablesReplacer.replace(tree, variables)\n return tree.body[0].body # type: ignore\n\n\ndef let(var: Any) -> None:\n \"\"\"Declares unique value in snippet. Code of snippet like:\n \n let(x)\n x += 1\n y = 1\n \n Will end up like:\n \n _py_backwards_x_0 += 1\n y = 1\n \"\"\"\n\n\ndef extend(var: Any) -> None:\n \"\"\"Extends code, so code like:\n \n extend(vars)\n print(x, y)\n \n When vars contains AST of assignments will end up:\n \n x = 1\n x = 2\n print(x, y)\n \"\"\"\n" }, { "path": "py_backwards/utils/tree.py", "content": "from weakref import WeakKeyDictionary\nfrom typing import Iterable, Type, TypeVar, Union, List\nfrom typed_ast import ast3 as ast\nfrom ..types import NodePosition\nfrom ..exceptions import NodeNotFound\n\n_parents = WeakKeyDictionary() # type: WeakKeyDictionary[ast.AST, ast.AST]\n\n\ndef _build_parents(tree: ast.AST) -> None:\n for node in ast.walk(tree):\n for child in ast.iter_child_nodes(node):\n _parents[child] = node\n\n\ndef get_parent(tree: ast.AST, node: ast.AST, rebuild: bool = False) -> ast.AST:\n \"\"\"Get parent of node in tree.\"\"\"\n if node not in _parents or rebuild:\n _build_parents(tree)\n\n try:\n return _parents[node]\n except IndexError:\n raise NodeNotFound('Parent for {} not found'.format(node))\n\n\ndef get_node_position(tree: ast.AST, node: ast.AST) -> NodePosition:\n \"\"\"Get node position with non-Exp parent.\"\"\"\n parent = get_parent(tree, node)\n\n while not hasattr(parent, 'body') and not hasattr(parent, 'orelse'):\n node = parent\n parent = get_parent(tree, parent)\n\n if node in parent.body: # type: ignore\n return NodePosition(parent, 'body', parent.body, # type: ignore\n parent.body.index(node)) # type: ignore\n else:\n return NodePosition(parent, 'orelse', parent.orelse, # type: ignore\n parent.orelse.index(node)) # type: ignore\n\n\nT = TypeVar('T', bound=ast.AST)\n\n\ndef find(tree: ast.AST, type_: Type[T]) -> Iterable[T]:\n \"\"\"Finds all nodes with type T.\"\"\"\n for node in ast.walk(tree):\n if isinstance(node, type_):\n yield node # type: ignore\n\n\ndef insert_at(index: int, parent: ast.AST,\n nodes: Union[ast.AST, List[ast.AST]],\n holder_attribute='body') -> None:\n \"\"\"Inserts nodes to parents body at index.\"\"\"\n if not isinstance(nodes, list):\n nodes = [nodes]\n\n for child in nodes[::-1]:\n getattr(parent, holder_attribute).insert(index, child) # type: ignore\n\n\ndef replace_at(index: int, parent: ast.AST,\n nodes: Union[ast.AST, List[ast.AST]],\n holder_attribute='body') -> None:\n \"\"\"Replaces node in parents body at index with nodes.\"\"\"\n getattr(parent, holder_attribute).pop(index) # type: ignore\n insert_at(index, parent, nodes, holder_attribute)\n\n\ndef get_closest_parent_of(tree: ast.AST, node: ast.AST,\n type_: Type[T]) -> T:\n \"\"\"Get a closest parent of passed type.\"\"\"\n parent = node\n\n while True:\n parent = get_parent(tree, parent)\n\n if isinstance(parent, type_):\n return parent # type: ignore\n" }, { "path": "requirements.txt", "content": "pytest\npytest-mock\npytest-docker-pexpect\nmypy\n" }, { "path": "setup.py", "content": "#!/usr/bin/env python\nfrom setuptools import setup, find_packages\n\nVERSION = '0.7'\n\ninstall_requires = ['typed-ast', 'autopep8', 'colorama', 'py-backwards-astunparse']\nextras_require = {':python_version<\"3.4\"': ['pathlib2'],\n ':python_version<\"3.5\"': ['typing']}\n\nsetup(name='py-backwards',\n version=VERSION,\n description=\"Translates python code for older versions\",\n author='Vladimir Iakovlev',\n author_email='nvbn.rm@gmail.com',\n url='https://github.com/nvbn/py-backwards',\n license='MIT',\n packages=find_packages(exclude=['ez_setup', 'example*', 'tests*']),\n include_package_data=True,\n zip_safe=False,\n install_requires=install_requires,\n extras_require=extras_require,\n entry_points={'console_scripts': [\n 'py-backwards = py_backwards.main:main']})\n" }, { "path": "tests/__init__.py", "content": "" }, { "path": "tests/conftest.py", "content": "import pytest\nfrom typed_ast import ast3 as ast\nfrom py_backwards.utils.helpers import VariablesGenerator, get_source\n\n\n@pytest.fixture(autouse=True)\ndef reset_variables_generator():\n VariablesGenerator._counter = 0\n\n\n@pytest.fixture\ndef as_str():\n def as_str(fn):\n return get_source(fn).strip()\n\n return as_str\n\n\n@pytest.fixture\ndef as_ast():\n def as_ast(fn):\n return ast.parse(get_source(fn))\n\n return as_ast\n\n\ndef pytest_addoption(parser):\n \"\"\"Adds `--enable-functional` argument.\"\"\"\n group = parser.getgroup(\"py_backwards\")\n group.addoption('--enable-functional', action=\"store_true\", default=False,\n help=\"Enable functional tests\")\n\n\n@pytest.fixture(autouse=True)\ndef functional(request):\n if request.node.get_marker('functional') \\\n and not request.config.getoption('enable_functional'):\n pytest.skip('functional tests are disabled')\n" }, { "path": "tests/functional/__init__.py", "content": "" }, { "path": "tests/functional/input.py", "content": "\"\"\"This file contains all supported python constructions.\"\"\"\n\n\n# Variables:\n\ndef test_variables():\n a = 1\n b: int = 2\n c: int\n c = 3\n print('test variables:', a, b, c)\n\n\ntest_variables()\n\n\n# Strings:\n\ndef test_strings():\n a = 'hi'\n e = f'{a}'\n b: str = 'there'\n c = f'{a}'\n d = f'{a} {b}!'\n print('test strings:', a, b, c, d, e)\n\n\ntest_strings()\n\n\n# Lists:\n\ndef test_lists():\n a = [1, 2]\n b = [*a]\n c = [4, *b, 5]\n d: list = [7, 8]\n e: list = [*d]\n print('test lists:', a, b, c, d, e)\n\n\ntest_lists()\n\n\n# Dicts:\n\ndef test_dicts():\n a = {1: 2}\n b = {'a': 'b', **a}\n c = {**a}\n d: dict = {4: 5}\n e: dict = {**d}\n key = '{0[0]}-{0[0]}'.format\n print('test dicts:',\n sorted(a.items(), key=key),\n sorted(b.items(), key=key),\n sorted(c.items(), key=key),\n sorted(d.items(), key=key),\n sorted(e.items(), key=key))\n\n\ntest_dicts()\n\n\n# Functions:\n\ndef test_functions():\n def inc(fn):\n def wrapper(x):\n return x + 1\n\n return wrapper\n\n @inc\n def fn_a(a: int) -> int:\n return a\n\n @inc\n def fn_b(b):\n return b\n\n def fn_c(a, *args, **kwargs):\n return a, args, kwargs\n\n print('test functions:', fn_a(1), fn_b(2), fn_c(1, 2, 3, b=4),\n fn_c(*[1, 2, 3], **{'b': 'c'}))\n\n\ntest_functions()\n\n\n# Cycles:\n\ndef test_cycles():\n xs = []\n for x in range(5):\n xs.append(x)\n\n for y in []:\n xs.append(y)\n else:\n xs.append('!')\n\n m = 0\n while m < 3:\n xs.append(m)\n m += 1\n\n print('test cycles:', xs)\n\n\ntest_cycles()\n\n\n# Class:\n\n\ndef test_class():\n class Base(type):\n def base_method(cls, x: int) -> int:\n return x + 1\n\n class First(metaclass=Base):\n def method_a(self):\n return 2\n\n @classmethod\n def method_b(cls):\n return 3\n\n @staticmethod\n def method_c():\n return 4\n\n class Second(First):\n def method_a(self):\n return super().method_a() * 10\n\n @classmethod\n def method_b(cls):\n return super().method_b() * 10\n\n print('test class:', First.base_method(1), First().method_a(),\n First.method_b(), First.method_c(), Second().method_a(),\n Second.method_b(), Second.method_c(), Second().method_c())\n\n\ntest_class()\n\n\n# Generators:\n\ndef test_generators():\n def gen_a():\n for x in range(10):\n yield x\n\n def gen_b():\n yield from gen_a()\n\n def gen_c():\n a = yield 10\n return a\n\n def gen_d():\n a = yield from gen_c()\n\n print('test generators:', list(gen_a()), list(gen_b()), list(gen_c()),\n list(gen_d()))\n\n\ntest_generators()\n\n\n# For-comprehension:\n\ndef test_for_comprehension():\n xs = [x ** 2 for x in range(5)]\n ys = (y + 1 for y in range(5))\n zs = {a: b for a, b in ({'x': 1}).items()}\n print('test for comprehension:', xs, list(ys), zs)\n\n\ntest_for_comprehension()\n\n\n# Exceptions:\n\ndef test_exceptions():\n result = []\n try:\n raise Exception()\n except Exception:\n result.append(1)\n else:\n result.append(2)\n finally:\n result.append(3)\n print('test exceptions:', *result)\n\n\ntest_exceptions()\n\n\n# Context manager:\n\ndef test_context_manager():\n result = []\n from contextlib import contextmanager\n\n @contextmanager\n def manager(x):\n try:\n yield x\n finally:\n result.append(x + 1)\n\n with manager(10) as v:\n result.append(v)\n\n print('test context manager:', result)\n\n\ntest_context_manager()\n\n\n# Imports:\n\ndef test_imports():\n from pathlib import Path\n import pathlib\n print('test import override:', Path.__name__, pathlib.PosixPath.__name__)\n\n\ntest_imports()\n" }, { "path": "tests/functional/test_compiled_code.py", "content": "import pytest\nimport os\nfrom py_backwards.compiler import compile_files\nfrom py_backwards.const import TARGETS\n\nexpected_output = '''\ntest variables: 1 2 3\ntest strings: hi there hi hi there! hi\ntest lists: [1, 2] [1, 2] [4, 1, 2, 5] [7, 8] [7, 8]\ntest dicts: [(1, 2)] [(1, 2), (u'a', u'b')] [(1, 2)] [(4, 5)] [(4, 5)]\ntest functions: 2 3 (1, (2, 3), {'b': 4}) (1, (2, 3), {u'b': u'c'})\ntest cycles: [0, 1, 2, 3, 4, u'!', 0, 1, 2]\ntest class: 2 2 3 4 20 30 4 4\ntest generators: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [10] [10]\ntest for comprehension: [0, 1, 4, 9, 16] [1, 2, 3, 4, 5] {u'x': 1}\ntest exceptions: 1 3\ntest context manager: [10, 11]\ntest import override: Path PosixPath\n'''.strip()\n\n# TODO: test also on 3.0, 3.1 and 3.2\ntargets = [(version, target) for version, target in TARGETS.items()\n if target < (3, 0) or target > (3, 2)]\n\n\n@pytest.mark.functional\n@pytest.mark.parametrize('version, target', targets)\ndef test_compiled_code(spawnu, TIMEOUT, version, target):\n root = os.path.abspath(os.path.dirname(__file__))\n output = 'output_{}.py'.format(version)\n\n proc = spawnu('py_backwards/python-{}'.format(version),\n 'FROM python:{}'.format(version),\n 'bash')\n try:\n result = compile_files(os.path.join(root, 'input.py'),\n os.path.join(root, output),\n target)\n if result.dependencies:\n proc.sendline('pip install {}'.format(\n ' '.join(result.dependencies)))\n assert proc.expect_exact([TIMEOUT, 'Successfully installed'])\n\n proc.sendline('python{} src/tests/functional/{}'.format(\n version, output))\n # Output of `input.py` and converted:\n for line in expected_output.split('\\n'):\n if target > (2, 7):\n line = line.replace(\"u'\", \"'\")\n print(line)\n assert proc.expect_exact([TIMEOUT, line], timeout=10)\n finally:\n try:\n os.remove(os.path.join(root, output))\n except Exception as e:\n print(\"Can't delete compiled\", e)\n proc.close(force=True)\n" }, { "path": "tests/test_compiler.py", "content": "from contextlib import contextmanager\n\nimport pytest\nfrom unittest.mock import Mock\nfrom io import StringIO\nfrom py_backwards import compiler\nfrom py_backwards.files import InputOutput\nfrom py_backwards.exceptions import CompilationError\n\n\nclass TestCompileFiles(object):\n @pytest.fixture\n def input_output(self, mocker):\n mock = mocker.patch('py_backwards.compiler.get_input_output_paths')\n io = InputOutput(Mock(), Mock())\n mock.return_value = [io]\n return io\n\n def test_syntax_error(self, input_output):\n input_output.input.as_posix.return_value = 'test.py'\n input_output.input.open.return_value = StringIO('a b c d')\n with pytest.raises(CompilationError):\n compiler.compile_files('test.py', 'lib/test.py', (2, 7))\n\n def test_compile(self, input_output):\n output = StringIO()\n\n @contextmanager\n def output_f(*_):\n yield output\n\n input_output.input.as_posix.return_value = 'test.py'\n input_output.input.open.return_value = StringIO(\"print('hello world')\")\n input_output.output.open = output_f\n\n result = compiler.compile_files('test.py', 'lib/test.py', (2, 7))\n assert result.files == 1\n assert result.target == (2, 7)\n assert result.time\n\n assert '# -*- coding: utf-8 -*-' in output.getvalue()\n assert \"print(u'hello world')\" in output.getvalue()\n" }, { "path": "tests/test_files.py", "content": "import pytest\n\ntry:\n from pathlib import Path\nexcept ImportError:\n from pathlib2 import Path\n\nfrom py_backwards.exceptions import InvalidInputOutput, InputDoesntExists\nfrom py_backwards import files\n\n\nclass TestGetInputPath(object):\n @pytest.fixture(autouse=True)\n def exists(self, mocker):\n exists_mock = mocker.patch('py_backwards.files.Path.exists')\n exists_mock.return_value = True\n return exists_mock\n\n def test_dir_to_file(self):\n with pytest.raises(InvalidInputOutput):\n list(files.get_input_output_paths('src/', 'out.py', None))\n\n def test_non_exists_input(self, exists):\n exists.return_value = False\n with pytest.raises(InputDoesntExists):\n list(files.get_input_output_paths('src/', 'out/', None))\n\n def test_file_to_dir(self):\n assert list(files.get_input_output_paths('test.py', 'out/', None)) == [\n files.InputOutput(Path('test.py'), Path('out/test.py'))]\n\n def test_file_to_file(self):\n assert list(files.get_input_output_paths('test.py', 'out.py', None)) == [\n files.InputOutput(Path('test.py'), Path('out.py'))]\n\n def test_dir_to_dir(self, mocker):\n glob_mock = mocker.patch('py_backwards.files.Path.glob')\n glob_mock.return_value = [Path('src/main.py'), Path('src/const/const.py')]\n assert list(files.get_input_output_paths('src', 'out', None)) == [\n files.InputOutput(Path('src/main.py'), Path('out/main.py')),\n files.InputOutput(Path('src/const/const.py'), Path('out/const/const.py'))]\n\n def test_file_to_dir_with_root(self):\n paths = list(files.get_input_output_paths('project/src/test.py',\n 'out',\n 'project'))\n assert paths == [files.InputOutput(Path('project/src/test.py'),\n Path('out/src/test.py'))]\n\n def test_dir_to_dir_with_root(self, mocker):\n glob_mock = mocker.patch('py_backwards.files.Path.glob')\n glob_mock.return_value = [Path('project/src/main.py'),\n Path('project/src/const/const.py')]\n paths = list(files.get_input_output_paths('project', 'out', 'project'))\n assert paths == [\n files.InputOutput(Path('project/src/main.py'),\n Path('out/src/main.py')),\n files.InputOutput(Path('project/src/const/const.py'),\n Path('out/src/const/const.py'))]\n" }, { "path": "tests/transformers/__init__.py", "content": "" }, { "path": "tests/transformers/conftest.py", "content": "import pytest\nfrom types import ModuleType\nfrom typed_ast.ast3 import parse, dump\nfrom astunparse import unparse, dump as dump_pretty\n\n\n@pytest.fixture\ndef transform():\n def transform(transformer, before):\n tree = parse(before)\n try:\n transformer.transform(tree)\n return unparse(tree).strip()\n except:\n print('Before:')\n print(dump_pretty(parse(before)))\n print('After:')\n print(dump_pretty(tree))\n raise\n\n return transform\n\n\n@pytest.fixture\ndef run_transformed(transform):\n def _get_latest_line(splitted):\n for n in range(-1, -1 - len(splitted), -1):\n if splitted[n][0] not in ')]} ':\n return n\n\n def run_transformed(transformer, code):\n transformed = transform(transformer, code)\n splitted = transformed.split('\\n')\n latest_line = _get_latest_line(splitted)\n splitted[latest_line] = '__result = ' + splitted[latest_line]\n compiled = compile('\\n'.join(splitted), '', 'exec')\n module = ModuleType('')\n exec(compiled, module.__dict__, )\n return module.__dict__['__result']\n\n return run_transformed\n\n\n@pytest.fixture\ndef ast():\n def ast(code):\n return dump(parse(code))\n\n return ast\n" }, { "path": "tests/transformers/test_class_without_bases.py", "content": "import pytest\nfrom py_backwards.transformers.class_without_bases import ClassWithoutBasesTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('''\nclass A:\n pass\n ''', '''\nclass A(object):\n pass\n '''),\n ('''\nclass A():\n pass\n ''', '''\nclass A(object):\n pass\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(ClassWithoutBasesTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_dict_unpacking.py", "content": "import pytest\nfrom py_backwards.transformers.dict_unpacking import DictUnpackingTransformer\n\n\nprefix = '''\ndef _py_backwards_merge_dicts(dicts):\n result = {}\n for dict_ in dicts:\n result.update(dict_)\n return result\n'''\n\n\n@pytest.mark.parametrize('before, after', [\n ('{1: 2, **{3: 4}}',\n prefix + '_py_backwards_merge_dicts([{1: 2}, dict({3: 4})])'),\n ('{**x}', prefix + '_py_backwards_merge_dicts([dict(x)])'),\n ('{1: 2, **a, 3: 4, **b, 5: 6}',\n prefix + '_py_backwards_merge_dicts([{1: 2}, dict(a), {3: 4}, dict(b), {5: 6}])')])\ndef test_transform(transform, ast, before, after):\n code = transform(DictUnpackingTransformer, before)\n assert ast(code) == ast(after)\n\n\n@pytest.mark.parametrize('code, result', [\n ('{1: 2, **{3: 4}}', {1: 2, 3: 4}),\n ('{**{5: 6}}', {5: 6}),\n ('{1: 2, **{7: 8}, 3: 4, **{9: 10}, 5: 6}',\n {1: 2, 7: 8, 3: 4, 9: 10, 5: 6})])\ndef test_run(run_transformed, code, result):\n assert run_transformed(DictUnpackingTransformer, code) == result\n" }, { "path": "tests/transformers/test_formatted_values.py", "content": "import pytest\nfrom py_backwards.transformers.formatted_values import FormattedValuesTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n (\"f'hi'\", \"'hi'\"),\n (\"f'hi {x}'\", \"''.join(['hi ', '{}'.format(x)])\"),\n (\"f'hi {x.upper()} {y:1}'\",\n \"''.join(['hi ', '{}'.format(x.upper()), ' ', '{:1}'.format(y)])\")])\ndef test_transform(transform, ast, before, after):\n code = transform(FormattedValuesTransformer, before)\n assert ast(code) == ast(after)\n\n\n@pytest.mark.parametrize('code, result', [\n (\"f'hi'\", 'hi'),\n (\"x = 12; f'hi {x}'\", 'hi 12'),\n (\"x = 'everyone'; y = 42; f'hi {x.upper()!r} {y:x}'\",\n 'hi EVERYONE 2a')])\ndef test_run(run_transformed, code, result):\n assert run_transformed(FormattedValuesTransformer, code) == result\n" }, { "path": "tests/transformers/test_functions_annotations.py", "content": "import pytest\nfrom py_backwards.transformers.functions_annotations import FunctionsAnnotationsTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('def fn(x: T) -> List[T]:\\n return [x]',\n 'def fn(x):\\n return [x]'),\n ('def fn(x: int) -> float:\\n return 1.5',\n 'def fn(x):\\n return 1.5')])\ndef test_transform(transform, ast, before, after):\n code = transform(FunctionsAnnotationsTransformer, before)\n assert ast(code) == ast(after)\n\n\n@pytest.mark.parametrize('code, result', [\n ('def fn(x: T) -> List[T]:\\n return [x]\\nfn(10)', [10]),\n ('def fn(x: int) -> float:\\n return 1.5\\nfn(10)', 1.5)])\ndef test_run(run_transformed, code, result):\n assert run_transformed(FunctionsAnnotationsTransformer, code) == result\n" }, { "path": "tests/transformers/test_import_dbm.py", "content": "import pytest\nfrom py_backwards.transformers.import_dbm import ImportDbmTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('import dbm',\n '''\nif __import__('six').PY2:\n import anydbm as dbm\nelse:\n import dbm\n '''),\n ('from dbm import ndbm',\n '''\nif __import__('six').PY2:\n import dbm as ndbm\nelse:\n from dbm import ndbm\n '''),\n ('from dbm.ndbm import library',\n '''\nif __import__('six').PY2:\n from dbm import library\nelse:\n from dbm.ndbm import library\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(ImportDbmTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_import_pathlib.py", "content": "import pytest\nfrom py_backwards.transformers.import_pathlib import ImportPathlibTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('import pathlib',\n '''\ntry:\n import pathlib\nexcept ImportError:\n import pathlib2 as pathlib\n '''),\n ('import pathlib as p',\n '''\ntry:\n import pathlib as p\nexcept ImportError:\n import pathlib2 as p\n '''),\n ('from pathlib import Path',\n '''\ntry:\n from pathlib import Path\nexcept ImportError:\n from pathlib2 import Path\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(ImportPathlibTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_metaclass.py", "content": "import pytest\nfrom py_backwards.transformers.metaclass import MetaclassTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('''\nclass A(metaclass=B):\n pass\n ''', '''\nfrom six import with_metaclass as _py_backwards_six_withmetaclass\n\nclass A(\n_py_backwards_six_withmetaclass(B, *[])):\n pass\n '''),\n ('''\nclass A(C, metaclass=B):\n pass\n ''', '''\nfrom six import with_metaclass as _py_backwards_six_withmetaclass\n\nclass A(\n_py_backwards_six_withmetaclass(B, *[C])):\n pass\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(MetaclassTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_python2_future.py", "content": "import pytest\nfrom py_backwards.transformers.python2_future import Python2FutureTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('print(10)', '''\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nfrom __future__ import unicode_literals\nprint(10)\n '''),\n ('a = 1', '''\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\nfrom __future__ import unicode_literals\na = 1\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(Python2FutureTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_return_from_generator.py", "content": "import pytest\nfrom py_backwards.transformers.return_from_generator import ReturnFromGeneratorTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('''\ndef fn():\n yield 1\n return 5\n ''', '''\ndef fn():\n (yield 1)\n _py_backwards_exc_0 = StopIteration()\n _py_backwards_exc_0.value = 5\n raise _py_backwards_exc_0\n '''),\n ('''\ndef fn():\n if True:\n x = yield from [1]\n return 5\n ''', '''\ndef fn():\n if True:\n x = (yield from [1])\n _py_backwards_exc_0 = StopIteration()\n _py_backwards_exc_0.value = 5\n raise _py_backwards_exc_0\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(ReturnFromGeneratorTransformer, before)\n assert ast(code) == ast(after)\n\n\nget_value = '''\ngen = fn()\nnext(gen)\nval = None\ntry:\n next(gen)\nexcept StopIteration as e:\n val = e.value\nval\n'''\n\n\n@pytest.mark.parametrize('code, result', [\n ('''\ndef fn():\n yield 1\n return 5\n{}\n '''.format(get_value), 5),\n ('''\ndef fn():\n yield from [1]\n return 6\n{}\n '''.format(get_value), 6),\n ('''\ndef fn():\n x = yield 1\n return 7\n{}\n '''.format(get_value), 7),\n ('''\ndef fn():\n x = yield from [1]\n return 8\n{}\n '''.format(get_value), 8)])\ndef test_run(run_transformed, code, result):\n assert run_transformed(ReturnFromGeneratorTransformer, code) == result\n" }, { "path": "tests/transformers/test_six_moves.py", "content": "import pytest\nfrom py_backwards.transformers.six_moves import SixMovesTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('from functools import reduce',\n '''\ntry:\n from functools import reduce\nexcept ImportError:\n from six.moves import reduce as reduce\n '''),\n ('from shlex import quote',\n '''\ntry:\n from shlex import quote\nexcept ImportError:\n from six.moves import shlex_quote as quote\n '''),\n ('from itertools import zip_longest',\n '''\ntry:\n from itertools import zip_longest\nexcept ImportError:\n from six.moves import zip_longest as zip_longest\n '''),\n ('from urllib.request import Request, pathname2url',\n '''\ntry:\n from urllib.request import Request, pathname2url\nexcept ImportError:\n from six.moves.urllib.request import Request as Request\n from six.moves.urllib.request import pathname2url as pathname2url\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(SixMovesTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_starred_unpacking.py", "content": "import pytest\nfrom py_backwards.transformers.starred_unpacking import StarredUnpackingTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('[1, 2, 3]', '[1, 2, 3]'),\n ('[1, 2, *range(5, 10), 3, 4]',\n '(([1, 2] + list(range(5, 10))) + [3, 4])'),\n ('[*range(5), *range(5, 10)]', '(list(range(5)) + list(range(5, 10)))'),\n ('[*range(5, 10)]', 'list(range(5, 10))'),\n ('print(1, 2, 3)', 'print(1, 2, 3)'),\n ('print(1, 2, *range(5, 10), 3, 4)',\n 'print(*(([1, 2] + list(range(5, 10))) + [3, 4]))'),\n ('print(*range(5), *range(5, 10))',\n 'print(*(list(range(5)) + list(range(5, 10))))'),\n ('print(*range(5, 10))',\n 'print(*list(range(5, 10)))'),\n])\ndef test_transform(transform, ast, before, after):\n code = transform(StarredUnpackingTransformer, before)\n assert ast(code) == ast(after)\n\n\n@pytest.mark.parametrize('code, result', [\n ('[1, 2, *range(5, 10), 3, 4]',\n [1, 2, 5, 6, 7, 8, 9, 3, 4]),\n ('[*range(5), *range(5, 10)]',\n [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),\n ('[*range(5, 10)]', [5, 6, 7, 8, 9]),\n ('to_tuple = lambda *xs: xs; to_tuple(1, 2, *range(5, 10), 3, 4)',\n (1, 2, 5, 6, 7, 8, 9, 3, 4)),\n ('to_tuple = lambda *xs: xs; to_tuple(*range(5), *range(5, 10))',\n (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)),\n ('to_tuple = lambda *xs: xs; to_tuple(*range(5, 10))',\n (5, 6, 7, 8, 9)),\n])\ndef test_run(run_transformed, code, result):\n assert run_transformed(StarredUnpackingTransformer, code) == result\n" }, { "path": "tests/transformers/test_string_types.py", "content": "import pytest\nfrom py_backwards.transformers.string_types import StringTypesTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('str(1)', 'unicode(1)'),\n ('str(\"hi\")', 'unicode(\"hi\")'),\n ('something.str()', 'something.str()')])\ndef test_transform(transform, ast, before, after):\n code = transform(StringTypesTransformer, before)\n assert ast(code) == ast(after)\n" }, { "path": "tests/transformers/test_super_without_arguments.py", "content": "import pytest\nfrom py_backwards.transformers.super_without_arguments import SuperWithoutArgumentsTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('''\nclass A:\n\n def method(self, x):\n return super().method(x)\n ''', '''\nclass A():\n\n def method(self, x):\n return super(A, self).method(x)\n '''),\n ('''\nclass A:\n\n @classmethod\n def method(cls, x):\n return super().method(x)\n ''', '''\nclass A():\n\n @classmethod\n def method(cls, x):\n return super(A, cls).method(x)\n ''')])\ndef test_transform(transform, ast, before, after):\n code = transform(SuperWithoutArgumentsTransformer, before)\n assert ast(code) == ast(after)\n\n\n@pytest.mark.parametrize('code, result', [\n ('''\nclass A:\n def x(self):\n return 5\n \nclass B(A):\n def x(self):\n return super().x()\n \nB().x() \n ''', 5),\n ('''\nclass A:\n @classmethod\n def x(cls):\n return 5\n \nclass B(A):\n @classmethod\n def x(cls):\n return super().x()\n \nB.x() \n ''', 5)])\ndef test_run(run_transformed, code, result):\n assert run_transformed(SuperWithoutArgumentsTransformer, code) == result\n" }, { "path": "tests/transformers/test_variables_annotations.py", "content": "import pytest\nfrom py_backwards.transformers.variables_annotations import VariablesAnnotationsTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('a: int = 10', 'a = 10'),\n ('a: int', '')])\ndef test_transform(transform, ast, before, after):\n code = transform(VariablesAnnotationsTransformer, before)\n assert ast(after) == ast(code)\n\n\n@pytest.mark.parametrize('code, result', [\n ('a: int = 10; a', 10),\n ('a: int; \"a\" in locals()', False)])\ndef test_run(run_transformed, code, result):\n assert run_transformed(VariablesAnnotationsTransformer, code) == result\n" }, { "path": "tests/transformers/test_yield_from.py", "content": "import pytest\nfrom py_backwards.transformers.yield_from import YieldFromTransformer\n\n\n@pytest.mark.parametrize('before, after', [\n ('''\ndef fn():\n yield from range(10)\n ''', '''\ndef fn():\n _py_backwards_iterable_1 = iter(range(10))\n while True:\n try:\n (yield next(_py_backwards_iterable_1))\n except StopIteration as _py_backwards_exc_0:\n break\n'''),\n ('''\ndef fn():\n a = yield from range(10)\n ''', '''\ndef fn():\n _py_backwards_iterable_1 = iter(range(10))\n while True:\n try:\n (yield next(_py_backwards_iterable_1))\n except StopIteration as _py_backwards_exc_0:\n if hasattr(_py_backwards_exc_0, 'value'):\n a = _py_backwards_exc_0.value\n break\n'''),\n])\ndef test_transform(transform, ast, before, after):\n code = transform(YieldFromTransformer, before)\n assert ast(code) == ast(after)\n\n\n@pytest.mark.parametrize('code, result', [\n ('''\ndef fn():\n yield from range(3)\n\nlist(fn())\n ''', [0, 1, 2]),\n ('''\ndef fn():\n def fake_gen():\n yield 0\n exc = StopIteration()\n exc.value = 5\n raise exc\n\n x = yield from fake_gen()\n yield x\n \nlist(fn())''', [0, 5])])\ndef test_run(run_transformed, code, result):\n assert run_transformed(YieldFromTransformer, code) == result\n" }, { "path": "tests/utils/__init__.py", "content": "" }, { "path": "tests/utils/test_helpers.py", "content": "from py_backwards.utils.helpers import VariablesGenerator, eager, get_source\n\n\ndef test_eager():\n @eager\n def fn():\n yield 1\n yield 2\n yield 3\n\n assert fn() == [1, 2, 3]\n\n\ndef test_variables_generator():\n assert VariablesGenerator.generate('x') == '_py_backwards_x_0'\n assert VariablesGenerator.generate('x') == '_py_backwards_x_1'\n\n\ndef test_get_source():\n def fn():\n x = 1\n\n source = '''\ndef fn():\n x = 1\n '''\n\n assert get_source(fn).strip() == source.strip()\n" }, { "path": "tests/utils/test_snippet.py", "content": "from typed_ast import ast3 as ast\nfrom astunparse import unparse\nfrom py_backwards.utils.snippet import (snippet, let, find_variables,\n VariablesReplacer, extend_tree)\n\n\ndef test_variables_finder():\n tree = ast.parse('''\nlet(a)\nx = 1\nlet(b)\n ''')\n assert find_variables(tree) == ['a', 'b']\n\n\ndef test_variables_replacer():\n tree = ast.parse('''\nfrom f.f import f as f\nimport f as f\n\nclass f(f):\n def f(f):\n f = f\n for f in f:\n with f as f:\n yield f\n return f\n\n ''')\n VariablesReplacer.replace(tree, {'f': 'x'})\n code = unparse(tree)\n\n expected = '''\nfrom x.x import x as x\nimport x as x\n\nclass x(x):\n\n def x(x):\n x = x\n for x in x:\n with x as x:\n (yield x)\n return x\n '''\n\n assert code.strip() == expected.strip()\n\n\n@snippet\ndef to_extend():\n y = 5\n\n\ndef test_extend_tree():\n tree = ast.parse('''\nx = 1\nextend(y)\n ''')\n extend_tree(tree, {'y': to_extend.get_body()})\n code = unparse(tree)\n expected = '''\nx = 1\ny = 5\n '''\n assert code.strip() == expected.strip()\n\n\n@snippet\ndef my_snippet(class_name, x_value):\n class class_name:\n pass\n\n let(x)\n x = x_value\n\n let(result)\n result = 0\n\n let(i)\n for i in range(x):\n result += i\n\n return result\n\n\ninitial_code = '''\ndef fn():\n pass\n \nresult = fn()\n'''\n\nexpected_code = '''\ndef fn():\n pass\n\n class MyClass():\n pass\n _py_backwards_x_0 = 10\n _py_backwards_result_1 = 0\n for _py_backwards_i_2 in range(_py_backwards_x_0):\n _py_backwards_result_1 += _py_backwards_i_2\n return _py_backwards_result_1\nresult = fn()\n'''\n\n\ndef _get_code():\n tree = ast.parse(initial_code)\n tree.body[0].body.extend(my_snippet.get_body(class_name='MyClass',\n x_value=ast.Num(10)))\n return unparse(tree)\n\n\ndef test_snippet_code():\n new_code = _get_code()\n assert new_code.strip() == expected_code.strip()\n\n\ndef test_snippet_run():\n new_code = _get_code()\n locals_ = {}\n exec(new_code, {}, locals_)\n assert locals_['result'] == 45\n" }, { "path": "tests/utils/test_tree.py", "content": "from typed_ast import ast3 as ast\nfrom astunparse import unparse\nfrom py_backwards.utils.snippet import snippet\nfrom py_backwards.utils.tree import (get_parent, get_node_position,\n find, insert_at, replace_at)\n\n\ndef test_get_parent(as_ast):\n @as_ast\n def tree():\n x = 1\n\n assignment = tree.body[0].body[0]\n assert get_parent(tree, assignment) == tree.body[0]\n\n\nclass TestGetNodePosition:\n def test_from_body(self, as_ast):\n @as_ast\n def tree():\n x = 1\n print(10)\n\n call = tree.body[0].body[1].value\n position = get_node_position(tree, call)\n assert position.index == 1\n assert position.parent == tree.body[0]\n assert position.attribute == 'body'\n\n def test_from_orelse(self, as_ast):\n @as_ast\n def tree():\n if True:\n print(0)\n else:\n print(1)\n\n call = tree.body[0].body[0].orelse[0].value\n position = get_node_position(tree, call)\n assert position.index == 0\n assert position.parent == tree.body[0].body[0]\n assert position.attribute == 'orelse'\n\n\ndef test_find(as_ast):\n @as_ast\n def tree():\n print('hi there')\n print(10)\n\n calls = list(find(tree, ast.Call))\n assert len(calls) == 2\n\n\n@snippet\ndef to_insert():\n print(10)\n\n\ndef test_insert_at(as_ast, as_str):\n def fn():\n print('hi there')\n\n tree = as_ast(fn)\n insert_at(0, tree.body[0], to_insert.get_body())\n\n def fn():\n print(10)\n print('hi there')\n\n expected_code = as_str(fn)\n assert unparse(tree).strip() == expected_code\n\n\ndef test_replace_at(as_ast, as_str):\n def fn():\n print('hi there')\n\n tree = as_ast(fn)\n replace_at(0, tree.body[0], to_insert.get_body())\n\n def fn():\n print(10)\n\n expected_code = as_str(fn)\n assert unparse(tree).strip() == expected_code\n" } ]