[ { "path": ".gitignore", "content": "__pycache__\n.hypothesis\n.test\ncoverage.info\n*.egg-info\n*.build.toml\n*.so\n*.o\n" }, { "path": ".gitmodules", "content": "[submodule \"misc/terryfy\"]\n\tpath = misc/terryfy\n\turl = https://github.com/MacPython/terryfy/\n" }, { "path": ".travis.yml", "content": "language: python\n\nsudo: required\n\nservices:\n - docker\n\nenv:\n global:\n - secure: \"HO3zCuv0FtNFTQ7kkBpIqKYAZW8sYZPfc1ROk6+ChoxufXcu529CTKNAr3KklfZCbMHiZKc3W83N7x9B/L2rtSuBQvJPPgVtIlaVKRyWWnY4nqrpwKEoOLUd3RjpAMfCB09sXQ2aTfQV8Ds5Zk+cF7R2toI6s2s4vymXvCLvfugrtO4sd91frSDv/fzjEEKOIeey8KXtPAPPFv6v64OScksPt1oCsVOPDtkZ7q0KSIzS9JN6BvM9oafPt9MaFPH84ITtdPMTjgQOQ+YFe8YBwgjkV/cX9rNs+vzSP6Bm2NQ9/xxd8XTDj6ukuEYD5HQi26IS6ddRyVGsn6/WRZx6/kQboJKh5r5pa4OAHPWnPirRWPQW46HI2iknAGTFWh8ARX2R208mK1vbQ66J+9zQDnkjMXGgX67gWyWQWfxwVHFsPyQEiSHHh2vEBkhs1+tqtvp7Ktnc+uCxXn0v/Humu3OvFSBxSXfyjvE9uUOGyB2zDwqmxLQQ5ftKAcGfLOaSqauJ1vQy1CWc5bROCn8aoch5iRf/tcX85TUDirAgAp3OUdt3VwcRNY+Fci7IU50gn2rghJWFzB5Zz9p1ShnZxIaD5GEPE45ju4UIpwYbs8iSqh+/RS8sR2Ffzx4M+6QJjj1BJABdtVPS9Jn5OkbuSdBW0K+MuLtmtbg4WLXv6+E=\"\n\njobs:\n include:\n - python: 3.5\n env: VERSION=3.5.3\n - python: 3.6\n env: VERSION=3.6.0\n - python: 3.7\n env: VERSION=3.7.1\n - python: 3.8\n env: VERSION=3.8.0\n\nbefore_install: source misc/travis/before_install.sh\n\ninstall: source misc/travis/install.sh\n\nscript: misc/travis/script.sh\n" }, { "path": "CHANGELOG.md", "content": "0.1.1 - Feb 9 2017\n------------------\n\n- Native support for OSX\n- Support for older hardware without SSE4.2\n- Better crash info with faulthandler\n" }, { "path": "LICENSE.txt", "content": "Copyright (c) 2017 Paweł Piotr Przeradowski\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "MANIFEST.in", "content": "include README.md LICENSE.txt src/picohttpparser/picohttpparser.c src/picohttpparser/picohttpparser.h\n" }, { "path": "README.md", "content": "# Japronto!\n\n[![irc: #japronto](https://img.shields.io/badge/irc-%23japronto-brightgreen.svg)](https://webchat.freenode.net/?channels=japronto)\n[![Gitter japronto/Lobby](https://badges.gitter.im/japronto/Lobby.svg)](https://gitter.im/japronto/Lobby) [![Build Status](https://travis-ci.org/squeaky-pl/japronto.svg?branch=master)](https://travis-ci.org/squeaky-pl/japronto) [![PyPI](https://img.shields.io/pypi/v/japronto.svg)](https://pypi.python.org/pypi/japronto) [![PyPI version](https://img.shields.io/pypi/pyversions/japronto.svg)](https://pypi.python.org/pypi/japronto/)\n\n__There is no new project development happening at the moment, but it's not abandoned either. Pull requests and new maintainers are welcome__.\n\n__If you are a novice Python programmer, you don't like plumbing yourself or you don't have basic understanding of C, this project is not probably what you are looking for__.\n\nJapronto (from Portuguese \"já pronto\" /ˈʒa pɾõtu/ meaning \"already done\") is a __screaming-fast__, __scalable__, __asynchronous__\nPython 3.5+ HTTP __toolkit__ integrated with __pipelining HTTP server__ based on [uvloop](https://github.com/MagicStack/uvloop) and [picohttpparser](https://github.com/h2o/picohttpparser). It's targeted at speed enthusiasts, people who like\nplumbing and early adopters.\n\nYou can read more in the [release announcement on medium](https://medium.com/@squeaky_pl/million-requests-per-second-with-python-95c137af319)\n\nPerformance\n-----------\n\nHere's a chart to help you imagine what kind of things you can do with Japronto:\n\n![Requests per second](benchmarks/results.png)\n\nAs user @heppu points out Go’s stdlib HTTP server can be 12% faster than the graph shows when written more carefully. Also there is the awesome fasthttp server for Go that apparently is only 18% slower than Japronto in this particular benchmark. Awesome! For details see https://github.com/squeaky-pl/japronto/pull/12 and https://github.com/squeaky-pl/japronto/pull/14.\n\nThese results of a simple \"Hello world\" application were obtained on AWS c4.2xlarge instance. To be fair all the contestants (including Go) were running single worker process. Servers were load tested using [wrk](https://github.com/wg/wrk) with 1 thread, 100 connections and 24 simultaneous (pipelined) requests per connection (cumulative parallelism of 2400 requests).\n\nThe source code for the benchmark can be found in [benchmarks](benchmarks) directory.\n\nThe server is written in hand tweaked C trying to take advantage of modern CPUs. It relies on picohttpparser for header &\nchunked-encoding parsing while uvloop provides asynchronous I/O. It also tries to save up on\nsystem calls by combining writes together when possible.\n\nEarly preview\n-------------\n\nThis is an early preview with alpha quality implementation. APIs are provisional meaning that they will change between versions and more testing is needed. Don't use it for anything serious for now and definitely don't use it in production. Please try it though and report back feedback. If you are shopping for your next project's framework I would recommend [Sanic](https://github.com/channelcat/sanic).\n\nAt the moment the work is focused on CPython but I have PyPy on my radar, though I am not gonna look into it until PyPy reaches 3.5 compatibility somewhere later this year and most known JIT regressions are removed.\n\nHello world\n-----------\n\nHere is how a simple web application looks like in Japronto:\n\n```python\nfrom japronto import Application\n\n\ndef hello(request):\n return request.Response(text='Hello world!')\n\n\napp = Application()\napp.router.add_route('/', hello)\napp.run(debug=True)\n```\n\nTutorial\n--------\n\n1. [Getting started](tutorial/1_hello.md)\n2. [Asynchronous handlers](tutorial/2_async.md)\n3. [Router](tutorial/3_router.md)\n4. [Request object](tutorial/4_request.md)\n5. [Response object](tutorial/5_response.md)\n6. [Handling exceptions](tutorial/6_exceptions.md)\n7. [Extending request](tutorial/7_extend.md)\n\nFeatures\n--------\n\n- HTTP 1.x implementation with support for chunked uploads\n- Full support for HTTP pipelining\n- Keep-alive connections with configurable reaper\n- Support for synchronous and asynchronous views\n- Master-multiworker model based on forking\n- Support for code reloading on changes\n- Simple routing\n\nLicense\n-------\n\nThis software is distributed under [MIT License](https://en.wikipedia.org/wiki/MIT_License). This is a very permissive license that lets you use this software for any\ncommercial and non-commercial work. Full text of the license is\nincluded in [LICENSE.txt](LICENSE.txt) file.\n\nThe source distribution of this software includes a copy of picohttpparser which is distributed under MIT license as well.\n" }, { "path": "benchmarks/aiohttp/micro.py", "content": "from aiohttp import web\nimport asyncio\nimport uvloop\n\n\nloop = uvloop.new_event_loop()\nasyncio.set_event_loop(loop)\n\n\nasync def hello(request):\n return web.Response(text='Hello world!')\n\napp = web.Application(loop=loop)\napp.router.add_route('GET', '/', hello)\n\nweb.run_app(app, port=8080, access_log=None)\n" }, { "path": "benchmarks/aiohttp/requirements.txt", "content": "aiohttp==1.2.0\n" }, { "path": "benchmarks/gevent/micro.py", "content": "from gevent.pywsgi import WSGIServer\n\n\ndef hello(environ, start_response):\n if(environ['PATH_INFO'] == '/' and environ['REQUEST_METHOD'] == 'GET'):\n status = '200 OK'\n text = \"Hello world!\"\n else:\n status = '404 Not Found'\n text = \"Not Found\"\n\n body = text.encode('utf-8')\n response_headers = [\n ('Content-type', 'text/plain; charset=utf-8'),\n ('Content-Length', str(len(body)))]\n start_response(status, response_headers)\n return [body]\n\n\nWSGIServer(('0.0.0.0', 8080), hello, log=None).serve_forever()\n" }, { "path": "benchmarks/gevent/requirements.txt", "content": "gevent==1.2.1\n" }, { "path": "benchmarks/golang/README.md", "content": "```\ngo build .\nGOMAXPROCS=1 ./bin\n```\n" }, { "path": "benchmarks/golang/micro.go", "content": "package main\n\nimport \"net/http\"\n\nvar (\n\thelloResp = []byte(\"Hello world!\")\n\tnotFoundResp = []byte(\"Not Found\")\n)\n\nfunc hello(w http.ResponseWriter, r *http.Request) {\n\tif r.URL.Path != \"/\" {\n\t\tw.WriteHeader(http.StatusNotFound)\n\t\tw.Write(notFoundResp)\n\t\treturn\n\t}\n\tw.Write(helloResp)\n}\n\nfunc main() {\n\thttp.HandleFunc(\"/\", hello)\n\thttp.ListenAndServe(\"0.0.0.0:8080\", nil)\n}\n" }, { "path": "benchmarks/golang-fasthttp/README.md", "content": "```\ngo build .\nGOMAXPROCS=1 ./bin\n```\n" }, { "path": "benchmarks/golang-fasthttp/micro.go", "content": "package main\n\nimport \"github.com/valyala/fasthttp\"\n\nfunc hello(ctx *fasthttp.RequestCtx) {\n\tif string(ctx.Path()) != \"/\" {\n\t\tctx.SetStatusCode(404)\n\t\tctx.WriteString(\"Not Found\")\n\t\treturn\n\t}\n\tctx.WriteString(\"Hello world!\")\n}\n\nfunc main() {\n\tfasthttp.ListenAndServe(\"0.0.0.0:8080\", hello)\n}\n" }, { "path": "benchmarks/japronto/micro.py", "content": "from japronto import Application\n\n\ndef hello(request):\n return request.Response(text='Hello world!')\n\n\napp = Application()\n\nr = app.router\nr.add_route('/', hello, method='GET')\n\napp.run()\n" }, { "path": "benchmarks/meinheld/micro.py", "content": "from meinheld import server\n\n\ndef hello(environ, start_response):\n if(environ['PATH_INFO'] == '/' and environ['REQUEST_METHOD'] == 'GET'):\n status = '200 OK'\n text = \"Hello world!\"\n else:\n status = '404 Not Found'\n text = \"Not Found\"\n\n body = text.encode('utf-8')\n response_headers = [\n ('Content-type', 'text/plain; charset=utf-8'),\n ('Content-Length', str(len(body)))]\n start_response(status, response_headers)\n return [body]\n\n\nserver.listen(('0.0.0.0', 8080))\nserver.set_access_logger(None)\nserver.set_keepalive(1)\nserver.run(hello)\n" }, { "path": "benchmarks/meinheld/requirements.txt", "content": "meinheld==0.6.1\n" }, { "path": "benchmarks/nodejs/micro.js", "content": "const http = require('http');\n\n\nvar srv = http.createServer( (req, res) => {\n res.sendDate = false;\n if(req.url == '/') {\n data = 'Hello world!'\n status = 200\n } else {\n data = 'Not Found'\n status = 404\n }\n res.writeHead(status, {\n 'Content-Type': 'text/plain; encoding=utf-8',\n 'Content-Length': data.length});\n res.end(data);\n});\n\n\nsrv.listen(8080, '0.0.0.0');\n" }, { "path": "benchmarks/sanic/micro.py", "content": "from sanic import Sanic\nfrom sanic.response import text\n\napp = Sanic(__name__)\n\n\n@app.route(\"/\")\nasync def hello(request):\n return text(\"Hello world!\")\n\napp.run(host=\"0.0.0.0\", port=8080)\n" }, { "path": "benchmarks/sanic/requirements.txt", "content": "sanic==0.2.0\n" }, { "path": "benchmarks/tornado/micro.py", "content": "from tornado import web\nfrom tornado.httputil import HTTPHeaders, responses\nfrom tornado.platform.asyncio import AsyncIOMainLoop\nimport asyncio\nimport uvloop\n\n\nloop = uvloop.new_event_loop()\nasyncio.set_event_loop(loop)\nAsyncIOMainLoop().install()\n\n\nclass MainHandler(web.RequestHandler):\n def get(self):\n self.write('Hello world!')\n\n # skip calculating ETag, ~8% faster\n def set_etag_header(self):\n pass\n\n def check_etag_header(self):\n return False\n\n # torando sends Server and Date headers by default, ~4% faster\n def clear(self):\n self._headers = HTTPHeaders(\n {'Content-Type': 'text/plain; charset=utf-8'})\n self._write_buffer = []\n self._status_code = 200\n self._reason = responses[200]\n\n\napp = web.Application([('/', MainHandler)])\n\napp.listen(8080)\n\nloop.run_forever()\n" }, { "path": "benchmarks/tornado/requirements.txt", "content": "tornado==4.4.2\n" }, { "path": "build.py", "content": "import argparse\nimport distutils\nfrom distutils.command.build_ext import build_ext, CompileError\nfrom distutils.core import Distribution\nfrom glob import glob\nimport os.path\nimport shutil\nimport sysconfig\nimport os\nimport sys\nimport subprocess\ntry:\n import pytoml\nexcept ImportError:\n pytoml = None\nimport runpy\n\n\nSRC_LOCATION = 'src'\nsys.path.insert(0, SRC_LOCATION)\n\n\nclass BuildSystem:\n def __init__(self, args, relative_source=False):\n self.args = args\n self.dest = self.args.dest\n self.relative_source = relative_source\n\n def get_extension_by_path(self, path):\n path = SRC_LOCATION + '/' + path\n result = runpy.run_path(path, {'system': self})\n extension = result['get_extension']()\n\n base_path = os.path.dirname(path)\n\n def fix_path(p):\n if os.path.isabs(p):\n return p\n\n return os.path.abspath(os.path.join(base_path, p))\n\n attrs = ['sources', 'include_dirs', 'library_dirs',\n 'runtime_library_dirs']\n for attr in attrs:\n val = getattr(extension, attr)\n if not val:\n continue\n\n if attr == 'sources' and self.relative_source:\n val = [\n (os.path.normpath(os.path.join(base_path, v))\n if not v.startswith('src')\n else v) for v in val]\n elif attr == 'runtime_library_dirs' and self.relative_source:\n pass\n else:\n val = [fix_path(v) for v in val]\n if attr == 'runtime_library_dirs':\n setattr(extension, attr, None)\n attr = 'extra_link_args'\n val = ['-Wl,-rpath,' + v for v in val]\n val = (getattr(extension, attr) or []) + val\n setattr(extension, attr, val)\n\n return extension\n\n def discover_extensions(self):\n self.extensions = []\n\n ext_files = glob(SRC_LOCATION + '/**/*_ext.py', recursive=True)\n ext_files = [os.path.relpath(p, SRC_LOCATION) for p in ext_files]\n self.extensions = [self.get_extension_by_path(f) for f in ext_files]\n\n return self.extensions\n\n def dest_folder(self, mod_name):\n return self.dest + '/' + '/'.join(mod_name.split('.')[:-1])\n\n def build_toml(self, mod_name):\n return self.dest + '/' + '/'.join(mod_name.split('.')) + '.build.toml'\n\n def get_so(self, ext):\n return self.dest + '/' + '/'.join(ext.name.split('.')) + '.' + \\\n sysconfig.get_config_var('SOABI') + '.so'\n\n def flags_changed(self, ext):\n toml = self.build_toml(ext.name)\n if not os.path.exists(toml):\n return True\n\n with open(toml) as f:\n flags = pytoml.load(f)\n\n ext_flags = {\n \"extra_compile_args\": ext.extra_compile_args,\n \"extra_link_args\": ext.extra_link_args,\n \"define_macros\": dict(ext.define_macros),\n \"sources\": ext.sources}\n\n return flags != ext_flags\n\n def should_rebuild(self, ext):\n so = self.get_so(ext)\n if not os.path.exists(so):\n return True\n\n so_mtime = os.stat(so).st_mtime\n\n includes = get_includes(ext)\n input_mtimes = [os.stat(s).st_mtime for s in ext.sources + includes]\n\n if max(input_mtimes) > so_mtime:\n return True\n\n if self.flags_changed(ext):\n return True\n\n return False\n\n\ndef prune(dest):\n paths = glob(os.path.join(dest, '.build/**/*.o'), recursive=True)\n paths.extend(glob(os.path.join(dest, '.build/**/*.so'), recursive=True))\n for path in paths:\n os.remove(path)\n\n\ndef profile_clean():\n paths = glob('build/**/*.gcda', recursive=True)\n for path in paths:\n os.remove(path)\n\n\ndef get_includes(ext):\n includes = []\n\n include_base = SRC_LOCATION + '/' + '/'.join(ext.name.split('.')[:-1])\n include_paths = [os.path.join(include_base, i) for i in ext.include_dirs]\n\n for source in ext.sources:\n with open(source) as f:\n for line in f:\n line = line.strip()\n if not line.startswith('#include'):\n continue\n\n header = line.split()[1][1:-1]\n for path in include_paths:\n if not os.path.exists(os.path.join(path, header)):\n continue\n\n includes.append(os.path.join(path, header))\n break\n\n return includes\n\n\ndef symlink_python_files(dest):\n if dest == SRC_LOCATION:\n return\n\n for parent, dirs, files in os.walk(SRC_LOCATION):\n if os.path.basename(parent) == '__pycache__':\n continue\n\n def _is_python_file(f):\n return f.endswith('.py') and not f.endswith('_ext.py') \\\n and not f.startswith('test_')\n\n files = [f for f in files if _is_python_file(f)]\n\n if not files:\n continue\n\n dest_parent = os.path.join(dest, *parent.split(os.sep)[1:])\n os.makedirs(dest_parent, exist_ok=True)\n for file in files:\n dst = os.path.join(dest_parent, file)\n src = os.path.relpath(os.path.join(parent, file), dest_parent)\n if os.path.exists(dst):\n os.unlink(dst)\n os.symlink(src, dst)\n\n\nkits = {\n 'platform': [\n 'japronto.request.crequest', 'japronto.protocol.cprotocol',\n 'japronto.protocol.creaper', 'japronto.router.cmatcher',\n 'japronto.response.cresponse']\n}\n\n\ndef get_parser():\n argparser = argparse.ArgumentParser('build')\n argparser.add_argument(\n '-d', dest='debug', const=True, action='store_const', default=False)\n argparser.add_argument(\n '--sanitize', dest='sanitize', const=True, action='store_const',\n default=False)\n argparser.add_argument(\n '--profile-generate', dest='profile_generate', const=True,\n action='store_const', default=False)\n argparser.add_argument('--dest', dest='dest', default='src')\n argparser.add_argument(\n '--profile-use', dest='profile_use', const=True,\n action='store_const', default=False)\n argparser.add_argument(\n '-flto', dest='flto', const=True,\n action='store_const', default=False)\n argparser.add_argument(\n '--profile-clean', dest='profile_clean', const=True,\n action='store_const', default=False)\n argparser.add_argument(\n '--disable-reaper', dest='enable_reaper', const=False,\n action='store_const', default=True)\n argparser.add_argument(\n '--disable-response-cache', dest='enable_response_cache', const=False,\n action='store_const', default=True)\n argparser.add_argument(\n '--coverage', dest='coverage', const=True,\n action='store_const', default=False)\n argparser.add_argument('-O1', dest='optimization', const='1',\n action='store_const')\n argparser.add_argument('-O2', dest='optimization', const='2',\n action='store_const')\n argparser.add_argument('-O3', dest='optimization', const='3',\n action='store_const')\n argparser.add_argument('-Os', dest='optimization', const='s',\n action='store_const')\n argparser.add_argument('-native', dest='native', const=True,\n action='store_const', default=False)\n argparser.add_argument('--path', dest='path')\n argparser.add_argument('--extra-compile', dest='extra_compile', default='')\n argparser.add_argument('--kit', dest='kit')\n\n return argparser\n\n\ndef get_platform():\n argparser = get_parser()\n args = argparser.parse_args([])\n system = BuildSystem(args, relative_source=True)\n\n ext_modules = system.discover_extensions()\n ext_modules = [e for e in ext_modules if e.name in kits['platform']]\n\n print({e.name: e.sources for e in ext_modules})\n return ext_modules\n\n\nclass custom_build_ext(build_ext):\n def build_extensions(self):\n if self.compiler.compiler_type == 'unix':\n for ext in self.extensions:\n if not ext.extra_compile_args:\n ext.extra_compiler_args = []\n extra_compile_args = ['-std=c99', '-UNDEBUG', '-D_GNU_SOURCE']\n if self.compiler.compiler_so[0].startswith('gcc') and sys.platform != 'darwin':\n extra_compile_args.append('-frecord-gcc-switches')\n ext.extra_compile_args.extend(extra_compile_args)\n compile_c(\n self.compiler,\n 'src/picohttpparser/picohttpparser.c',\n 'src/picohttpparser/ssepicohttpparser.o',\n options={'unix': ['-msse4.2']})\n compile_c(\n self.compiler,\n 'src/picohttpparser/picohttpparser.c',\n 'src/picohttpparser/picohttpparser.o')\n build_ext.build_extensions(self)\n\n\ndef compile_c(compiler, cfile, ofile, *, options=None):\n if not options:\n options = {}\n\n options = options.get(compiler.compiler_type, [])\n cmd = [*compiler.compiler_so, *options, '-c', '-o', ofile, cfile]\n print(\"building '{}'\".format(ofile))\n print(' '.join(cmd))\n subprocess.check_call(cmd)\n\n\ndef main():\n argparser = get_parser()\n args = argparser.parse_args(sys.argv[1:])\n\n if args.profile_clean:\n profile_clean()\n return\n\n distutils.log.set_verbosity(1)\n\n system = BuildSystem(args)\n\n if args.path:\n ext_modules = [system.get_extension_by_path(args.path)]\n else:\n ext_modules = system.discover_extensions()\n\n if args.kit:\n ext_modules = [e for e in ext_modules if e.name in kits[args.kit]]\n\n def add_args(arg_name, values, append=True):\n for ext_module in ext_modules:\n arg_value = getattr(ext_module, arg_name) or []\n if append:\n arg_value.extend(values)\n else:\n newvalues = list(values)\n newvalues.extend(arg_value)\n arg_value = newvalues\n setattr(ext_module, arg_name, arg_value)\n\n def append_compile_args(*values):\n add_args('extra_compile_args', values)\n\n def append_link_args(*values):\n add_args('extra_link_args', values)\n\n def prepend_libraries(*values):\n add_args('libraries', values, append=False)\n\n if args.native:\n append_compile_args('-march=native')\n if args.optimization:\n append_compile_args('-O' + args.optimization)\n if args.debug:\n append_compile_args('-g3', '-O0', '-Wp,-U_FORTIFY_SOURCE')\n if args.sanitize:\n append_compile_args('-g3', '-fsanitize=address',\n '-fsanitize=undefined', '-fno-common',\n '-fno-omit-frame-pointer')\n prepend_libraries('asan', 'ubsan')\n if args.profile_generate:\n append_compile_args('--profile-generate')\n append_link_args('-lgcov')\n if args.profile_use:\n for ext_module in ext_modules:\n if ext_module.name in ('parser.cparser', 'pipeline.cpipeline'):\n continue\n ext_module.extra_compile_args.append('--profile-use')\n if args.flto:\n append_compile_args('-flto')\n append_link_args('-flto')\n if args.coverage:\n append_compile_args('--coverage')\n append_link_args('-lgcov')\n if args.extra_compile:\n append_compile_args(args.extra_compile)\n\n ext_modules = [e for e in ext_modules if system.should_rebuild(e)]\n if not ext_modules:\n return\n\n dist = Distribution(dict(ext_modules=ext_modules))\n\n prune(args.dest)\n\n cmd = custom_build_ext(dist)\n cmd.build_lib = os.path.join(args.dest, '.build/lib')\n cmd.build_temp = os.path.join(args.dest, '.build/temp')\n cmd.finalize_options()\n\n try:\n cmd.run()\n except CompileError:\n sys.exit(1)\n\n symlink_python_files(args.dest)\n\n for ext_module in ext_modules:\n os.makedirs(system.dest_folder(ext_module.name), exist_ok=True)\n shutil.copy(\n cmd.get_ext_fullpath(ext_module.name),\n system.dest_folder(ext_module.name))\n\n for ext_module in ext_modules:\n with open(system.build_toml(ext_module.name), 'w') as f:\n build_info = {\n 'extra_compile_args': ext_module.extra_compile_args,\n 'extra_link_args': ext_module.extra_link_args,\n 'define_macros': dict(ext_module.define_macros),\n 'sources': ext_module.sources\n }\n pytoml.dump(build_info, f)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "cases/__init__.py", "content": "from collections import namedtuple\nimport glob\nimport os.path\n\nimport pytoml\nimport pytest\n\n\ntestcase_fields = 'data,method,path,version,headers,body,error,disconnect'\n\nHttpTestCase = namedtuple('HTTPTestCase', testcase_fields)\n\n\ndef parse_casesel(suite, casesel):\n for casespec in casesel.split('+'):\n *transforms, case = casespec.split(':')\n if case.endswith('!'):\n transforms.append('!')\n case = case[:-1]\n case = suite[case]\n\n for transform in reversed(transforms):\n func, *args = transform.split()\n case = transorm_dict[func](case, *args)\n\n yield case\n\n\ndef parametrize_cases(suite, *args):\n suite = suites[suite]\n cases_list = [\n list(parse_casesel(suite, sel)) for sel in args]\n return pytest.mark.parametrize('cases', cases_list, ids=args)\n\n\ndef load_casefile(path):\n result = {}\n\n with open(path) as casefile:\n cases = pytoml.load(casefile)\n\n for case_name, case_data in cases.items():\n case_data['data'] = case_data['data'].encode('utf-8')\n case_data['body'] = case_data['body'].encode('utf-8') \\\n if 'body' in case_data else None\n case_data['disconnect'] = False\n case = HttpTestCase._make(\n case_data.get(f) for f in testcase_fields.split(','))\n result[case_name] = case\n\n return result\n\n\ndef load_cases():\n cases = {}\n\n for filename in glob.glob('cases/*.toml'):\n suite_name, _ = os.path.splitext(os.path.basename(filename))\n cases[suite_name] = load_casefile(filename)\n\n return cases\n\n\ndef keep_alive(case):\n headers = case.headers.copy()\n headers['Connection'] = 'keep-alive'\n# if case.body is not None \\\n# and headers.get('Transfer-Encoding', 'identity') == 'identity':\n# headers['Content-Length'] = str(len(case.body))\n\n return update_case(case, headers)\n\ndef close(case):\n headers = case.headers.copy()\n headers['Connection'] = 'close'\n\n return update_case(case, headers)\n\n\ndef should_keep_alive(case):\n return case.headers.get(\n 'Connection',\n 'close' if case.version == '1.0' else 'keep-alive') == 'keep-alive'\n\n\ndef set_error(case, error):\n return update_case(case, error=error)\n\n\ndef disconnect(case):\n return update_case(case, disconnect=True)\n\n\ndef update_case(case, headers=False, error=False, disconnect=None):\n data = False\n if headers:\n data = bytearray()\n status = case.method + ' ' + case.path + ' HTTP/' + case.version + '\\r\\n'\n data += status.encode('ascii')\n for name, value in headers.items():\n data += name.encode('ascii') + b': ' + value.encode('latin1') + b'\\r\\n'\n data += b'\\r\\n'\n if case.body:\n data += case.body\n\n headers = headers or case.headers\n data = data or case.data\n error = error or case.error\n disconnect = disconnect if disconnect is not None else case.disconnect\n\n return case._replace(\n headers=headers, error=error, disconnect=disconnect,\n data=bytes(data))\n\n\ntransorm_dict = {\n 'keep': keep_alive,\n 'close': close,\n 'e': set_error,\n '!': disconnect\n}\n\n\nsuites = load_cases()\nglobals().update(suites)\n" }, { "path": "cases/base.toml", "content": "[10msg]\ndata = \"\"\"\\\nPOST \\\n/wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg \\\nHTTP/1.0\\r\\n\\\nHOST: www.kittyhell.com\\r\\n\\\nUser-Agent: Mozilla/5.0 \\\n (Macintosh; U; Intel Mac OS X 10.6; ja-JP-mac; rv:1.9.2.3) \\\n Gecko/20100401 Firefox/3.6.3 Pathtraq/0.9\\r\\n\\\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\\r\\n\\\nAccept-Language: ja,en-us;q=0.7,en;q=0.3\\r\\n\\\nAccept-Encoding: gzip,deflate\\r\\n\\\nAccept-Charset: Shift_JIS,utf-8;q=0.7,*;q=0.7\\r\\n\\\nKeep-Alive: 115\\r\\n\\\nCookie: wp_ozh_wsa_visits=2; \\\n wp_ozh_wsa_visit_lasttime=xxxxxxxxxx; \\\n __utma=xxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.x; \\\n __utmz=xxxxxxxxx.xxxxxxxxxx.x.x.utmccn=(referral)|utmcsr=reader.livedoor.com|utmcct=/reader/|utmcmd=referral\\r\\n\\\nContent-Length: 11\\r\\n\\\n\\r\\n\\\nHello there\\\n\"\"\"\n\nmethod = \"POST\"\npath = \"/wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg\"\nversion = \"1.0\"\nbody = \"Hello there\"\n[10msg.headers]\nHost = \"www.kittyhell.com\"\nUser-Agent = \"\"\"Mozilla/5.0 \\\n (Macintosh; U; Intel Mac OS X 10.6; ja-JP-mac; rv:1.9.2.3) \\\n Gecko/20100401 Firefox/3.6.3 Pathtraq/0.9\"\"\"\nAccept = \"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\"\nAccept-Language = \"ja,en-us;q=0.7,en;q=0.3\"\nAccept-Encoding = \"gzip,deflate\"\nAccept-Charset = \"Shift_JIS,utf-8;q=0.7,*;q=0.7\"\nKeep-Alive = \"115\"\nCookie = \"\"\"wp_ozh_wsa_visits=2; \\\n wp_ozh_wsa_visit_lasttime=xxxxxxxxxx; \\\n __utma=xxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.x; \\\n __utmz=xxxxxxxxx.xxxxxxxxxx.x.x.utmccn=(referral)|utmcsr=reader.livedoor.com|utmcct=/reader/|utmcmd=referral\"\"\"\nContent-Length = \"11\"\n\n[10get]\ndata = \"GET / HTTP/1.0\\r\\nHost: www.example.com\\r\\n\\r\\n\"\nmethod = \"GET\"\npath = \"/\"\nversion = \"1.0\"\nheaders = { Host = \"www.example.com\" }\n\n\n[10malformed_headers1]\ndata = \"GET / HTTP 1.0\"\nerror = \"malformed_headers\"\n\n\n[10malformed_headers2]\ndata = \"GET / HTTP/2\"\nerror = \"malformed_headers\"\n\n\n[10incomplete_headers]\ndata = \"GET / HTTP/1.0\\r\\nH\"\nerror = \"incomplete_headers\"\n\n\n[11get]\ndata = \"GET /index.html HTTP/1.1\\r\\n\\r\\n\"\nmethod = \"GET\"\npath = \"/index.html\"\nversion = \"1.1\"\nheaders = {}\n# body = null\n\n\n[11getmsg]\ndata = \"\"\"\\\nGET /body HTTP/1.1\\r\\n\\\nContent-Length: 12\\r\\n\\\n\\r\\n\\\nHello World!\\\n\"\"\"\nmethod = \"GET\"\npath = \"/body\"\nversion = \"1.1\"\nheaders = { Content-Length = \"12\" }\nbody = \"Hello World!\"\n\n\n[11msg]\ndata = \"POST /login HTTP/1.1\\r\\nContent-Length: 5\\r\\n\\r\\nHello\"\nmethod = \"POST\"\npath = \"/login\"\nversion = \"1.1\"\nheaders = { Content-Length = \"5\" }\nbody = \"Hello\"\n\n\n[11msgzero]\ndata = \"POST /zero HTTP/1.1\\r\\nContent-Length: 0\\r\\n\\r\\n\"\nmethod = \"POST\"\npath = \"/zero\"\nversion = \"1.1\"\nheaders = { Content-Length = \"0\" }\nbody = \"\"\n\n\n[11clincomplete_headers]\ndata = \"\"\"\\\nPOST / HTTP/1.1\\r\\n\\\nContent-Length: 3\\r\\n\\\nI\"\"\"\nerror = \"incomplete_headers\"\n\n\n[11clincomplete_body]\ndata = \"POST / HTTP/1.1\\r\\nContent-Length: 5\\r\\n\\r\\nI\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Content-Length = \"5\" }\nerror = \"incomplete_body\"\n\n[11clinvalid1]\ndata = \"POST / HTTP/1.1\\r\\nContent-Length: asd\\r\\n\\r\\n\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Content-Length = \"asd\" }\nerror = \"invalid_headers\"\n\n[11clinvalid2]\ndata = \"POST / HTTP/1.1\\r\\nContent-Length: +5\\r\\n\\r\\n\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Content-Length = \"+5\" }\nerror = \"invalid_headers\"\n\n[11clinvalid3]\ndata = \"POST / HTTP/1.1\\r\\nContent-Length: -5\\r\\n\\r\\n\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Content-Length = \"+5\" }\nerror = \"invalid_headers\"\n\n[11clinvalid4]\ndata = \"POST / HTTP/1.1\\r\\nContent-Length: 4f\\r\\n\\r\\n\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Content-Length = \"4f\" }\nerror = \"invalid_headers\"\n\n[11clinvalid5]\ndata = \"POST / HTTP/1.1\\r\\nContent-Length: \\r\\n\\r\\n\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Content-Length = \"\" }\nerror = \"invalid_headers\"\n\n\n[11chunked1]\ndata = \"\"\"\\\nPOST /chunked HTTP/1.1\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n4\\r\\n\\\nWiki\\r\\n\\\n5\\r\\n\\\npedia\\r\\n\\\nE\\r\\n in\\r\\n\\\n\\r\\n\\\nchunks.\\r\\n\\\n0\\r\\n\\\n\\r\\n\\\n\"\"\"\nmethod = \"POST\"\npath = \"/chunked\"\nversion = \"1.1\"\nheaders = { Transfer-Encoding = \"chunked\" }\nbody = \"Wikipedia in\\r\\n\\r\\nchunks.\"\n\n[11chunkedzero]\ndata = \"\"\"\nPUT /zero HTTP/1.1\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n0\\r\\n\\\n\\r\\n\\\n\"\"\"\nmethod = \"PUT\"\npath = \"/zero\"\nversion = \"1.1\"\nheaders = { Transfer-Encoding = \"chunked\" }\nbody = \"\"\n\n[11chunked2]\ndata = \"\"\"\\\nPOST /chunked HTTP/1.1\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n1;token=123;x=3\\r\\n\\\nr\\r\\n\\\n0\\r\\n\\\n\\r\\n\\\n\"\"\"\nmethod = \"POST\"\npath = \"/chunked\"\nversion = \"1.1\"\nheaders = { Transfer-Encoding = \"chunked\" }\nbody = \"r\"\n\n\n[11chunked3]\ndata = \"\"\"\\\nPOST / HTTP/1.1\\r\\n\\\nTransfer-Encoding: chunked\\r\\n\\\n\\r\\n\\\n000002\\r\\n\\\nab\\r\\n\\\n0;q=1\\r\\n\\\nThis: is trailer header\\r\\n\\\n\\r\\n\\\n\"\"\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Transfer-Encoding = \"chunked\" }\nbody = \"ab\"\n\n\n[11chunkedincomplete_body]\ndata = \"POST / HTTP/1.1\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n10\\r\\nasd\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Transfer-Encoding = \"chunked\" }\nerror = \"incomplete_body\"\n\n\n[11chunkedmalformed_body]\ndata = \"POST / HTTP/1.1\\r\\nTransfer-Encoding: chunked\\r\\n\\r\\n1x\\r\\nhello\"\nmethod = \"POST\"\npath = \"/\"\nversion = \"1.1\"\nheaders = { Transfer-Encoding = \"chunked\" }\nerror = \"malformed_body\"\n" }, { "path": "cases/websites.toml", "content": "[github]\ndata = \"\"\"\\\nGET / HTTP/1.1\\r\\n\\\nHost: github.com\\r\\n\\\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0\\r\\n\\\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\\r\\n\\\nAccept-Language: en-US,en;q=0.5\\r\\n\\\nAccept-Encoding: gzip, deflate, br\\r\\n\\\nCookie: _octo=GHx; logged_in=yes; _ga=GAx; user_session=x; dotcom_user=x; __Host-user_session_same_site=x; _gh_sess=x; tz=America%2FSao_Paulo; _gat=x\\r\\n\\\nConnection: keep-alive\\r\\n\\\nUpgrade-Insecure-Requests: 1\\r\\n\\\nPragma: no-cache\\r\\n\\\nCache-Control: no-cache\\r\\n\\\n\\r\\n\\\n\"\"\"\n\n[google]\ndata = \"\"\"\\\nGET / HTTP/1.1\\r\\n\\\nHost: google.com\\r\\n\\\nConnection: keep-alive\\r\\n\\\nUpgrade-Insecure-Requests: 1\\r\\n\\\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36\\r\\n\\\nX-Client-Data: Qwerty\\r\\n\\\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\\r\\n\\\nAccept-Encoding: gzip, deflate, sdch\\r\\n\\\nAccept-Language: en-US,en;q=0.8,es;q=0.6\\r\\n\\\nCookie: NID=x; SID=x; HSID=x; APISID=x\\r\\n\\\n\\r\\n\\\n\"\"\"\n\n[amazon]\ndata = \"\"\"\\\nGET / HTTP/1.1\\r\\n\\\nHost: www.amazon.com\\r\\n\\\nConnection: keep-alive\\r\\n\\\nUpgrade-Insecure-Requests: 1\\r\\n\\\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.50 Safari/537.36 OPR/41.0.2353.23 (Edition beta)\\r\\n\\\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\\r\\n\\\nAccept-Encoding: gzip, deflate, lzma, sdch, br\\r\\n\\\nAccept-Language: en-US,en;q=0.8\\r\\n\\\n\\r\\n\\\n\"\"\"\n\n[xkcd]\ndata = \"\"\"\\\nGET /comics/mushrooms.png HTTP/1.1\\r\\n\\\nHost: imgs.xkcd.com\\r\\n\\\nConnection: keep-alive\\r\\n\\\nUpgrade-Insecure-Requests: 1\\r\\n\\\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.50 Safari/537.36 OPR/41.0.2353.23 (Edition beta)\\r\\n\\\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\\r\\n\\\nAccept-Encoding: gzip, deflate, lzma, sdch\\r\\n\\\nAccept-Language: en-US,en;q=0.8\\r\\n\\\n\\r\\n\\\n\"\"\"\n\n[4chan]\ndata = \"\"\"\\\nGET /image/favicon.ico HTTP/1.1\\r\\n\\\nHost: s.4cdn.org\\r\\n\\\nConnection: keep-alive\\r\\n\\\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36\\r\\n\\\nAccept: */*\\r\\n\\\nReferer: http://www.4chan.org/\\r\\n\\\nAccept-Encoding: gzip, deflate, sdch\\r\\n\\\nAccept-Language: en-US,en;q=0.8,es;q=0.6\\r\\n\\\n\\r\\n\\\n\"\"\"\n" }, { "path": "conftest.py", "content": "import subprocess\nimport sys\nimport os\nimport shutil\n\n\nbuilds = []\ncoverages = set()\n\n\ndef add_build(mark):\n global builds\n args, kwargs = list(mark.args), mark.kwargs.copy()\n kwargs.pop('coverage', None)\n cfg = args, kwargs\n if cfg not in builds:\n builds.append(cfg)\n\n\ndef execute_builds():\n common_options = ['--coverage', '-d', '--sanitize']\n for args, kwargs in builds:\n build_options = args[:]\n build_options.extend(['--dest', kwargs.get('dest', '.test')])\n if 'kit' not in kwargs:\n build_options.extend(['--kit', 'platform'])\n build_options.extend(common_options)\n\n print('Executing build', *build_options)\n subprocess.check_call([sys.executable, 'build.py', *build_options])\n\n\ndef add_coverage(mark):\n dest = mark.kwargs.get('dest', '.test')\n coverages.add(dest)\n\n\ndef setup_coverage():\n if coverages:\n print('Setting up C coverage for', *coverages)\n\n for dest in coverages:\n subprocess.check_call([\n 'lcov', '--base-directory', '.', '--directory',\n dest + '/.build/temp', '--zerocounters', '-q'])\n\n\ndef make_coverage():\n for dest in coverages:\n try:\n os.unlink(dest + '/coverage.info')\n except FileNotFoundError:\n pass\n\n subprocess.check_call([\n 'lcov', '--base-directory', '.', '--directory',\n dest + '/.build/temp', '-c', '-o', dest + '/coverage.info', '-q'])\n subprocess.check_call([\n 'lcov', '--remove', dest + '/coverage.info',\n '/usr*', '-o', 'coverage.info', '-q'])\n\n try:\n shutil.rmtree(dest + '/coverage_report')\n except FileNotFoundError:\n pass\n\n subprocess.check_call([\n 'genhtml', '-o', dest + '/coverage_report',\n dest + '/coverage.info', '-q'\n ])\n\n print('C coverage report saved in',\n dest + '/coverage_report/index.html')\n\n\ndef pytest_itemcollected(item):\n needs_build = item.get_closest_marker('needs_build')\n if needs_build:\n add_build(needs_build)\n if needs_build and needs_build.kwargs.get('coverage'):\n add_coverage(needs_build)\n\n\ndef pytest_collection_modifyitems(config, items):\n execute_builds()\n setup_coverage()\n\n\ndef pytest_unconfigure():\n make_coverage()\n" }, { "path": "do_wrk.py", "content": "import argparse\nimport sys\nimport asyncio as aio\nimport os\nfrom asyncio.subprocess import PIPE, STDOUT\nimport statistics\n\nimport uvloop\nimport psutil\n\nfrom misc import cpu\nfrom misc import buggers\n\n\ndef run_wrk(loop, endpoint=None):\n endpoint = endpoint or 'http://localhost:8080'\n wrk_fut = aio.create_subprocess_exec(\n './wrk', '-t', '1', '-c', '100', '-d', '2', '-s', 'misc/pipeline.lua',\n endpoint, stdout=PIPE, stderr=STDOUT)\n\n wrk = loop.run_until_complete(wrk_fut)\n\n lines = []\n while 1:\n line = loop.run_until_complete(wrk.stdout.readline())\n if line:\n line = line.decode('utf-8')\n lines.append(line)\n if line.startswith('Requests/sec:'):\n rps = float(line.split()[-1])\n else:\n break\n\n retcode = loop.run_until_complete(wrk.wait())\n if retcode != 0:\n print('\\r\\n'.join(lines))\n\n return rps\n\n\ndef cpu_usage(p):\n return p.cpu_percent() + sum(c.cpu_percent() for c in p.children())\n\n\ndef connections(process):\n return len(\n set(c.fd for c in process.connections()) |\n set(c.fd for p in process.children() for c in p.connections()))\n\n\ndef memory(p):\n return p.memory_percent('uss') \\\n + sum(c.memory_percent('uss') for c in p.children())\n\n\nif __name__ == '__main__':\n buggers.silence()\n loop = uvloop.new_event_loop()\n\n argparser = argparse.ArgumentParser('do_wrk')\n argparser.add_argument('-s', dest='server', default='')\n argparser.add_argument('-e', dest='endpoint')\n argparser.add_argument('--pid', dest='pid', type=int)\n argparser.add_argument(\n '--no-cpu', dest='cpu_change', default=True,\n action='store_const', const=False)\n\n args = argparser.parse_args(sys.argv[1:])\n\n if args.cpu_change:\n cpu.change('userspace', cpu.min_freq())\n cpu.dump()\n\n aio.set_event_loop(loop)\n\n if not args.endpoint:\n os.putenv('PYTHONPATH', 'src')\n server_fut = aio.create_subprocess_exec(\n 'python', 'benchmarks/japronto/micro.py', *args.server.split())\n server = loop.run_until_complete(server_fut)\n os.unsetenv('PYTHONPATH')\n if not args.endpoint:\n process = psutil.Process(server.pid)\n elif args.pid:\n process = psutil.Process(args.pid)\n else:\n process = None\n\n cpu_p = 100\n while cpu_p > 5:\n cpu_p = psutil.cpu_percent(interval=1)\n print('CPU usage in 1 sec:', cpu_p)\n\n results = []\n cpu_usages = []\n process_cpu_usages = []\n mem_usages = []\n conn_cnt = []\n if process:\n cpu_usage(process)\n for _ in range(10):\n results.append(run_wrk(loop, args.endpoint))\n cpu_usages.append(psutil.cpu_percent())\n if process:\n process_cpu_usages.append(cpu_usage(process))\n conn_cnt.append(connections(process))\n mem_usages.append(round(memory(process), 2))\n print('.', end='')\n sys.stdout.flush()\n\n if not args.endpoint:\n server.terminate()\n loop.run_until_complete(server.wait())\n\n if args.cpu_change:\n cpu.change('ondemand')\n\n print()\n print('RPS', results)\n print('Mem', mem_usages)\n print('Conn', conn_cnt)\n print('Server', process_cpu_usages)\n print('System', cpu_usages)\n median = statistics.median_grouped(results)\n stdev = round(statistics.stdev(results), 2)\n p = round((stdev / median) * 100, 2)\n print('median:', median, 'stdev:', stdev, '%', p)\n" }, { "path": "examples/1_hello/hello.py", "content": "from japronto import Application\n\n\n# Views handle logic, take request as a parameter and\n# return the Response object back to the client\ndef hello(request):\n return request.Response(text='Hello world!')\n\n\n# The Application instance is a fundamental concept.\n# It is a parent to all the resources and all the settings\n# can be tweaked there.\napp = Application()\n\n# The Router instance lets you register your handlers and execute\n# them depending on the url path and methods.\napp.router.add_route('/', hello)\n\n# Finally, start our server and handle requests until termination is\n# requested. Enabling debug lets you see request logs and stack traces.\napp.run(debug=True)\n" }, { "path": "examples/2_async/async.py", "content": "import asyncio\nfrom japronto import Application\n\n\n# This is a synchronous handler.\ndef synchronous(request):\n return request.Response(text='I am synchronous!')\n\n\n# This is an asynchronous handler. It spends most of the time in the event loop.\n# It wakes up every second 1 to print and finally returns after 3 seconds.\n# This lets other handlers execute in the same processes while\n# from the point of view of the client it took 3 seconds to complete.\nasync def asynchronous(request):\n for i in range(1, 4):\n await asyncio.sleep(1)\n print(i, 'seconds elapsed')\n\n return request.Response(text='3 seconds elapsed')\n\n\napp = Application()\n\nr = app.router\nr.add_route('/sync', synchronous)\nr.add_route('/async', asynchronous)\n\napp.run()\n" }, { "path": "examples/3_router/router.py", "content": "from japronto import Application\n\n\napp = Application()\nr = app.router\n\n\n# Requests with the path set exactly to `/` and whatever method\n# will be directed here.\ndef slash(request):\n return request.Response(text='Hello {} /!'.format(request.method))\n\n\nr.add_route('/', slash)\n\n\n# Requests with the path set exactly to '/love' and the method\n# set exactly to `GET` will be directed here.\ndef get_love(request):\n return request.Response(text='Got some love')\n\n\nr.add_route('/love', get_love, 'GET')\n\n\n# Requests with the path set exactly to '/methods' and the method\n# set to `POST` or `DELETE` will be directed here.\ndef methods(request):\n return request.Response(text=request.method)\n\n\nr.add_route('/methods', methods, methods=['POST', 'DELETE'])\n\n\n# Requests with the path starting with `/params/` segment and followed\n# by two additional segments will be directed here.\n# Values of the additional segments will be stored inside `request.match_dict`\n# dictionary with keys taken from {} placeholders. A request to `/params/1/2`\n# would leave `match_dict` set to `{'p1': 1, 'p2': '2'}`.\ndef params(request):\n return request.Response(text=str(request.match_dict))\n\n\nr.add_route('/params/{p1}/{p2}', params)\n\napp.run()\n" }, { "path": "examples/4_request/request.py", "content": "from json import JSONDecodeError\n\nfrom japronto import Application\n\n\n# Request line and headers.\n# This represents the part of a request that comes before message body.\n# Given an HTTP 1.1 `GET` request to `/basic?a=1` this would yield\n# `method` set to `GET`, `path` set to `/basic`, `version` set to `1.1`\n# `query_string` set to `a=1` and `query` set to `{'a': '1'}`.\n# Additionally if headers are sent they will be present in `request.headers`\n# dictionary. The keys are normalized to standard `Camel-Cased` convention.\ndef basic(request):\n text = \"\"\"Basic request properties:\n Method: {0.method}\n Path: {0.path}\n HTTP version: {0.version}\n Query string: {0.query_string}\n Query: {0.query}\"\"\".format(request)\n\n if request.headers:\n text += \"\\nHeaders:\\n\"\n for name, value in request.headers.items():\n text += \" {0}: {1}\\n\".format(name, value)\n\n return request.Response(text=text)\n\n\n# Message body\n# If there is a message body attached to a request (as in a case of `POST`)\n# the following attributes can be used to examine it.\n# Given a `POST` request with body set to `b'J\\xc3\\xa1'`, `Content-Length` header set\n# to `3` and `Content-Type` header set to `text/plain; charset=utf-8` this\n# would yield `mime_type` set to `'text/plain'`, `encoding` set to `'utf-8'`,\n# `body` set to `b'J\\xc3\\xa1'` and `text` set to `'Já'`.\n# `form` and `files` attributes are dictionaries respectively used for HTML forms and\n# HTML file uploads. The `json` helper property will try to decode `body` as a\n# JSON document and give you resulting Python data type.\ndef body(request):\n text = \"\"\"Body related properties:\n Mime type: {0.mime_type}\n Encoding: {0.encoding}\n Body: {0.body}\n Text: {0.text}\n Form parameters: {0.form}\n Files: {0.files}\n \"\"\".format(request)\n\n try:\n json = request.json\n except JSONDecodeError:\n pass\n else:\n text += \"\\nJSON:\\n\"\n text += str(json)\n\n return request.Response(text=text)\n\n\n# Miscellaneous\n# `route` will point to an instance of `Route` object representing\n# route chosen by router to handle this request. `hostname` and `port`\n# represent parsed `Host` header if any. `remote_addr` is the address of\n# a client or reverse proxy. If `keep_alive` is true the client requested to\n# keep the connection open after the response is delivered. `match_dict` contains\n# route placeholder values as documented in `2_router.md`. `cookies` contains\n# a dictionary of HTTP cookies if any.\ndef misc(request):\n text = \"\"\"Miscellaneous:\n Matched route: {0.route}\n Hostname: {0.hostname}\n Port: {0.port}\n Remote address: {0.remote_addr},\n HTTP Keep alive: {0.keep_alive}\n Match parameters: {0.match_dict}\n \"\"\".strip().format(request)\n\n if request.cookies:\n text += \"\\nCookies:\\n\"\n for name, value in request.cookies.items():\n text += \" {0}: {1}\\n\".format(name, value)\n\n return request.Response(text=text)\n\n\napp = Application()\napp.router.add_route('/basic', basic)\napp.router.add_route('/body', body)\napp.router.add_route('/misc', misc)\napp.run()\n" }, { "path": "examples/5_response/response.py", "content": "import random\nfrom http.cookies import SimpleCookie\n\nfrom japronto.app import Application\n\n\n# Providing just a text argument yields a `text/plain` response\n# encoded with `utf8` codec (charset set accordingly)\ndef text(request):\n return request.Response(text='Hello world!')\n\n\n# You can override encoding by providing the `encoding` attribute.\ndef encoding(request):\n return request.Response(text='Já pronto!', encoding='iso-8859-1')\n\n\n# You can also set a custom MIME type.\ndef mime(request):\n return request.Response(\n mime_type=\"image/svg+xml\",\n text=\"\"\"\n \n \n \n \"\"\")\n\n\n# Or serve binary data. `Content-Type` is set to `application/octet-stream`\n# automatically but you can always provide your own `mime_type`.\ndef body(request):\n return request.Response(body=b'\\xde\\xad\\xbe\\xef')\n\n\n# There exist a shortcut `json` argument. This automatically encodes the\n# provided object as JSON and servers it with `Content-Type` set to\n# `application/json; charset=utf8`\ndef json(request):\n return request.Response(json={'hello': 'world'})\n\n\n# You can change the default 200 status `code` for another\ndef code(request):\n return request.Response(code=random.choice([200, 201, 400, 404, 500]))\n\n\n# And of course you can provide custom `headers`.\ndef headers(request):\n return request.Response(\n text='headers',\n headers={'X-Header': 'Value',\n 'Refresh': '5; url=https://xkcd.com/353/'})\n\n\n# Or `cookies` by using Python standard library `http.cookies.SimpleCookie`.\ndef cookies(request):\n cookies = SimpleCookie()\n cookies['hello'] = 'world'\n cookies['hello']['domain'] = 'localhost'\n cookies['hello']['path'] = '/'\n cookies['hello']['max-age'] = 3600\n cookies['city'] = 'São Paulo'\n\n return request.Response(text='cookies', cookies=cookies)\n\n\napp = Application()\nrouter = app.router\nrouter.add_route('/text', text)\nrouter.add_route('/encoding', encoding)\nrouter.add_route('/mime', mime)\nrouter.add_route('/body', body)\nrouter.add_route('/json', json)\nrouter.add_route('/code', code)\nrouter.add_route('/headers', headers)\nrouter.add_route('/cookies', cookies)\napp.run()\n" }, { "path": "examples/6_exceptions/exceptions.py", "content": "from japronto import Application, RouteNotFoundException\n\n\n# These are our custom exceptions we want to turn into 200 response.\nclass KittyError(Exception):\n def __init__(self):\n self.greet = 'meow'\n\n\nclass DoggieError(Exception):\n def __init__(self):\n self.greet = 'woof'\n\n\n# The two handlers below raise exceptions which will be turned\n# into 200 responses by the handlers registered later\ndef cat(request):\n raise KittyError()\n\n\ndef dog(request):\n raise DoggieError()\n\n\n# This handler raises ZeroDivisionError which doesn't have an error\n# handler registered so it will result in 500 Internal Server Error\ndef unhandled(request):\n 1 / 0\n\n\napp = Application()\n\nr = app.router\nr.add_route('/cat', cat)\nr.add_route('/dog', dog)\nr.add_route('/unhandled', unhandled)\n\n\n# These two are handlers for `Kitty` and `DoggyError`s.\ndef handle_cat(request, exception):\n return request.Response(text='Just a kitty, ' + exception.greet)\n\n\ndef handle_dog(request, exception):\n return request.Response(text='Just a doggie, ' + exception.greet)\n\n\n# You can also override default 404 handler if you want\ndef handle_not_found(request, exception):\n return request.Response(code=404, text=\"Are you lost, pal?\")\n\n\n# register all the error handlers so they are actually effective\napp.add_error_handler(KittyError, handle_cat)\napp.add_error_handler(DoggieError, handle_dog)\napp.add_error_handler(RouteNotFoundException, handle_not_found)\n\napp.run()\n" }, { "path": "examples/7_extend/extend.py", "content": "from japronto import Application\n\n\n# This view accesses custom method host_startswith\n# and a custom property reversed_agent. Both are registered later.\ndef extended_hello(request):\n if request.host_startswith('api.'):\n text = 'Hello ' + request.reversed_agent\n else:\n text = 'Hello stranger'\n\n return request.Response(text=text)\n\n\n# This view registers a callback, such callbacks are executed after handler\n# exits and the response is ready to be sent over the wire.\ndef with_callback(request):\n def cb(r):\n print('Done!')\n\n request.add_done_callback(cb)\n\n return request.Response(text='cb')\n\n\n# This is a body for reversed_agent property\ndef reversed_agent(request):\n return request.headers['User-Agent'][::-1]\n\n\n# This is a body for host_startswith method\n# Custom methods and properties always accept request\n# object.\ndef host_startswith(request, prefix):\n return request.headers['Host'].startswith(prefix)\n\n\napp = Application()\n# Finally register the custom property and method\n# By default the names are taken from function names\n# unelss you provide `name` keyword parameter.\napp.extend_request(reversed_agent, property=True)\napp.extend_request(host_startswith)\n\nr = app.router\nr.add_route('/', extended_hello)\nr.add_route('/callback', with_callback)\n\n\napp.run()\n" }, { "path": "examples/8_template/index.html", "content": "\n\n\n\n \n\n japronto\n \n\n\n\n

Hello World!

\n\n

Behold, the power of japronto!

\n\n\n" }, { "path": "examples/8_template/template.py", "content": "# examples/8_template/template.py\nfrom japronto import Application\nfrom jinja2 import Template\n\n\n# A view can read HTML from a file\ndef index(request):\n with open('index.html') as html_file:\n return request.Response(text=html_file.read(), mime_type='text/html')\n\n\n# A view could also return a raw HTML string\ndef example(request):\n return request.Response(text='

Some HTML!

', mime_type='text/html')\n\n\ntemplate = Template('

Hello {{ name }}!

')\n\n# A view could also return a rendered jinja2 template\ndef jinja(request):\n return request.Response(text=template.render(name='World'), \n mime_type='text/html')\n\n\n# Create the japronto application\napp = Application()\n\n# Add routes to the app\napp.router.add_route('/', index)\napp.router.add_route('/example', example)\napp.router.add_route('/jinja2', jinja)\n\n# Start the server\napp.run(debug=True)\n\n" }, { "path": "examples/todo_api/.gitignore", "content": "todo.sqlite\n" }, { "path": "examples/todo_api/todo_api.py", "content": "import os.path\nimport sqlite3\nfrom functools import partial\n\nfrom japronto import Application\n\n\ndef add_todo(request):\n cur = request.cursor\n todo = request.json[\"todo\"]\n cur.execute(\"\"\"INSERT INTO todos (todo) VALUES (?)\"\"\", (todo,))\n last_id = cur.lastrowid\n cur.connection.commit()\n\n return request.Response(json={\"id\": last_id, \"todo\": todo})\n\n\ndef list_todos(request):\n cur = request.cursor\n cur.execute(\"\"\"SELECT id, todo FROM todos\"\"\")\n todos = [{\"id\": id, \"todo\": todo} for id, todo in cur]\n\n return request.Response(json={\"results\": todos})\n\n\ndef show_todo(request):\n cur = request.cursor\n id = int(request.match_dict['id'])\n cur.execute(\"\"\"SELECT id, todo FROM todos WHERE id = ?\"\"\", (id,))\n todo = cur.fetchone()\n if not todo:\n return request.Response(code=404, json={\"error\": \"not found\"})\n todo = {\"id\": todo[0], \"todo\": todo[1]}\n\n return request.Response(json=todo)\n\n\ndef delete_todo(request):\n cur = request.cursor\n id = int(request.match_dict['id'])\n cur.execute(\"\"\"DELETE FROM todos WHERE id = ?\"\"\", (id,))\n if not cur.rowcount:\n return request.Response(code=404, json={\"error\": \"not found\"})\n cur.connection.commit()\n\n return request.Response(json={})\n\n\nDB_FILE = os.path.abspath(\n os.path.join(os.path.dirname(__file__), 'todo.sqlite'))\ndb_connect = partial(sqlite3.connect, DB_FILE)\n\n\ndef maybe_create_schema():\n db = db_connect()\n db.execute(\"\"\"\n CREATE TABLE IF NOT EXISTS todos\n (id INTEGER PRIMARY KEY, todo TEXT)\"\"\")\n db.close()\n\n\nmaybe_create_schema()\napp = Application()\n\n\ndef cursor(request):\n def done_cb(request):\n request.extra['conn'].close()\n\n if 'conn' not in request.extra:\n request.extra['conn'] = db_connect()\n request.add_done_callback(done_cb)\n\n return request.extra['conn'].cursor()\n\n\napp.extend_request(cursor, property=True)\nrouter = app.router\nrouter.add_route('/todos', list_todos, method='GET')\nrouter.add_route('/todos/{id}', show_todo, method='GET')\nrouter.add_route('/todos/{id}', delete_todo, method='DELETE')\nrouter.add_route('/todos', add_todo, method='POST')\n\napp.run()\n" }, { "path": "integration_tests/__init__.py", "content": "" }, { "path": "integration_tests/common.py", "content": "import os\nimport subprocess\nimport ctypes.util\nimport sys\nimport time\n\nimport psutil\n\n\ndef start_server(script, *, stdout=None, path=None, sanitize=True, wait=True,\n return_process=False, buffer=False):\n if not isinstance(script, list):\n script = [script]\n if path:\n os.putenv('PYTHONPATH', path)\n if sanitize:\n os.putenv('LD_PRELOAD', ctypes.util.find_library('asan'))\n os.putenv('LSAN_OPTIONS', 'suppressions=misc/suppr.txt')\n if not buffer:\n os.putenv('PYTHONUNBUFFERED', '1')\n server = subprocess.Popen([sys.executable, *script], stdout=stdout)\n if not buffer:\n os.unsetenv('PYTHONUNBUFFERED')\n if sanitize:\n os.unsetenv('LSAN_OPTIONS')\n os.unsetenv('LD_PRELOAD')\n if path:\n os.unsetenv('PYTHONPATH')\n\n process = psutil.Process(server.pid)\n if wait:\n # wait until the server socket is open\n while 1:\n assert server.poll() is None\n conn_num = len(process.connections())\n for child in process.children():\n conn_num += len(child.connections())\n if conn_num:\n break\n time.sleep(.001)\n\n assert server.poll() is None\n\n if return_process:\n return server, process\n else:\n return server\n" }, { "path": "integration_tests/drain.py", "content": "import asyncio\n\nfrom japronto.app import Application\n\n\ndef slash(request):\n return request.Response()\n\n\nasync def sleep(request):\n await asyncio.sleep(int(request.match_dict['sleep']))\n return request.Response()\n\n\napp = Application()\n\nr = app.router\nr.add_route('/', slash)\nr.add_route('/sleep/{sleep}', sleep)\n\n\nif __name__ == '__main__':\n app.run()\n" }, { "path": "integration_tests/dump.py", "content": "import base64\nimport asyncio\n\n\nfrom japronto.app import Application\n\n\nclass ForcedException(Exception):\n pass\n\n\ndef dump(request, exception=None):\n if not exception and 'Force-Raise' in request.headers:\n raise ForcedException(request.headers['Force-Raise'])\n\n body = request.body\n if body is not None:\n body = base64.b64encode(body).decode('ascii')\n\n result = {\n \"method\": request.method,\n \"path\": request.path,\n \"query_string\": request.query_string,\n \"headers\": request.headers,\n \"match_dict\": request.match_dict,\n \"body\": body,\n \"route\": request.route and request.route.pattern\n }\n\n if exception:\n result['exception'] = {\n \"type\": type(exception).__name__,\n \"args\": \", \".join(str(a) for a in exception.args)\n }\n\n return request.Response(code=500 if exception else 200, json=result)\n\n\nasync def adump(request):\n sleep = float(request.query.get('sleep', 0))\n await asyncio.sleep(sleep)\n\n return dump(request)\n\n\napp = Application()\n\nr = app.router\nr.add_route('/dump/{p1}/{p2}', dump)\nr.add_route('/dump1/{p1}/{p2}', dump)\nr.add_route('/dump2/{p1}/{p2}', dump)\nr.add_route('/async/dump/{p1}/{p2}', adump)\nr.add_route('/async/dump1/{p1}/{p2}', adump)\nr.add_route('/async/dump2/{p1}/{p2}', adump)\napp.add_error_handler(None, dump)\n\n\nif __name__ == '__main__':\n app.run()\n" }, { "path": "integration_tests/experiments.py", "content": "import pytest\n\n\n@pytest.fixture(autouse=True)\ndef my_fix(request):\n print('auto')\n pytest.set_trace()\n\n\n@pytest.fixture(params=[1, 2])\ndef size_k(request):\n return request.param\n\n# @pytest.fixture(autouse=True, scope='module')\n# def a():\n# print('a')\n\n\n# @pytest.fixture(scope='function', params=[1,2])\n# def fix():\n# print('> fix')\n# yield 3\n# print('< fix')\n\n\n# def test(my_fix):\n# print(test)\n\ndef test1(size_k):\n print(test1)\n" }, { "path": "integration_tests/generators.py", "content": "from integration_tests import strategies as st\n\nfrom hypothesis.strategies import SearchStrategy\n\n\ndef generate_body(body, size_k):\n if size_k and body:\n if isinstance(body, list):\n length = sum(len(b) for b in body)\n else:\n length = len(body)\n body = body * ((size_k * 1024) // length + 1)\n\n return body\n\n\ndef makeval(v, default_st, default=None):\n if isinstance(v, SearchStrategy):\n return v.example()\n\n if v is True:\n return default_st.example()\n\n if v is not None:\n return v\n\n return default\n\n\ndef print_request(request):\n body = request['body']\n if body:\n if isinstance(body, list):\n body = '{} chunks'.format(len(body))\n else:\n if len(body) > 32:\n body = body[:32] + b'...'\n print(repr(request['method']), repr(request['path']),\n repr(request['query_string']), body)\n\n\ndef generate_request(*, method=None, path=None, query_string=None,\n headers=None, body=None, size_k=None):\n request = {}\n request['method'] = makeval(method, st.method, 'GET')\n request['path'] = makeval(path, st.path, '/')\n request['query_string'] = makeval(query_string, st.query_string)\n request['headers'] = makeval(headers, st.headers)\n request['body'] = generate_body(makeval(body, st.body), size_k)\n\n return request\n\n\ndef generate_combinations(reverse=False):\n props = ['method', 'path', 'query_string', 'headers', 'body']\n sizes = [None, 8, 32, 64]\n if reverse:\n props = reversed(props)\n sizes = reversed(sizes)\n for prop in props:\n if prop == 'body':\n for size_k in sizes:\n yield {'body': True, 'size_k': size_k}\n else:\n yield {prop: True}\n\n\ndef send_requests(conn, number, **kwargs):\n for _ in range(number):\n request = generate_request(**kwargs)\n print_request(request)\n conn.request(**request)\n conn.getresponse()\n" }, { "path": "integration_tests/longrun.py", "content": "import subprocess\nimport sys\nimport signal\nimport atexit\nimport os\nimport time\n\nsys.path.insert(0, '.')\n\nimport integration_tests.common # noqa\nimport integration_tests.generators # noqa\n\nfrom misc import client # noqa\n\n\ndef setup():\n subprocess.check_call([\n sys.executable, 'build.py', '--dest', '.test/longrun',\n '--kit', 'platform', '--disable-response-cache'])\n\n os.putenv('MALLOC_TRIM_THRESHOLD_', '0')\n server = integration_tests.common.start_server(\n 'integration_tests/dump.py', path='.test/longrun', sanitize=False)\n os.unsetenv('MALLOC_TRIM_THRESHOLD_')\n\n os.makedirs('.collector', exist_ok=True)\n\n os.putenv('COLLECTOR_FILE', '.collector/{}.json'.format(server.pid))\n collector = subprocess.Popen([\n sys.executable, 'misc/collector.py', str(server.pid)])\n os.unsetenv('COLLECTOR_FILE')\n\n def cleanup(*args):\n try:\n server.terminate()\n assert server.wait() == 0\n finally:\n atexit.unregister(cleanup)\n\n atexit.register(cleanup)\n signal.signal(signal.SIGINT, cleanup)\n\n\ndef run():\n time.sleep(2)\n for reverse in [True, False]:\n for combination in integration_tests.generators.generate_combinations(\n reverse=reverse):\n conn = client.Connection('localhost:8080')\n time.sleep(2)\n integration_tests.generators.send_requests(\n conn, 200, **combination)\n time.sleep(2)\n conn.close()\n time.sleep(2)\n\n\ndef main():\n setup()\n run()\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "integration_tests/noleak.py", "content": "import sys\n\n\nfrom japronto.app import Application\n\n\nprop = sys.argv[1]\n\nif prop == 'method':\n def noleak(request):\n return request.Response(text=request.method)\nelif prop == 'path':\n def noleak(request):\n return request.Response(text=request.path)\nelif prop == 'match_dict':\n def noleak(request):\n return request.Response(json=request.match_dict)\nelif prop == 'query_string':\n def noleak(request):\n return request.Response(text=request.query_string)\nelif prop == 'headers':\n def noleak(request):\n return request.Response(json=request.headers)\nelif prop == 'body':\n def noleak(request):\n return request.Response(body=request.body)\nelif prop == 'keep_alive':\n def noleak(request):\n return request.Response(text=str(request.keep_alive))\nelif prop == 'route':\n def noleak(request):\n return request.Response(text=str(request.route))\n\napp = Application()\n\nr = app.router\nr.add_route('/noleak/{p1}/{p2}', noleak)\n\n\nif __name__ == '__main__':\n app.run()\n" }, { "path": "integration_tests/reaper.py", "content": "import sys\n\nfrom japronto.app import Application\n\nreaper_settings = {\n 'check_interval': int(sys.argv[1]),\n 'idle_timeout': int(sys.argv[2])\n}\n\napp = Application(reaper_settings=reaper_settings)\n\nif __name__ == '__main__':\n app.run()\n" }, { "path": "integration_tests/strategies.py", "content": "import string\nimport re\n\nfrom hypothesis import strategies as st\nsampled_from = st.sampled_from\nfixed_dictionaries = st.fixed_dictionaries\nlists = st.lists\nbuilds = st.builds\nintegers = st.integers\n\n_method_alphabet = ''.join(chr(x) for x in range(33, 256) if x != 127)\nmethod = st.text(_method_alphabet, min_size=1)\n\n\n_path_alphabet = st.characters(\n blacklist_characters='?', blacklist_categories=['Cs'])\npath = st.text(_path_alphabet).map(lambda x: '/' + x)\n\n_param_alphabet = st.characters(\n blacklist_characters='/?', blacklist_categories=['Cs'])\nparam = st.text(_param_alphabet, min_size=1)\n\nquery_string = st.one_of(st.text(), st.none())\n\n_name_alphabet = string.digits + string.ascii_letters + '!#$%&\\'*+-.^_`|~'\n_names = st.text(_name_alphabet, min_size=1).map(lambda x: 'X-' + x)\n_value_alphabet = ''.join(chr(x) for x in range(ord(' '), 256) if x != 127)\n_is_illegal_value = re.compile(r'\\n(?![ \\t])|\\r(?![ \\t\\n])').search\n_values = st.text(_value_alphabet, min_size=1) \\\n .filter(lambda x: not _is_illegal_value(x)).map(lambda x: x.strip())\nheaders = st.lists(st.tuples(_names, _values), max_size=48)\n\nidentity_body = st.one_of(st.binary(), st.none())\nchunked_body = st.lists(st.binary(min_size=24))\nbody = st.one_of(st.binary(), st.none(), chunked_body)\n" }, { "path": "integration_tests/test_drain.py", "content": "import pytest\nimport subprocess\nimport time\n\nfrom misc import client\nimport integration_tests.common\n\n\npytestmark = pytest.mark.needs_build\n\n\n@pytest.fixture(scope='function')\ndef server():\n return integration_tests.common.start_server(\n 'integration_tests/drain.py', stdout=subprocess.PIPE, path='.test')\n\n\n@pytest.fixture(scope='function')\ndef server_terminate(server):\n def terminate():\n server.terminate()\n assert server.wait() == 0\n\n stdout = server.stdout.read()\n\n return [l.decode('utf-8').strip() for l in stdout.splitlines()]\n\n yield terminate\n\n server.terminate()\n\n\n@pytest.fixture(scope='function')\ndef connect():\n connections = []\n\n def _connect():\n conn = client.Connection('localhost:8080')\n conn.maybe_connect()\n\n connections.append(conn)\n\n return conn\n\n yield _connect\n\n for c in connections:\n c.close()\n\n\ndef test_no_connections(server_terminate):\n lines = server_terminate()\n\n assert lines[-1] == 'Termination request received'\n\n\n@pytest.mark.parametrize('num', range(1, 5))\ndef test_unclosed_connections(num, connect, server_terminate):\n for _ in range(num):\n connect()\n\n lines = server_terminate()\n\n assert lines[-1] == '{} idle connections closed immediately'.format(num)\n\n\n@pytest.mark.parametrize('num', range(1, 5))\ndef test_closed_connections(num, connect, server_terminate):\n for _ in range(num):\n con = connect()\n con.close()\n\n lines = server_terminate()\n\n assert lines[-1] == 'Termination request received'\n\n\n@pytest.mark.parametrize('num', range(1, 5))\ndef test_unclosed_requests(num, connect, server_terminate):\n for _ in range(num):\n con = connect()\n con.putrequest('GET', '/')\n con.endheaders()\n\n lines = server_terminate()\n\n assert lines[-1] == '{} idle connections closed immediately'.format(num)\n\n\n@pytest.mark.parametrize('num', range(1, 5))\ndef test_closed_requests(num, connect, server_terminate):\n for _ in range(num):\n con = connect()\n con.putrequest('GET', '/')\n con.endheaders()\n con.getresponse()\n con.close()\n\n lines = server_terminate()\n\n assert lines[-1] == 'Termination request received'\n\n\n@pytest.mark.parametrize('num', range(1, 3))\ndef test_pipelined(num, connect, server_terminate):\n connections = []\n\n for _ in range(num):\n con = connect()\n connections.append(con)\n con.putrequest('GET', '/sleep/1')\n con.endheaders()\n\n lines = server_terminate()\n\n assert '{} connections busy, read-end closed'.format(num) in lines\n assert not any(l.startswith('Forcefully killing') for l in lines)\n\n assert all(c.getresponse().status == 200 for c in connections)\n\n\n@pytest.mark.parametrize('num', range(1, 3))\ndef test_pipelined_timeout(num, connect, server_terminate):\n connections = []\n\n for _ in range(num):\n con = connect()\n connections.append(con)\n con.putrequest('GET', '/sleep/10')\n con.endheaders()\n\n lines = server_terminate()\n\n assert '{} connections busy, read-end closed'.format(num) in lines\n assert 'Forcefully killing {} connections'.format(num) in lines\n\n assert all(c.getresponse().status == 503 for c in connections)\n\n\ndef test_refuse(connect, server):\n con = connect()\n con.putrequest('GET', '/sleep/10')\n con.endheaders()\n\n server.terminate()\n\n # give time for the signal to propagate\n time.sleep(1)\n\n with pytest.raises(ConnectionRefusedError):\n con = connect()\n\n assert server.wait() == 0\n" }, { "path": "integration_tests/test_noleak.py", "content": "import pytest\n\nfrom misc import client\nimport integration_tests.common\n\n\npytestmark = pytest.mark.needs_build(\n '--extra-compile=-DPROTOCOL_TRACK_REFCNT=1', dest='.test/noleak')\n\n\n@pytest.fixture(scope='function')\ndef server(request):\n arg = request.node.get_marker('arg').args[0]\n\n server = integration_tests.common.start_server([\n 'integration_tests/noleak.py', arg], path='.test/noleak')\n\n yield server\n\n server.terminate()\n server.wait() == 0\n\n\n@pytest.fixture(scope='function')\ndef connection(server):\n conn = client.Connection('localhost:8080')\n yield conn\n conn.close()\n\n\n@pytest.mark.arg('method')\ndef test_method(connection):\n methods = ['GET', 'POST', 'PATCH', 'DELETE', 'PUT']\n\n for method in methods:\n connection.putrequest(method, '/noleak/1/2')\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('path')\ndef test_path(connection):\n paths = ['/noleak/1/2', '/noleak/3/4', '/noleak/5/4', '/noleak/6/7']\n\n for path in paths:\n connection.putrequest('GET', path)\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('match_dict')\ndef test_match_dict(connection):\n paths = ['/noleak/1/2', '/noleak/3/4', '/noleak/5/4', '/noleak/6/7']\n\n for path in paths:\n connection.putrequest('GET', path)\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('query_string')\ndef test_query_string(connection):\n query_strings = ['?', None, '?a', None, '?', None, '?b', None, '?']\n\n for query_string in query_strings:\n connection.putrequest('GET', '/noleak/1/2', query_string)\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('headers')\ndef test_headers(connection):\n header_list = [{}, {\"X-a\": \"b\"}, {}, {\"X-b\": \"c\"}, {}, {\"X-c\": \"d\"}]\n\n for headers in header_list:\n connection.putrequest('GET', '/noleak/1/2')\n for name, value in headers.items():\n connection.putheader(name, value)\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('body')\ndef test_body(connection):\n bodies = [None, b'a', None, b'b', None, b'c', None, b'd', None, b'e']\n\n for body in bodies:\n connection.putrequest('GET', '/noleak/1/2')\n if body:\n connection.putheader('Content-Length', str(len(body)))\n connection.endheaders(body)\n\n response = connection.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('keep_alive')\ndef test_keep_alive(request):\n keep_alives = [True, False, True, True, False, False, True, False]\n\n still_open = False\n\n for keep_alive in keep_alives:\n if not still_open:\n connection_gen = connection(request.getfixturevalue('server'))\n conn = next(connection_gen)\n still_open = keep_alive\n conn.putrequest('GET', '/noleak/1/2')\n if not keep_alive:\n conn.putheader('Connection', 'close')\n conn.endheaders()\n\n response = conn.getresponse()\n assert response.status == 200\n\n\n@pytest.mark.arg('route')\ndef test_route(connection):\n for _ in range(7):\n connection.putrequest('GET', '/noleak/1/2')\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 200\n" }, { "path": "integration_tests/test_perror.py", "content": "import pytest\nfrom hypothesis import given, strategies as st, settings, Verbosity\nimport subprocess\nimport queue\nimport threading\nfrom functools import partial\n\nfrom misc import client\nimport integration_tests.common\n\n\npytestmark = pytest.mark.needs_build\n\n\n@pytest.fixture(autouse=True, scope='module')\ndef server():\n server = integration_tests.common.start_server(\n ['-u', 'integration_tests/dump.py'],\n stdout=subprocess.PIPE, path='.test')\n\n accepting = server.stdout.readline().decode('utf-8')\n assert accepting.startswith('Accepting connections ')\n\n yield server\n\n server.terminate()\n server.wait() == 0\n\n\n@pytest.fixture\ndef line_getter(server):\n q = queue.Queue()\n\n def enqueue_output():\n q.put(server.stdout.readline().strip().decode('utf-8'))\n\n class LineGetter:\n def start(self):\n self.thread = threading.Thread(target=enqueue_output)\n self.thread.start()\n\n def wait(self):\n self.thread.join()\n return q.get()\n\n return LineGetter()\n\n\n@pytest.fixture()\ndef connect(request):\n return partial(client.Connection, 'localhost:8080')\n\n\nfull_request_line = 'GET /asd?qwe HTTP/1.1'\n\n\ndef make_truncated_request_line(cut):\n return full_request_line[:-cut]\n\n\nst_request_cut = st.integers(min_value=1, max_value=len(full_request_line) - 1)\nst_request_line = st.builds(make_truncated_request_line, st_request_cut)\n\n\n@given(request_line=st_request_line)\n@settings(verbosity=Verbosity.verbose, max_examples=20)\ndef test_truncated_request_line(line_getter, connect, request_line):\n connection = connect()\n line_getter.start()\n\n connection.putline(request_line)\n\n assert line_getter.wait() == 'malformed_headers'\n\n response = connection.getresponse()\n assert response.status == 400\n assert response.text == 'malformed_headers'\n\n\n@given(request_line=st_request_line)\n@settings(verbosity=Verbosity.verbose, max_examples=20)\ndef test_truncated_request_line_disconnect(line_getter, connect, request_line):\n connection = connect()\n line_getter.start()\n\n connection.putclose(request_line)\n\n assert line_getter.wait() == 'incomplete_headers'\n\n\nfull_header = 'X-Header: asd'\n\n\ndef make_truncated_header(cut):\n return full_header[:-cut]\n\n\nst_header_cut = st.integers(min_value=5, max_value=len(full_header) - 1)\nst_header_line = st.builds(make_truncated_header, st_header_cut)\n\n\n@given(header_line=st_header_line)\n@settings(verbosity=Verbosity.verbose, max_examples=20)\ndef test_truncated_header(line_getter, connect, header_line):\n connection = connect()\n line_getter.start()\n connection.putline(full_request_line)\n connection.putline(header_line)\n connection.putline()\n\n assert line_getter.wait() == 'malformed_headers'\n\n response = connection.getresponse()\n assert response.status == 400\n assert response.text == 'malformed_headers'\n\n\n@given(header_line=st_header_line)\n@settings(verbosity=Verbosity.verbose, max_examples=20)\ndef test_truncated_header_disconnect(line_getter, connect, header_line):\n connection = connect()\n line_getter.start()\n connection.putline(full_request_line)\n connection.putclose(header_line)\n\n assert line_getter.wait() == 'incomplete_headers'\n\n\n@pytest.mark.parametrize('value', [\n '',\n '+5',\n '-5',\n '0x12',\n '12a'\n])\ndef test_invalid_content_length(line_getter, connect, value):\n connection = connect()\n line_getter.start()\n connection.putline(full_request_line)\n connection.putheader('Content-Length', value)\n connection.putline()\n\n assert line_getter.wait() == 'invalid_headers'\n\n response = connection.getresponse()\n assert response.status == 400\n assert response.text == 'invalid_headers'\n" }, { "path": "integration_tests/test_reaper.py", "content": "from functools import partial\nimport time\n\nimport pytest\n\nfrom misc import client\nimport integration_tests.common\n\n\npytestmark = pytest.mark.needs_build\n\n\n@pytest.fixture(scope='function', params=[2, 3, 4])\ndef get_connections_and_wait(request):\n server, process = integration_tests.common.start_server([\n 'integration_tests/reaper.py', '1', str(request.param)], path='.test',\n return_process=True)\n\n def connection_num():\n return len(\n set(c.fd for c in process.connections()) |\n set(c.fd for p in process.children() for c in p.connections()))\n\n yield connection_num, partial(time.sleep, request.param)\n\n server.terminate()\n assert server.wait() == 0\n\n\ndef test_empty(get_connections_and_wait):\n get_connections, wait = get_connections_and_wait\n conn = client.Connection('localhost:8080')\n\n assert get_connections() == 1\n\n conn.maybe_connect()\n time.sleep(.1)\n\n assert get_connections() == 2\n\n wait()\n\n assert get_connections() == 1\n\n\ndef test_request(get_connections_and_wait):\n get_connections, wait = get_connections_and_wait\n conn = client.Connection('localhost:8080')\n\n assert get_connections() == 1\n\n conn.putrequest('GET', '/')\n conn.endheaders()\n\n assert get_connections() == 2\n\n wait()\n time.sleep(1)\n\n assert get_connections() == 1\n" }, { "path": "integration_tests/test_request.py", "content": "import pytest\nimport json\nimport base64\nfrom functools import partial\n\nfrom hypothesis import given, settings, Verbosity, HealthCheck\n\nimport integration_tests.common\nfrom integration_tests import strategies as st\nfrom misc import client\n\n\npytestmark = pytest.mark.needs_build(coverage=True)\n\n\n@pytest.fixture(autouse=True, scope='module')\ndef server():\n server = integration_tests.common.start_server(\n 'integration_tests/dump.py', path='.test')\n\n yield server\n\n server.terminate()\n server.wait() == 0\n\n\n@pytest.fixture(params=['example', 'test'])\ndef connect(request):\n if request.param == 'example':\n yield partial(client.Connection, 'localhost:8080')\n elif request.param == 'test':\n connection = client.Connection('localhost:8080')\n close = connection.close\n connection.close = lambda: None\n yield lambda: connection\n close()\n\n\n@pytest.fixture(params=['', '/async'], ids=['sync', 'async'])\ndef prefix(request):\n return request.param\n\n\n@given(method=st.method)\n@settings(verbosity=Verbosity.verbose)\ndef test_method(prefix, connect, method):\n connection = connect()\n connection.request(method, prefix + '/dump/1/2')\n response = connection.getresponse()\n json_body = json.loads(response.body)\n\n assert response.status == 200\n assert json_body['method'] == method\n\n connection.close()\n\n\nst_route_prefix = st.sampled_from(['/dump/', '/dump1/', '/dump2/'])\n\n\n@given(route_prefix=st_route_prefix)\n@settings(verbosity=Verbosity.verbose)\ndef test_route(prefix, connect, route_prefix):\n connection = connect()\n connection.request('GET', prefix + route_prefix + '1/2')\n response = connection.getresponse()\n json_body = json.loads(response.body)\n\n assert response.status == 200\n assert json_body['route'].startswith(prefix + route_prefix)\n\n connection.close()\n\n\n@given(param1=st.param, param2=st.param)\n@settings(verbosity=Verbosity.verbose)\ndef test_match_dict(prefix, connect, param1, param2):\n connection = connect()\n connection.request('GET', prefix + '/dump/{}/{}'.format(param1, param2))\n response = connection.getresponse()\n json_body = json.loads(response.body)\n\n assert response.status == 200\n assert json_body['match_dict'] == {'p1': param1, 'p2': param2}\n\n connection.close()\n\n\n@given(query_string=st.query_string)\n@settings(verbosity=Verbosity.verbose)\ndef test_query_string(prefix, connect, query_string):\n connection = connect()\n connection.request('GET', prefix + '/dump/1/2', query_string)\n response = connection.getresponse()\n json_body = json.loads(response.body)\n\n assert response.status == 200\n assert json_body['query_string'] == query_string\n\n connection.close()\n\n\n@given(headers=st.headers)\n@settings(\n verbosity=Verbosity.verbose,\n suppress_health_check=[HealthCheck.too_slow]\n)\ndef test_headers(prefix, connect, headers):\n connection = connect()\n connection.request('GET', prefix + '/dump/1/2', headers=headers)\n response = connection.getresponse()\n json_body = json.loads(response.body)\n\n assert response.status == 200\n headers = {k.title(): v for k, v in headers}\n assert json_body['headers'] == headers\n\n connection.close()\n\n\nst_errors = st.sampled_from(['not-found', 'forced-1', 'forced-2'])\n\n\n@given(error=st_errors)\n@settings(\n verbosity=Verbosity.verbose,\n suppress_health_check=[HealthCheck.too_slow]\n)\ndef test_error(prefix, connect, error):\n connection = connect()\n connection.putrequest(\n 'GET', prefix + '/not-found' if error == 'not-found' else '/dump/1/2')\n if error != 'not-found':\n connection.putheader('Force-Raise', error)\n connection.endheaders()\n\n response = connection.getresponse()\n assert response.status == 500\n json_body = json.loads(response.body)\n assert json_body['exception']['type'] == \\\n 'RouteNotFoundException' if error == 'not-found' else 'ForcedException'\n assert json_body['exception']['args'] == \\\n '' if error == 'not-found' else error\n\n\n@given(body=st.identity_body)\n@settings(verbosity=Verbosity.verbose)\n@pytest.mark.parametrize(\n 'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k'])\ndef test_body(prefix, connect, size_k, body):\n if size_k and body:\n body = body * ((size_k * 1024) // len(body) + 1)\n\n connection = connect()\n connection.request('GET', prefix + '/dump/1/2', body=body)\n response = connection.getresponse()\n\n assert response.status == 200\n json_body = json.loads(response.body)\n\n if body is not None:\n assert base64.b64decode(json_body['body']) == body\n else:\n assert json_body['body'] == body\n\n connection.close()\n\n\n@given(body=st.chunked_body)\n@settings(verbosity=Verbosity.verbose)\n@pytest.mark.parametrize(\n 'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k'])\ndef test_chunked(prefix, connect, size_k, body):\n length = sum(len(b) for b in body)\n if size_k and length:\n body = body * ((size_k * 1024) // length + 1)\n\n connection = connect()\n connection.request('POST', prefix + '/dump/1/2', body=body)\n response = connection.getresponse()\n assert response.status == 200\n json_body = json.loads(response.body)\n\n assert base64.b64decode(json_body['body']) == b''.join(body)\n\n connection.close()\n\n\nst_errors = st.sampled_from([None, None, None, 'not-found', 'forced-1'])\n\n\n@given(\n method=st.method,\n error=st_errors,\n route_prefix=st_route_prefix,\n param1=st.param, param2=st.param,\n query_string=st.query_string,\n headers=st.headers,\n body=st.identity_body\n)\n@settings(\n verbosity=Verbosity.verbose,\n suppress_health_check=[HealthCheck.too_slow]\n)\n@pytest.mark.parametrize(\n 'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k'])\ndef test_all(prefix, connect, size_k, method, error, route_prefix,\n param1, param2, query_string, headers, body):\n connection = connect()\n if size_k and body:\n body = body * ((size_k * 1024) // len(body) + 1)\n url = prefix + ('/not-found' if error == 'not-found' else '') \\\n + route_prefix + '{}/{}'.format(param1, param2)\n connection.putrequest(method, url, query_string)\n for name, value in headers:\n connection.putheader(name, value)\n if body is not None:\n headers.append(('Content-Length', str(len(body))))\n connection.putheader('Content-Length', str(len(body)))\n if error == 'forced-1':\n headers.append(('Force-Raise', 'forced-1'))\n connection.putheader('Force-Raise', 'forced-1')\n connection.endheaders(body)\n response = connection.getresponse()\n\n assert response.status == 500 if error else 200\n json_body = json.loads(response.body)\n assert json_body['method'] == method\n if error != 'not-found':\n assert json_body['route'].startswith(prefix + route_prefix)\n else:\n assert json_body['route'] is None\n assert json_body['match_dict'] == \\\n {} if error == 'not-found' else {'p1': param1, 'p2': param2}\n assert json_body['query_string'] == query_string\n headers = {k.title(): v for k, v in headers}\n assert json_body['headers'] == headers\n if body is not None:\n assert base64.b64decode(json_body['body']) == body\n else:\n assert json_body['body'] is None\n if error:\n assert json_body['exception']['type'] == \\\n 'RouteNotFoundException' if error == 'not-found' \\\n else 'ForcedException'\n assert json_body['exception']['args'] == \\\n '' if error == 'not-found' else error\n else:\n assert 'exception' not in json_body\n\n connection.close()\n\n\nst_request = st.fixed_dictionaries({\n 'method': st.method,\n 'error': st_errors,\n 'route_prefix': st_route_prefix,\n 'param1': st.param,\n 'param2': st.param,\n 'query_string': st.query_string,\n 'headers': st.headers,\n 'body': st.identity_body\n})\nst_requests = st.lists(st_request, min_size=2)\n\n\n@given(requests=st_requests)\n@settings(\n verbosity=Verbosity.verbose,\n suppress_health_check=[HealthCheck.too_slow])\ndef test_pipeline(requests):\n connection = client.Connection('localhost:8080')\n\n for request in requests:\n connection.putrequest(\n request['method'], request['route_prefix'] +\n ('/not-found' if request['error'] == 'not-found' else '') +\n '{param1}/{param2}'.format_map(request),\n request['query_string'])\n for name, value in request['headers']:\n connection.putheader(name, value)\n if request['body'] is not None:\n body_len = str(len(request['body']))\n request['headers'].append(('Content-Length', body_len))\n connection.putheader('Content-Length', body_len)\n if request['error'] == 'forced-1':\n request['headers'].append(('Force-Raise', 'forced-1'))\n connection.putheader('Force-Raise', 'forced-1')\n connection.endheaders(request['body'])\n\n for request in requests:\n response = connection.getresponse()\n assert response.status == 500 if request['error'] else 200\n json_body = response.json\n assert json_body['method'] == request['method']\n if request['error'] != 'not-found':\n assert json_body['route'].startswith(request['route_prefix'])\n else:\n assert json_body['route'] is None\n assert json_body['match_dict'] == \\\n {} if request['error'] == 'not-found' else \\\n {'p1': request['param1'], 'p2': request['param2']}\n assert json_body['query_string'] == request['query_string']\n assert json_body['headers'] == \\\n {k.title(): v for k, v in request['headers']}\n if request['body'] is not None:\n assert base64.b64decode(json_body['body']) == request['body']\n else:\n assert json_body['body'] is None\n if request['error']:\n assert json_body['exception']['type'] == \\\n 'RouteNotFoundException' if request['error'] == 'not-found' \\\n else 'ForcedException'\n assert json_body['exception']['args'] == \\\n '' if request['error'] == 'not-found' else request['error']\n else:\n assert 'exception' not in json_body\n\n connection.close()\n\n\ndef format_sleep_qs(val):\n return 'sleep=' + str(val / 100)\n\n\nst_sleep = st.builds(format_sleep_qs, st.integers(min_value=0, max_value=10))\nst_prefix = st.sampled_from(['/dump', '/async/dump'])\nst_async_request = st.fixed_dictionaries({\n 'query_string': st_sleep,\n 'prefix': st_prefix,\n 'error': st_errors\n})\nst_async_requests = st.lists(st_async_request, min_size=2, max_size=5) \\\n .filter(lambda rs: any(r['prefix'].startswith('/async') for r in rs))\n\n\n@given(requests=st_async_requests)\n@settings(verbosity=Verbosity.verbose)\ndef test_async_pipeline(requests):\n connection = client.Connection('localhost:8080')\n\n for request in requests:\n connection.putrequest(\n 'GET', request['prefix'] +\n ('/not-found' if request['error'] == 'not-found' else '') +\n '/1/2', request['query_string'])\n if request['error'] == 'forced-1':\n connection.putheader('Force-Raise', 'forced-1')\n\n connection.endheaders()\n\n for request in requests:\n response = connection.getresponse()\n assert response.status == 500 if request['error'] else 200\n json_body = response.json\n assert json_body['query_string'] == request['query_string']\n\n connection.close()\n" }, { "path": "misc/__init__.py", "content": "" }, { "path": "misc/bootstrap.sh", "content": "#!/bin/bash\n\ncd ~\n\nsudo apt-get update\nsudo apt-get install -y python3 git libbz2-dev libz-dev libsqlite3-dev libssl-dev gcc make libffi-dev lcov\ngit clone https://github.com/yyuu/pyenv .pyenv\n.pyenv/bin/pyenv install -v 3.6.0\nwget https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz\ntar xvfz virtualenv-15.1.0.tar.gz\npython3 virtualenv-15.1.0/virtualenv.py -p .pyenv/versions/3.6.0/bin/python japronto-env\ngit clone https://github.com/squeaky-pl/japronto\n\ncd japronto/src/picohttpparser\n./build\ncd -\n\ncd japronto\n../japronto-env/bin/pip install -r requirements.txt\n../japronto-env/bin/python build.py --kit=platform\ncd -\n\ngit clone https://github.com/wg/wrk\ncd wrk\nmake\ncd -\n\ncp wrk/wrk japronto\n" }, { "path": "misc/buggers.py", "content": "import atexit\nimport psutil\n\nnoisy = ['atom', 'chrome', 'firefox', 'dropbox', 'opera', 'spotify',\n 'gnome-documents']\n\n\ndef silence():\n for proc in psutil.process_iter():\n if proc.name() in noisy:\n proc.suspend()\n\n def noise():\n for proc in psutil.process_iter():\n if proc.name() in noisy:\n proc.resume()\n atexit.register(noise)\n" }, { "path": "misc/cleanup_script.py", "content": "import sys\n\n\ndef main():\n fp = open(sys.argv[1])\n\n for line in fp:\n line = line.rstrip()\n if line.startswith('\\t'):\n rest = line[18:]\n name_addr, _, rest = rest.partition(' ')\n name, _, addr = name_addr.partition('+')\n line = line[:18] + name + ' ' + rest\n\n print(line)\n\n fp.close()\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "misc/client.py", "content": "import socket\nimport urllib.parse\nimport json\n\n\ndef readline(sock):\n line = b''\n while not line.endswith(b'\\r\\n'):\n line += sock.recv(1)\n\n return line\n\n\ndef readexact(sock, size):\n data = b''\n while size:\n chunk = sock.recv(size)\n data += chunk\n size -= len(chunk)\n\n return data\n\n\nclass Response:\n def __init__(self, sock):\n self.sock = sock\n\n self.read_status_line()\n self.read_headers()\n self.read_body()\n\n def read_status_line(self):\n status_line = b''\n while not status_line:\n status_line = readline(self.sock).strip()\n _, self.status, self.reason = status_line.split(None, 2)\n self.status = int(self.status)\n\n def read_headers(self):\n self.headers = {}\n\n while 1:\n line = readline(self.sock).strip()\n if not line:\n break\n\n name, value = line.split(b':')\n name = name.strip().decode('ascii').title()\n value = value.strip().decode('latin1')\n self.headers[name] = value\n\n @property\n def encoding(self):\n content_type = self.headers.get('Content-Type')\n if not content_type:\n return 'latin1'\n\n _, *rest = [v.split('=') for v in content_type.split(';')]\n\n rest = {k.strip(): v.strip() for k, v in rest}\n\n return rest.get('charset', 'iso-8859-1')\n\n def read_body(self):\n self.body = readexact(self.sock, int(self.headers['Content-Length']))\n self.text = self.body.decode(self.encoding)\n\n @property\n def json(self):\n return json.loads(self.text)\n\n\nclass Connection:\n def __init__(self, addr):\n self.addr = addr\n self.sock = None\n\n def maybe_connect(self):\n if self.sock:\n return self.sock\n\n addr = self.addr.split(':')\n addr[1] = int(addr[1])\n addr = tuple(addr)\n\n self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n self.sock.connect(addr)\n\n return self.sock\n\n def putline(self, line=None):\n line = line or b''\n sock = self.maybe_connect()\n if not isinstance(line, bytes):\n line = str(line).encode('latin1')\n sock.sendall(line + b'\\r\\n')\n\n def putclose(self, data):\n sock = self.maybe_connect()\n if not isinstance(data, bytes):\n data = str(data).encode('latin1')\n sock.sendall(data)\n self.close()\n\n def putrequest(self, method, path, query_string=None):\n url = urllib.parse.quote(path)\n if query_string is not None:\n url += '?' + urllib.parse.quote(query_string)\n\n request_line = \"{method} {url} HTTP/1.1\" \\\n .format(method=method, url=url)\n self.putline(request_line)\n\n def request(self, method, path, query_string=None, headers=None,\n body=None):\n self.putrequest(method, path, query_string)\n headers = headers or []\n for name, value in headers:\n self.putheader(name, value)\n if body is not None:\n if isinstance(body, list):\n self.putheader('Transfer-Encoding', 'chunked')\n else:\n self.putheader('Content-Length', str(len(body)))\n\n self.endheaders(body)\n\n def putheader(self, name, value):\n header_line = name + ': ' + value\n self.putline(header_line)\n\n def endheaders(self, body=None):\n self.putline()\n if body is not None:\n sock = self.maybe_connect()\n if isinstance(body, list):\n for chunk in chunked_encoder(body):\n sock.sendall(chunk)\n else:\n sock.sendall(body)\n\n def getresponse(self):\n return Response(self.sock)\n\n def close(self):\n self.sock.close()\n\n\ndef chunked_encoder(data):\n for chunk in data:\n if not chunk:\n continue\n yield '{:X}\\r\\n'.format(len(chunk)).encode('ascii')\n yield chunk + b'\\r\\n'\n yield b'0\\r\\n\\r\\n'\n" }, { "path": "misc/collector.py", "content": "import psutil\nimport sys\nimport time\nimport os\nimport json\nfrom functools import partial\n\n\ndef get_connections(process):\n return len(\n set(c.fd for c in process.connections()) |\n set(c.fd for p in process.children() for c in p.connections()))\n\n\ndef get_memory(p):\n return p.memory_full_info().uss \\\n + sum(c.memory_full_info().uss for c in p.children())\n\n\ndef sample_process(pid):\n process = psutil.Process(pid)\n samples = []\n\n while 1:\n try:\n uss = get_memory(process)\n conn = get_connections(process)\n except (psutil.NoSuchProcess, psutil.AccessDenied):\n break\n\n samples.append({\n 't': time.monotonic(),\n 'uss': uss, 'conn': conn, 'type': 'proc'})\n time.sleep(.5)\n\n return samples\n\n\ndef main():\n pid = int(sys.argv[1])\n\n samples = sample_process(pid)\n\n with open(os.environ['COLLECTOR_FILE'], 'a') as fp:\n for sample in samples:\n fp.write(json.dumps(sample) + '\\n')\n\n print('Collector info written to', os.environ['COLLECTOR_FILE'])\n\nif __name__ == '__main__':\n main()\n" }, { "path": "misc/cpu.py", "content": "\"\"\"\nCPU file\n\"\"\"\n\n\n# module imports\nimport subprocess\n\n\n# cpu location\nCPU_PREFIX = '/sys/devices/system/cpu/'\n\n\ndef save():\n \"\"\"\n save function\n \"\"\"\n results = {}\n cpu_number = 0\n\n while True:\n try:\n _file = open(\n CPU_PREFIX + 'cpu{}/cpufreq/scaling_governor'.format(cpu_number))\n except:\n break\n\n governor = _file.read().strip()\n results.setdefault(cpu_number, {})['governor'] = governor\n\n _file.close()\n\n try:\n _file = open(\n CPU_PREFIX + 'cpu{}/cpufreq/scaling_cur_freq'.format(cpu_number))\n except:\n break\n\n results[cpu_number]['freq'] = _file.read().strip()\n\n _file.close()\n\n cpu_number += 1\n\n return results\n\n\ndef change(governor, freq=None):\n \"\"\"\n change function\n \"\"\"\n cpu_number = 0\n\n while True:\n try:\n subprocess.check_output([\n \"sudo\", \"bash\", \"-c\",\n \"echo {governor} > {CPU_PREFIX}cpu{cpu_number}/cpufreq/scaling_governor\"\n .format(governor=governor,\n CPU_PREFIX=CPU_PREFIX,\n cpu_number=cpu_number)],\n stderr=subprocess.STDOUT)\n except:\n break\n\n if freq:\n subprocess.check_output([\n \"sudo\", \"bash\", \"-c\",\n \"echo {freq} > {CPU_PREFIX}cpu{cpu_number}/cpufreq/scaling_setspeed\"\n .format(freq=freq,\n CPU_PREFIX=CPU_PREFIX,\n cpu_number=cpu_number)],\n stderr=subprocess.STDOUT)\n\n cpu_number += 1\n\n\ndef available_freq():\n \"\"\"\n function for checking available frequency\n \"\"\"\n _file = open(CPU_PREFIX + 'cpu0/cpufreq/scaling_available_frequencies')\n\n freq = [int(_file) for _file in _file.read().strip().split()]\n\n _file.close()\n\n return freq\n\n\ndef min_freq():\n \"\"\"\n function for returning minimum available frequency\n \"\"\"\n return min(available_freq())\n\n\ndef max_freq():\n \"\"\"\n function for returning maximum avaliable frequency\n \"\"\"\n return max(available_freq())\n\n\ndef dump():\n \"\"\"\n dump function\n \"\"\"\n\n try:\n sensors = subprocess.check_output('sensors').decode('utf-8')\n\n except (FileNotFoundError, subprocess.CalledProcessError):\n print(\"Couldn't read CPU temp\")\n\n else:\n cores = []\n\n for line in sensors.splitlines():\n if line.startswith('Core '):\n core, rest = line.split(':')\n temp = rest.strip().split()[0]\n cores.append((core, temp))\n\n for core, temp in cores:\n print(core + ':', temp)\n\n cpu_number = 0\n\n while True:\n try:\n _file = open(\n CPU_PREFIX + 'cpu{}/cpufreq/scaling_governor'.format(cpu_number))\n except:\n break\n\n print('Core ' + str(cpu_number) + ':', _file.read().strip(), end=', ')\n\n _file.close()\n\n try:\n _file = open(\n CPU_PREFIX + 'cpu{}/cpufreq/scaling_cur_freq'.format(cpu_number))\n except:\n break\n\n freq = round(int(_file.read()) / 10 ** 6, 2)\n\n print(freq, 'GHz')\n\n cpu_number += 1\n" }, { "path": "misc/do_perf.py", "content": "import subprocess\nimport os\nimport sys\nimport argparse\n\nimport parsers\nimport cases\nimport buggers\nimport cpu\n\n\ndef get_http10long():\n return cases.base['10long'].data\n\n\ndef get_websites(size=2 ** 18):\n data = b''\n while len(data) < size:\n for c in cases.websites.values():\n data += c.data\n\n return data\n\n\nif __name__ == '__main__':\n print('pid', os.getpid())\n\n cpu.dump()\n buggers.silence()\n\n argparser = argparse.ArgumentParser(description='do_perf')\n argparser.add_argument(\n '-p', '--parsers', dest='parsers', default='cffi,cext')\n argparser.add_argument(\n '-b', '--benchmarks', dest='benchmarks',\n default='http10long,websites,websitesn')\n\n result = argparser.parse_args(sys.argv[1:])\n parsers = result.parsers.split(',')\n benchmarks = result.benchmarks.split(',')\n\n one_shot = [b for b in benchmarks if b in ['http10long', 'websites']]\n multi_shot = [b for b in benchmarks if b in ['websitesn']]\n\n setup = \"\"\"\nimport parsers\nimport do_perf\nparser, _ = parsers.make_{}(parsers.NullProtocol)\ndata = do_perf.get_{}()\n\"\"\"\n\n loop = \"\"\"\nparser.feed(data)\nparser.feed_disconnect()\n\"\"\"\n\n for dataset in one_shot:\n for parser in parsers:\n print('-- {} {} --'.format(dataset, parser))\n subprocess.check_call([\n 'python', '-m', 'perf', 'timeit', '-s',\n setup.format(parser, dataset), loop])\n print()\n\n setup += \"\"\"\nimport parts\np = parts.make_parts(data, parts.fancy_series(1450))\n\"\"\"\n\n loop = \"\"\"\nfor i in p:\n parser.feed(i)\nparser.feed_disconnect()\n\"\"\"\n\n if multi_shot:\n for parser in parsers:\n print('-- website parts {} --'.format(parser))\n subprocess.check_call([\n 'python', '-m', 'perf', 'timeit', '-s',\n setup.format(parser, 'websites'), loop])\n print()\n" }, { "path": "misc/docker/Dockerfile", "content": "FROM python:3.6.0-slim\n\nRUN pip3 install japronto\nENV PYTHONUNBUFFERED=1\nENTRYPOINT [\"japronto\"]\n" }, { "path": "misc/parts.py", "content": "from functools import partial\nimport types\nimport math\n\n\ndef make_parts(value, get_size, dir=1):\n parts = []\n\n left = bytearray(value)\n while left:\n if isinstance(get_size, types.GeneratorType):\n size = next(get_size)\n else:\n size = get_size\n\n if dir == 1:\n parts.append(bytes(left[:size]))\n left = left[size:]\n else:\n parts.append(bytes(left[-size:]))\n left = left[:-size]\n\n return parts if dir == 1 else list(reversed(parts))\n\n\ndef one_part(value):\n return [value]\n\n\ndef geometric_series():\n s = 2\n while 1:\n yield s\n s *= 2\n\n\ndef fancy_series(minimum=2):\n x = 0\n while 1:\n yield int(minimum + abs(math.sin(x / 3)) * 64)\n x += 1\n" }, { "path": "misc/perf.md", "content": "Capturing performance data with perf\n====================================\n\nFor best results the C source should be built with `-g -O0`.\n\nRun the server, record performance events with `-F` frequency `997 Hz`, `-a` all processes and `-g` take stack info.\n\n```\npython examples/simple/simple.py -p c & \\\nperf record -F 997 -a -g -- sleep 59 & \\\nsleep 1 && ./wrk -c 100 -t 1 -d 60 http://localhost:8080 &\n```\n\nWrite data to a text file filtering for pid `2836` (should be server process)\n\n```\nperf script --pid 2836 > out.perf\n```\n\nRemove address info to make graphs easier to read\n\n```\npython cleanup_script.py out.perf > out2.perf\n```\n\nVisualize data with a flame graph\n=================================\n\n```\n./stackcollapse-perf.pl out2.perf > out.folded\n./flamegraph.pl out.folded > test.svg\n```\n" }, { "path": "misc/pipeline.lua", "content": "-- example script demonstrating HTTP pipelining\n\ninit = function(args)\n local r = {}\n r[1] = wrk.format(nil, \"/\")\n r[2] = wrk.format(nil, \"/\")\n r[3] = wrk.format(nil, \"/\")\n r[4] = wrk.format(nil, \"/\")\n r[5] = wrk.format(nil, \"/\")\n r[6] = wrk.format(nil, \"/\")\n r[7] = wrk.format(nil, \"/\")\n r[8] = wrk.format(nil, \"/\")\n r[9] = wrk.format(nil, \"/\")\n r[10] = wrk.format(nil, \"/\")\n r[11] = wrk.format(nil, \"/\")\n r[12] = wrk.format(nil, \"/\")\n r[13] = wrk.format(nil, \"/\")\n r[14] = wrk.format(nil, \"/\")\n r[15] = wrk.format(nil, \"/\")\n r[16] = wrk.format(nil, \"/\")\n r[17] = wrk.format(nil, \"/\")\n r[18] = wrk.format(nil, \"/\")\n r[19] = wrk.format(nil, \"/\")\n r[20] = wrk.format(nil, \"/\")\n r[21] = wrk.format(nil, \"/\")\n r[22] = wrk.format(nil, \"/\")\n r[23] = wrk.format(nil, \"/\")\n r[24] = wrk.format(nil, \"/\")\n req = table.concat(r)\nend\n\nrequest = function()\n return req\nend\n" }, { "path": "misc/report.py", "content": "import matplotlib.pyplot as plt\nimport sys\nimport os\nimport json\n\n\ndef report(samples, pid):\n plt.figure(figsize=(25, 10))\n\n x = [s['t'] for s in samples if s['type'] == 'proc']\n\n lines = [s for s in samples if s['type'] == 'event']\n\n # minuss = min(s['uss'] for s in samples if s['type'] == 'proc')\n ussplot = plt.subplot(211)\n ussplot.set_title('uss')\n ussplot.plot(\n x, [s['uss'] for s in samples if s['type'] == 'proc'], '.')\n for l in lines:\n # ussplot.text(l['t'], minuss, l['event'], horizontalalignment='right',\n # rotation=-90, rotation_mode='anchor')\n ussplot.axvline(l['t'])\n\n connplot = plt.subplot(212)\n connplot.set_title('conn')\n connplot.plot(\n x, [s['conn'] for s in samples if s['type'] == 'proc'], '.')\n\n os.makedirs('.reports', exist_ok=True)\n path = '.reports/{}.png'.format(pid)\n plt.savefig(path)\n\n return path\n\n\ndef load(filepath):\n samples = []\n with open(filepath) as fp:\n for line in fp:\n line = line.strip()\n samples.append(json.loads(line))\n\n return samples\n\n\ndef order(samples):\n return sorted(samples, key=lambda x: x['t'])\n\n\ndef normalize_time(samples):\n if not samples:\n return []\n\n base_time = samples[0]['t']\n\n return [{**s, 't': s['t'] - base_time} for s in samples]\n\n\ndef main():\n samples = load(sys.argv[1])\n pid, _ = os.path.splitext(os.path.basename(sys.argv[1]))\n samples = order(samples)\n samples = normalize_time(samples)\n report(samples, pid)\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "misc/requirements-test.txt", "content": "-r requirements.txt\nhypothesis==3.80.0\npsutil==5.6.6\npytest==3.9.2\npytoml==0.1.20\nperf==1.5.1\ncffi==1.11.5\n" }, { "path": "misc/requirements.txt", "content": "uvloop>=0.11.3\n" }, { "path": "misc/rpm-requirements.txt", "content": "libasan\nlcov\nlibubsan\n" }, { "path": "misc/runpytest.py", "content": "from pytest import main\n\nmain()\n" }, { "path": "misc/simple.py", "content": "import asyncio\nimport argparse\nimport os.path\nimport sys\nimport socket\n\nsys.path.insert(0, os.path.abspath(os.path.dirname(__file__) + '/../../src'))\n\nimport japronto.protocol.handler # noqa\nfrom japronto.router.cmatcher import Matcher # noqa\nfrom japronto.router import Router # noqa\nfrom japronto.app import Application # noqa\n\n\ndef slash(request):\n return request.Response(text='Hello slash!')\n\n\ndef hello(request):\n return request.Response(text='Hello hello!')\n\n\nasync def sleep(request):\n await asyncio.sleep(3)\n return request.Response(text='I am sleepy')\n\n\nasync def loop(request):\n i = 0\n while i < 10:\n await asyncio.sleep(1)\n print(i)\n i += 1\n\n return request.Response(text='Loop finished')\n\n\ndef dump(request):\n sock = request.transport.get_extra_info('socket')\n no_delay = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY)\n text = \"\"\"\nMethod: {0.method}\nPath: {0.path}\nVersion: {0.version}\nHeaders: {0.headers}\nMatch: {0.match_dict}\nBody: {0.body}\nQS: {0.query_string}\nquery: {0.query}\nmime_type: {0.mime_type}\nencoding: {0.encoding}\nform: {0.form}\nkeep_alive: {0.keep_alive}\nno_delay: {1}\nroute: {0.route}\n\"\"\".strip().format(request, no_delay)\n\n return request.Response(text=text, headers={'X-Version': '123'})\n\n\napp = Application()\n\nr = app.router\nr.add_route('/', slash)\nr.add_route('/hello', hello)\nr.add_route('/dump/{this}/{that}', dump)\nr.add_route('/sleep/{pinch}', sleep)\nr.add_route('/loop', loop)\n\n\nif __name__ == '__main__':\n argparser = argparse.ArgumentParser('server')\n argparser.add_argument(\n '-p', dest='flavor', default='block')\n args = argparser.parse_args(sys.argv[1:])\n\n app.run(protocol_factory=japronto.protocol.handler.make_class(args.flavor))\n" }, { "path": "misc/suppr.txt", "content": "# This is a known leak.\n# Python leaks a little, we use malloc directly\nleak:PyMem_RawMalloc\nleak:_PyObject_GC_Resize\nleak:PyThread_allocate_lock\nleak:resize_compact\n" }, { "path": "misc/travis/before_install.sh", "content": "#!/bin/bash\n\nexport JAPR_MSG=`git show -s --format=%B | xargs`\nexport JAPR_WHEEL=`[[ $JAPR_MSG == *\"[travis-wheel]\"* ]] && echo 1 || echo 0`\nexport JAPR_OS=`uname`\n\nif [[ $VERSION == \"3.5.\"* ]]; then\n export PYTHON_TAG=cp35-cp35m\nelif [[ $VERSION == \"3.6.\"* ]]; then\n export PYTHON_TAG=cp36-cp36m\nelif [[ $VERSION == \"3.7.\"* ]]; then\n export PYTHON_TAG=cp37-cp37m\nelif [[ $VERSION == \"3.8.\"* ]]; then\n export PYTHON_TAG=cp38-cp38m\nfi\n\nenv | grep \"^JAPR_\"\n" }, { "path": "misc/travis/install.sh", "content": "#!/bin/bash\n\nset -ex\n\nif [[ $JAPR_OS == \"Darwin\" ]]; then\n /usr/bin/clang --version\n source misc/terryfy/travis_tools.sh\n source misc/terryfy/library_installers.sh\n clean_builds\n get_python_environment macpython $VERSION venv\nfi\n\nif [[ $JAPR_WHEEL == \"1\" ]]; then\n pip install twine\n\n if [[ $JAPR_OS == \"Linux\" ]]; then\n docker info\n docker pull quay.io/pypa/manylinux1_x86_64\n fi\nfi\n" }, { "path": "misc/travis/script.sh", "content": "#!/bin/bash\n\nset -ex\n\nif [[ $JAPR_WHEEL == \"1\" ]]; then\n if [[ $JAPR_OS == \"Linux\" ]]; then\n docker run --rm -u `id -u` -w /io -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /opt/python/$PYTHON_TAG/bin/python setup.py bdist_wheel\n docker run --rm -u `id -u` -w /io -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 auditwheel repair dist/*-$PYTHON_TAG-linux_x86_64.whl\n rm -r dist/*\n cp wheelhouse/*.whl dist\n fi\n\n if [[ $JAPR_OS == \"Darwin\" ]]; then\n python setup.py bdist_wheel\n fi\n\n ls -lha dist\n unzip -l dist/*.whl\n twine upload -u squeaky dist/*.whl\nfi\n" }, { "path": "setup.py", "content": "\"\"\"\nJapronto\n\"\"\"\nimport codecs\nimport os\nimport re\n\nfrom setuptools import setup, find_packages\n\nimport build\n\n\nwith codecs.open(os.path.join(os.path.abspath(os.path.dirname(\n __file__)), 'src', 'japronto', '__init__.py'), 'r', 'latin1') as fp:\n try:\n version = re.findall(r\"^__version__ = '([^']+)'\\r?$\",\n fp.read(), re.M)[0]\n except IndexError:\n raise RuntimeError('Unable to determine version.')\n\n\nsetup(\n name='japronto',\n version=version,\n url='http://github.com/squeaky-pl/japronto/',\n license='MIT',\n author='Paweł Piotr Przeradowski',\n author_email='przeradowski@gmail.com',\n description='A HTTP application toolkit and server bundle ' +\n 'based on uvloop and picohttpparser',\n package_dir={'': 'src'},\n packages=find_packages('src'),\n keywords=['web', 'asyncio'],\n platforms='x86_64 Linux and MacOS X',\n install_requires=[\n 'uvloop>=0.11.3',\n ],\n entry_points=\"\"\"\n [console_scripts]\n japronto = japronto.__main__:main\n \"\"\",\n classifiers=[\n 'Development Status :: 2 - Pre-Alpha',\n 'Intended Audience :: Developers',\n 'Environment :: Web Environment',\n 'License :: OSI Approved :: MIT License',\n 'Operating System :: MacOS :: MacOS X',\n 'Operating System :: POSIX :: Linux',\n 'Programming Language :: C',\n 'Programming Language :: Python :: 3.5',\n 'Programming Language :: Python :: 3.6',\n 'Programming Language :: Python :: 3.7',\n 'Programming Language :: Python :: 3.8',\n 'Programming Language :: Python :: Implementation :: CPython',\n 'Topic :: Internet :: WWW/HTTP'\n ],\n zip_safe=False,\n include_package_data=True,\n package_data={'picohttpparser': ['*.so']},\n ext_modules=build.get_platform(),\n cmdclass={'build_ext': build.custom_build_ext}\n)\n" }, { "path": "src/japronto/__init__.py", "content": "from .app import Application # noqa\nfrom .router import RouteNotFoundException # noqa\n\n\n__version__ = '0.1.2'\n" }, { "path": "src/japronto/__main__.py", "content": "import sys\nimport os\n\nfrom .runner import get_parser, verify, run\n\n\ndef main():\n parser = get_parser()\n args = parser.parse_args()\n\n if not args.script:\n os.putenv('_JAPR_IGNORE_RUN', '1')\n\n if args.reload:\n os.execv(\n sys.executable,\n [sys.executable, '-m', 'japronto.reloader', *sys.argv[1:]])\n\n if not args.script:\n os.environ['_JAPR_IGNORE_RUN'] = '1'\n\n attribute = verify(args)\n if not attribute:\n return 1\n\n run(attribute, args)\n\n\nsys.exit(main())\n" }, { "path": "src/japronto/app/__init__.py", "content": "import signal\nimport asyncio\nimport traceback\nimport socket\nimport os\nimport sys\nimport multiprocessing\nimport faulthandler\n\nimport uvloop\n\nfrom japronto.router import Router, RouteNotFoundException\nfrom japronto.protocol.cprotocol import Protocol\nfrom japronto.protocol.creaper import Reaper\n\n\nsignames = {\n int(v): v.name for k, v in signal.__dict__.items()\n if isinstance(v, signal.Signals)}\n\n\nclass Application:\n def __init__(self, *, reaper_settings=None, log_request=None,\n protocol_factory=None, debug=False):\n self._router = None\n self._loop = None\n self._connections = set()\n self._reaper_settings = reaper_settings or {}\n self._error_handlers = []\n self._log_request = log_request\n self._request_extensions = {}\n self._protocol_factory = protocol_factory or Protocol\n self._debug = debug\n\n @property\n def loop(self):\n if not self._loop:\n self._loop = uvloop.new_event_loop()\n\n return self._loop\n\n @property\n def router(self):\n if not self._router:\n self._router = Router()\n\n return self._router\n\n def __finalize(self):\n self.loop\n self.router\n\n self._reaper = Reaper(self, **self._reaper_settings)\n self._matcher = self._router.get_matcher()\n\n def protocol_error_handler(self, error):\n print(error)\n\n error = error.encode('utf-8')\n\n response = [\n 'HTTP/1.0 400 Bad Request\\r\\n',\n 'Content-Type: text/plain; charset=utf-8\\r\\n',\n 'Content-Length: {}\\r\\n\\r\\n'.format(len(error))]\n\n return ''.join(response).encode('utf-8') + error\n\n def default_request_logger(self, request):\n print(request.remote_addr, request.method, request.path)\n\n def add_error_handler(self, typ, handler):\n self._error_handlers.append((typ, handler))\n\n def default_error_handler(self, request, exception):\n if isinstance(exception, RouteNotFoundException):\n return request.Response(code=404, text='Not Found')\n if isinstance(exception, asyncio.CancelledError):\n return request.Response(code=503, text='Service unavailable')\n\n tb = traceback.format_exception(\n None, exception, exception.__traceback__)\n tb = ''.join(tb)\n print(tb, file=sys.stderr, end='')\n return request.Response(\n code=500,\n text=tb if self._debug else 'Internal Server Error')\n\n def error_handler(self, request, exception):\n for typ, handler in self._error_handlers:\n if typ is not None and not isinstance(exception, typ):\n continue\n\n try:\n return handler(request, exception)\n except:\n print('-- Exception in error_handler occured:')\n traceback.print_exc()\n\n print('-- while handling:')\n traceback.print_exception(None, exception, exception.__traceback__)\n return request.Response(\n code=500, text='Internal Server Error')\n\n return self.default_error_handler(request, exception)\n\n def _get_idle_and_busy_connections(self):\n # FIXME if there is buffered data in gather the connections should be\n # considered busy, now it's idle\n return \\\n [c for c in self._connections if c.pipeline_empty], \\\n [c for c in self._connections if not c.pipeline_empty]\n\n async def drain(self):\n # TODO idle connections will close connection with half-read requests\n idle, busy = self._get_idle_and_busy_connections()\n for c in idle:\n c.transport.close()\n# for c in busy_connections:\n# need to implement something that makes protocol.on_data\n# start rejecting incoming data\n# this closes transport unfortunately\n# sock = c.transport.get_extra_info('socket')\n# sock.shutdown(socket.SHUT_RD)\n\n if idle or busy:\n print('Draining connections...')\n else:\n return\n\n if idle:\n print('{} idle connections closed immediately'.format(len(idle)))\n if busy:\n print('{} connections busy, read-end closed'.format(len(busy)))\n\n for x in range(5, 0, -1):\n await asyncio.sleep(1)\n idle, busy = self._get_idle_and_busy_connections()\n for c in idle:\n c.transport.close()\n if not busy:\n break\n else:\n print(\n \"{} seconds remaining, {} connections still busy\"\n .format(x, len(busy)))\n\n _, busy = self._get_idle_and_busy_connections()\n if busy:\n print('Forcefully killing {} connections'.format(len(busy)))\n for c in busy:\n c.pipeline_cancel()\n\n def extend_request(self, handler, *, name=None, property=False):\n if not name:\n name = handler.__name__\n\n self._request_extensions[name] = (handler, property)\n\n def serve(self, *, sock, host, port, reloader_pid):\n faulthandler.enable()\n self.__finalize()\n\n loop = self.loop\n asyncio.set_event_loop(loop)\n\n server_coro = loop.create_server(\n lambda: self._protocol_factory(self), sock=sock)\n\n server = loop.run_until_complete(server_coro)\n\n loop.add_signal_handler(signal.SIGTERM, loop.stop)\n loop.add_signal_handler(signal.SIGINT, loop.stop)\n\n if reloader_pid:\n from japronto.reloader import ChangeDetector\n detector = ChangeDetector(loop)\n detector.start()\n\n print('Accepting connections on http://{}:{}'.format(host, port))\n\n try:\n loop.run_forever()\n finally:\n server.close()\n loop.run_until_complete(server.wait_closed())\n loop.run_until_complete(self.drain())\n self._reaper.stop()\n loop.close()\n\n # break reference and cleanup matcher buffer\n del self._matcher\n\n def _run(self, *, host, port, worker_num=None, reloader_pid=None,\n debug=None):\n self._debug = debug or self._debug\n if self._debug and not self._log_request:\n self._log_request = self._debug\n\n sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n sock.bind((host, port))\n os.set_inheritable(sock.fileno(), True)\n\n workers = set()\n\n terminating = False\n\n def stop(sig, frame):\n nonlocal terminating\n if reloader_pid and sig == signal.SIGHUP:\n print('Reload request received')\n elif not terminating:\n terminating = True\n print('Termination request received')\n for worker in workers:\n worker.terminate()\n\n signal.signal(signal.SIGINT, stop)\n signal.signal(signal.SIGTERM, stop)\n signal.signal(signal.SIGHUP, stop)\n\n for _ in range(worker_num or 1):\n worker = multiprocessing.Process(\n target=self.serve,\n kwargs=dict(sock=sock, host=host, port=port,\n reloader_pid=reloader_pid))\n worker.daemon = True\n worker.start()\n workers.add(worker)\n\n # prevent further operations on socket in parent\n sock.close()\n\n for worker in workers:\n worker.join()\n\n if worker.exitcode > 0:\n print('Worker exited with code {}'.format(worker.exitcode))\n elif worker.exitcode < 0:\n try:\n signame = signames[-worker.exitcode]\n except KeyError:\n print(\n 'Worker crashed with unknown code {}!'\n .format(worker.exitcode))\n else:\n print('Worker crashed on signal {}!'.format(signame))\n\n def run(self, host='0.0.0.0', port=8080, *, worker_num=None, reload=False,\n debug=False):\n if os.environ.get('_JAPR_IGNORE_RUN'):\n return\n\n reloader_pid = None\n if reload:\n if '_JAPR_RELOADER' not in os.environ:\n from japronto.reloader import exec_reloader\n exec_reloader(host=host, port=port, worker_num=worker_num)\n else:\n reloader_pid = int(os.environ['_JAPR_RELOADER'])\n\n self._run(\n host=host, port=port, worker_num=worker_num,\n reloader_pid=reloader_pid, debug=debug)\n" }, { "path": "src/japronto/capsule.c", "content": "#include \n\n\nvoid* get_ptr_from_mod(const char* module_name, const char* attr_name,\n const char* capsule_name)\n{\n void* ptr;\n PyObject* module = NULL;\n PyObject* capsule = NULL;\n\n module = PyImport_ImportModule(module_name);\n if(!module)\n goto error;\n\n capsule = PyObject_GetAttrString(module, attr_name);\n if(!capsule)\n goto error;\n\n ptr = PyCapsule_GetPointer(capsule, capsule_name);\n if(!ptr)\n goto error;\n\n goto finally;\n\n error:\n ptr = NULL;\n\n finally:\n Py_XDECREF(capsule);\n Py_XDECREF(module);\n return ptr;\n}\n\n\nPyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name,\n const char* capsule_name)\n{\n PyObject* capsule = NULL;\n\n capsule = PyCapsule_New(ptr, capsule_name, NULL);\n if(!capsule)\n goto error;\n\n if(PyModule_AddObject(m, attr_name, capsule) == -1)\n goto error;\n\n Py_INCREF(capsule);\n goto finally;\n\n error:\n Py_XDECREF(capsule);\n capsule = NULL;\n\n finally:\n return capsule;\n}\n" }, { "path": "src/japronto/capsule.h", "content": "#pragma once\n\n\nvoid* get_ptr_from_mod(const char* module_name, const char* attr_name,\n const char* capsule_name);\n\nPyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name,\n const char* capsule_name);\n\n#define import_capi(module_name) \\\n get_ptr_from_mod(module_name, \"_capi\", module_name \"._capi\")\n\n#define export_capi(m, module_name, capi) \\\n put_ptr_in_mod(m, capi, \"_capi\", module_name \"._capi\")\n" }, { "path": "src/japronto/common.h", "content": "#pragma once\n\n\ntypedef enum {\n KEEP_ALIVE_UNSET,\n KEEP_ALIVE_TRUE,\n KEEP_ALIVE_FALSE\n} KEEP_ALIVE;\n" }, { "path": "src/japronto/cpu_features.c", "content": "#include \n#include \n\n#include \"cpu_features.h\"\n\nint supports_x86_sse42(void)\n{\n#if defined(__clang__)\n unsigned int eax = 0, ebx = 0, ecx = 0, edx = 0;\n __get_cpuid(1, &eax, &ebx, &ecx, &edx);\n return ecx & bit_SSE42;\n#else\n __builtin_cpu_init();\n return __builtin_cpu_supports(\"sse4.2\");\n#endif\n}\n" }, { "path": "src/japronto/cpu_features.h", "content": "#pragma once\n\nint supports_x86_sse42(void);\n" }, { "path": "src/japronto/parser/.gitignore", "content": "libpicohttpparser.c\n" }, { "path": "src/japronto/parser/__init__.py", "content": "header_errors = [\n 'malformed_headers', 'incomplete_headers', 'invalid_headers',\n 'excessive_data']\nbody_errors = ['malformed_body', 'incomplete_body']\n" }, { "path": "src/japronto/parser/build_libpicohttpparser.py", "content": "import distutils.log\ndistutils.log.set_verbosity(distutils.log.DEBUG)\n\nimport os.path\n\nimport cffi\nffibuilder = cffi.FFI()\n\nshared_path = os.path.abspath(os.path.join(os.path.dirname(__file__),\n '../../picohttpparser'))\n\nprint(shared_path)\n\nffibuilder.set_source(\"libpicohttpparser\", \"\"\"\n #include \"picohttpparser.h\"\n \"\"\", libraries=['picohttpparser'], include_dirs=[shared_path],\n library_dirs=[shared_path],\n extra_link_args=['-Wl,-rpath=' + shared_path])\n\n# extra_objects=[os.path.join(shared_path, 'picohttpparser.o')],\n# or a list of libraries to link with\n# (more arguments like setup.py's Extension class:\n# include_dirs=[..], extra_objects=[..], and so on)\n\nffibuilder.cdef(\"\"\"\n struct phr_header {\n const char *name;\n size_t name_len;\n const char *value;\n size_t value_len;\n };\n\n struct phr_chunked_decoder {\n size_t bytes_left_in_chunk; /* number of bytes left in current chunk */\n char consume_trailer; /* if trailing headers should be consumed */\n char _hex_count;\n char _state;\n };\n\n int phr_parse_request(const char *buf, size_t len, const char **method,\n size_t *method_len, const char **path,\n size_t *path_len, int *minor_version,\n struct phr_header *headers, size_t *num_headers,\n size_t last_len);\n\n ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf,\n size_t *bufsz);\n\"\"\")\n\n\nif __name__ == \"__main__\":\n ffibuilder.compile(verbose=True)\n" }, { "path": "src/japronto/parser/cffiparser.py", "content": "from parser.libpicohttpparser import ffi, lib\n\n\nclass HttpRequestParser(object):\n def __init__(self, on_headers, on_body, on_error):\n self.on_headers = on_headers\n self.on_body = on_body\n self.on_error = on_error\n\n self.c_method = ffi.new('char **')\n self.method_len = ffi.new('size_t *')\n self.c_path = ffi.new('char **')\n self.path_len = ffi.new('size_t *')\n self.minor_version = ffi.new('int *')\n self.c_headers = ffi.new('struct phr_header[10]')\n self.num_headers = ffi.new('size_t *')\n self.chunked_offset = ffi.new('size_t*')\n\n self._reset_state(True)\n\n def _reset_state(self, disconnect=False):\n self.state = 'headers'\n self.transfer = None\n self.content_length = None\n self.chunked_decoder = None\n self.chunked_offset[0] = 0\n if disconnect:\n self.connection = None\n self.buffer = bytearray()\n\n def _parse_headers(self):\n if self.connection == 'close':\n self.on_error('excessive_data')\n self._reset_state(True)\n\n return -1\n\n self.num_headers[0] = 10\n\n # FIXME: More than 10 headers\n\n result = lib.phr_parse_request(\n ffi.from_buffer(self.buffer), len(self.buffer),\n self.c_method, self.method_len,\n self.c_path, self.path_len,\n self.minor_version, self.c_headers, self.num_headers, 0)\n\n if result == -2:\n return result\n elif result == -1:\n self.on_error('malformed_headers')\n self._reset_state(True)\n\n return result\n else:\n self._reset_state()\n\n method = ffi.cast(\n 'char[{}]'.format(self.method_len[0]), self.c_method[0])\n path = ffi.cast(\n 'char[{}]'.format(self.path_len[0]), self.c_path[0])\n headers = ffi.cast(\n \"struct phr_header[{}]\".format(self.num_headers[0]),\n self.c_headers)\n\n if ffi.buffer(method)[:] in (b'GET', b'DELETE', b'HEAD'):\n self.no_semantics = True\n\n if self.minor_version[0] == 0:\n self.connection = 'close'\n else:\n self.connection = 'keep-alive'\n\n for header in headers:\n header_name = ffi.string(header.name, header.name_len).title()\n # maybe len + strcasecmp C style is faster?\n if header_name == b'Transfer-Encoding':\n self.transfer = ffi.string(\n header.value, header.value_len).decode('ascii')\n # FIXME comma separated and invalid values\n elif header_name == b'Connection':\n self.connection = ffi.string(\n header.value, header.value_len).decode('ascii')\n # FIXME other options for Connection like updgrade\n elif header_name == b'Content-Length':\n content_length_error = False\n\n if not header.value_len:\n content_length_error = True\n\n if not content_length_error:\n content_length = ffi.buffer(header.value, header.value_len)\n\n if not content_length_error and content_length[0] in b'+-':\n content_length_error = True\n\n if not content_length_error:\n try:\n self.content_length = int(content_length[:])\n except ValueError:\n content_length_error = True\n\n if content_length_error:\n self.on_error('invalid_headers')\n self._reset_state(True)\n\n return -1\n\n self.on_headers(method, path, self.minor_version[0], headers)\n\n self.buffer = self.buffer[result:]\n\n return result\n\n def _parse_body(self):\n if self.content_length is None and self.transfer is None:\n self.on_body(None)\n return 0\n elif self.content_length == 0:\n self.on_body(ffi.from_buffer(b\"\"))\n return 0\n elif self.content_length is not None:\n if self.content_length > len(self.buffer):\n return -2\n\n body = memoryview(self.buffer)[:self.content_length]\n self.on_body(ffi.from_buffer(body))\n self.buffer = self.buffer[self.content_length:]\n\n result = self.content_length\n\n return result\n elif self.transfer == 'chunked':\n if not self.chunked_decoder:\n self.chunked_decoder = ffi.new('struct phr_chunked_decoder*')\n self.chunked_decoder.consume_trailer = b'\\x01'\n\n chunked_offset_start = self.chunked_offset[0]\n self.chunked_offset[0] = len(self.buffer) - self.chunked_offset[0]\n result = lib.phr_decode_chunked(\n self.chunked_decoder,\n ffi.from_buffer(self.buffer) + chunked_offset_start,\n self.chunked_offset)\n self.chunked_offset[0] = self.chunked_offset[0] \\\n + chunked_offset_start\n\n if result == -2:\n self.buffer = self.buffer[:self.chunked_offset[0]]\n return result\n elif result == -1:\n self.on_error('malformed_body')\n self._reset_state(True)\n\n return result\n\n body = memoryview(self.buffer)[:self.chunked_offset[0]]\n self.on_body(ffi.from_buffer(body))\n self.buffer = self.buffer[\n self.chunked_offset[0]:self.chunked_offset[0] + result]\n self._reset_state()\n\n return result\n\n def feed(self, data):\n self.buffer += data\n\n while self.buffer:\n if self.state == 'headers':\n result = self._parse_headers()\n\n if result <= 0:\n return None\n\n self.state = 'body'\n\n if self.state == 'body':\n result = self._parse_body()\n\n if result < 0:\n return None\n\n self.state = 'headers'\n\n def feed_disconnect(self):\n if self.state == 'headers' and self.buffer:\n self.on_error('incomplete_headers')\n elif self.state == 'body':\n self.on_error('incomplete_body')\n\n self._reset_state(True)\n" }, { "path": "src/japronto/parser/cparser.c", "content": "#include \n#include \n\n#include \"cparser.h\"\n#include \"cpu_features.h\"\n\n#ifndef PARSER_STANDALONE\n#include \"cprotocol.h\"\n#endif\n\nstatic PyObject* malformed_headers;\nstatic PyObject* malformed_body;\nstatic PyObject* incomplete_headers;\nstatic PyObject* invalid_headers;\nstatic PyObject* incomplete_body;\nstatic PyObject* excessive_data;\n//static PyObject* empty_body;\n\nconst char zero_body[] = \"\";\n\n/*static PyObject* GET;\nstatic PyObject* POST;\nstatic PyObject* DELETE;\nstatic PyObject* HEAD;\nstatic PyObject* Host;\nstatic PyObject* User_Agent;\nstatic PyObject* Accept;\nstatic PyObject* Accept_Language;\nstatic PyObject* Accept_Encoding;\nstatic PyObject* Accept_Charset;\nstatic PyObject* Connection;\nstatic PyObject* Cookie;\nstatic PyObject* Content_Length;\nstatic PyObject* Transfer_Encoding;\nstatic PyObject* val_close;\nstatic PyObject* keep_alive;*/\n\n\nstatic unsigned long const CONTENT_LENGTH_UNSET = ULONG_MAX;\n\n\nstatic void _reset_state(Parser* self, bool disconnect) {\n self->state = PARSER_HEADERS;\n self->transfer = PARSER_TRANSFER_UNSET;\n self->content_length = CONTENT_LENGTH_UNSET;\n memset(&self->chunked_decoder, 0, sizeof(struct phr_chunked_decoder));\n self->chunked_decoder.consume_trailer = 1;\n self->chunked_offset = 0;\n if(disconnect) {\n self->connection = PARSER_CONNECTION_UNSET;\n self->buffer_start = 0;\n self->buffer_end = 0;\n }\n}\n\n#ifdef PARSER_STANDALONE\nstatic PyObject *\nParser_new(PyTypeObject *type, PyObject *args, PyObject *kwds)\n#else\nvoid\nParser_new(Parser* self)\n#endif\n{\n#ifdef PARSER_STANDALONE\n Parser *self = NULL;\n\n self = (Parser *)type->tp_alloc(type, 0);\n if (!self)\n goto finally;\n\n self->on_headers = NULL;\n self->on_body = NULL;\n self->on_error = NULL;\n#endif\n\n#ifdef PARSER_STANDALONE\n finally:\n return (PyObject *)self;\n#endif\n}\n\n#ifdef PARSER_STANDALONE\nstatic int\nParser_init(Parser *self, PyObject *args, PyObject *kwds)\n#else\nint\nParser_init(Parser* self, void* protocol)\n#endif\n{\n#ifdef PARSER_STANDALONE\n#ifdef DEBUG_PRINT\n printf(\"__init__\\n\");\n#endif\n // FIXME: __init__ can be called many times\n\n // FIXME: check argument types\n int result = PyArg_ParseTuple(\n args, \"OOO\", &self->on_headers, &self->on_body, &self->on_error);\n if(!result)\n return -1;\n Py_INCREF(self->on_headers);\n Py_INCREF(self->on_body);\n Py_INCREF(self->on_error);\n#else\n self->protocol = protocol;\n#endif\n\n _reset_state(self, true);\n\n self->buffer_capacity = PARSER_INITIAL_BUFFER_SIZE;\n self->buffer = self->inline_buffer;\n\n return 0;\n}\n\n#ifdef PARSER_STANDALONE\nstatic void\nParser_dealloc(Parser* self)\n#else\nvoid\nParser_dealloc(Parser* self)\n#endif\n{\n#ifdef PARSER_STANDALONE\n#ifdef DEBUG_PRINT\n printf(\"__del__\\n\");\n#endif\n#endif\n\n if(self->buffer != self->inline_buffer)\n free(self->buffer);\n\n#ifdef PARSER_STANDALONE\n Py_XDECREF(self->on_error);\n Py_XDECREF(self->on_body);\n Py_XDECREF(self->on_headers);\n Py_TYPE(self)->tp_free((PyObject*)self);\n#endif\n}\n\nstatic int\n(*_phr_parse_request)(\n const char *, size_t, const char **, size_t *, const char **, size_t *,\n int *, struct phr_header *, size_t *, size_t);\n\nstatic int _parse_headers(Parser* self) {\n#ifdef PARSER_STANDALONE\n PyObject* method_view = NULL;\n PyObject* path_view = NULL;\n PyObject* minor_version_long = NULL;\n PyObject* headers_view = NULL;\n#endif\n PyObject* error;\n\n int result = -1;\n if(self->connection == PARSER_CLOSE) {\n error = excessive_data;\n goto on_error;\n }\n\n char* method;\n size_t method_len;\n char* path;\n size_t path_len;\n int minor_version;\n struct phr_header headers[50];\n size_t num_headers = sizeof(headers) / sizeof(struct phr_header);\n\n result = _phr_parse_request(\n self->buffer + self->buffer_start, self->buffer_end - self->buffer_start,\n (const char**)&method, &method_len,\n (const char**)&path, &path_len,\n &minor_version, headers, &num_headers, 0);\n\n // FIXME: More than 10 headers\n#ifdef DEBUG_PRINT\n printf(\"result: %d\\n\", result);\n#endif\n\n if(result == -2)\n goto finally;\n\n if(result == -1) {\n error = malformed_headers;\n goto on_error;\n }\n\n if(minor_version == 0) {\n self->connection = PARSER_CLOSE;\n } else {\n self->connection = PARSER_KEEP_ALIVE;\n }\n\n#define header_name_equal(val) \\\n header->name_len == strlen(val) && strncasecmp(header->name, val, header->name_len) == 0\n#define header_value_equal(val) \\\n header->value_len == strlen(val) && strncasecmp(header->value, val, header->value_len) == 0\n/*#define cmp_and_set_header_name(name, val) \\\n if(header_name_equal(val)) { \\\n py_header_name = name; \\\n Py_INCREF(name); \\\n }\n#define cmp_and_set_header_value(name, val) \\\n if(header_value_equal(val)) { \\\n py_header_value = name; \\\n Py_INCREF(name); \\\n }*/\n\n for(struct phr_header* header = headers;\n header < headers + num_headers;\n header++) {\n\n // TODO: common names and values static\n /*PyObject* py_header_name = NULL;\n PyObject* py_header_value = NULL;*/\n\n if(header_name_equal(\"Transfer-Encoding\")) {\n if(header_value_equal(\"chunked\"))\n self->transfer = PARSER_CHUNKED;\n else if(header_value_equal(\"identity\"))\n self->transfer = PARSER_IDENTITY;\n else\n /*TODO: handle incorrept values for protocol version, also comma sep*/;\n\n /*py_header_name = Transfer_Encoding;\n Py_INCREF(Transfer_Encoding);*/\n } else if(header_name_equal(\"Content-Length\")) {\n if(!header->value_len) {\n error = invalid_headers;\n goto on_error;\n }\n\n if(*header->value == '+' || *header->value == '-') {\n error = invalid_headers;\n goto on_error;\n }\n\n char * endptr = (char *)header->value + header->value_len;\n self->content_length = strtol(header->value, &endptr, 10);\n // FIXME: overflow?\n\n if(endptr != (char*)header->value + header->value_len) {\n error = invalid_headers;\n goto on_error;\n }\n } else if(header_name_equal(\"Connection\")) {\n if(header_value_equal(\"close\"))\n self->connection = PARSER_CLOSE;\n else if(header_value_equal(\"keep-alive\"))\n self->connection = PARSER_KEEP_ALIVE;\n else\n /* FIXME: on_error*/;\n /*py_header_name = Content_Length;\n Py_INCREF(Content_Length);*/\n }\n /*else cmp_and_set_header_name(Host, \"Host\")\n else cmp_and_set_header_name(User_Agent, \"User-Agent\")\n else cmp_and_set_header_name(Accept, \"Accept\")\n else cmp_and_set_header_name(Accept_Language, \"Accept-Language\")\n else cmp_and_set_header_name(Accept_Encoding, \"Accept-Encoding\")\n else cmp_and_set_header_name(Accept_Charset, \"Accept-Charset\")\n else cmp_and_set_header_name(Connection, \"Connection\")\n else cmp_and_set_header_name(Cookie, \"Cookie\")\n else {\n bool prev_alpha = false;\n for(char* c = (char*)header.name; c < header.name + header.name_len; c++) {\n if(*c >= 'A' && *c <= 'Z') {\n if(prev_alpha)\n *c |= 0x20;\n prev_alpha = true;\n } else if (*c >= 'a' && *c <= 'z')\n prev_alpha = true;\n else\n prev_alpha = false;\n }\n\n // FIXME this should accept only ascii\n py_header_name = PyUnicode_FromStringAndSize(\n header.name, header.name_len);\n if(!py_header_name) {\n result = -3;\n goto finally_loop;\n }\n }\n\n if(py_header_name == Connection) {\n cmp_and_set_header_value(keep_alive, \"keep-alive\")\n else cmp_and_set_header_value(val_close, \"close\")\n else FIXME: invalid Connection value;\n } else {\n // FIXME: this can return NULL on codec error\n py_header_value = PyUnicode_DecodeLatin1(\n header.value, header.value_len, NULL);\n if(!py_header_value) {\n result = -3;\n goto finally_loop;\n }\n }\n\n if(PyDict_SetItem(py_headers, py_header_name, py_header_value) == -1)\n result = -3;\n\n#ifdef DEBUG_PRINT\n PyObject_Print(py_header_name, stdout, 0); printf(\": \");\n PyObject_Print(py_header_value, stdout, 0); printf(\"\\n\");\n#endif\n\n finally_loop:\n Py_XDECREF(py_header_value);\n Py_XDECREF(py_header_name);\n\n if(result == -3)\n goto finally;*/\n }\n\n#ifdef DEBUG_PRINT\n if(self->content_length != CONTENT_LENGTH_UNSET)\n printf(\"self->content_length: %ld\\n\", self->content_length);\n if(self->transfer == PARSER_IDENTITY)\n printf(\"self->transfer: identity\\n\");\n else if(self->transfer == PARSER_CHUNKED)\n printf(\"self->transfer: chunked\\n\");\n#endif\n\n#ifdef PARSER_STANDALONE\n method_view = PyMemoryView_FromMemory(method, method_len, PyBUF_READ);\n path_view = PyMemoryView_FromMemory(path, path_len, PyBUF_READ);\n minor_version_long = PyLong_FromLong(minor_version);\n headers_view = PyMemoryView_FromMemory((char*)headers, sizeof(struct phr_header) * num_headers, PyBUF_READ);\n // FIXME the functions above can fail\n PyObject* on_headers_result = PyObject_CallFunctionObjArgs(\n self->on_headers, method_view, path_view, minor_version_long, headers_view, NULL);\n if(!on_headers_result)\n goto error;\n Py_DECREF(on_headers_result);\n#else\n if(!Protocol_on_headers(\n self->protocol, method, method_len,\n path, path_len, minor_version, headers, num_headers))\n goto error;\n#endif\n\n self->buffer_start += (size_t)result;\n\n goto finally;\n\n#ifdef PARSER_STANDALONE\n PyObject* on_error_result;\n on_error:\n on_error_result = PyObject_CallFunctionObjArgs(\n self->on_error, error, NULL);\n if(!on_error_result)\n goto error;\n Py_DECREF(on_error_result);\n#else\n on_error:\n if(!Protocol_on_error(self->protocol, error))\n goto error;\n#endif\n\n _reset_state(self, true);\n\n result = -1;\n\n goto finally;\n\n error:\n result = -3;\n\n finally:\n#ifdef PARSER_STANDALONE\n Py_XDECREF(headers_view);\n Py_XDECREF(minor_version_long);\n Py_XDECREF(path_view);\n Py_XDECREF(method_view);\n#endif\n\n return result;\n}\n\nstatic int _parse_body(Parser* self) {\n#ifdef PARSER_STANDALONE\n PyObject* body_view = NULL;\n#endif\n\n char* body = NULL;\n size_t body_len = 0;\n int result = -2;\n if(self->content_length == CONTENT_LENGTH_UNSET\n && self->transfer == PARSER_TRANSFER_UNSET) {\n result = 0;\n goto on_body;\n }\n\n if(self->content_length == 0) {\n body = (char*)zero_body;\n result = 0;\n goto on_body;\n }\n\n if(self->content_length != CONTENT_LENGTH_UNSET) {\n if(self->content_length > self->buffer_end - self->buffer_start) {\n result = -2;\n goto finally;\n }\n\n body = self->buffer + self->buffer_start;\n body_len = self->content_length;\n\n self->buffer_start += self->content_length;\n\n // TODO result = self->content_length (long)\n result = 1;\n\n goto on_body;\n }\n\n if(self->transfer == PARSER_CHUNKED) {\n size_t chunked_offset_start = self->chunked_offset;\n self->chunked_offset = self->buffer_end - self->buffer_start - self->chunked_offset;\n result = phr_decode_chunked(\n &self->chunked_decoder,\n self->buffer + self->buffer_start + chunked_offset_start,\n &self->chunked_offset);\n self->chunked_offset = self->chunked_offset + chunked_offset_start;\n\n if(result == -2) {\n self->buffer_end = self->buffer_start + self->chunked_offset;\n goto finally;\n }\n\n if(result == -1)\n goto on_error;\n\n body = self->buffer + self->buffer_start;\n body_len = self->chunked_offset;\n\n self->buffer_start += self->chunked_offset;\n self->buffer_end = self->buffer_start + (size_t)result;\n\n goto on_body;\n }\n\n goto finally;\n\n on_body:\n\n if(body) {\n#if 0\n if(PyObject_SetAttrString(self->request, \"body\", body) == -1) {\n result = -3;\n goto finally;\n }\n#else\n /*((Request*)(self->request))->body = body;\n Py_INCREF(body);*/\n#endif\n\n#ifdef DEBUG_PRINT\n printf(\"body: \"); PyObject_Print(body, stdout, 0); printf(\"\\n\");\n#endif\n }\n\n#ifdef PARSER_STANDALONE\n if(body) {\n body_view = PyMemoryView_FromMemory(body, body_len, PyBUF_READ);\n if(!body_view)\n goto error;\n } else {\n body_view = Py_None;\n Py_INCREF(body_view);\n }\n PyObject* on_body_result = PyObject_CallFunctionObjArgs(\n self->on_body, body_view, NULL);\n if(!on_body_result)\n goto error;\n Py_DECREF(on_body_result);\n#else\n if(!Protocol_on_body(self->protocol, body, body_len, self->buffer_end - self->buffer_start))\n goto error;\n#endif\n\n _reset_state(self, false);\n\n goto finally;\n\n#ifdef PARSER_STANDALONE\n PyObject* on_error_result;\n on_error:\n on_error_result = PyObject_CallFunctionObjArgs(\n self->on_error, malformed_body, NULL);\n if(!on_error_result)\n goto error;\n Py_DECREF(on_error_result);\n#else\n on_error:\n if(!Protocol_on_error(self->protocol, malformed_body))\n goto error;\n#endif\n\n _reset_state(self, true);\n\n result = -1;\n\n goto finally;\n\n error:\n result = -3;\n\n finally:\n#ifdef PARSER_STANDALONE\n Py_XDECREF(body_view);\n#endif\n return result;\n}\n\n\n#ifdef PARSER_STANDALONE\nstatic PyObject *\nParser_feed(Parser* self, PyObject *args)\n#else\nParser*\nParser_feed(Parser* self, PyObject* py_data)\n#endif\n{\n char* data;\n int iresult = 0;\n#ifdef PARSER_STANDALONE\n PyObject* result = Py_None;\n // FIXME: can be called without __init__\n#ifdef DEBUG_PRINT\n printf(\"feed\\n\");\n#endif\n int data_len;\n if(!PyArg_ParseTuple(args, \"y#\", &data, &data_len))\n goto error;\n#else\n Parser* result = self;\n Py_ssize_t data_len;\n if(PyBytes_AsStringAndSize(py_data, &data, &data_len) == -1)\n goto error;\n#endif\n\n if(self->buffer_start == self->buffer_end) {\n self->buffer_start = 0;\n self->buffer_end = 0;\n } else if((size_t)data_len > self->buffer_capacity - self->buffer_end) {\n memmove(self->buffer, self->buffer + self->buffer_start, self->buffer_end - self->buffer_start);\n self->buffer_end -= self->buffer_start;\n self->buffer_start = 0;\n }\n\n if((size_t)data_len > self->buffer_capacity - (self->buffer_end - self->buffer_start)) {\n self->buffer_capacity = MAX(\n self->buffer_capacity * 2,\n self->buffer_end - self->buffer_start + data_len);\n if(self->buffer == self->inline_buffer) {\n self->buffer = malloc(self->buffer_capacity);\n memcpy(self->buffer + self->buffer_start,\n self->inline_buffer + self->buffer_start,\n self->buffer_end - self->buffer_start);\n } else\n self->buffer = realloc(self->buffer, self->buffer_capacity);\n if(!self->buffer)\n goto error;\n }\n\n memcpy(self->buffer + self->buffer_end, data, (size_t)data_len);\n self->buffer_end += (size_t)data_len;\n\n while(self->buffer_start != self->buffer_end) {\n if(self->state == PARSER_HEADERS) {\n iresult = _parse_headers(self);\n if(iresult == -3)\n goto error;\n\n if(iresult <= 0)\n break;\n\n self->state = PARSER_BODY;\n }\n\n if(self->state == PARSER_BODY) {\n iresult = _parse_body(self);\n if(iresult == -3)\n goto error;\n\n if(iresult < 0)\n break;\n\n self->state = PARSER_HEADERS;\n }\n }\n\n#ifndef PARSER_STANDALONE\n if(iresult == -2)\n Protocol_on_incomplete(self->protocol);\n#endif\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n#ifdef PARSER_STANDALONE\n if(result)\n Py_INCREF(result);\n#endif\n return result;\n}\n\n#ifdef PARSER_STANDALONE\nstatic PyObject *\nParser_feed_disconnect(Parser* self)\n#else\nParser*\nParser_feed_disconnect(Parser* self)\n#endif\n{\n // FIXME: can be called without __init__\n#ifdef DEBUG_PRINT\n printf(\"feed_disconnect\\n\");\n#endif\n\n PyObject* error;\n\n if(self->state == PARSER_HEADERS\n && self->buffer_start != self->buffer_end) {\n error = incomplete_headers;\n goto on_error;\n }\n\n if(self->state == PARSER_BODY) {\n error = incomplete_body;\n goto on_error;\n }\n\n goto finally;\n\n#ifdef PARSER_STANDALONE\n PyObject* on_error_result;\n on_error:\n on_error_result = PyObject_CallFunctionObjArgs(\n self->on_error, error, NULL);\n if(!on_error_result)\n return NULL; /*FIXME maybe leak */\n Py_DECREF(on_error_result);\n#else\n on_error:\n if(!Protocol_on_error(self->protocol, error)) {\n return NULL; /*FIXME maybe leak */\n }\n#endif\n\n finally:\n _reset_state(self, true);\n\n #ifdef PARSER_STANDALONE\n Py_RETURN_NONE;\n #else\n return self;\n #endif\n}\n\n#ifdef PARSER_STANDALONE\nstatic PyObject *\nParser_dump_buffer(Parser* self) {\n // printf(\"buffer: \"); PyObject_Print(self->buffer, stdout, 0); printf(\"\\n\");\n\n Py_RETURN_NONE;\n}\n\n\nstatic PyMethodDef Parser_methods[] = {\n {\"feed\", (PyCFunction)Parser_feed, METH_VARARGS, \"feed\"},\n {\"feed_disconnect\", (PyCFunction)Parser_feed_disconnect,\n METH_NOARGS,\n \"feed_disconnect\"\n },\n {\n \"_dump_buffer\", (PyCFunction)Parser_dump_buffer,\n METH_NOARGS,\n \"_dump_buffer\"\n },\n {NULL} /* Sentinel */\n};\n\n\nstatic PyTypeObject ParserType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"cparser.HttpRequestParser\", /* tp_name */\n sizeof(Parser), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Parser_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"HttpRequestParser\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Parser_methods, /* tp_methods */\n 0, /* tp_members */\n 0, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Parser_init, /* tp_init */\n 0, /* tp_alloc */\n Parser_new, /* tp_new */\n};\n\nstatic PyModuleDef cparser = {\n PyModuleDef_HEAD_INIT,\n \"cparser\",\n \"cparser\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n#endif\n\n#ifdef PARSER_STANDALONE\nPyMODINIT_FUNC\nPyInit_cparser(void)\n#else\nint\ncparser_init(void)\n#endif\n{\n if(supports_x86_sse42()) {\n _phr_parse_request = phr_parse_request_sse42;\n } else {\n printf(\"Warning: Host CPU doesn't support SSE 4.2, selecting slower implementation\\n\");\n _phr_parse_request = phr_parse_request;\n }\n\n malformed_headers = NULL;\n invalid_headers = NULL;\n malformed_body = NULL;\n incomplete_headers = NULL;\n incomplete_body = NULL;\n excessive_data = NULL;\n /*empty_body = NULL;\n GET = NULL;\n POST = NULL;\n DELETE = NULL;\n HEAD = NULL;\n Host = NULL;\n User_Agent = NULL;\n Accept = NULL;\n Accept_Language = NULL;\n Accept_Encoding = NULL;\n Accept_Charset = NULL;\n Connection = NULL;\n Cookie = NULL;\n Content_Length = NULL;\n Transfer_Encoding = NULL;\n val_close = NULL;\n keep_alive = NULL;*/\n#ifdef PARSER_STANDALONE\n PyObject* m = NULL;\n#else\n int m = 0;\n#endif\n\n#ifdef PARSER_STANDALONE\n if (PyType_Ready(&ParserType) < 0)\n goto error;\n\n m = PyModule_Create(&cparser);\n if (!m)\n goto error;\n#endif\n\n#define alloc_static(name) \\\n name = PyUnicode_FromString(#name); \\\n if(!name) \\\n goto error;\n#define alloc_static2(name, val) \\\n name = PyUnicode_FromString(val); \\\n if(!name) \\\n goto error;\n\n alloc_static(malformed_headers)\n alloc_static(malformed_body)\n alloc_static(incomplete_headers)\n alloc_static(invalid_headers)\n alloc_static(incomplete_body)\n alloc_static(excessive_data)\n\n /*empty_body = PyBytes_FromString(\"\");\n if(!empty_body)\n goto error;\n\n alloc_static(GET)\n alloc_static(POST)\n alloc_static(DELETE)\n alloc_static(HEAD)\n\n alloc_static(Host)\n alloc_static2(User_Agent, \"User-Agent\")\n alloc_static(Accept)\n alloc_static2(Accept_Language, \"Accept-Language\")\n alloc_static2(Accept_Encoding, \"Accept-Encoding\")\n alloc_static2(Accept_Charset, \"Accept-Charset\")\n alloc_static(Connection)\n alloc_static(Cookie)\n alloc_static2(Content_Length, \"Content-Length\")\n alloc_static2(Transfer_Encoding, \"Transfer-Encoding\")\n\n alloc_static2(val_close, \"close\")\n alloc_static2(keep_alive, \"keep-alive\")*/\n\n#undef alloc_static\n#undef alloc_static2\n\n#ifdef PARSER_STANDALONE\n Py_INCREF(&ParserType);\n PyModule_AddObject(\n m, \"HttpRequestParser\", (PyObject *)&ParserType);\n#endif\n\n goto finally;\n\n error:\n /*Py_XDECREF(keep_alive);\n Py_XDECREF(val_close);\n\n Py_XDECREF(Transfer_Encoding);\n Py_XDECREF(Content_Length);\n Py_XDECREF(Cookie);\n Py_XDECREF(Connection);\n Py_XDECREF(Accept_Charset);\n Py_XDECREF(Accept_Encoding);\n Py_XDECREF(Accept_Language);\n Py_XDECREF(Accept);\n Py_XDECREF(User_Agent);\n Py_XDECREF(Host);\n\n Py_XDECREF(HEAD);\n Py_XDECREF(DELETE);\n Py_XDECREF(POST);\n Py_XDECREF(GET);\n\n Py_XDECREF(empty_body);*/\n Py_XDECREF(incomplete_body);\n Py_XDECREF(invalid_headers);\n Py_XDECREF(incomplete_headers);\n Py_XDECREF(malformed_body);\n Py_XDECREF(malformed_headers);\n\n#ifndef PARSER_STANDALONE\n m = -1;\n#endif\n finally:\n return m;\n}\n" }, { "path": "src/japronto/parser/cparser.h", "content": "#pragma once\n\n#include \n#include \n\n#include \"picohttpparser.h\"\n\n\nenum Parser_state {\n PARSER_HEADERS,\n PARSER_BODY\n};\n\n\nenum Parser_transfer {\n PARSER_TRANSFER_UNSET,\n PARSER_IDENTITY,\n PARSER_CHUNKED\n};\n\n\nenum Parser_connection {\n PARSER_CONNECTION_UNSET,\n PARSER_CLOSE,\n PARSER_KEEP_ALIVE\n};\n\n#define PARSER_INITIAL_BUFFER_SIZE 4096\n\ntypedef struct {\n#ifdef PARSER_STANDALONE\n PyObject_HEAD\n#endif\n\n enum Parser_state state;\n enum Parser_transfer transfer;\n enum Parser_connection connection;\n\n unsigned long content_length;\n struct phr_chunked_decoder chunked_decoder;\n size_t chunked_offset;\n\n char* buffer;\n size_t buffer_start;\n size_t buffer_end;\n size_t buffer_capacity;\n char inline_buffer[PARSER_INITIAL_BUFFER_SIZE];\n\n#ifdef PARSER_STANDALONE\n PyObject* on_headers;\n PyObject* on_body;\n PyObject* on_error;\n#else\n void* protocol;\n#endif\n} Parser;\n\n#ifndef PARSER_STANDALONE\nvoid\nParser_new(Parser* self);\n\nint\nParser_init(Parser* self, void* protocol);\n\nvoid\nParser_dealloc(Parser* self);\n\nParser*\nParser_feed(Parser* self, PyObject* py_data);\n\nParser*\nParser_feed_disconnect(Parser* self);\n\nint\ncparser_init(void);\n#endif\n" }, { "path": "src/japronto/parser/cparser_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.parser.cparser',\n sources=['cparser.c', '../cpu_features.c'],\n include_dirs=['../../picohttpparser', '..'],\n extra_objects=[\n 'src/picohttpparser/picohttpparser.o',\n 'src/picohttpparser/ssepicohttpparser.o'],\n define_macros=[('PARSER_STANDALONE', 1)])\n" }, { "path": "src/japronto/parser/test_parser.py", "content": "from functools import partial\nfrom itertools import zip_longest\n\nimport pytest\n\nfrom cases import parametrize_cases\nfrom parts import one_part, make_parts, geometric_series, fancy_series\nfrom protocol.tracing import CTracingProtocol, CffiTracingProtocol\nfrom parser import cffiparser, header_errors, body_errors\ntry:\n from parser import cparser\nexcept ImportError:\n cparser = None\n\n\nif cparser:\n def make_c(protocol_factory=CTracingProtocol):\n protocol = protocol_factory()\n parser = cparser.HttpRequestParser(\n protocol.on_headers, protocol.on_body, protocol.on_error)\n\n return parser, protocol\n\n\ndef make_cffi(protocol_factory=CffiTracingProtocol):\n protocol = protocol_factory()\n parser = cffiparser.HttpRequestParser(\n protocol.on_headers, protocol.on_body, protocol.on_error)\n\n return parser, protocol\n\n\n@pytest.mark.parametrize('data,get_size,dir,parts', [\n (b'abcde', 2, 1, [b'ab', b'cd', b'e']),\n (b'abcde', 2, -1, [b'a', b'bc', b'de']),\n (b'aaBBBBccccCCCCd', geometric_series(), 1,\n [b'aa', b'BBBB', b'ccccCCCC', b'd']),\n (b'dCCCCccccBBBBaa', geometric_series(), -1,\n [b'd', b'CCCCcccc', b'BBBB', b'aa'])\n])\ndef test_make_parts(data, get_size, dir, parts):\n assert make_parts(data, get_size, dir) == parts\n\n\ndef parametrize_make_parser():\n ids = []\n factories = []\n if 'make_c' in globals():\n factories.append(make_c)\n ids.append('c')\n\n factories.append(make_cffi)\n ids.append('cffi')\n\n return pytest.mark.parametrize('make_parser', factories, ids=ids)\n\n\ndef parametrize_do_parts():\n funcs = [\n one_part,\n partial(make_parts, get_size=15),\n partial(make_parts, get_size=geometric_series()),\n partial(make_parts, get_size=geometric_series(), dir=-1),\n partial(make_parts, get_size=fancy_series())\n ]\n\n ids = ['one', 'const', 'geom', 'invgeom', 'fancy']\n\n return pytest.mark.parametrize('do_parts', funcs, ids=ids)\n\n\n_begin = object()\n_end = object()\n\n\n@parametrize_do_parts()\n@parametrize_cases(\n 'base',\n '10msg', '10msg!', '10get', '10get!', 'keep:10msg+10get',\n 'keep:10get+10msg',\n\n '10malformed_headers1', '10malformed_headers2', '10incomplete_headers!',\n 'keep:10msg+10malformed_headers2', 'keep:10msg+10incomplete_headers!',\n 'keep:10get+10malformed_headers1', 'keep:10get+10malformed_headers2',\n\n '10msg!+10get!', '10get!+10msg!',\n '10msg!+keep:10get+keep:10msg+10get',\n '10msg+e excessive_data:10get', '10get+e excessive_data:10msg')\n@parametrize_make_parser()\ndef test_http10(make_parser, do_parts, cases):\n parser, protocol = make_parser()\n\n def flush():\n nonlocal data\n if not data:\n return\n\n parts = do_parts(data)\n\n for part in parts:\n parser.feed(part)\n if protocol.error:\n break\n\n data = b''\n\n data = b''\n for case in cases:\n data += case.data\n\n if case.disconnect:\n flush()\n parser.feed_disconnect()\n flush()\n\n header_count = 0\n error_count = 0\n body_count = 0\n\n for case, request in zip_longest(cases, protocol.requests):\n if case.error:\n assert protocol.error == case.error\n\n if case.error in header_errors:\n error_count += 1\n break\n\n header_count += 1\n\n assert request.method == case.method\n assert request.path == case.path\n assert request.version == case.version\n assert request.headers == case.headers\n\n if case.error in body_errors:\n error_count += 1\n break\n\n body_count += 1\n\n assert request.body == case.body\n\n assert protocol.on_headers_call_count == header_count\n assert protocol.on_error_call_count == error_count\n assert protocol.on_body_call_count == body_count\n\n\n@parametrize_make_parser()\ndef test_empty(make_parser):\n parser, protocol = make_parser()\n\n parser.feed_disconnect()\n parser.feed(b'')\n parser.feed(b'')\n parser.feed_disconnect()\n parser.feed_disconnect()\n parser.feed(b'')\n\n assert not protocol.on_headers_call_count\n assert not protocol.on_error_call_count\n assert not protocol.on_body_call_count\n\n\n@parametrize_do_parts()\n@parametrize_cases(\n 'base',\n '11get', '11getmsg', '11msg', '11msgzero', 'close:11get', 'close:11msg',\n '11get!', '11getmsg!', '11msg!', 'close:11msgzero!',\n '11msg+close:11msg', '11msg+11msg',\n 'close:11msg!+11msg', 'close:11msg!+close:11msg',\n '11msg!+close:11msg', '11msg!+11msg',\n '11get+close:11msg', '11msg+11get', '11getmsg+11get',\n '11get+close:11msg!', '11msg!+11get', '11getmsg!+11get!',\n '11msg+11msg+close:11msg',\n '11msg+11msg+11msg',\n '11msg+11msgzero+11msg',\n '11msgzero+11msg+11msgzero',\n '11msg+11get+11msgzero',\n '11msgzero+11msgzero',\n '11get+11getmsg+11get',\n\n 'close:11msg+e excessive_data:11msg',\n 'close:11msg+e excessive_data:close:11msg',\n 'close:11msg+e excessive_data:close:11msg+11msg',\n '11msg+close:11msgzero+e excessive_data:11get',\n\n '11clincomplete_headers!', '11clincomplete_body!',\n '11clinvalid1', '11clinvalid2', '11clinvalid3',\n '11clinvalid4', '11clinvalid5',\n '11msg+11clincomplete_headers!', 'close:11msg!+11clincomplete_body!',\n '11msgzero+11clincomplete_headers!', '11msgzero+11clincomplete_body!',\n 'close:11msg!+11msg+11clincomplete_body!',\n '11get+11clincomplete_body!',\n '11getmsg+11clincomplete_headers!'\n)\n@parametrize_make_parser()\ndef test_http11(make_parser, do_parts, cases):\n parser, protocol = make_parser()\n\n def flush():\n nonlocal data\n if not data:\n return\n\n parts = do_parts(data)\n\n for part in parts:\n parser.feed(part)\n if protocol.error:\n break\n\n data = b''\n\n data = b''\n for case in cases:\n data += case.data\n\n if case.disconnect:\n flush()\n parser.feed_disconnect()\n flush()\n\n header_count = 0\n error_count = 0\n body_count = 0\n\n for case, request in zip_longest(cases, protocol.requests):\n if case.error:\n assert protocol.error == case.error\n\n if case.error in header_errors:\n error_count += 1\n break\n\n header_count += 1\n\n assert request.method == case.method\n assert request.path == case.path\n assert request.version == case.version\n assert request.headers == case.headers\n\n if case.error in body_errors:\n error_count += 1\n break\n\n body_count += 1\n\n assert request.body == case.body\n\n assert protocol.on_headers_call_count == header_count\n assert protocol.on_error_call_count == error_count\n assert protocol.on_body_call_count == body_count\n\n\n@parametrize_do_parts()\n@parametrize_cases(\n 'base',\n '11chunked1', '11chunked2', '11chunked3', '11chunkedzero',\n '11chunked1+11chunked1',\n '11chunked1+11chunked2',\n '11chunked2+11chunked1',\n '11chunked2+11chunked3',\n '11chunked1+11chunked2+11chunked3',\n '11chunked3+11chunked2+11chunked1',\n '11chunked3+11chunked3+11chunked3',\n\n '11chunkedincomplete_body!', '11chunkedmalformed_body',\n '11chunked1+11chunkedincomplete_body!',\n '11chunked1+11chunkedmalformed_body',\n '11chunked2+11chunkedincomplete_body!',\n '11chunked2+11chunkedmalformed_body',\n '11chunked2+11chunked2+11chunkedincomplete_body!',\n '11chunked3+11chunked1+11chunkedmalformed_body'\n)\n@parametrize_make_parser()\ndef test_http11_chunked(make_parser, do_parts, cases):\n parser, protocol = make_parser()\n\n def flush():\n nonlocal data\n if not data:\n return\n\n parts = do_parts(data)\n\n for part in parts:\n parser.feed(part)\n if protocol.error:\n break\n\n data = b''\n\n data = b''\n for case in cases:\n data += case.data\n\n if case.disconnect:\n flush()\n parser.feed_disconnect()\n flush()\n\n header_count = 0\n error_count = 0\n body_count = 0\n\n for case, request in zip_longest(cases, protocol.requests):\n if case.error:\n assert protocol.error == case.error\n\n if case.error in header_errors:\n error_count += 1\n break\n\n header_count += 1\n\n assert request.method == case.method\n assert request.path == case.path\n assert request.version == case.version\n assert request.headers == case.headers\n\n if case.error in body_errors:\n error_count += 1\n break\n\n body_count += 1\n\n assert request.body == case.body\n\n assert protocol.on_headers_call_count == header_count\n assert protocol.on_error_call_count == error_count\n assert protocol.on_body_call_count == body_count\n\n\n@parametrize_do_parts()\n@parametrize_cases(\n 'base',\n '11chunked1+11msgzero',\n '11msg+11chunked2',\n '11chunked2+close:11msg',\n '11msgzero+11chunked3',\n 'close:11msg+e excessive_data:11chunked1+11chunked3',\n '11chunked3+11msg+close:11msg',\n '11chunked3+11chunked3+close:11msg'\n)\n@parametrize_make_parser()\ndef test_http11_mixed(make_parser, do_parts, cases):\n parser, protocol = make_parser()\n\n def flush():\n nonlocal data\n if not data:\n return\n\n parts = do_parts(data)\n\n for part in parts:\n parser.feed(part)\n if protocol.error:\n break\n\n data = b''\n\n data = b''\n for case in cases:\n data += case.data\n\n if case.disconnect:\n flush()\n parser.feed_disconnect()\n flush()\n\n header_count = 0\n error_count = 0\n body_count = 0\n\n for case, request in zip_longest(cases, protocol.requests):\n if case.error:\n assert protocol.error == case.error\n\n if case.error in header_errors:\n error_count += 1\n break\n\n header_count += 1\n\n assert request.method == case.method\n assert request.path == case.path\n assert request.version == case.version\n assert request.headers == case.headers\n\n if case.error in body_errors:\n error_count += 1\n break\n\n body_count += 1\n\n assert request.body == case.body\n\n assert protocol.on_headers_call_count == header_count\n assert protocol.on_error_call_count == error_count\n assert protocol.on_body_call_count == body_count\n" }, { "path": "src/japronto/pipeline/__init__.py", "content": "class Pipeline:\n def __init__(self, ready):\n self._queue = []\n self._ready = ready\n\n @property\n def empty(self):\n return not self._queue\n\n def queue(self, task):\n print(\"queued\")\n\n self._queue.append(task)\n\n task.add_done_callback(self._task_done)\n\n def _task_done(self, task):\n print('Done', task.result())\n\n pop_idx = 0\n for task in self._queue:\n if not task.done():\n break\n\n self.write(task)\n\n pop_idx += 1\n\n if pop_idx:\n self._queue[:pop_idx] = []\n\n def write(self, task):\n self._ready(task)\n print('Written', task.result())\n\n\nif __name__ == '__main__':\n import asyncio\n\n async def coro(sleep):\n await asyncio.sleep(sleep)\n\n return sleep\n\n from uvloop import new_event_loop\n\n loop = new_event_loop()\n asyncio.set_event_loop(loop)\n\n pipeline = Pipeline()\n\n def queue(x):\n t = loop.create_task(coro(x))\n pipeline.queue(t)\n\n loop.call_later(2, lambda: queue(2))\n loop.call_later(12, lambda: queue(2))\n\n queue(1)\n queue(10)\n queue(5)\n queue(1)\n\n loop.run_forever()\n" }, { "path": "src/japronto/pipeline/cpipeline.c", "content": "#include \n#include \"structmember.h\"\n\n#include \"cpipeline.h\"\n\nstatic PyTypeObject PipelineType;\n\n#ifdef PIPELINE_OPAQUE\nstatic PyObject*\nPipeline_new(PyTypeObject* type, PyObject* args, PyObject* kw)\n#else\nPyObject*\nPipeline_new(Pipeline* self)\n#endif\n{\n#ifdef PIPELINE_OPAQUE\n Pipeline* self = NULL;\n\n self = (Pipeline*)type->tp_alloc(type, 0);\n if(!self)\n goto finally;\n#else\n ((PyObject*)self)->ob_refcnt = 1;\n ((PyObject*)self)->ob_type = &PipelineType;\n#endif\n\n self->ready = NULL;\n self->task_done = NULL;\n\n#ifdef PIPELINE_OPAQUE\n finally:\n#endif\n return (PyObject*)self;\n}\n\n#ifdef PIPELINE_OPAQUE\nstatic void\n#else\nvoid\n#endif\nPipeline_dealloc(Pipeline* self)\n{\n#ifdef PIPELINE_OPAQUE\n Py_XDECREF(self->ready);\n#endif\n Py_XDECREF(self->task_done);\n\n#ifdef PIPELINE_OPAQUE\n Py_TYPE(self)->tp_free((PyObject*)self);\n#endif\n}\n\n#ifdef PIPELINE_OPAQUE\nstatic int\nPipeline_init(Pipeline* self, PyObject *args, PyObject* kw)\n#else\nint\nPipeline_init(Pipeline* self, void* (*ready)(PipelineEntry, PyObject*), PyObject* protocol)\n#endif\n{\n int result = 0;\n\n#ifdef PIPELINE_OPAQUE\n if(!PyArg_ParseTuple(args, \"O\", &self->ready))\n goto error;\n\n Py_INCREF(self->ready);\n#else\n self->ready = ready;\n self->protocol = protocol;\n#endif\n\n if(!(self->task_done = PyObject_GetAttrString((PyObject*)self, \"_task_done\")))\n goto error;\n\n self->queue_start = 0;\n self->queue_end = 0;\n\n goto finally;\n\n error:\n result = -1;\n\n finally:\n return result;\n}\n\n\nstatic PyObject*\nPipeline__task_done(Pipeline* self, PyObject* task)\n{\n PyObject* result = Py_True;\n\n PipelineEntry *queue_entry;\n for(queue_entry = self->queue + self->queue_start;\n queue_entry < self->queue + self->queue_end; queue_entry++) {\n PyObject* done = NULL;\n PyObject* done_result = NULL;\n result = Py_True;\n\n if(PipelineEntry_is_task(*queue_entry)) {\n task = PipelineEntry_get_task(*queue_entry);\n\n if(!(done = PyObject_GetAttrString(task, \"done\")))\n goto loop_error;\n\n if(!(done_result = PyObject_CallFunctionObjArgs(done, NULL)))\n goto loop_error;\n\n if(done_result == Py_False) {\n result = Py_False;\n goto loop_finally;\n }\n }\n\n#ifdef PIPELINE_OPAQUE\n PyObject* tmp;\n if(!(tmp = PyObject_CallFunctionObjArgs(self->ready, *queue_entry, NULL)))\n goto loop_error;\n Py_DECREF(tmp);\n#else\n if(!self->ready(*queue_entry, self->protocol))\n goto loop_error;\n#endif\n\n PipelineEntry_DECREF(*queue_entry);\n\n goto loop_finally;\n\n loop_error:\n result = NULL;\n\n loop_finally:\n Py_XDECREF(done_result);\n Py_XDECREF(done);\n if(!result)\n goto error;\n if(result == Py_False)\n break;\n }\n\n self->queue_start = queue_entry - self->queue;\n\n#ifndef PIPELINE_OPAQUE\n if(PIPELINE_EMPTY(self))\n // we became empty so release protocol\n Py_DECREF(self->protocol);\n#endif\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XINCREF(result);\n return result;\n}\n\n#ifdef PIPELINE_OPAQUE\nstatic PyObject*\n#else\nPyObject*\n#endif\nPipeline_queue(Pipeline* self, PipelineEntry entry)\n{\n PyObject* result = Py_None;\n PyObject* add_done_callback = NULL;\n\n if(PIPELINE_EMPTY(self)) {\n self->queue_start = self->queue_end = 0;\n#ifndef PIPELINE_OPAQUE\n // we will become non empty so hold a reference to protocol\n Py_INCREF(self->protocol);\n#endif\n }\n\n assert(self->queue_end < sizeof(self->queue) / sizeof(self->queue[0]));\n\n PipelineEntry* queue_entry = self->queue + self->queue_end;\n *queue_entry = entry;\n PipelineEntry_INCREF(*queue_entry);\n\n self->queue_end++;\n\n if(PipelineEntry_is_task(entry)) {\n PyObject* task = PipelineEntry_get_task(entry);\n if(!(add_done_callback = PyObject_GetAttrString(task, \"add_done_callback\")))\n goto error;\n\n PyObject* tmp;\n if(!(tmp = PyObject_CallFunctionObjArgs(add_done_callback, self->task_done, NULL)))\n goto error;\n Py_DECREF(tmp);\n }\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(add_done_callback);\n#ifdef PIPELINE_OPAQUE\n Py_XINCREF(result);\n#endif\n return result;\n}\n\n\n#ifndef PIPELINE_OPAQUE\nvoid*\nPipeline_cancel(Pipeline* self)\n{\n void* result = self;\n\n PipelineEntry *queue_entry;\n for(queue_entry = self->queue + self->queue_start;\n queue_entry < self->queue + self->queue_end; queue_entry++) {\n if(!PipelineEntry_is_task(*queue_entry))\n continue;\n\n PyObject* task = PipelineEntry_get_task(*queue_entry);\n PyObject* cancel = NULL;\n\n if(!(cancel = PyObject_GetAttrString(task, \"cancel\")))\n goto loop_error;\n\n PyObject* tmp;\n if(!(tmp = PyObject_CallFunctionObjArgs(cancel, NULL)))\n goto loop_error;\n Py_DECREF(tmp);\n\n goto loop_finally;\n\n loop_error:\n result = NULL;\n\n loop_finally:\n Py_XDECREF(cancel);\n\n if(!result)\n break;\n }\n\n return result;\n}\n#endif\n\n\n#ifdef PIPELINE_OPAQUE\nstatic PyObject*\nPipeline_get_empty(Pipeline* self, void* closure) {\n PyObject* result = PIPELINE_EMPTY(self) ? Py_True : Py_False;\n\n Py_INCREF(result);\n return result;\n}\n#endif\n\nstatic PyMethodDef Pipeline_methods[] = {\n#ifdef PIPELINE_OPAQUE\n {\"queue\", (PyCFunction)Pipeline_queue, METH_O, \"\"},\n#endif\n {\"_task_done\", (PyCFunction)Pipeline__task_done, METH_O, \"\"},\n {NULL}\n};\n\n#ifdef PIPELINE_OPAQUE\nstatic PyGetSetDef Pipeline_getset[] = {\n {\"empty\", (getter)Pipeline_get_empty, NULL, \"\", NULL},\n {NULL}\n};\n#endif\n\n\nstatic PyTypeObject PipelineType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"cpipeline.Pipeline\", /* tp_name */\n sizeof(Pipeline), /* tp_basicsize */\n 0, /* tp_itemsize */\n#ifdef PIPELINE_OPAQUE\n (destructor)Pipeline_dealloc, /* tp_dealloc */\n#else\n 0, /* tp_dealloc */\n#endif\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Pipeline\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Pipeline_methods, /* tp_methods */\n 0, /* tp_members */\n#ifdef PIPELINE_OPAQUE\n Pipeline_getset, /* tp_getset */\n#else\n 0, /* tp_getset */\n#endif\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n#ifdef PIPELINE_OPAQUE\n (initproc)Pipeline_init, /* tp_init */\n#else\n 0, /* tp_init */\n#endif\n 0, /* tp_alloc */\n#ifdef PIPELINE_OPAQUE\n Pipeline_new, /* tp_new */\n#else\n 0, /* tp_new */\n#endif\n};\n\n#ifdef PIPELINE_OPAQUE\nstatic PyModuleDef cpipeline = {\n PyModuleDef_HEAD_INIT,\n \"cpipeline\",\n \"cpipeline\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n#endif\n\n\n#ifdef PIPELINE_OPAQUE\nPyMODINIT_FUNC\nPyInit_cpipeline(void)\n#else\nvoid*\ncpipeline_init(void)\n#endif\n{\n#ifdef PIPELINE_OPAQUE\n PyObject* m = NULL;\n#else\n void* m = &PipelineType;\n#endif\n\n if(PyType_Ready(&PipelineType) < 0)\n goto error;\n\n#ifdef PIPELINE_OPAQUE\n if(!(m = PyModule_Create(&cpipeline)))\n goto error;\n\n Py_INCREF(&PipelineType);\n PyModule_AddObject(m, \"Pipeline\", (PyObject*)&PipelineType);\n#endif\n\n goto finally;\n\n error:\n m = NULL;\n\n finally:\n return m;\n}\n" }, { "path": "src/japronto/pipeline/cpipeline.h", "content": "#pragma once\n\n#include \n\n#include \n\n#ifdef PIPELINE_PAIR\n\ntypedef struct {\n bool is_task;\n PyObject* request;\n PyObject* task;\n} PipelineEntry;\n\nstatic inline bool\nPipelineEntry_is_task(PipelineEntry entry)\n{\n return entry.is_task;\n}\n\nstatic inline void\nPipelineEntry_DECREF(PipelineEntry entry)\n{\n Py_DECREF(entry.request);\n // if not real task this was response,\n // that was inside request that was already freed above\n if(entry.is_task)\n Py_XDECREF(entry.task);\n}\n\nstatic inline void\nPipelineEntry_INCREF(PipelineEntry entry)\n{\n Py_INCREF(entry.request);\n Py_XINCREF(entry.task);\n}\n\nstatic inline PyObject*\nPipelineEntry_get_task(PipelineEntry entry)\n{\n return entry.task;\n}\n#else\ntypedef PyObject* PipelineEntry;\n\nstatic inline bool\nPipelineEntry_is_task(PipelineEntry entry)\n{\n return true;\n}\n\nstatic inline void\nPipelineEntry_DECREF(PipelineEntry entry)\n{\n Py_DECREF(entry);\n}\n\nstatic inline void\nPipelineEntry_INCREF(PipelineEntry entry)\n{\n Py_INCREF(entry);\n}\n\nstatic inline PyObject*\nPipelineEntry_get_task(PipelineEntry entry)\n{\n return entry;\n}\n#endif\n\n\ntypedef struct {\n PyObject_HEAD\n#ifdef PIPELINE_OPAQUE\n PyObject* ready;\n#else\n void* (*ready)(PipelineEntry, PyObject*);\n PyObject* protocol;\n#endif\n PyObject* task_done;\n PipelineEntry queue[10];\n size_t queue_start;\n size_t queue_end;\n} Pipeline;\n\n\n#define PIPELINE_EMPTY(p) ((p)->queue_start == (p)->queue_end)\n\n#ifndef PIPELINE_OPAQUE\nPyObject*\nPipeline_new(Pipeline* self);\n\nvoid\nPipeline_dealloc(Pipeline* self);\n\nint\nPipeline_init(Pipeline* self, void* (*ready)(PipelineEntry, PyObject*), PyObject* protocol);\n\nPyObject*\nPipeline_queue(Pipeline* self, PipelineEntry entry);\n\nvoid*\nPipeline_cancel(Pipeline* self);\n\nvoid*\ncpipeline_init(void);\n#endif\n" }, { "path": "src/japronto/pipeline/cpipeline_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.pipeline.cpipeline',\n sources=['cpipeline.c'],\n include_dirs=[],\n libraries=[], library_dirs=[],\n extra_link_args=[],\n define_macros=[('PIPELINE_OPAQUE', 1)])\n" }, { "path": "src/japronto/pipeline/test_pipeline.py", "content": "import asyncio\nimport gc\nimport sys\nfrom collections import namedtuple\nfrom functools import partial\n\nimport pytest\nimport uvloop\n\nfrom japronto.pipeline import Pipeline\nfrom japronto.pipeline.cpipeline import Pipeline as CPipeline\n\n\nExample = namedtuple('Example', 'value,delay')\n\n\nclass FakeLoop:\n def call_soon(self, callback, val):\n callback(val)\n\n def get_debug(self):\n return False\n\n def create_future(self):\n return asyncio.Future(loop=self)\n\n\nclass FakeFuture:\n cnt = 0\n\n def __new__(cls):\n print('new')\n cls.cnt += 1\n return object.__new__(cls)\n\n def __del__(self):\n type(self).cnt -= 1\n print('del')\n\n def __init__(self):\n self.callbacks = []\n\n def add_done_callback(self, cb):\n self.callbacks.append(cb)\n\n def done(self):\n return hasattr(self, '_result')\n\n def result(self):\n return self._result\n\n def set_result(self, result):\n self._result = result\n\n for cb in self.callbacks:\n cb(self)\n\n self.callbacks = []\n\n\ndef parametrize_make_pipeline():\n def make_pipeline(cls):\n results = []\n\n def append(task):\n results.append(task.result())\n\n return cls(append), results\n\n return pytest.mark.parametrize(\n 'make_pipeline',\n [partial(make_pipeline, CPipeline), partial(make_pipeline, Pipeline)],\n ids=['c', 'py'])\n\n\ndef parametrize_case(examples):\n cases = [parse_case(i) for i in examples]\n\n return pytest.mark.parametrize('case', cases, ids=examples)\n\n\ndef parse_example(e, accum):\n value, delay = map(int, e.split('@')) if '@' in e else (int(e), 0)\n\n return Example(value, delay + accum)\n\n\ndef parse_case(case):\n results = []\n delay = 0\n for c in case.split('+'):\n e = parse_example(c, delay)\n results.append(e)\n delay = e.delay\n\n return results\n\n\ndef create_futures(resolves, case):\n futures = [None] * len(case)\n case = case[:]\n for c in sorted(case):\n idx = case.index(c)\n futures[idx] = resolves[idx]()\n case[idx] = None\n\n return tuple(futures)\n\n\n@parametrize_case([\n '1',\n '1+5', '5+1',\n '1+5+10', '10+5+1', '5+1+10', '10+1+5',\n '1+10+5+1', '1+1+10+5', '10+5+1+1', '1+1+5+10'\n])\n@parametrize_make_pipeline()\ndef test_fake_future(make_pipeline, case):\n pipeline, results = make_pipeline()\n\n def queue(x):\n fut = FakeFuture()\n pipeline.queue(fut)\n\n def resolve():\n fut.set_result(x)\n return fut\n\n return resolve\n\n resolves = tuple(queue(v) for v in case)\n futures = create_futures(resolves, case)\n\n assert pipeline.empty\n\n del resolves\n\n # this loop is not pythonic on purpose\n # carefully don't create extra references\n for i in range(len(futures)):\n print(sys.getrefcount(futures[i]))\n del i\n\n assert results == case\n\n gc.collect()\n\n del futures\n\n gc.set_debug(gc.DEBUG_LEAK)\n gc.collect()\n\n print(gc.garbage)\n gc.set_debug(0)\n\n assert FakeFuture.cnt == 0\n\n\ndef parametrize_loop():\n return pytest.mark.parametrize(\n 'loop', [uvloop.new_event_loop(), asyncio.new_event_loop()],\n ids=['uv', 'aio'])\n\n\n@parametrize_case([\n '1', '1@1',\n '1+2', '2+1', '2+1@1', '1@1+2',\n '1+2+3', '3+2+1', '2+1+3', '3+1+2',\n '1+3+2+1', '1+3+2+1+1@1', '1+1+3+2', '3+2+1+1', '1+1+2+3'\n])\n@parametrize_make_pipeline()\n@parametrize_loop()\ndef test_real_task(loop, make_pipeline, case):\n DIVISOR = 1000\n pipeline, results = make_pipeline()\n\n async def coro(example):\n await asyncio.sleep(example.value / DIVISOR, loop=loop)\n\n return example\n\n def queue(x):\n task = loop.create_task(coro(x))\n pipeline.queue(task)\n\n for v in case:\n if v.delay:\n loop.call_later(v.delay / DIVISOR, partial(queue, v))\n else:\n queue(v)\n\n duration = max((e.value + e.delay) / DIVISOR for e in case)\n loop.run_until_complete(asyncio.sleep(duration, loop=loop))\n\n # timing issue, wait a little bit more so we collect all the results\n if len(results) < len(case):\n loop.run_until_complete(asyncio.sleep(10 / DIVISOR, loop=loop))\n\n assert pipeline.empty\n assert results == case\n" }, { "path": "src/japronto/protocol/__init__.py", "content": "" }, { "path": "src/japronto/protocol/cprotocol.c", "content": "#include \n\n\n#include \"cprotocol.h\"\n#include \"cmatcher.h\"\n#include \"crequest.h\"\n#include \"cresponse.h\"\n#include \"capsule.h\"\n#include \"match_dict.h\"\n\n#ifdef PARSER_STANDALONE\nstatic PyObject* Parser;\n#endif\nstatic PyObject* PyRequest;\nstatic PyObject* RouteNotFoundException;\n\nstatic Request_CAPI* request_capi;\nstatic Matcher_CAPI* matcher_capi;\nstatic Response_CAPI* response_capi;\n\n\nstatic PyObject *\nProtocol_new(PyTypeObject *type, PyObject *args, PyObject *kwds)\n{\n Protocol* self = NULL;\n\n self = (Protocol*)type->tp_alloc(type, 0);\n if(!self)\n goto finally;\n\n#ifdef PARSER_STANDALONE\n self->feed = NULL;\n self->feed_disconnect = NULL;\n#else\n Parser_new(&self->parser);\n#endif\n Pipeline_new(&self->pipeline);\n Request_new(request_capi->RequestType, &self->static_request);\n self->app = NULL;\n self->matcher = NULL;\n self->error_handler = NULL;\n self->transport = NULL;\n self->write = NULL;\n self->create_task = NULL;\n self->request_logger = NULL;\n\n self->gather.prev_buffer = NULL;\n\n finally:\n return (PyObject*)self;\n}\n\n\nstatic void\nProtocol_dealloc(Protocol* self)\n{\n Py_XDECREF(self->gather.prev_buffer);\n Py_XDECREF(self->request_logger);\n Py_XDECREF(self->create_task);\n Py_XDECREF(self->write);\n Py_XDECREF(self->writelines);\n Py_XDECREF(self->transport);\n Py_XDECREF(self->error_handler);\n Py_XDECREF(self->matcher);\n Py_XDECREF(self->app);\n Request_dealloc(&self->static_request);\n Pipeline_dealloc(&self->pipeline);\n#ifdef PARSER_STANDALONE\n Py_XDECREF(self->feed_disconnect);\n Py_XDECREF(self->feed);\n#else\n Parser_dealloc(&self->parser);\n#endif\n\n Py_TYPE(self)->tp_free((PyObject*)self);\n}\n\n\nstatic void* Protocol_pipeline_ready(PipelineEntry entry, PyObject* protocol);\n\n\nstatic int\nProtocol_init(Protocol* self, PyObject *args, PyObject *kw)\n{\n int result = 0;\n PyObject* loop = NULL;\n PyObject* log_request = NULL;\n#ifdef PARSER_STANDALONE\n PyObject* parser = NULL;\n\n PyObject* on_headers = PyObject_GetAttrString((PyObject*)self, \"on_headers\");\n if(!on_headers) // FIXME leak\n goto error;\n PyObject* on_body = PyObject_GetAttrString((PyObject*)self, \"on_body\");\n if(!on_body) // FIXME leak\n goto error;\n PyObject* on_error = PyObject_GetAttrString((PyObject*)self, \"on_error\");\n if(!on_error) // FIXME leak\n goto error;\n\n parser = PyObject_CallFunctionObjArgs(\n Parser, on_headers, on_body, on_error, NULL);\n if(!parser)\n goto error;\n\n self->feed = PyObject_GetAttrString(parser, \"feed\");\n if(!self->feed)\n goto error;\n\n self->feed_disconnect = PyObject_GetAttrString(parser, \"feed_disconnect\");\n if(!self->feed_disconnect)\n goto error;\n#else\n if(Parser_init(&self->parser, self) == -1)\n goto error;\n#endif\n\n if(Pipeline_init(&self->pipeline, Protocol_pipeline_ready, (PyObject*)self) == -1)\n goto error;\n\n if(!PyArg_ParseTuple(args, \"O\", &self->app))\n goto error;\n Py_INCREF(self->app);\n\n self->matcher = PyObject_GetAttrString(self->app, \"_matcher\");\n if(!self->matcher)\n goto error;\n\n self->error_handler = PyObject_GetAttrString(self->app, \"error_handler\");\n if(!self->error_handler)\n goto error;\n\n loop = PyObject_GetAttrString(self->app, \"_loop\");\n if(!loop)\n goto error;\n\n self->create_task = PyObject_GetAttrString(loop, \"create_task\");\n if(!self->create_task)\n goto error;\n\n if(!(log_request = PyObject_GetAttrString(self->app, \"_log_request\")))\n goto error;\n\n if(log_request == Py_True) {\n if(!(self->request_logger = PyObject_GetAttrString(self->app, \"default_request_logger\")))\n goto error;\n }\n\n self->gather.responses_end = 0;\n self->gather.len = 0;\n\n goto finally;\n\n error:\n result = -1;\n finally:\n Py_XDECREF(log_request);\n Py_XDECREF(loop);\n#ifdef PARSER_STANDALONE\n Py_XDECREF(parser);\n#endif\n return result;\n}\n\n\nstatic PyObject*\nProtocol_connection_made(Protocol* self, PyObject* transport)\n{\n#ifdef PROTOCOL_TRACK_REFCNT\n printf(\"made: %ld, %ld, %ld, \",\n (size_t)Py_REFCNT(Py_None), (size_t)Py_REFCNT(Py_True), (size_t)Py_REFCNT(Py_False));\n self->none_cnt = Py_REFCNT(Py_None);\n self->true_cnt = Py_REFCNT(Py_True);\n self->false_cnt = Py_REFCNT(Py_False);\n#endif\n\n PyObject* connections = NULL;\n self->transport = transport;\n Py_INCREF(self->transport);\n\n if(!(self->write = PyObject_GetAttrString(transport, \"write\")))\n goto error;\n\n if(!(self->writelines = PyObject_GetAttrString(transport, \"writelines\")))\n goto error;\n\n if(!(connections = PyObject_GetAttrString(self->app, \"_connections\")))\n goto error;\n\n#ifdef REAPER_ENABLED\n self->idle_time = 0;\n self->read_ops = 0;\n self->last_read_ops = 0;\n#endif\n\n if(PySet_Add(connections, (PyObject*)self) == -1)\n goto error;\n\n self->closed = false;\n\n goto finally;\n\n error:\n return NULL;\n\n finally:\n Py_XDECREF(connections);\n Py_RETURN_NONE;\n}\n\n\nstatic void*\nProtocol_close(Protocol* self)\n{\n void* result = self;\n\n PyObject* close = NULL;\n close = PyObject_GetAttrString(self->transport, \"close\");\n if(!close)\n goto error;\n PyObject* tmp = PyObject_CallFunctionObjArgs(close, NULL);\n if(!tmp)\n goto error;\n Py_DECREF(tmp);\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(close);\n return result;\n}\n\n\nstatic PyObject*\nProtocol_connection_lost(Protocol* self, PyObject* args)\n{\n self->closed = true;\n\n PyObject* connections = NULL;\n PyObject* result = Py_None;\n#ifdef PARSER_STANDALONE\n PyObject* result = PyObject_CallFunctionObjArgs(\n self->feed_disconnect, NULL);\n if(!result)\n goto error;\n Py_DECREF(result); // FIXME: result can leak\n#else\n if(!Parser_feed_disconnect(&self->parser))\n goto error;\n#endif\n\n if(!(connections = PyObject_GetAttrString(self->app, \"_connections\")))\n goto error;\n\n if(PySet_Discard(connections, (PyObject*)self) == -1)\n goto error;\n\n if(!Pipeline_cancel(&self->pipeline))\n goto error;\n\n#ifdef PROTOCOL_TRACK_REFCNT\nprintf(\"lost: %ld, %ld, %ld\\n\",\n (size_t)Py_REFCNT(Py_None), (size_t)Py_REFCNT(Py_True), (size_t)Py_REFCNT(Py_False));\n assert(Py_REFCNT(Py_None) == self->none_cnt);\n assert(Py_REFCNT(Py_True) == self->true_cnt);\n assert(Py_REFCNT(Py_False) >= self->false_cnt);\n#endif\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(connections);\n Py_XINCREF(result);\n return result;\n}\n\n\nstatic PyObject*\nProtocol_data_received(Protocol* self, PyObject* data)\n{\n#ifdef REAPER_ENABLED\n self->read_ops++;\n#endif\n\n#ifdef PARSER_STANDALONE\n PyObject* result = PyObject_CallFunctionObjArgs(\n self->feed, data, NULL);\n if(!result)\n goto error;\n Py_DECREF(result);\n#else\n if(!Parser_feed(&self->parser, data))\n goto error;\n#endif\n\n goto finally;\n\n error:\n return NULL;\n finally:\n Py_RETURN_NONE;\n}\n\n\nstatic inline PyObject* Gather_flush(Gather* gather);\n\n#ifndef PARSER_STANDALONE\nProtocol*\nProtocol_on_incomplete(Protocol* self)\n{\n Gather* gather = &self->gather;\n PyObject* gather_buffer = NULL;\n\n if(!gather->len)\n goto finally;\n\n if(!(gather_buffer = Gather_flush(gather)))\n goto error;\n\n PyObject* tmp;\n if(!(tmp = PyObject_CallFunctionObjArgs(self->write, gather_buffer, NULL)))\n goto error;\n Py_DECREF(tmp);\n\n goto finally;\n\n error:\n self = NULL;\n\n finally:\n Py_XDECREF(gather_buffer);\n return self;\n}\n#endif\n\n\n#ifdef PARSER_STANDALONE\nstatic PyObject*\nProtocol_on_headers(Protocol* self, PyObject *args)\n{\n Py_RETURN_NONE;\n}\n#else\nProtocol*\nProtocol_on_headers(Protocol* self, char* method, size_t method_len,\n char* path, size_t path_len, int minor_version,\n void* headers, size_t num_headers)\n{\n Protocol* result = self;\n\n Request_dealloc(&self->static_request);\n Request_new(request_capi->RequestType, &self->static_request);\n\n request_capi->Request_from_raw(\n &self->static_request, method, method_len, path, path_len, minor_version,\n headers, num_headers);\n\n goto finally;\n\n finally:\n return result;\n}\n#endif\n\n\n#define Protocol_catch_exception(request) \\\n{ \\\n PyObject* etype; \\\n PyObject* evalue; \\\n PyObject* etraceback; \\\n \\\n PyErr_Fetch(&etype, &evalue, &etraceback); \\\n PyErr_NormalizeException(&etype, &evalue, &etraceback); \\\n if(etraceback) { \\\n PyException_SetTraceback(evalue, etraceback); \\\n Py_DECREF(etraceback); \\\n } \\\n Py_DECREF(etype); \\\n \\\n ((Request*)request)->exception = evalue; \\\n}\n\n\nstatic inline PyBytesObject*\nBytes_FromSize(size_t size)\n{\n PyBytesObject* result;\n if(!(result = malloc(sizeof(PyBytesObject) + GATHER_MAX_LEN)))\n return (PyBytesObject*)PyErr_NoMemory();\n\n result->ob_base.ob_base.ob_refcnt = 1;\n result->ob_base.ob_base.ob_type = &PyBytes_Type;\n result->ob_base.ob_size = (Py_ssize_t)size;\n\n return result;\n}\n\n\nstatic inline\nPyObject* Gather_flush(Gather* gather)\n{\n PyBytesObject* gather_buffer = NULL;\n\n if(gather->responses_end == 1) {\n gather_buffer = (PyBytesObject*)gather->responses[0];\n goto reset;\n }\n\n if(gather->prev_buffer) {\n if(Py_REFCNT(gather->prev_buffer) == 1) {\n gather_buffer = gather->prev_buffer;\n Py_SIZE(gather_buffer) = (ssize_t)gather->len;\n } else {\n Py_DECREF(gather->prev_buffer);\n gather->prev_buffer = NULL;\n }\n }\n\n if(!gather_buffer && !(gather_buffer = Bytes_FromSize(gather->len)))\n goto error;\n\n size_t gather_offset = 0;\n for(size_t i = 0; i < gather->responses_end; i++) {\n PyObject* item = gather->responses[i];\n memcpy(\n gather_buffer->ob_sval + gather_offset, PyBytes_AS_STRING(item),\n Py_SIZE(item));\n gather_offset += Py_SIZE(item);\n Py_DECREF(item);\n }\n\n gather->prev_buffer = gather_buffer;\n\n reset:\n gather->responses_end = 0;\n gather->len = 0;\n\n goto finally;\n\n error:\n return NULL;\n\n finally:\n if(gather_buffer == gather->prev_buffer)\n Py_INCREF(gather_buffer);\n return (PyObject*)gather_buffer;\n}\n\n\nstatic inline Protocol*\nProtocol_write_response_or_err(Protocol* self, PyObject* request, Response* response)\n{\n Protocol* result = self;\n PyObject* response_bytes = NULL;\n PyObject* error_result = NULL;\n PyObject* gather_buffer = NULL;\n\n if(response && Py_TYPE(response) != response_capi->ResponseType)\n {\n PyErr_SetString(PyExc_ValueError, \"View did not return Response instance\");\n Protocol_catch_exception(request);\n response = NULL;\n }\n\n if(!response) {\n error_result = PyObject_CallFunctionObjArgs(\n self->error_handler, request, ((Request*)request)->exception, NULL);\n if(!error_result)\n goto error;\n\n ((Request*)request)->simple = false;\n if(!Protocol_write_response_or_err(self, request, (Response*)error_result))\n goto error;\n\n goto finally;\n }\n\n if(!(response_bytes =\n response_capi->Response_render(response, ((Request*)request)->simple)))\n goto error;\n\n PyObject* tmp;\n\n PyObject* done_callbacks = ((Request*)request)->done_callbacks;\n for(Py_ssize_t i = 0; done_callbacks && i < PyList_GET_SIZE(done_callbacks); i++) {\n PyObject* callback = PyList_GET_ITEM(done_callbacks, i);\n\n if(!(tmp = PyObject_CallFunctionObjArgs(callback, request, NULL)))\n goto error;\n Py_DECREF(tmp);\n }\n\n Gather* gather = &self->gather;\n\n if(!gather->enabled)\n goto maybe_flush;\n\n if(gather->responses_end == GATHER_MAX_RESP)\n goto maybe_flush;\n\n if(gather->len + Py_SIZE(response_bytes) > GATHER_MAX_LEN)\n goto maybe_flush;\n\n gather->responses[gather->responses_end] = response_bytes;\n gather->responses_end++;\n gather->len += Py_SIZE(response_bytes);\n response_bytes = NULL;\n\n goto dont_flush;\n\n maybe_flush:\n if(!gather->len)\n goto dont_flush;\n\n if(!(gather_buffer = Gather_flush(gather)))\n goto error;\n\n if(!(tmp = PyObject_CallFunctionObjArgs(self->write, gather_buffer, NULL)))\n goto error;\n Py_DECREF(tmp);\n\n dont_flush:\n\n if(response_bytes) {\n if(!(tmp = PyObject_CallFunctionObjArgs(self->write, response_bytes, NULL)))\n goto error;\n Py_DECREF(tmp);\n }\n\n if(self->request_logger) {\n if(!(tmp = PyObject_CallFunctionObjArgs(self->request_logger, request, NULL)))\n goto error;\n Py_DECREF(tmp);\n }\n\n if(response->keep_alive == KEEP_ALIVE_FALSE) {\n if(!Protocol_close(self))\n goto error;\n }\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(gather_buffer);\n Py_XDECREF(error_result);\n Py_XDECREF(response_bytes);\n return result;\n}\n\n\nstatic void* Protocol_pipeline_ready(PipelineEntry entry, PyObject* protocol)\n{\n Protocol* self = (Protocol*)protocol;\n PyObject* get_result = NULL;\n PyObject* response = NULL;\n PyObject* request = entry.request;\n PyObject* task = entry.task;\n\n if(PipelineEntry_is_task(entry)) {\n if(!(get_result = PyObject_GetAttrString(task, \"result\")))\n goto error;\n\n if(!(response = PyObject_CallFunctionObjArgs(get_result, NULL)))\n Protocol_catch_exception(request);\n } else {\n response = task;\n }\n\n if(!self->closed) {\n if(!Protocol_write_response_or_err(self, request, (Response*)response))\n goto error;\n } else {\n // TODO: Send that to protocol_error\n printf(\"Connection closed, response dropped\\n\");\n }\n\n // important: this breaks a cycle in case of an exception\n Py_CLEAR(((Request*)request)->exception);\n\n goto finally;\n\n error:\n self = NULL;\n\n finally:\n if(PipelineEntry_is_task(entry))\n Py_XDECREF(response);\n Py_XDECREF(get_result);\n return self;\n}\n\n\nstatic inline Protocol*\nProtocol_handle_coro(Protocol* self, PyObject* request, PyObject* coro)\n{\n Protocol* result = self;\n PyObject* task = NULL;\n\n if(!(task = PyObject_CallFunctionObjArgs(self->create_task, coro, NULL)))\n goto error;\n\n if(!Pipeline_queue(&self->pipeline, (PipelineEntry){true, request, task}))\n goto error;\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(task);\n return result;\n}\n\n\n#ifdef PARSER_STANDALONE\nstatic PyObject*\nProtocol_on_body(Protocol* self, PyObject *args)\n#else\nProtocol*\nProtocol_on_body(Protocol* self, char* body, size_t body_len, size_t tail_len)\n#endif\n{\n#ifdef PARSER_STANDALONE\n PyObject* result = Py_None;\n#else\n Protocol* result = self;\n#endif\n PyObject* request = NULL;\n PyObject* handler_result = NULL;\n MatchDictEntry* entries;\n MatcherEntry* matcher_entry;\n size_t entries_length;\n#ifdef PARSER_STANDALONE\n/* PyObject* request;\n if(!PyArg_ParseTuple(args, \"O\", &request))\n goto error;\n*/ // FIXME implement body setting\n#endif\n\n matcher_entry = matcher_capi->Matcher_match_request(\n (Matcher*)self->matcher, (PyObject*)&self->static_request,\n &entries, &entries_length);\n\n request_capi->Request_set_match_dict_entries(\n &self->static_request, entries, entries_length);\n\n request_capi->Request_set_body(\n &self->static_request, body, body_len);\n\n self->static_request.simple = matcher_entry && matcher_entry->simple;\n\n request = (PyObject*)&self->static_request;\n if((matcher_entry && matcher_entry->coro_func) || !PIPELINE_EMPTY(&self->pipeline)) {\n self->gather.enabled = false;\n\n if(!(request = request_capi->Request_clone(&self->static_request)))\n goto error;\n } else\n // TODO: should be tweaked to minimal request length\n self->gather.enabled = tail_len > 0;\n\n ((Request*)request)->transport = self->transport;\n Py_INCREF(self->transport);\n\n ((Request*)request)->app = self->app;\n Py_INCREF(self->app);\n\n ((Request*)request)->matcher_entry = matcher_entry;\n\n if(!matcher_entry) {\n if(!(((Request*)request)->exception = PyObject_CallFunctionObjArgs(\n RouteNotFoundException, NULL)))\n goto error;\n\n goto queue_or_write;\n }\n\n if(!(handler_result = PyObject_CallFunctionObjArgs(\n matcher_entry->handler, request, NULL))) {\n Protocol_catch_exception(request);\n goto queue_or_write;\n }\n\n if(matcher_entry->coro_func) {\n if(!Protocol_handle_coro(self, request, handler_result))\n goto error;\n\n goto finally;\n }\n\n queue_or_write:\n\n if(!PIPELINE_EMPTY(&self->pipeline))\n {\n if(!Pipeline_queue(&self->pipeline, (PipelineEntry){false, request, handler_result}))\n goto error;\n\n goto finally;\n }\n\n if(!Protocol_write_response_or_err(\n self, (PyObject*)&self->static_request, (Response*)handler_result))\n goto error;\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n if(request != (PyObject*)&self->static_request)\n Py_XDECREF(request);\n Py_XDECREF(handler_result);\n#ifdef PARSER_STANDALONE\n if(result)\n Py_INCREF(result);\n#endif\n return result;\n}\n\n#ifdef PARSER_STANDALONE\nstatic PyObject*\nProtocol_on_error(Protocol* self, PyObject *args)\n{\n Py_RETURN_NONE;\n}\n#else\nProtocol*\nProtocol_on_error(Protocol* self, PyObject* error)\n{\n PyObject* protocol_error_handler = NULL;\n PyObject* response = NULL;\n\n if(!(protocol_error_handler =\n PyObject_GetAttrString(self->app, \"protocol_error_handler\")))\n goto error;\n\n if(!(response =\n PyObject_CallFunctionObjArgs(protocol_error_handler, error, NULL)))\n goto error;\n\n PyObject* tmp;\n if(!(tmp = PyObject_CallFunctionObjArgs(self->write, response, NULL)))\n goto error;\n Py_DECREF(tmp);\n\n if(!Protocol_close(self))\n goto error;\n\n goto finally;\n\n error:\n self = NULL;\n\n finally:\n Py_XDECREF(response);\n Py_XDECREF(protocol_error_handler);\n return self;\n}\n#endif\n\n\nstatic PyObject*\nProtocol_pipeline_cancel(Protocol* self)\n{\n if(!Pipeline_cancel(&self->pipeline))\n return NULL;\n\n Py_RETURN_NONE;\n}\n\n\nstatic PyMethodDef Protocol_methods[] = {\n {\"connection_made\", (PyCFunction)Protocol_connection_made, METH_O, \"\"},\n {\"connection_lost\", (PyCFunction)Protocol_connection_lost, METH_VARARGS, \"\"},\n {\"data_received\", (PyCFunction)Protocol_data_received, METH_O, \"\"},\n {\"pipeline_cancel\", (PyCFunction)Protocol_pipeline_cancel, METH_NOARGS, \"\"},\n#ifdef PARSER_STANDALONE\n {\"on_headers\", (PyCFunction)Protocol_on_headers, METH_VARARGS, \"\"},\n {\"on_body\", (PyCFunction)Protocol_on_body, METH_VARARGS, \"\"},\n {\"on_error\", (PyCFunction)Protocol_on_error, METH_VARARGS, \"\"},\n#endif\n {NULL}\n};\n\n\nstatic PyObject*\nProtocol_get_pipeline_empty(Protocol* self)\n{\n if(PIPELINE_EMPTY(&self->pipeline))\n Py_RETURN_TRUE;\n\n Py_RETURN_FALSE;\n}\n\n\nstatic PyObject*\nProtocol_get_transport(Protocol* self)\n{\n Py_INCREF(self->transport);\n\n return self->transport;\n}\n\n\nstatic PyGetSetDef Protocol_getset[] = {\n {\"pipeline_empty\", (getter)Protocol_get_pipeline_empty, NULL, \"\", NULL},\n {\"transport\", (getter)Protocol_get_transport, NULL, \"\", NULL},\n {NULL}\n};\n\n\nstatic PyTypeObject ProtocolType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"cprotocol.Protocol\", /* tp_name */\n sizeof(Protocol), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Protocol_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Protocol\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Protocol_methods, /* tp_methods */\n 0, /* tp_members */\n Protocol_getset, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Protocol_init, /* tp_init */\n 0, /* tp_alloc */\n Protocol_new, /* tp_new */\n};\n\n\nstatic PyModuleDef cprotocol = {\n PyModuleDef_HEAD_INIT,\n \"cprotocol\",\n \"cprotocol\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n\n\nPyMODINIT_FUNC\nPyInit_cprotocol(void)\n{\n PyObject* m = NULL;\n#ifdef PARSER_STANDALONE\n PyObject* cparser = NULL;\n Parser = NULL;\n#endif\n PyObject* api_capsule = NULL;\n PyObject* crequest = NULL;\n PyObject* route = NULL;\n\n if (PyType_Ready(&ProtocolType) < 0)\n goto error;\n\n m = PyModule_Create(&cprotocol);\n if(!m)\n goto error;\n\n#ifdef PARSER_STANDALONE\n cparser = PyImport_ImportModule(\"japronto.parser.cparser\");\n if(!cparser)\n goto error;\n\n Parser = PyObject_GetAttrString(cparser, \"HttpRequestParser\");\n if(!Parser)\n goto error;\n#else\n if(cparser_init() == -1)\n goto error;\n#endif\n\n if(!cpipeline_init())\n goto error;\n\n if(!crequest_init())\n goto error;\n\n crequest = PyImport_ImportModule(\"japronto.request.crequest\");\n if(!crequest)\n goto error;\n\n PyRequest = PyObject_GetAttrString(crequest, \"Request\");\n if(!PyRequest)\n goto error;\n\n if(!(route = PyImport_ImportModule(\"japronto.router.route\")))\n goto error;\n\n if(!(RouteNotFoundException = PyObject_GetAttrString(\n route, \"RouteNotFoundException\")))\n goto error;\n\n request_capi = import_capi(\"japronto.request.crequest\");\n if(!request_capi)\n goto error;\n\n matcher_capi = import_capi(\"japronto.router.cmatcher\");\n if(!matcher_capi)\n goto error;\n\n response_capi = import_capi(\"japronto.response.cresponse\");\n if(!response_capi)\n goto error;\n\n Py_INCREF(&ProtocolType);\n PyModule_AddObject(m, \"Protocol\", (PyObject*)&ProtocolType);\n\n static Protocol_CAPI capi = {\n Protocol_close\n };\n api_capsule = export_capi(m, \"japronto.protocol.cprotocol\", &capi);\n if(!api_capsule)\n goto error;\n\n goto finally;\n\n error:\n Py_XDECREF(PyRequest);\n#ifdef PARSER_STANDALONE\n Py_XDECREF(Parser);\n#endif\n m = NULL;\n finally:\n Py_XDECREF(api_capsule);\n Py_XDECREF(crequest);\n Py_XDECREF(route);\n#ifdef PARSER_STANDALONE\n Py_XDECREF(cparser);\n#endif\n return m;\n}\n" }, { "path": "src/japronto/protocol/cprotocol.h", "content": "#pragma once\n\n#ifndef PARSER_STANDALONE\n#include \"cparser.h\"\n#endif\n\n#include \"cpipeline.h\"\n#include \"crequest.h\"\n#include \n\n#define GATHER_MAX_RESP 24\n\ntypedef struct {\n PyObject* responses[GATHER_MAX_RESP];\n size_t responses_end;\n size_t len;\n PyBytesObject* prev_buffer;\n bool enabled;\n} Gather;\n\n\ntypedef struct {\n PyObject_HEAD\n\n#ifdef PARSER_STANDALONE\n PyObject* feed;\n PyObject* feed_disconnect;\n#else\n Parser parser;\n#endif\n Request static_request;\n Pipeline pipeline;\n#ifdef REAPER_ENABLED\n unsigned long idle_time;\n unsigned long read_ops;\n unsigned long last_read_ops;\n#endif\n PyObject* app;\n PyObject* matcher;\n PyObject* error_handler;\n PyObject* transport;\n PyObject* write;\n PyObject* writelines;\n PyObject* create_task;\n PyObject* request_logger;\n#ifdef PROTOCOL_TRACK_REFCNT\n Py_ssize_t none_cnt;\n Py_ssize_t true_cnt;\n Py_ssize_t false_cnt;\n#endif\n bool closed;\n Gather gather;\n} Protocol;\n\n#define GATHER_MAX_LEN (4096 - sizeof(PyBytesObject))\n\n#ifndef PARSER_STANDALONE\nProtocol* Protocol_on_incomplete(Protocol* self);\nProtocol* Protocol_on_headers(Protocol*, char* method, size_t method_len,\n char* path, size_t path_len, int minor_version,\n void* headers, size_t num_headers);\nProtocol* Protocol_on_body(Protocol*, char* body, size_t body_len, size_t tail_len);\nProtocol* Protocol_on_error(Protocol*, PyObject*);\n#endif\n\ntypedef struct {\n void* (*Protocol_close)\n (Protocol* self);\n} Protocol_CAPI;\n" }, { "path": "src/japronto/protocol/cprotocol_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n cparser = system.get_extension_by_path('japronto/parser/cparser_ext.py')\n cpipeline = system.get_extension_by_path(\n 'japronto/pipeline/cpipeline_ext.py')\n\n define_macros = [('PIPELINE_PAIR', 1)]\n if system.args.enable_reaper:\n define_macros.append(('REAPER_ENABLED', 1))\n\n return Extension(\n 'japronto.protocol.cprotocol',\n sources=[\n 'cprotocol.c', '../capsule.c', '../request/crequest.c',\n '../response/cresponse.c',\n *cparser.sources, *cpipeline.sources],\n include_dirs=[\n '.', '..', '../parser', '../pipeline',\n '../router', '../request',\n '../response', *cparser.include_dirs],\n extra_objects=cparser.extra_objects,\n define_macros=define_macros)\n" }, { "path": "src/japronto/protocol/creaper.c", "content": "#include \n\n#include \"cprotocol.h\"\n#include \"capsule.h\"\n\ntypedef struct {\n PyObject_HEAD\n\n PyObject* connections;\n PyObject* call_later;\n PyObject* check_idle;\n PyObject* check_idle_handle;\n PyObject* check_interval;\n unsigned long idle_timeout;\n} Reaper;\n\n#ifdef REAPER_DEBUG_PRINT\n#define debug_print(format, ...) printf(\"reaper: \" format \"\\n\", __VA_ARGS__)\n#else\n#define debug_print(format, ...)\n#endif\n\nstatic Protocol_CAPI* protocol_capi;\n\nconst long DEFAULT_CHECK_INTERVAL = 10;\nconst unsigned long DEFAULT_IDLE_TIMEOUT = 60;\n\nstatic PyObject* default_check_interval;\n\nstatic PyObject*\nReaper_new(PyTypeObject* type, PyObject* args, PyObject* kwds)\n{\n Reaper* self = NULL;\n\n self = (Reaper*)type->tp_alloc(type, 0);\n if(!self)\n goto finally;\n\n self->connections = NULL;\n self->call_later = NULL;\n self->check_idle = NULL;\n self->check_idle_handle = NULL;\n self->check_interval = NULL;\n\n finally:\n return (PyObject*)self;\n}\n\n\nstatic void\nReaper_dealloc(Reaper* self)\n{\n Py_XDECREF(self->check_interval);\n Py_XDECREF(self->check_idle_handle);\n Py_XDECREF(self->check_idle);\n Py_XDECREF(self->call_later);\n Py_XDECREF(self->connections);\n\n Py_TYPE(self)->tp_free((PyObject*)self);\n}\n\n#ifdef REAPER_ENABLED\nstatic inline void*\nReaper_schedule_check_idle(Reaper* self)\n{\n Py_XDECREF(self->check_idle_handle);\n self->check_idle_handle = PyObject_CallFunctionObjArgs(\n self->call_later, self->check_interval, self->check_idle, NULL);\n\n return self->check_idle_handle;\n}\n#endif\n\n\nstatic PyObject*\nReaper_stop(Reaper* self)\n{\n#ifdef REAPER_ENABLED\n void* result = Py_None;\n PyObject* cancel = NULL;\n\n if(!(cancel = PyObject_GetAttrString(self->check_idle_handle, \"cancel\")))\n goto error;\n\n PyObject* tmp;\n if(!(tmp = PyObject_CallFunctionObjArgs(cancel, NULL)))\n goto error;\n Py_DECREF(tmp);\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XINCREF(result);\n Py_XDECREF(cancel);\n return result;\n#else\n Py_RETURN_NONE;\n#endif\n}\n\n\nstatic int\nReaper_init(Reaper* self, PyObject* args, PyObject* kwds)\n{\n PyObject* loop = NULL;\n int result = 0;\n\n PyObject* app = NULL;\n PyObject* idle_timeout = NULL;\n\n static char* kwlist[] = {\"app\", \"check_interval\", \"idle_timeout\", NULL};\n\n if (!PyArg_ParseTupleAndKeywords(\n args, kwds, \"|OOO\", kwlist, &app, &self->check_interval, &idle_timeout))\n goto error;\n\n assert(app);\n\n if(!self->check_interval)\n self->check_interval = default_check_interval;\n Py_INCREF(self->check_interval);\n\n assert(PyLong_AsLong(self->check_interval) >= 0);\n\n if(!idle_timeout)\n self->idle_timeout = DEFAULT_IDLE_TIMEOUT;\n else\n self->idle_timeout = PyLong_AsLong(idle_timeout);\n\n assert(self->idle_timeout >= 0);\n\n debug_print(\"check_interval %ld\", PyLong_AsLong(self->check_interval));\n debug_print(\"idle_timeout %ld\", self->idle_timeout);\n\n if(!(loop = PyObject_GetAttrString(app, \"_loop\")))\n goto error;\n\n if(!(self->call_later = PyObject_GetAttrString(loop, \"call_later\")))\n goto error;\n\n if(!(self->connections = PyObject_GetAttrString(app, \"_connections\")))\n goto error;\n\n#ifdef REAPER_ENABLED\n if(!(self->check_idle = PyObject_GetAttrString((PyObject*)self, \"_check_idle\")))\n goto error;\n\n if(!Reaper_schedule_check_idle(self))\n goto error;\n#endif\n\n goto finally;\n\n error:\n result = -1;\n\n finally:\n Py_XDECREF(loop);\n return result;\n}\n\n\n#ifdef REAPER_ENABLED\nstatic PyObject*\nReaper__check_idle(Reaper* self, PyObject* args)\n{\n PyObject* result = Py_None;\n PyObject* iterator = NULL;\n Protocol* conn = NULL;\n\n if(!(iterator = PyObject_GetIter(self->connections)))\n goto error;\n\n unsigned long check_interval = PyLong_AsLong(self->check_interval);\n while((conn = (Protocol*)PyIter_Next(iterator))) {\n debug_print(\n \"conn %p, idle_time %ld, read_ops %ld, last_read_ops %ld\",\n conn, conn->idle_time, conn->read_ops, conn->last_read_ops);\n\n if(conn->read_ops == conn->last_read_ops) {\n conn->idle_time += check_interval;\n\n if(conn->idle_time >= self->idle_timeout) {\n if(!protocol_capi->Protocol_close(conn))\n goto error;\n }\n } else {\n conn->idle_time = 0;\n conn->last_read_ops = conn->read_ops;\n }\n\n Py_DECREF(conn);\n }\n\n if(!Reaper_schedule_check_idle(self))\n goto error;\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(conn);\n Py_XDECREF(iterator);\n Py_XINCREF(result);\n return result;\n}\n#endif\n\n\nstatic PyMethodDef Reaper_methods[] = {\n#ifdef REAPER_ENABLED\n {\"_check_idle\", (PyCFunction)Reaper__check_idle, METH_NOARGS, \"\"},\n#endif\n {\"stop\", (PyCFunction)Reaper_stop, METH_NOARGS, \"\"},\n {NULL}\n};\n\n\nstatic PyTypeObject ReaperType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"creaper.Reaper\", /* tp_name */\n sizeof(Reaper), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Reaper_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Reaper\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Reaper_methods, /* tp_methods */\n 0, /* tp_members */\n 0, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Reaper_init, /* tp_init */\n 0, /* tp_alloc */\n Reaper_new, /* tp_new */\n};\n\n\nstatic PyModuleDef creaper = {\n PyModuleDef_HEAD_INIT,\n \"creaper\",\n \"creaper\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n\n\nPyMODINIT_FUNC\nPyInit_creaper(void)\n{\n PyObject* m = NULL;\n default_check_interval = NULL;\n\n if (PyType_Ready(&ReaperType) < 0)\n goto error;\n\n m = PyModule_Create(&creaper);\n if(!m)\n goto error;\n\n Py_INCREF(&ReaperType);\n PyModule_AddObject(m, \"Reaper\", (PyObject*)&ReaperType);\n\n if(!(default_check_interval = PyLong_FromLong(DEFAULT_CHECK_INTERVAL)))\n goto error;\n\n protocol_capi = import_capi(\"japronto.protocol.cprotocol\");\n if(!protocol_capi)\n goto error;\n\n goto finally;\n\n error:\n Py_XDECREF(default_check_interval);\n m = NULL;\n\n finally:\n return m;\n}\n" }, { "path": "src/japronto/protocol/creaper_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n define_macros = [('PIPELINE_PAIR', 1)]\n if system.args.enable_reaper:\n define_macros.append(('REAPER_ENABLED', 1))\n\n return Extension(\n 'japronto.protocol.creaper',\n sources=['creaper.c', '../capsule.c'],\n include_dirs=[\n '../parser', '../../picohttpparser',\n '../pipeline', '../request',\n '../router', '../response', '..'],\n define_macros=define_macros)\n" }, { "path": "src/japronto/protocol/generator.c", "content": "#include \n\n#include \"generator.h\"\n\n\ntypedef struct _Generator {\n PyObject_HEAD\n\n PyObject* object;\n} Generator;\n\n\nstatic PyTypeObject GeneratorType;\n\n\n#ifdef GENERATOR_OPAQUE\nstatic PyObject*\nGenerator_new(PyTypeObject* type, PyObject* args, PyObject* kw)\n#else\nPyObject*\nGenerator_new(void)\n#endif\n{\n Generator* self = NULL;\n\n#ifdef GENERATOR_OPAQUE\n self = (Generator*)type->tp_alloc(type, 0);\n#else\n self = (Generator*)GeneratorType.tp_alloc(&GeneratorType, 0);\n#endif\n if(!self)\n goto finally;\n\n self->object = NULL;\n\n finally:\n return (PyObject*)self;\n}\n\n\n#ifdef GENERATOR_OPAQUE\nstatic void\n#else\nvoid\n#endif\nGenerator_dealloc(Generator* self)\n{\n Py_XDECREF(self->object);\n\n Py_TYPE(self)->tp_free((PyObject*)self);\n}\n\n\n#ifdef GENERATOR_OPAQUE\nstatic int\nGenerator_init(Generator* self, PyObject *args, PyObject *kw)\n#else\nint\nGenerator_init(Generator* self, PyObject* object)\n#endif\n{\n int result = 0;\n\n#ifdef GENERATOR_OPAQUE\n if(!PyArg_ParseTuple(args, \"O\", &self->object))\n goto error;\n#else\n self->object = object;\n#endif\n\n Py_INCREF(self->object);\n\n goto finally;\n\n#ifdef GENERATOR_OPAQUE\n error:\n result = -1;\n#endif\n finally:\n return result;\n}\n\n\nstatic PyObject*\nGenerator_next(Generator* self)\n{\n PyErr_SetObject(PyExc_StopIteration, self->object);\n\n return NULL;\n}\n\n\nstatic PyObject*\nGenerator_send(Generator* self, PyObject* arg)\n{\n return Generator_next(self);\n}\n\n\nstatic PyMethodDef Generator_methods[] = {\n {\"send\", (PyCFunction)Generator_send, METH_O, \"\"},\n {NULL}\n};\n\n\nstatic PyTypeObject GeneratorType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"protocol.Generator\", /* tp_name */\n sizeof(Generator), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Generator_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Generator\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n PyObject_SelfIter, /* tp_iter */\n (iternextfunc)Generator_next, /* tp_iternext */\n Generator_methods, /* tp_methods */\n#ifdef GENERATOR_OPAQUE\n 0, /* tp_members */\n 0, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Generator_init, /* tp_init */\n 0, /* tp_alloc */\n Generator_new, /* tp_new */\n#endif\n};\n\n\n#ifdef GENERATOR_OPAQUE\nstatic PyModuleDef generator = {\n PyModuleDef_HEAD_INIT,\n \"generator\",\n \"generator\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n#endif\n\n\n#ifdef GENERATOR_OPAQUE\nPyMODINIT_FUNC\nPyInit_generator(void)\n#else\nvoid*\ngenerator_init(void)\n#endif\n{\n#ifdef GENERATOR_OPAQUE\n PyObject* m = NULL;\n#else\n void* m = &GeneratorType;\n#endif\n\n if(PyType_Ready(&GeneratorType) < 0)\n goto error;\n\n#ifdef GENERATOR_OPAQUE\n m = PyModule_Create(&generator);\n if(!m)\n goto error;\n\n Py_INCREF(&GeneratorType);\n PyModule_AddObject(m, \"Generator\", (PyObject*)&GeneratorType);\n#endif\n\n goto finally;\n\n error:\n m = NULL;\n\n finally:\n return m;\n}\n" }, { "path": "src/japronto/protocol/generator.h", "content": "#pragma once\n\n#ifndef GENERATOR_OPAQUE\nstruct _Generator;\n\n#define GENERATOR struct _Generator\n\nPyObject*\nGenerator_new(void);\n\nvoid\nGenerator_dealloc(struct _Generator* self);\n\nint\nGenerator_init(struct _Generator* self, PyObject* object);\n\nvoid*\ngenerator_init(void);\n#endif\n" }, { "path": "src/japronto/protocol/generator_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'protocol.generator',\n sources=['generator.c'],\n include_dirs=[],\n define_macros=[('GENERATOR_OPAQUE', 1)])\n" }, { "path": "src/japronto/protocol/handler.py", "content": "import asyncio\nfrom asyncio.queues import Queue\n\n\nfrom japronto.response.cresponse import Response\nfrom japronto.protocol.cprotocol import Protocol as CProtocol\n\nstatic_response = b\"\"\"HTTP/1.1 200 OK\\r\nConnection: keep-alive\\r\nContent-Length: 12\\r\nContent-Type: text/plain; encoding=utf-8\\r\n\\r\nHello statc!\n\"\"\"\n\n\ndef make_class(flavor):\n if flavor == 'c':\n return CProtocol\n\n from japronto.parser import cparser\n\n class HttpProtocol(asyncio.Protocol):\n def __init__(self, loop, handler):\n self.parser = cparser.HttpRequestParser(\n self.on_headers, self.on_body, self.on_error)\n self.loop = loop\n self.response = Response()\n\n if flavor == 'queue':\n def connection_made(self, transport):\n self.transport = transport\n self.queue = Queue(loop=self.loop)\n self.loop.create_task(handle_requests(self.queue, transport))\n else:\n def connection_made(self, transport):\n self.transport = transport\n\n def connection_lost(self, exc):\n self.parser.feed_disconnect()\n\n def data_received(self, data):\n self.parser.feed(data)\n\n def on_headers(self, request):\n return\n\n if flavor == 'block':\n def on_body(self, request):\n handle_request_block(request, self.transport, self.response)\n elif flavor == 'dump':\n def on_body(self, request):\n handle_dump(request, self.transport, self.response)\n elif flavor == 'task':\n def on_body(self, request):\n self.loop.create_task(handle_request(request, self.transport))\n elif flavor == 'queue':\n def on_body(self, request):\n self.queue.put_nowait(request)\n elif flavor == 'inline':\n def on_body(self, request):\n body = 'Hello inlin!'\n status_code = 200\n mime_type = 'text/plain'\n encoding = 'utf-8'\n text = [b'HTTP/1.1 ']\n text.extend([str(status_code).encode(), b' OK\\r\\n'])\n text.append(b'Connection: keep-alive\\r\\n')\n text.append(b'Content-Length: ')\n text.extend([str(len(body)).encode(), b'\\r\\n'])\n text.extend([\n b'Content-Type: ', mime_type.encode(),\n b'; encoding=', encoding.encode(), b'\\r\\n\\r\\n'])\n text.append(body.encode())\n\n self.transport.write(b''.join(text))\n\n elif flavor == 'static':\n def on_body(self, request):\n self.transport.write(static_response)\n\n def on_error(self, error):\n print(error)\n\n return HttpProtocol\n\n\nasync def handle_requests(queue, transport):\n while 1:\n await queue.get()\n\n response = Response(text='Hello queue!')\n\n transport.write(response.render())\n\n\nasync def handle_request(request, transport):\n response = Response(text='Hello ttask!')\n\n transport.write(response.render())\n\n\ndef handle_request_block(request, transport, response):\n response.__init__(404, text='Hello block')\n\n transport.write(response.render())\n\n\ndef handle_dump(request, transport, response):\n text = request.path\n response.__init__(text=text)\n\n transport.write(response.render())\n" }, { "path": "src/japronto/protocol/null.py", "content": "class NullProtocol:\n def on_headers(self, *args):\n pass\n\n def on_body(self, body):\n pass\n\n def on_error(self, error):\n pass\n" }, { "path": "src/japronto/protocol/tracing.py", "content": "from functools import partial\n\nfrom parser.libpicohttpparser import ffi\nfrom request import HttpRequest\n\n\nclass TracingProtocol:\n def __init__(self, on_headers_adapter: callable,\n on_body_adapter: callable):\n self.requests = []\n self.error = None\n\n self.on_headers_adapter = on_headers_adapter\n self.on_body_adapter = on_body_adapter\n\n self.on_headers_call_count = 0\n self.on_body_call_count = 0\n self.on_error_call_count = 0\n\n def on_headers(self, *args):\n self.request = self.on_headers_adapter(*args)\n\n self.requests.append(self.request)\n\n self.on_headers_call_count += 1\n\n def on_body(self, body):\n self.request.body = self.on_body_adapter(body)\n\n self.on_body_call_count += 1\n\n def on_error(self, error: str):\n self.error = error\n\n self.on_error_call_count += 1\n\n\ndef _request_from_cprotocol(method: memoryview, path: memoryview, version: int,\n headers: memoryview):\n method = method.tobytes().decode('ascii')\n path = path.tobytes().decode('ascii')\n version = \"1.0\" if version == 0 else \"1.1\"\n headers_len = headers.nbytes // ffi.sizeof(\"struct phr_header\")\n headers_cdata = ffi.from_buffer(headers)\n headers_cdata = ffi.cast(\n 'struct phr_header[{}]'.format(headers_len), headers_cdata)\n\n headers = _extract_headers(headers_cdata)\n\n return HttpRequest(method, path, version, headers)\n\n\ndef _body_from_cprotocol(body: memoryview):\n return None if body is None else body.tobytes()\n\n\ndef _request_from_cffiprotocol(method: \"char[]\", path: \"char[]\", version: int,\n headers: \"struct phr_header[]\"):\n method = ffi.buffer(method)[:].decode('ascii')\n path = ffi.buffer(path)[:].decode('ascii')\n version = \"1.0\" if version == 0 else \"1.1\"\n\n headers = _extract_headers(headers)\n\n return HttpRequest(method, path, version, headers)\n\n\ndef _body_from_cffiprotocol(body: \"char[]\"):\n return None if body is None else ffi.buffer(body)[:]\n\n\ndef _extract_headers(headers_cdata: \"struct phr_header[]\"):\n headers = {}\n for header in headers_cdata:\n name = ffi.string(header.name, header.name_len).decode('ascii').title()\n value = ffi.string(header.value, header.value_len).decode('latin1')\n headers[name] = value\n\n return headers\n\n\nCTracingProtocol = partial(\n TracingProtocol, on_headers_adapter=_request_from_cprotocol,\n on_body_adapter=_body_from_cprotocol)\n\n\nCffiTracingProtocol = partial(\n TracingProtocol, on_headers_adapter=_request_from_cffiprotocol,\n on_body_adapter=_body_from_cffiprotocol)\n" }, { "path": "src/japronto/reloader.py", "content": "import sys\nimport os.path\nimport os\nimport time\nimport threading\nimport signal\n\n\ndef main():\n import subprocess\n\n terminating = False\n\n def signal_received(sig, frame):\n nonlocal terminating\n\n if sig == signal.SIGHUP:\n child.send_signal(signal.SIGHUP)\n else:\n terminating = True\n child.terminate()\n\n signal.signal(signal.SIGINT, signal_received)\n signal.signal(signal.SIGTERM, signal_received)\n signal.signal(signal.SIGHUP, signal_received)\n\n os.putenv('_JAPR_RELOADER', str(os.getpid()))\n\n while not terminating:\n child = subprocess.Popen([\n sys.executable, '-m', 'japronto',\n '--reloader-pid', str(os.getpid()),\n *(v for v in sys.argv[1:] if v != '--reload')])\n child.wait()\n if child.returncode != 0:\n break\n\n\ndef exec_reloader(*, host, port, worker_num):\n args = ['--host', host, '--port', str(port)]\n if worker_num:\n args.extend(['--worker-num', str(worker_num)])\n os.execv(\n sys.executable,\n [sys.executable, '-m', 'japronto.reloader',\n *args, '--script', *sys.argv])\n\n\ndef change_detector():\n previous_mtimes = {}\n\n while 1:\n changed = False\n current_mtimes = {}\n\n for name, module in list(sys.modules.items()):\n try:\n filename = module.__file__\n except AttributeError:\n continue\n\n if not filename.endswith('.py'):\n continue\n\n mtime = os.path.getmtime(filename)\n previous_mtime = previous_mtimes.get(name)\n if previous_mtime and previous_mtime != mtime:\n changed = True\n\n current_mtimes[name] = mtime\n\n yield changed\n\n previous_mtimes = current_mtimes\n\n\nclass ChangeDetector(threading.Thread):\n def __init__(self, loop):\n super().__init__(daemon=True)\n self.loop = loop\n\n def run(self):\n for changed in change_detector():\n if changed:\n self.loop.call_soon_threadsafe(self.loop.stop)\n os.kill(os.getppid(), signal.SIGHUP)\n return\n time.sleep(.5)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "src/japronto/request/__init__.py", "content": "import urllib.parse\nfrom json import loads as json_loads\nimport cgi\nimport encodings.idna\nimport collections\nfrom http.cookies import _unquote as unquote_cookie\n\n\nclass HttpRequest(object):\n __slots__ = ('path', 'method', 'version', 'headers', 'body')\n\n def __init__(self, method, path, version, headers):\n self.path = path\n self.method = method\n self.version = version\n self.headers = headers\n self.body = None\n\n def dump_headers(self):\n print('path', self.path)\n print('method', self.method)\n print('version', self.version)\n for n, v in self.headers.items():\n print(n, v)\n\n def __repr__(self):\n return '' \\\n .format(self, len(self.headers))\n\n\ndef memoize(func):\n def wrapper(request):\n ns = request.extra.setdefault('_japronto', {})\n try:\n return ns[func.__name__]\n except KeyError:\n pass\n\n result = func(request)\n ns[func.__name__] = result\n\n return result\n\n return wrapper\n\n\n@memoize\ndef text(request):\n if request.body is None:\n return None\n\n return request.body.decode(request.encoding or 'utf-8')\n\n\n@memoize\ndef json(request):\n if request.body is None:\n return None\n\n return json_loads(request.text)\n\n\n@memoize\ndef query(request):\n qs = request.query_string\n if not qs:\n return {}\n return dict(urllib.parse.parse_qsl(qs))\n\n\ndef remote_addr(request):\n return request.transport.get_extra_info('peername')[0]\n\n\n@memoize\ndef parsed_content_type(request):\n content_type = request.headers.get('Content-Type')\n if not content_type:\n return None, {}\n\n return cgi.parse_header(content_type)\n\n\ndef mime_type(request):\n return parsed_content_type(request)[0]\n\n\ndef encoding(request):\n return parsed_content_type(request)[1].get('charset')\n\n\n@memoize\ndef parsed_form_and_files(request):\n if request.mime_type == 'application/x-www-form-urlencoded':\n return dict(urllib.parse.parse_qsl(request.text)), None\n elif request.mime_type == 'multipart/form-data':\n boundary = parsed_content_type(request)[1]['boundary'].encode('utf-8')\n return parse_multipart_form(request.body, boundary)\n\n return None, None\n\n\ndef form(request):\n return parsed_form_and_files(request)[0]\n\n\ndef files(request):\n return parsed_form_and_files(request)[1]\n\n\n@memoize\ndef hostname_and_port(request):\n host = request.headers.get('Host')\n if not host:\n return None, None\n\n hostname, *rest = host.split(':', 1)\n port = rest[0] if rest else None\n\n return encodings.idna.ToUnicode(hostname), int(port)\n\n\ndef port(request):\n return hostname_and_port(request)[1]\n\n\ndef hostname(request):\n return hostname_and_port(request)[0]\n\n\ndef parse_cookie(cookie):\n \"\"\"Parse a ``Cookie`` HTTP header into a dict of name/value pairs.\n This function attempts to mimic browser cookie parsing behavior;\n it specifically does not follow any of the cookie-related RFCs\n (because browsers don't either).\n The algorithm used is identical to that used by Django version 1.9.10.\n \"\"\"\n cookiedict = {}\n for chunk in cookie.split(str(';')):\n if str('=') in chunk:\n key, val = chunk.split(str('='), 1)\n else:\n # Assume an empty name per\n # https://bugzilla.mozilla.org/show_bug.cgi?id=169091\n key, val = str(''), chunk\n key, val = key.strip(), val.strip()\n if key or val:\n # unquote using Python's algorithm.\n cookiedict[key] = unquote_cookie(val)\n return cookiedict\n\n\n@memoize\ndef cookies(request):\n if 'Cookie' not in request.headers:\n return {}\n\n try:\n cookies = parse_cookie(request.headers['Cookie'])\n except Exception:\n return {}\n\n return {k: urllib.parse.unquote(v) for k, v in cookies.items()}\n\n\nFile = collections.namedtuple('File', ['type', 'body', 'name'])\n\n\ndef parse_multipart_form(body, boundary):\n files = {}\n fields = {}\n\n form_parts = body.split(boundary)\n for form_part in form_parts[1:-1]:\n file_name = None\n file_type = None\n field_name = None\n line_index = 2\n line_end_index = 0\n while not line_end_index == -1:\n line_end_index = form_part.find(b'\\r\\n', line_index)\n form_line = form_part[line_index:line_end_index].decode('utf-8')\n line_index = line_end_index + 2\n\n if not form_line:\n break\n\n colon_index = form_line.index(':')\n form_header_field = form_line[0:colon_index]\n form_header_value, form_parameters = cgi.parse_header(\n form_line[colon_index + 2:])\n\n if form_header_field == 'Content-Disposition':\n if 'filename' in form_parameters:\n file_name = form_parameters['filename']\n field_name = form_parameters.get('name')\n elif form_header_field == 'Content-Type':\n file_type = form_header_value\n\n post_data = form_part[line_index:-4]\n if file_name or file_type:\n file = File(type=file_type, name=file_name, body=post_data)\n files[field_name] = file\n else:\n value = post_data.decode('utf-8')\n fields[field_name] = value\n\n return fields, files\n" }, { "path": "src/japronto/request/crequest.c", "content": "#include \n#include \n#include \n#include \n\n#include \"crequest.h\"\n\n#include \"cresponse.h\"\n#ifdef REQUEST_OPAQUE\n#include \"picohttpparser.h\"\n#endif\n#include \"capsule.h\"\n\nstatic PyObject* PyResponse;\nstatic PyObject* partial;\n\n\n#ifdef REQUEST_OPAQUE\nstatic PyObject* HTTP10;\nstatic PyObject* HTTP11;\nstatic PyObject* request;\n#endif\n\nstatic Response_CAPI* response_capi;\n\n#ifdef REQUEST_OPAQUE\nstatic PyObject*\nRequest_new(PyTypeObject *type, PyObject *args, PyObject *kwds)\n#else\nPyObject*\nRequest_new(PyTypeObject* type, Request* self)\n#endif\n{\n#ifdef REQUEST_OPAQUE\n Request* self = NULL;\n\n self = (Request*)type->tp_alloc(type, 0);\n if(!self)\n goto finally;\n#else\n ((PyObject*)self)->ob_refcnt = 1;\n ((PyObject*)self)->ob_type = type;\n#endif\n self->response_called = false;\n\n self->matcher_entry = NULL;\n self->exception = NULL;\n self->app = NULL;\n\n self->transport = NULL;\n self->py_method = NULL;\n self->py_path = NULL;\n self->py_qs = NULL;\n self->py_headers = NULL;\n self->py_match_dict = NULL;\n self->py_body = NULL;\n self->extra = NULL;\n self->done_callbacks = NULL;\n\n Response_new(response_capi->ResponseType, &self->response);\n\n self->buffer = self->inline_buffer;\n self->buffer_len = REQUEST_INITIAL_BUFFER_LEN;\n#ifdef REQUEST_OPAQUE\n finally:\n#endif\n return (PyObject*)self;\n}\n\n\n#ifdef REQUEST_OPAQUE\nstatic void\n#else\nvoid\n#endif\nRequest_dealloc(Request* self)\n{\n if(self->buffer != self->inline_buffer)\n free(self->buffer);\n\n Response_dealloc(&self->response);\n Py_XDECREF(self->app);\n Py_XDECREF(self->done_callbacks);\n Py_XDECREF(self->extra);\n Py_XDECREF(self->py_body);\n Py_XDECREF(self->py_match_dict);\n Py_XDECREF(self->py_headers);\n Py_XDECREF(self->py_qs);\n Py_XDECREF(self->py_path);\n Py_XDECREF(self->py_method);\n Py_XDECREF(self->transport);\n\n Py_XDECREF(self->exception);\n#ifdef REQUEST_OPAQUE\n Py_TYPE(self)->tp_free((PyObject*)self);\n#endif\n}\n\n\n#ifdef REQUEST_OPAQUE\nstatic int\nRequest_init(Request* self, PyObject *args, PyObject* kw)\n#else\nint\nRequest_init(Request* self)\n#endif\n{\n return 0;\n}\n\n\n#ifdef REQUEST_OPAQUE\n\nstatic PyTypeObject RequestType;\n\nstatic PyObject*\nRequest_clone(Request* original)\n{\n Request* clone = NULL;\n\n if(!(clone = (Request*)Request_new(&RequestType, NULL, NULL)))\n goto error;\n\n if(Request_init(clone, NULL, NULL) == -1)\n goto error;\n\n const size_t offset = offsetof(Request, method);\n const size_t length = offsetof(Request, transport) - offset;\n\n memcpy((char*)clone + offset, (char*)original + offset, length);\n\n if(original->buffer == original->inline_buffer) {\n clone->buffer = clone->inline_buffer;\n\n ptrdiff_t shift = (char*)clone - (char*)original;\n clone->method += shift;\n clone->path += shift;\n clone->headers = (struct phr_header*)((char*)clone->headers + shift);\n for(struct phr_header* header = clone->headers;\n header < clone->headers + clone->num_headers;\n header++) {\n header->name += shift;\n header->value += shift;\n }\n clone->match_dict_entries =\n (MatchDictEntry*)((char*)clone->match_dict_entries + shift);\n for(MatchDictEntry* entry = clone->match_dict_entries;\n entry < clone->match_dict_entries + clone->match_dict_length; entry++) {\n // the keys didnt move, they reference immutable memory from the router\n entry->value += shift;\n }\n if(clone->body)\n // body can be NULL\n clone->body += shift;\n } else {\n // just steal the buffer since the original request will be destroyed anyway\n clone->buffer = original->buffer;\n original->buffer = original->inline_buffer;\n }\n\n goto finally;\n\n error:\n Py_XDECREF(clone);\n clone = NULL;\n\n finally:\n return (PyObject*)clone;\n}\n\n\nstatic KEEP_ALIVE\n_Request_get_keep_alive(Request* self);\n\n\nstatic PyObject*\nRequest_Response(Request* self, PyObject *args, PyObject* kw)\n{\n if(self->response_called) {\n PyErr_SetString(\n PyExc_RuntimeError,\n \"Request.Response can only be called once per request\");\n goto error;\n }\n\n self->response_called = true;\n Response* result = &self->response;\n\n if(response_capi->Response_init(result, args, kw) == -1)\n goto error;\n\n result->minor_version = self->minor_version;\n result->keep_alive = _Request_get_keep_alive(self);\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XINCREF(result);\n return (PyObject*)result;\n}\n\n\ntypedef enum {\n REQUEST_HEADERS,\n REQUEST_MATCH_DICT,\n REQUEST_BODY\n} RequestCopy;\n\n\n#define ROUNDTO8(v) (((v) + 7) & ~7)\n\n\nstatic inline char*\nbfrcpy(Request* self, const RequestCopy what)\n{\n size_t len;\n char* dst;\n char* old_buffer = self->buffer;\n size_t headers_len;\n size_t header_entries_len;\n\n if(self->num_headers) {\n struct phr_header* last_header = &self->headers[self->num_headers - 1];\n headers_len = last_header->value + last_header->value_len - self->method;\n header_entries_len = sizeof(struct phr_header) * self->num_headers;\n } else {\n headers_len = self->path + self->path_len + self->qs_len - self->method;\n header_entries_len = 0;\n }\n\n switch(what) {\n case REQUEST_HEADERS:\n len = ROUNDTO8(headers_len) + header_entries_len;\n dst = self->buffer;\n break;\n\n case REQUEST_MATCH_DICT:\n len = sizeof(MatchDictEntry) * self->match_dict_length;\n dst = self->buffer + ROUNDTO8(headers_len + header_entries_len);\n break;\n\n case REQUEST_BODY:\n len = self->body_length;\n dst = (char*)self->match_dict_entries \\\n + sizeof(MatchDictEntry) * self->match_dict_length;\n break;\n\n default:\n assert(0);\n }\n\n if(dst + len > self->buffer + self->buffer_len)\n {\n self->buffer_len = MAX(self->buffer_len * 2, self->buffer_len + len);\n\n if(self->buffer == self->inline_buffer)\n {\n self->buffer = malloc(self->buffer_len);\n if(!self->buffer)\n assert(0);\n // TODO, propagate memory error\n memcpy(self->buffer, self->inline_buffer, REQUEST_INITIAL_BUFFER_LEN);\n } else {\n self->buffer = realloc(self->buffer, self->buffer_len);\n if(!self->buffer)\n assert(0);\n // TODO, propagate memory error\n }\n }\n\n ptrdiff_t buffer_shift = self->buffer - old_buffer;\n dst += buffer_shift;\n ptrdiff_t shift;\n\n if(what == REQUEST_HEADERS) {\n shift = dst - self->method;\n\n memcpy(dst, self->method, headers_len);\n self->method += shift;\n self->path += shift;\n memcpy(dst + ROUNDTO8(headers_len), (char*)self->headers, header_entries_len);\n self->headers = (struct phr_header*)((char*)dst + ROUNDTO8(headers_len));\n for(struct phr_header* header = self->headers;\n header < self->headers + self->num_headers;\n header++) {\n header->name += shift;\n header->value += shift;\n }\n\n goto finally;\n }\n\n if(buffer_shift) {\n self->method += buffer_shift;\n self->path += buffer_shift;\n self->headers = (struct phr_header*)((char*)self->headers + buffer_shift);\n for(struct phr_header* header = self->headers;\n header < self->headers + self->num_headers;\n header++) {\n header->name += buffer_shift;\n header->value += buffer_shift;\n }\n }\n\n if(what == REQUEST_HEADERS)\n goto finally;\n\n if(what == REQUEST_MATCH_DICT) {\n shift = dst - (char*)self->match_dict_entries;\n\n memcpy(dst, (char*)self->match_dict_entries, len);\n self->match_dict_entries =\n (MatchDictEntry*)((char*)self->match_dict_entries + shift);\n /* match_dict_entires values don't need moving by shift because the block\n * they reference couldn't move (the previous call)\n */\n }\n\n if(buffer_shift) {\n if(what != REQUEST_MATCH_DICT)\n self->match_dict_entries =\n (MatchDictEntry*)((char*)self->match_dict_entries + buffer_shift);\n for(MatchDictEntry* entry = self->match_dict_entries;\n entry < self->match_dict_entries + self->match_dict_length; entry++) {\n // the keys didnt move, they reference immutable memory from the router\n entry->value += buffer_shift;\n }\n }\n\n if(what == REQUEST_MATCH_DICT)\n goto finally;\n\n if(what == REQUEST_BODY) {\n shift = dst - self->body;\n\n memcpy(dst, self->body, len);\n self->body += shift;\n\n goto finally;\n }\n\n assert(0);\n\n finally:\n return self->buffer;\n}\n\n\nstatic void\nRequest_from_raw(Request* self, char* method, size_t method_len, char* path, size_t path_len,\n int minor_version,\n struct phr_header* headers, size_t num_headers)\n{\n // fill\n self->method = method;\n self->method_len = method_len;\n self->path = path;\n self->path_decoded = false;\n self->path_len = path_len;\n self->qs_len = 0;\n self->qs_decoded = false;\n self->minor_version = minor_version;\n self->headers = headers;\n self->num_headers = num_headers;\n self->keep_alive = KEEP_ALIVE_UNSET;\n\n bfrcpy(self, REQUEST_HEADERS);\n}\n\n\nstatic void\nRequest_set_match_dict_entries(Request* self, MatchDictEntry* entries,\n size_t length)\n{\n self->match_dict_entries = entries;\n self->match_dict_length = length;\n bfrcpy(self, REQUEST_MATCH_DICT);\n}\n\n\nstatic void\nRequest_set_body(Request* self, char* body, size_t body_len)\n{\n if(!body) {\n self->body = NULL;\n return;\n }\n\n self->body = body;\n self->body_length = body_len;\n bfrcpy(self, REQUEST_BODY);\n}\n\n\n#define hex_to_dec(x) \\\n ((x <= '9' ? 0 : 9) + (x & 0x0f))\n#define is_hex(x) ((x >= '0' && x <= '9') || (x >= 'A' && x <= 'F'))\nstatic inline size_t percent_decode(char* data, ssize_t length, size_t* shifted_bytes, const char* stopchr) {\n if(shifted_bytes)\n *shifted_bytes = 0;\n\n for(char* end = data + length; data < end; data++) {\n if(stopchr && *data == *stopchr) {\n length -= end - data;\n break;\n }\n\n if(end - data < 3)\n continue;\n\n if(*data == '%' && is_hex(*(data + 1)) && is_hex(*(data + 2))) {\n *data = (hex_to_dec(*(data + 1)) << 4) + hex_to_dec(*(data + 2));\n length -= 2;\n if(shifted_bytes)\n *shifted_bytes += 2;\n memmove(data + 1, data + 3, end - (data + 3));\n end -= 2;\n }\n }\n\n return length;\n}\n#undef hex_to_dec\n#undef is_hex\n\n\nchar*\nRequest_get_decoded_path(Request* self, size_t* path_len) {\n if(!self->path_decoded) {\n size_t shifted_bytes;\n const char stopchr = '?';\n *path_len = percent_decode(\n self->path, self->path_len, &shifted_bytes, &stopchr);\n self->path_decoded = true;\n\n self->qs_len = self->path_len - *path_len - shifted_bytes;\n self->path_len = *path_len;\n }\n\n *path_len = self->path_len;\n return self->path;\n}\n\n\nstatic char*\nRequest_get_decoded_qs(Request* self, size_t* qs_len) {\n if(!self->qs_len) {\n *qs_len = 0;\n return NULL;\n }\n\n char* qs = self->path + self->path_len;\n\n if(!self->qs_decoded) {\n self->qs_len = percent_decode(qs, self->qs_len, NULL, NULL);\n self->qs_decoded = true;\n }\n\n *qs_len = self->qs_len;\n return qs;\n}\n\n\nstatic inline void title_case(char* data, size_t len)\n{\n bool prev_alpha = false;\n for(char* c = data; c < data + len; c++) {\n if(*c >= 'A' && *c <= 'Z') {\n if(prev_alpha)\n *c ^= 0x20;\n prev_alpha = true;\n } else if (*c >= 'a' && *c <= 'z') {\n if(!prev_alpha)\n *c ^= 0x20;\n prev_alpha = true;\n } else\n prev_alpha = false;\n }\n}\n\n\nstatic inline PyObject*\nRequest_decode_headers(Request* self)\n{\n PyObject* result = NULL;\n PyObject* headers = PyDict_New();\n if(!headers)\n goto error;\n result = headers;\n\n for(struct phr_header* header = self->headers;\n header < self->headers + self->num_headers;\n header++) {\n\n PyObject* name = NULL;\n PyObject* value = NULL;\n\n title_case((char*)header->name, header->name_len);\n // TODO by inserting 0 byte we could call PyDict_SetItemString\n\n // FIXME check ASCII\n name = PyUnicode_FromStringAndSize(header->name, header->name_len);\n if(!name)\n goto loop_error;\n\n // FIXME this can fail on codec errors\n value = PyUnicode_DecodeLatin1(header->value, header->value_len, NULL);\n if(!value)\n goto loop_error;\n\n if(PyDict_SetItem(headers, name, value) == -1)\n goto loop_error;\n\n goto loop_finally;\n\n loop_error:\n result = NULL;\n\n loop_finally:\n Py_XDECREF(name);\n Py_XDECREF(value);\n\n if(!result)\n goto error;\n }\n\n goto finally;\n\n error:\n Py_XDECREF(headers);\n result = NULL;\n\n finally:\n return result;\n}\n\n\nstatic PyObject*\nRequest_get_method(Request* self, void* closure)\n{\n if(!self->py_method) {\n self->py_method = PyUnicode_DecodeLatin1(\n REQUEST_METHOD(self), self->method_len, NULL);\n }\n\n Py_XINCREF(self->py_method);\n return self->py_method;\n}\n\n\nstatic PyObject*\nRequest_get_path(Request* self, void* closure)\n{\n if(!self->py_path) {\n size_t path_len;\n char* path = Request_get_decoded_path(self, &path_len);\n self->py_path = PyUnicode_FromStringAndSize(path, path_len);\n }\n\n Py_XINCREF(self->py_path);\n return self->py_path;\n}\n\n\nstatic PyObject*\nRequest_get_qs(Request* self, void* closure)\n{\n if(!self->py_qs) {\n size_t qs_len;\n char* qs = Request_get_decoded_qs(self, &qs_len);\n if(!qs)\n Py_RETURN_NONE;\n\n // skip the ? char\n self->py_qs = PyUnicode_FromStringAndSize(qs + 1, qs_len - 1);\n }\n\n Py_XINCREF(self->py_qs);\n return self->py_qs;\n}\n\n\nstatic PyObject*\nRequest_get_version(Request* self, void* closure) {\n PyObject* result = self->minor_version ? HTTP11 : HTTP10;\n\n Py_INCREF(result);\n return result;\n}\n\n\nstatic PyObject*\nRequest_get_headers(Request* self, void* closure) {\n if(!self->py_headers)\n self->py_headers = Request_decode_headers(self);\n\n Py_XINCREF(self->py_headers);\n return self->py_headers;\n}\n\n\nstatic PyObject*\nRequest_get_match_dict(Request* self, void* closure)\n{\n if(!self->py_match_dict)\n self->py_match_dict = MatchDict_entries_to_dict(\n self->match_dict_entries, self->match_dict_length);\n\n Py_XINCREF(self->py_match_dict);\n return self->py_match_dict;\n}\n\n\nstatic PyObject*\nRequest_get_body(Request* self, void* closure)\n{\n if(!self->body)\n Py_RETURN_NONE;\n\n if(!self->py_body)\n self->py_body = PyBytes_FromStringAndSize(self->body, self->body_length);\n\n Py_XINCREF(self->py_body);\n return self->py_body;\n}\n\n\nstatic PyObject*\nRequest_get_transport(Request* self, void* closure)\n{\n Py_INCREF(self->transport);\n return self->transport;\n}\n\n\nstatic KEEP_ALIVE\n_Request_get_keep_alive(Request* self)\n{\n if(self->keep_alive == KEEP_ALIVE_UNSET) {\n struct phr_header* Connection = NULL;\n for(struct phr_header* header = self->headers;\n header < self->headers + self->num_headers;\n header++) {\n if(header->name_len == strlen(\"Connection\")\n && strncasecmp(header->name, \"Connection\", header->name_len) == 0) {\n Connection = header;\n break;\n }\n }\n\n if(self->minor_version == 0) {\n // FIXME: this should check what's before and after\n if(Connection &&\n memmem(Connection->value, Connection->value_len,\n \"keep-alive\", strlen(\"keep-alive\")))\n self->keep_alive = KEEP_ALIVE_TRUE;\n else\n self->keep_alive = KEEP_ALIVE_FALSE;\n } else {\n if(Connection &&\n memmem(Connection->value, Connection->value_len,\n \"close\", strlen(\"close\")))\n self->keep_alive = KEEP_ALIVE_FALSE;\n else\n self->keep_alive = KEEP_ALIVE_TRUE;\n }\n }\n\n return self->keep_alive;\n}\n\n\nstatic PyObject*\nRequest_get_keep_alive(Request* self, void* closure)\n{\n if(_Request_get_keep_alive(self) == KEEP_ALIVE_TRUE)\n Py_RETURN_TRUE;\n else\n Py_RETURN_FALSE;\n}\n\n\nstatic PyObject*\nRequest_get_route(Request* self, void* closure)\n{\n if(!self->matcher_entry)\n Py_RETURN_NONE;\n\n Py_INCREF(self->matcher_entry->route);\n return self->matcher_entry->route;\n}\n\n\nstatic PyObject*\nRequest_get_extra(Request* self, void* closure)\n{\n if(!self->extra)\n self->extra = PyDict_New();\n\n Py_XINCREF(self->extra);\n return self->extra;\n}\n\n\nstatic PyObject*\nRequest_get_app(Request* self, void* app)\n{\n Py_INCREF(self->app);\n return self->app;\n}\n\n\nstatic PyObject*\nRequest_get_proxy(Request* self, char* attr)\n{\n PyObject* callable = NULL;\n PyObject* result = NULL;\n if(!(callable = PyObject_GetAttrString(request, attr)))\n goto error;\n\n if(!(result = PyObject_CallFunctionObjArgs(callable, self, NULL)))\n goto error;\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n Py_XDECREF(callable);\n\n return result;\n}\n\n\n#define PROXY(attr) \\\n {#attr, (getter)Request_get_proxy, NULL, \"\", #attr}\n\nstatic PyGetSetDef Request_getset[] = {\n {\"method\", (getter)Request_get_method, NULL, \"\", NULL},\n {\"path\", (getter)Request_get_path, NULL, \"\", NULL},\n {\"query_string\", (getter)Request_get_qs, NULL, \"\", NULL},\n {\"version\", (getter)Request_get_version, NULL, \"\", NULL},\n {\"headers\", (getter)Request_get_headers, NULL, \"\", NULL},\n {\"match_dict\", (getter)Request_get_match_dict, NULL, \"\", NULL},\n {\"body\", (getter)Request_get_body, NULL, \"\", NULL},\n {\"transport\", (getter)Request_get_transport, NULL, \"\", NULL},\n {\"keep_alive\", (getter)Request_get_keep_alive, NULL, \"\", NULL},\n {\"route\", (getter)Request_get_route, NULL, \"\", NULL},\n {\"extra\", (getter)Request_get_extra, NULL, \"\", NULL},\n {\"app\", (getter)Request_get_app, NULL, \"\", NULL},\n PROXY(text),\n PROXY(json),\n PROXY(query),\n PROXY(mime_type),\n PROXY(encoding),\n PROXY(form),\n PROXY(files),\n PROXY(remote_addr),\n PROXY(hostname),\n PROXY(port),\n PROXY(cookies),\n {NULL}\n};\n\n#undef PROXY\n\nstatic PyObject*\nRequest_getattro(Request* self, PyObject* name)\n{\n PyObject* result;\n\n if((result = PyObject_GenericGetAttr((PyObject*)self, name)))\n return result;\n\n PyObject* extensions = NULL;\n if(!(extensions = PyObject_GetAttrString(self->app, \"_request_extensions\")))\n goto error;\n\n PyObject* entry;\n if(!(entry = PyDict_GetItem(extensions, name)))\n goto error;\n else\n PyErr_Clear();\n\n PyObject* handler;\n PyObject* property;\n if(!(handler = PyTuple_GetItem(entry, 0)))\n goto error;\n\n if(!(property = PyTuple_GetItem(entry, 1)))\n goto error;\n\n if(property == Py_True) {\n if(!(result = PyObject_CallFunctionObjArgs(handler, self, NULL)))\n goto error;\n } else {\n if(!(result = PyObject_CallFunctionObjArgs(partial, handler, self, NULL)))\n goto error;\n }\n\n error:\n Py_XDECREF(extensions);\n\n return result;\n}\n\n\nstatic PyObject*\nRequest_add_done_callback(Request* self, PyObject* callback)\n{\n if(!self->done_callbacks) {\n if(!(self->done_callbacks = PyList_New(0)))\n goto error;\n }\n\n if(PyList_Append(self->done_callbacks, callback) == -1)\n goto error;\n\n Py_RETURN_NONE;\n\n error:\n return NULL;\n}\n\n\nstatic PyMethodDef Request_methods[] = {\n {\"Response\", (PyCFunction)Request_Response, METH_VARARGS | METH_KEYWORDS, \"\"},\n {\"add_done_callback\", (PyCFunction)Request_add_done_callback, METH_O, \"\"},\n {NULL}\n};\n\n\nstatic PyTypeObject RequestType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"crequest.Request\", /* tp_name */\n sizeof(Request), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Request_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n (getattrofunc)Request_getattro, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Request\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Request_methods, /* tp_methods */\n 0, /* tp_members */\n Request_getset, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Request_init, /* tp_init */\n 0, /* tp_alloc */\n Request_new, /* tp_new */\n};\n\n\nstatic PyModuleDef crequest = {\n PyModuleDef_HEAD_INIT,\n \"crequest\",\n \"crequest\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n#endif\n\n#ifdef REQUEST_OPAQUE\nPyMODINIT_FUNC\nPyInit_crequest(void)\n#else\nvoid*\ncrequest_init(void)\n#endif\n{\n#ifdef REQUEST_OPAQUE\n PyObject* m = NULL;\n PyObject* api_capsule = NULL;\n\n HTTP10 = NULL;\n HTTP11 = NULL;\n#else\n void* m = (void*)1;\n#endif\n PyObject* cresponse = NULL;\n PyObject* functools = NULL;\n PyResponse = NULL;\n\n#ifdef REQUEST_OPAQUE\n if (PyType_Ready(&RequestType) < 0)\n goto error;\n\n#define alloc_static2(name, val) \\\n name = PyUnicode_FromString(val); \\\n if(!name) \\\n goto error;\n\n alloc_static2(HTTP10, \"1.0\")\n alloc_static2(HTTP11, \"1.1\")\n\n m = PyModule_Create(&crequest);\n if(!m)\n goto error;\n#endif\n\n cresponse = PyImport_ImportModule(\"japronto.response.cresponse\");\n if(!cresponse)\n goto error;\n\n if(!(functools = PyImport_ImportModule(\"functools\")))\n goto error;\n\n if(!(partial = PyObject_GetAttrString(functools, \"partial\")))\n goto error;\n\n PyResponse = PyObject_GetAttrString(cresponse, \"Response\");\n if(!PyResponse)\n goto error;\n\n#ifdef REQUEST_OPAQUE\n request = PyImport_ImportModule(\"japronto.request\");\n if(!request)\n goto error;\n\n Py_INCREF(&RequestType);\n PyModule_AddObject(m, \"Request\", (PyObject*)&RequestType);\n\n static Request_CAPI capi = {\n &RequestType,\n Request_clone,\n Request_from_raw,\n Request_get_decoded_path,\n Request_set_match_dict_entries,\n Request_set_body\n };\n api_capsule = export_capi(m, \"japronto.request.crequest\", &capi);\n if(!api_capsule)\n goto error;\n\n#endif\n response_capi = import_capi(\"japronto.response.cresponse\");\n if(!response_capi)\n goto error;\n\n goto finally;\n\n error:\n Py_XDECREF(PyResponse);\n#ifdef REQUEST_OPAQUE\n Py_XDECREF(HTTP10);\n Py_XDECREF(HTTP11);\n#endif\n m = NULL;\n\n finally:\n Py_XDECREF(functools);\n Py_XDECREF(cresponse);\n#ifdef REQUEST_OPAQUE\n Py_XDECREF(api_capsule);\n#endif\n return m;\n}\n" }, { "path": "src/japronto/request/crequest.h", "content": "#pragma once\n\n#include \n#include \n\n#include \"cmatcher.h\"\n#include \"cresponse.h\"\n#include \"common.h\"\n\n#define REQUEST_INITIAL_BUFFER_LEN 1024\n\ntypedef struct {\n PyObject_HEAD\n\n char* method;\n size_t method_len;\n char* path;\n bool path_decoded;\n size_t path_len;\n bool qs_decoded;\n size_t qs_len;\n int minor_version;\n struct phr_header* headers;\n size_t num_headers;\n MatchDictEntry* match_dict_entries;\n size_t match_dict_length;\n char* body;\n size_t body_length;\n char* buffer;\n size_t buffer_len;\n char inline_buffer[REQUEST_INITIAL_BUFFER_LEN];\n KEEP_ALIVE keep_alive;\n bool simple;\n bool response_called;\n MatcherEntry* matcher_entry;\n PyObject* exception;\n\n PyObject* transport;\n PyObject* app;\n PyObject* py_method;\n PyObject* py_path;\n PyObject* py_qs;\n PyObject* py_headers;\n PyObject* py_match_dict;\n PyObject* py_body;\n PyObject* extra;\n PyObject* done_callbacks;\n Response response;\n} Request;\n\n#define REQUEST(r) \\\n ((Request*)r)\n\n#define REQUEST_METHOD(r) \\\n REQUEST(r)->buffer\n\n#define REQUEST_PATH(r) \\\n REQUEST(r)->path\n\n\ntypedef struct {\n PyTypeObject* RequestType;\n\n PyObject* (*Request_clone)\n (Request* original);\n\n void (*Request_from_raw)\n (Request* self, char* method, size_t method_len,\n char* path, size_t path_len,\n int minor_version,\n struct phr_header* headers, size_t num_headers);\n\n char* (*Request_get_decoded_path)\n (Request* self, size_t* path_len);\n\n void (*Request_set_match_dict_entries)\n (Request* self, MatchDictEntry* entries, size_t length);\n\n void (*Request_set_body)\n (Request* self, char* body, size_t body_len);\n} Request_CAPI;\n\n\n#ifndef REQUEST_OPAQUE\nPyObject*\nRequest_new(PyTypeObject* type, Request* self);\n\nvoid\nRequest_dealloc(Request* self);\n\nint\nRequest_init(Request* self);\n\nvoid*\ncrequest_init(void);\n#endif\n" }, { "path": "src/japronto/request/crequest_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.request.crequest',\n sources=['crequest.c', '../response/cresponse.c',\n '../router/match_dict.c', '../capsule.c'],\n include_dirs=['../../picohttpparser', '..',\n '../response', '../router'],\n define_macros=[('REQUEST_OPAQUE', 1)])\n" }, { "path": "src/japronto/response/__init__.py", "content": "" }, { "path": "src/japronto/response/cresponse.c", "content": "#include \n#include \n\n#include \"cresponse.h\"\n#include \"capsule.h\"\n#include \"reasons.h\"\n\n#ifdef RESPONSE_OPAQUE\nstatic PyObject* json_dumps;\nstatic const size_t reason_offset = 13;\nstatic const size_t minor_offset = 7;\n#endif\n\n\n\nstatic const char header[] = \"HTTP/1.1 200 OK\\r\\n\"\n \"Content-Length: \";\n\n\n#ifdef RESPONSE_OPAQUE\nstatic PyObject *\nResponse_new(PyTypeObject *type, PyObject *args, PyObject *kwds)\n#else\nPyObject*\nResponse_new(PyTypeObject* type, Response* self)\n#endif\n{\n#ifdef RESPONSE_OPAQUE\n Response* self = NULL;\n\n self = (Response*)type->tp_alloc(type, 0);\n if(!self)\n goto finally;\n self->opaque = true;\n#else\n ((PyObject*)self)->ob_refcnt = 1;\n ((PyObject*)self)->ob_type = type;\n self->opaque = false;\n#endif\n\n self->code = NULL;\n self->mime_type = NULL;\n self->body = NULL;\n self->encoding = NULL;\n self->headers = NULL;\n self->cookies = NULL;\n\n self->buffer = self->inline_buffer;\n self->buffer_len = RESPONSE_INITIAL_BUFFER_LEN;\n memcpy(self->buffer, header, strlen(header));\n\n#ifdef RESPONSE_OPAQUE\n finally:\n#endif\n return (PyObject*)self;\n}\n\n\n#ifdef RESPONSE_OPAQUE\nstatic void\n#else\nvoid\n#endif\nResponse_dealloc(Response* self)\n{\n if(self->buffer != self->inline_buffer)\n free(self->buffer);\n\n Py_XDECREF(self->cookies);\n Py_XDECREF(self->headers);\n Py_XDECREF(self->encoding);\n Py_XDECREF(self->body);\n Py_XDECREF(self->mime_type);\n Py_XDECREF(self->code);\n\n#ifdef RESPONSE_OPAQUE\n if(self->opaque)\n Py_TYPE(self)->tp_free((PyObject*)self);\n#endif\n}\n\n#ifdef RESPONSE_OPAQUE\nstatic const size_t code_offset = 9;\n\n#define empty(v) (!v || v == Py_None)\n\nstatic PyObject* application_json;\nstatic PyObject* application_octet;\n\nint\nResponse_init(Response* self, PyObject *args, PyObject *kw)\n{\n static char *kwlist[] = {\"text\", \"code\", \"body\", \"json\", \"mime_type\", \"encoding\", \"headers\", \"cookies\", NULL};\n\n PyObject* code = NULL;\n PyObject* body = NULL;\n PyObject* text = NULL;\n PyObject* json = NULL;\n PyObject* mime_type = NULL;\n PyObject* encoding = NULL;\n PyObject* headers = NULL;\n PyObject* cookies = NULL;\n\n // FIXME: check argument types\n if (!PyArg_ParseTupleAndKeywords(\n args, kw, \"|OOOOOOOO\", kwlist,\n &text, &code, &body, &json,\n &mime_type, &encoding, &headers, &cookies))\n goto error;\n\n if(!empty(code)) {\n self->code = code;\n Py_INCREF(self->code);\n }\n\n if(!empty(json)) {\n assert(empty(text) && empty(body));\n\n if(!(text = PyObject_CallFunctionObjArgs(json_dumps, json, NULL)))\n goto error;\n } else if(!empty(text)) {\n Py_INCREF(text);\n }\n\n if(!empty(text)) {\n assert(empty(body));\n\n if(!encoding) {\n if(!(self->body = PyUnicode_AsUTF8String(text)))\n goto error;\n } else {\n char* cencoding;\n if(!(cencoding = PyUnicode_AsUTF8(encoding)))\n goto error;\n\n if(!(self->body = PyUnicode_AsEncodedString(text, cencoding, NULL)))\n goto error;\n }\n\n Py_DECREF(text);\n }\n\n if(!empty(body)) {\n self->body = body;\n Py_INCREF(self->body);\n }\n\n if(!empty(mime_type)) {\n self->mime_type = mime_type;\n Py_INCREF(self->mime_type);\n } else {\n if(!empty(json)) {\n self->mime_type = application_json;\n Py_INCREF(self->mime_type);\n } else if(!empty(body)) {\n self->mime_type = application_octet;\n Py_INCREF(self->mime_type);\n }\n }\n\n if(!empty(encoding)) {\n self->encoding = encoding;\n Py_INCREF(self->encoding);\n }\n\n if(!empty(headers)) {\n self->headers = headers;\n Py_INCREF(self->headers);\n }\n\n if(!empty(cookies)) {\n self->cookies = cookies;\n Py_INCREF(self->cookies);\n }\n\n goto finally;\n\n error:\n return -1;\n finally:\n return 0;\n}\n\n\nstatic const char Content_Type[] = \"Content-Type: \";\nstatic const char charset[] = \"; charset=\";\nstatic const char utf8[] = \"utf-8\";\nstatic const char text_plain[] = \"text/plain\";\n\n\n#define CRLF \\\n *(self->buffer + buffer_offset) = '\\r'; \\\n buffer_offset++; \\\n *(self->buffer + buffer_offset) = '\\n'; \\\n buffer_offset++;\n\n\n#define bfrcpy(data, len) \\\n if(buffer_offset + len > self->buffer_len) \\\n { \\\n self->buffer_len = MAX(self->buffer_len * 2, self->buffer_len + len); \\\n \\\n if(self->buffer == self->inline_buffer) \\\n { \\\n self->buffer = malloc(self->buffer_len); \\\n if(!self->buffer) \\\n assert(0); \\\n memcpy(self->buffer, self->inline_buffer, buffer_offset); \\\n } else { \\\n self->buffer = realloc(self->buffer, self->buffer_len); \\\n if(!self->buffer) \\\n assert(0); \\\n } \\\n } \\\n \\\n memcpy(self->buffer + buffer_offset, data, len); \\\n buffer_offset += len;\n\n#ifdef RESPONSE_CACHE\n\ntypedef struct {\n PyObject* body;\n PyObject* response_bytes;\n} CacheEntry;\n\n#define CACHE_LEN 10\n#define CACHE_CUTOFF 4096\n\ntypedef struct {\n size_t end;\n CacheEntry entries[CACHE_LEN];\n} Cache;\n\nstatic Cache cache = {0};\n\n#define Bytes_AS_STRING(op) ((PyBytesObject *)op)->ob_sval\n\n#define Response_cacheable(r, simple) \\\n simple && r->body && Py_SIZE(r->body) < CACHE_CUTOFF \\\n && !r->code && !r->headers && !r->cookies && !r->mime_type \\\n && !r->encoding && r->minor_version == 1 && r->keep_alive == KEEP_ALIVE_TRUE\n\nstatic inline PyObject*\nResponse_from_cache(PyObject* body)\n{\n CacheEntry* cache_entry;\n for(cache_entry = cache.entries; cache_entry < cache.entries + cache.end;\n cache_entry++) {\n if(Py_SIZE(cache_entry->body) != Py_SIZE(body))\n continue;\n\n if(memcmp(Bytes_AS_STRING(cache_entry->body), Bytes_AS_STRING(body), Py_SIZE(body)) != 0)\n continue;\n\n Py_INCREF(cache_entry->response_bytes);\n return cache_entry->response_bytes;\n }\n\n return NULL;\n}\n\n\nstatic inline void\nResponse_cache(PyObject* body, PyObject* response_bytes)\n{\n if(cache.end == CACHE_LEN)\n return;\n\n CacheEntry* entry = cache.entries + cache.end;\n entry->body = body;\n entry->response_bytes = response_bytes;\n\n Py_INCREF(body);\n Py_INCREF(response_bytes);\n cache.end++;\n}\n\n#endif\n\n\nPyObject*\nResponse_render(Response* self, bool simple)\n{\n PyObject* response_bytes = NULL;\n PyObject* cookies_str = NULL;\n PyObject* cookies_bytes = NULL;\n\n#ifdef RESPONSE_CACHE\n bool cacheable = Response_cacheable(self, simple);\n if(cacheable && (response_bytes = Response_from_cache(self->body)))\n return response_bytes;\n#endif\n\n size_t buffer_offset;\n Py_ssize_t body_len = 0;\n const char* body = NULL;\n\n *(self->buffer + minor_offset) = '0' + (char)self->minor_version;\n\n if(self->code) {\n unsigned long code = PyLong_AsUnsignedLong(self->code);\n\n if(code < 100 || code > 599) {\n PyErr_SetString(PyExc_ValueError, \"Invalid status code\");\n goto error;\n }\n\n unsigned int status_category = code / 100 - 1;\n unsigned int status_rest = code % 100;\n\n const ReasonRange* reason_range = reason_ranges + status_category;\n if(status_rest > reason_range->maximum) {\n PyErr_SetString(PyExc_ValueError, \"Invalid status code\");\n goto error;\n }\n\n /* TODO these are always 3 digit, maybe modulus would be faster */\n snprintf(self->buffer + code_offset, 4, \"%ld\", code);\n *(self->buffer + code_offset + 3) = ' ';\n\n const char* reason = reason_range->reasons[status_rest];\n size_t reason_len = strlen(reason);\n\n\n memcpy(self->buffer + reason_offset, reason, reason_len);\n buffer_offset = reason_offset + reason_len;\n\n CRLF\n\n memcpy(self->buffer + buffer_offset, \"Content-Length: \", strlen(\"Content-Length: \"));\n buffer_offset += strlen(\"Content-Length: \");\n } else {\n memcpy(self->buffer + code_offset, \"200\", 3);\n buffer_offset = strlen(header);\n }\n\n if(self->body) {\n if(PyBytes_AsStringAndSize(self->body, (char**)&body, &body_len) == -1)\n goto error;\n\n int result = sprintf(\n self->buffer + buffer_offset, \"%ld\", (unsigned long)body_len);\n buffer_offset += result;\n } else {\n *(self->buffer + buffer_offset) = '0';\n buffer_offset++;\n }\n\n CRLF\n\n if(self->minor_version == 1 && self->keep_alive == KEEP_ALIVE_FALSE) {\n memcpy(\n self->buffer + buffer_offset, \"Connection: close\\r\\n\",\n strlen(\"Connection: close\\r\\n\"));\n buffer_offset += strlen(\"Connection: close\\r\\n\");\n } else if(self->minor_version == 0 && self->keep_alive == KEEP_ALIVE_TRUE) {\n memcpy(\n self->buffer + buffer_offset, \"Connection: keep-alive\\r\\n\",\n strlen(\"Connection: keep-aplive\\r\\n\"));\n buffer_offset += strlen(\"Connection: keep-alive\\r\\n\");\n }\n\n // dont output Content-Type if there is no body\n if(!self->body)\n goto headers;\n\n memcpy(self->buffer + buffer_offset, Content_Type, strlen(Content_Type));\n buffer_offset += strlen(Content_Type);\n\n Py_ssize_t mime_type_len = strlen(text_plain);\n const char* mime_type = text_plain;\n if(self->mime_type) {\n mime_type = PyUnicode_AsUTF8AndSize(self->mime_type, &mime_type_len);\n if(!mime_type)\n goto error;\n\n }\n memcpy(self->buffer + buffer_offset, mime_type, (size_t)mime_type_len);\n buffer_offset += mime_type_len;\n\n Py_ssize_t encoding_len = strlen(utf8);\n const char* encoding = utf8;\n if(self->encoding) {\n encoding = PyUnicode_AsUTF8AndSize(self->encoding, &encoding_len);\n if(!encoding)\n goto error;\n }\n\n #define text_or_json \\\n (mime_type_len >= 5 \\\n && (memcmp(mime_type, \"text/\", 5) == 0 \\\n || memcmp(mime_type + mime_type_len - 5, \"/json\", 5) == 0))\n\n if(self->encoding || text_or_json) {\n memcpy(self->buffer + buffer_offset, charset, strlen(charset));\n buffer_offset += strlen(charset);\n\n memcpy(self->buffer + buffer_offset, encoding, (size_t)encoding_len);\n buffer_offset += (size_t)encoding_len;\n }\n\n CRLF\n\n headers:\n\n if(!self->headers)\n goto empty_headers;\n\n Py_ssize_t headers_len;\n if((headers_len = PyDict_Size(self->headers)) < 0)\n goto error;\n\n if(!headers_len)\n goto empty_headers;\n\n PyObject *name, *value;\n Py_ssize_t pos = 0;\n\n while (PyDict_Next(self->headers, &pos, &name, &value)) {\n const char* cname;\n Py_ssize_t name_len;\n const char* cvalue;\n Py_ssize_t value_len;\n\n if(!(cname = PyUnicode_AsUTF8AndSize(name, &name_len)))\n goto error;\n\n memcpy(self->buffer + buffer_offset, cname, (size_t)name_len);\n buffer_offset += (size_t)name_len;\n\n *(self->buffer + buffer_offset) = ':';\n buffer_offset++;\n *(self->buffer + buffer_offset) = ' ';\n buffer_offset++;\n\n if(!(cvalue = PyUnicode_AsUTF8AndSize(value, &value_len)))\n goto error;\n\n memcpy(self->buffer + buffer_offset, cvalue, (size_t)value_len);\n buffer_offset += (size_t)value_len;\n\n CRLF\n }\n\n empty_headers:\n\n if(!self->cookies)\n goto empty_cookies;\n\n Py_ssize_t cookies_len;\n if((cookies_len = PyObject_Size(self->cookies)) < 0)\n goto error;\n\n if(!cookies_len)\n goto empty_cookies;\n\n if(!(cookies_str = PyObject_Str(self->cookies)))\n goto error;\n\n if(!(cookies_bytes = PyUnicode_AsASCIIString(cookies_str)))\n goto error;\n\n char* ccookies;\n Py_ssize_t ccookies_len;\n if(PyBytes_AsStringAndSize(cookies_bytes, &ccookies, &ccookies_len) == -1)\n goto error;\n\n memcpy(self->buffer + buffer_offset, ccookies, (size_t)ccookies_len);\n buffer_offset += (size_t)ccookies_len;\n\n CRLF\n\n empty_cookies:\n CRLF\n\n if(body) {\n bfrcpy(body, (size_t)body_len)\n }\n\n#undef CRLF\n\n if(!(response_bytes = PyBytes_FromStringAndSize(self->buffer, buffer_offset)))\n goto error;\n\n#ifdef RESPONSE_CACHE\n if(cacheable)\n Response_cache(self->body, response_bytes);\n#endif\n\n goto finally;\n\n error:\n Py_XDECREF(response_bytes);\n response_bytes = NULL;\n\n finally:\n Py_XDECREF(cookies_str);\n Py_XDECREF(cookies_bytes);\n\n return response_bytes;\n}\n\n\nstatic PyMethodDef Response_methods[] = {\n //{\"render\", (PyCFunction)Response_render, METH_NOARGS, \"render\"},\n {NULL}\n};\n\n\nstatic PyTypeObject ResponseType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"cresponse.Response\", /* tp_name */\n sizeof(Response), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Response_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Response\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Response_methods, /* tp_methods */\n 0, /* tp_members */\n 0, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Response_init, /* tp_init */\n 0, /* tp_alloc */\n Response_new, /* tp_new */\n};\n\n\nstatic PyModuleDef cresponse = {\n PyModuleDef_HEAD_INIT,\n \"cresponse\",\n \"cresponse\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n\n\nPyMODINIT_FUNC\nPyInit_cresponse(void)\n{\n PyObject* m = NULL;\n PyObject* api_capsule = NULL;\n PyObject* json = NULL;\n\n if (PyType_Ready(&ResponseType) < 0)\n goto error;\n\n m = PyModule_Create(&cresponse);\n if(!m)\n goto error;\n\n Py_INCREF(&ResponseType);\n PyModule_AddObject(m, \"Response\", (PyObject*)&ResponseType);\n\n if(!(json = PyImport_ImportModule(\"json\")))\n goto error;\n\n if(!(json_dumps = PyObject_GetAttrString(json, \"dumps\")))\n goto error;\n\n if(!(application_json = PyUnicode_FromString(\"application/json\")))\n goto error;\n\n if(!(application_octet = PyUnicode_FromString(\"application/octet-stream\")))\n goto error;\n\n static Response_CAPI capi = {\n &ResponseType,\n Response_render,\n Response_init\n };\n api_capsule = export_capi(m, \"japronto.response.cresponse\", &capi);\n if(!api_capsule)\n goto error;\n\n goto finally;\n\n error:\n m = NULL;\n finally:\n Py_XDECREF(json);\n Py_XDECREF(api_capsule);\n return m;\n}\n#endif\n" }, { "path": "src/japronto/response/cresponse.h", "content": "#pragma once\n\n#include \n#include \n#include \"common.h\"\n\n\n#define RESPONSE_INITIAL_BUFFER_LEN 1024\n\ntypedef struct {\n PyObject_HEAD\n\n bool opaque;\n int minor_version;\n KEEP_ALIVE keep_alive;\n\n PyObject* code;\n PyObject* mime_type;\n PyObject* body;\n PyObject* encoding;\n PyObject* headers;\n PyObject* cookies;\n\n char* buffer;\n size_t buffer_len;\n char inline_buffer[RESPONSE_INITIAL_BUFFER_LEN];\n} Response;\n\n\ntypedef struct {\n PyTypeObject* ResponseType;\n PyObject* (*Response_render)(Response*, bool);\n int (*Response_init)(Response* self, PyObject *args, PyObject *kw);\n} Response_CAPI;\n\n#ifndef RESPONSE_OPAQUE\nPyObject*\nResponse_new(PyTypeObject* type, Response* self);\n\nvoid\nResponse_dealloc(Response* self);\n#endif\n" }, { "path": "src/japronto/response/cresponse_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n define_macros = [('RESPONSE_OPAQUE', 1)]\n if system.args.enable_response_cache:\n define_macros.append(('RESPONSE_CACHE', 1))\n\n return Extension(\n 'japronto.response.cresponse',\n sources=['cresponse.c', '../capsule.c'],\n include_dirs=['..'],\n define_macros=define_macros)\n" }, { "path": "src/japronto/response/py.py", "content": "_responses = None\n\n\ndef factory(status_code=200, text='', mime_type='text/plain',\n encoding='utf-8'):\n global _responses\n if _responses is None:\n _responses = []\n for _ in range(100):\n _responses.append(Response())\n\n response = _responses.pop()\n\n response.status_code = status_code\n response.mime_type = mime_type\n response.text = text\n response.encoding = encoding\n\n return response\n\n\ndef dispose(response):\n _responses.append(response)\n\n\nclass Response:\n __slots__ = ('status_code', 'mime_type', 'text', 'encoding')\n\n def __init__(self, status_code=200, text='', mime_type='text/plain',\n encoding='utf-8'):\n self.status_code = status_code\n self.mime_type = mime_type\n self.text = text\n self.encoding = encoding\n\n def render(self):\n data = ['HTTP/1.1 ', str(self.status_code), ' OK\\r\\n']\n data.append('Connection: keep-alive\\r\\n')\n body = self.text.encode(self.encoding)\n data.extend([\n 'Content-Type: ', self.mime_type,\n '; encoding=', self.encoding, '\\r\\n'])\n data.extend(['Content-Length: ', str(len(body)), '\\r\\n\\r\\n'])\n\n return ''.join(data).encode(self.encoding) + body\n" }, { "path": "src/japronto/response/reasons.h", "content": "#pragma once\n\nstatic const char* reasons_1xx[] = {\n \"Continue\", //100\n \"Switching Protocols\", //101\n};\n\nstatic const char* reasons_2xx[] = {\n \"OK\", //200\n \"Created\", //201\n \"Accepted\", //202\n \"Non-Authoritative Information\", //203\n \"No Content\", // 204\n \"Reset Content\", //205\n \"Partial Content\", //206\n};\n\nstatic const char* reasons_3xx[] = {\n \"Multiple Choices\", //300\n \"Moved Permanently\", //301\n \"Found\", //302\n \"See Other\", //303,\n \"Not Modified\", //304\n \"Use Proxy\", //305\n \"Proxy Switch\", //306\n \"Temporary Redirect\", //307\n};\n\nstatic const char* reasons_4xx[] = {\n \"Bad Request\", //400\n \"Unauthorized\", //401\n \"Payment Required\", //402\n \"Forbidden\", //403\n \"Not Found\", //404\n \"Method Not Allowed\", //405\n \"Not acceptable\", //406\n \"Proxy Authentication Required\", //407\n \"Request Timeout\", //408\n \"Conflict\", //409\n \"Gone\", //410\n \"Length Required\", //411\n \"Precondition Failed\", //412\n \"Request Entity Too Large\", //413\n \"Request-URI Too Long\", //414\n \"Unsupported Media Type\", //415\n \"Requested Range Not Satisfiable\", //416\n \"Expectation Failed\", //417\n};\n\nstatic const char* reasons_5xx[] = {\n \"Internal Server Error\", //500\n \"Not Implemented\", //501\n \"Bad Gateway\", //502\n \"Service Unavailable\", //503\n \"Gateway Timeout\", //504\n \"HTTP Version Not Supported\", //505\n};\n\ntypedef struct {\n const char** reasons;\n size_t maximum;\n} ReasonRange;\n\nstatic const ReasonRange reason_ranges[] = {\n {reasons_1xx, 1},\n {reasons_2xx, 6},\n {reasons_3xx, 7},\n {reasons_4xx, 17},\n {reasons_5xx, 5}\n};\n" }, { "path": "src/japronto/router/__init__.py", "content": "from .route import Route, RouteNotFoundException\nfrom .cmatcher import Matcher\n\n\nclass Router:\n def __init__(self, matcher_factory=Matcher):\n self._routes = []\n self.matcher_factory = matcher_factory\n\n def add_route(self, pattern, handler, method=None, methods=None):\n assert not(method and methods), \"Cannot use method and methods\"\n\n if method:\n methods = [method]\n\n if not methods:\n methods = []\n\n methods = {m.upper() for m in methods}\n route = Route(pattern, handler, methods)\n\n self._routes.append(route)\n\n return route\n\n def get_matcher(self):\n return self.matcher_factory(self._routes)\n" }, { "path": "src/japronto/router/analyzer.py", "content": "import dis\nimport functools\nimport types\n\n\nFLAG_COROUTINE = 128\n\n\ndef is_simple(fun):\n \"\"\"A heuristic to find out if a function is simple enough.\"\"\"\n seen_load_fast_0 = False\n seen_load_response = False\n seen_call_fun = False\n\n for instruction in dis.get_instructions(fun):\n if instruction.opname == 'LOAD_FAST' and instruction.arg == 0:\n seen_load_fast_0 = True\n continue\n\n if instruction.opname == 'LOAD_ATTR' \\\n and instruction.argval == 'Response':\n seen_load_response = True\n continue\n\n if instruction.opname.startswith('CALL_FUNCTION'):\n if seen_call_fun:\n return False\n\n seen_call_fun = True\n continue\n\n return seen_call_fun and seen_load_fast_0 and seen_load_response\n\n\ndef is_pointless_coroutine(fun):\n for instruction in dis.get_instructions(fun):\n if instruction.opname in ('GET_AWAITABLE', 'YIELD_FROM'):\n return False\n\n return True\n\n\ndef coroutine_to_func(f):\n # Based on http://stackoverflow.com/questions/13503079/\n # how-to-create-a-copy-of-a-python-function\n oc = f.__code__\n code = types.CodeType(\n oc.co_argcount, oc.co_kwonlyargcount, oc.co_nlocals, oc.co_stacksize,\n oc.co_flags & ~FLAG_COROUTINE,\n oc.co_code, oc.co_consts, oc.co_names, oc.co_varnames, oc.co_filename,\n oc.co_name, oc.co_firstlineno, oc.co_lnotab, oc.co_freevars,\n oc.co_cellvars)\n g = types.FunctionType(\n code, f.__globals__, name=f.__name__, argdefs=f.__defaults__,\n closure=f.__closure__)\n g = functools.update_wrapper(g, f)\n g.__kwdefaults__ = f.__kwdefaults__\n\n return g\n" }, { "path": "src/japronto/router/cmatcher.c", "content": "#include \n#include \n\n#include \"cmatcher.h\"\n#include \"crequest.h\"\n#include \"capsule.h\"\n\n\nstruct _Matcher {\n PyObject_HEAD\n\n size_t buffer_len;\n char* buffer;\n};\n\n\ntypedef enum {\n SEGMENT_EXACT,\n SEGMENT_PLACEHOLDER\n} SegmentType;\n\n\ntypedef struct {\n size_t data_length;\n char data[];\n} ExactSegment;\n\n\ntypedef struct {\n size_t name_length;\n char name[];\n} PlaceholderSegment;\n\n\ntypedef struct {\n SegmentType type;\n\n union {\n ExactSegment exact;\n PlaceholderSegment placeholder;\n };\n} Segment;\n\n\nstatic MatchDictEntry _match_dict_entries[10];\n\nstatic Request_CAPI* request_capi;\nstatic PyObject* compile_all;\n\nstatic PyObject *\nMatcher_new(PyTypeObject *type, PyObject *args, PyObject *kwds)\n{\n Matcher* self = NULL;\n\n self = (Matcher*)type->tp_alloc(type, 0);\n if(!self)\n goto finally;\n\n self->buffer = NULL;\n self->buffer_len = 0;\n\n finally:\n return (PyObject*)self;\n}\n\n\n#define ROUNDTO8(v) (((v) + 7) & ~7)\n\n\n#define ENTRY_LOOP \\\nchar* entry_end = self->buffer + self->buffer_len; \\\nfor(MatcherEntry* entry = (MatcherEntry*)self->buffer; \\\n (char*)entry < entry_end; \\\n entry = (MatcherEntry*)((char*)entry + sizeof(MatcherEntry) + \\\n ROUNDTO8(entry->pattern_len) + ROUNDTO8(entry->methods_len)))\n\n#define SEGMENT_LOOP \\\nchar* segments_end = entry->buffer + entry->pattern_len; \\\nfor(Segment* segment = (Segment*)entry->buffer; \\\n (char*)segment < segments_end; \\\n segment = (Segment*)((char*)segment + sizeof(Segment) + \\\n ROUNDTO8(segment->type == SEGMENT_EXACT ? \\\n segment->exact.data_length : segment->placeholder.name_length)))\n\n\nstatic void\nMatcher_dealloc(Matcher* self)\n{\n if(self->buffer) {\n ENTRY_LOOP {\n Py_DECREF(entry->handler);\n Py_DECREF(entry->route);\n }\n free(self->buffer);\n }\n\n Py_TYPE(self)->tp_free((PyObject*)self);\n}\n\n\nstatic int\nMatcher_init(Matcher* self, PyObject *args, PyObject *kw)\n{\n int result = 0;\n PyObject* compiled = NULL;\n\n PyObject* routes;\n if(!PyArg_ParseTuple(args, \"O\", &routes))\n goto error;\n\n if(!(compiled = PyObject_CallFunctionObjArgs(compile_all, routes, NULL)))\n goto error;\n\n char* compiled_buffer;\n if(PyBytes_AsStringAndSize(compiled, &compiled_buffer, (Py_ssize_t*)&self->buffer_len) == -1)\n goto error;\n\n if(!(self->buffer = malloc(self->buffer_len)))\n goto error;\n\n memcpy(self->buffer, compiled_buffer, self->buffer_len);\n\n ENTRY_LOOP {\n Py_INCREF(entry->handler);\n Py_INCREF(entry->route);\n }\n\n goto finally;\n\n error:\n result = -1;\n finally:\n Py_XDECREF(compiled);\n return result;\n}\n\n// borrows route and handler in matcher entry\nMatcherEntry*\nMatcher_match_request(Matcher* self, PyObject* request,\n MatchDictEntry** match_dict_entries,\n size_t* match_dict_length)\n{\n MatcherEntry* result = NULL;\n PyObject* path = NULL;\n PyObject* method = NULL;\n\n size_t method_len;\n char* method_str;\n size_t path_len;\n char* path_str;\n if(Py_TYPE(request) != request_capi->RequestType) {\n path = PyObject_GetAttrString(request, \"path\");\n if(!path)\n goto error;\n\n path_str = PyUnicode_AsUTF8AndSize(path, (Py_ssize_t*)&path_len);\n if(!path_str)\n goto error;\n\n method = PyObject_GetAttrString(request, \"method\");\n if(!method)\n goto error;\n\n method_str = PyUnicode_AsUTF8AndSize(method, (Py_ssize_t*)&method_len);\n if(!method_str)\n goto error;\n } else {\n method_len = REQUEST(request)->method_len;\n method_str = REQUEST_METHOD(request);\n path_str = request_capi->Request_get_decoded_path(\n REQUEST(request), &path_len);\n }\n\n ENTRY_LOOP {\n char* rest = path_str;\n size_t rest_len = path_len;\n\n MatchDictEntry* current_mde = _match_dict_entries;\n size_t value_length = 1;\n\n SEGMENT_LOOP {\n if(segment->type == SEGMENT_EXACT) {\n if(rest_len < segment->exact.data_length)\n break;\n\n if(memcmp(rest, segment->exact.data, segment->exact.data_length) != 0)\n break;\n\n rest += segment->exact.data_length;\n rest_len -= segment->exact.data_length;\n } else if(segment->type == SEGMENT_PLACEHOLDER) {\n assert(((size_t)(current_mde - _match_dict_entries)) < sizeof(_match_dict_entries) / sizeof(MatchDictEntry));\n\n char* slash = memchr(rest, '/', rest_len);\n current_mde->value = rest;\n if(slash) {\n value_length = current_mde->value_length = slash - rest;\n rest_len -= current_mde->value_length;\n rest = slash;\n } else {\n value_length = current_mde->value_length = rest_len;\n rest_len = 0;\n }\n\n if(!value_length)\n break;\n\n current_mde->key = segment->placeholder.name;\n current_mde->key_length = segment->placeholder.name_length;\n\n current_mde++;\n } else {\n assert(0);\n }\n }\n\n if(rest_len)\n continue;\n\n if(!value_length)\n continue;\n\n if((size_t)(current_mde - _match_dict_entries) != entry->placeholder_cnt)\n continue;\n\n if(!entry->methods_len)\n goto loop_finally;\n\n char* method_found = memmem(\n entry->buffer + entry->pattern_len, entry->methods_len,\n method_str, (size_t)method_len);\n if(!method_found)\n continue;\n\n if(*(method_found + (size_t)method_len) != ' ')\n continue;\n\n loop_finally:\n result = entry;\n\n if(match_dict_entries)\n *match_dict_entries = _match_dict_entries;\n if(match_dict_length)\n *match_dict_length = current_mde - _match_dict_entries;\n goto finally;\n }\n\n if(match_dict_length)\n *match_dict_length = 0;\n\n goto finally;\n\n error:\n result = NULL;\n\n finally:\n\n if(Py_TYPE(request) != request_capi->RequestType) {\n Py_XDECREF(method);\n Py_XDECREF(path);\n }\n\n return result;\n}\n\n\nstatic PyObject*\n_Matcher_match_request(Matcher* self, PyObject* request)\n{\n MatcherEntry* matcher_entry;\n MatchDictEntry* entries;\n PyObject* route = NULL;\n size_t length;\n PyObject* match_dict = NULL;\n PyObject* route_dict = NULL;\n\n if(!(matcher_entry = Matcher_match_request(\n self, request, &entries, &length)))\n Py_RETURN_NONE;\n\n route = matcher_entry->route;\n\n if(!(match_dict = MatchDict_entries_to_dict(entries, length)))\n goto error;\n\n if(!(route_dict = PyTuple_New(2)))\n goto error;\n\n PyTuple_SET_ITEM(route_dict, 0, route);\n PyTuple_SET_ITEM(route_dict, 1, match_dict);\n\n goto finally;\n\n error:\n Py_XDECREF(match_dict);\n route = NULL;\n\n finally:\n if(route)\n Py_INCREF(route);\n return route_dict;\n}\n\n\nstatic PyMethodDef Matcher_methods[] = {\n {\"match_request\", (PyCFunction)_Matcher_match_request, METH_O, \"\"},\n {NULL}\n};\n\n\nstatic PyTypeObject MatcherType = {\n PyVarObject_HEAD_INIT(NULL, 0)\n \"cmatcher.Matcher\", /* tp_name */\n sizeof(Matcher), /* tp_basicsize */\n 0, /* tp_itemsize */\n (destructor)Matcher_dealloc, /* tp_dealloc */\n 0, /* tp_print */\n 0, /* tp_getattr */\n 0, /* tp_setattr */\n 0, /* tp_reserved */\n 0, /* tp_repr */\n 0, /* tp_as_number */\n 0, /* tp_as_sequence */\n 0, /* tp_as_mapping */\n 0, /* tp_hash */\n 0, /* tp_call */\n 0, /* tp_str */\n 0, /* tp_getattro */\n 0, /* tp_setattro */\n 0, /* tp_as_buffer */\n Py_TPFLAGS_DEFAULT, /* tp_flags */\n \"Matcher\", /* tp_doc */\n 0, /* tp_traverse */\n 0, /* tp_clear */\n 0, /* tp_richcompare */\n 0, /* tp_weaklistoffset */\n 0, /* tp_iter */\n 0, /* tp_iternext */\n Matcher_methods, /* tp_methods */\n 0, /* tp_members */\n 0, /* tp_getset */\n 0, /* tp_base */\n 0, /* tp_dict */\n 0, /* tp_descr_get */\n 0, /* tp_descr_set */\n 0, /* tp_dictoffset */\n (initproc)Matcher_init, /* tp_init */\n 0, /* tp_alloc */\n Matcher_new, /* tp_new */\n};\n\n\nstatic PyModuleDef cmatcher = {\n PyModuleDef_HEAD_INIT,\n \"cmatcher\",\n \"cmatcher\",\n -1,\n NULL, NULL, NULL, NULL, NULL\n};\n\n\nPyMODINIT_FUNC\nPyInit_cmatcher(void)\n{\n PyObject* m = NULL;\n PyObject* api_capsule = NULL;\n PyObject* router_route = NULL;\n\n if (PyType_Ready(&MatcherType) < 0)\n goto error;\n\n m = PyModule_Create(&cmatcher);\n if(!m)\n goto error;\n\n request_capi = import_capi(\"japronto.request.crequest\");\n if(!request_capi)\n goto error;\n\n if(!(router_route = PyImport_ImportModule(\"japronto.router.route\")))\n goto error;\n\n if(!(compile_all = PyObject_GetAttrString(router_route, \"compile_all\")))\n goto error;\n\n Py_INCREF(&MatcherType);\n PyModule_AddObject(m, \"Matcher\", (PyObject*)&MatcherType);\n\n static Matcher_CAPI capi = { Matcher_match_request };\n api_capsule = export_capi(m, \"japronto.router.cmatcher\", &capi);\n if(!api_capsule)\n goto error;\n\n goto finally;\n\n error:\n m = NULL;\n\n finally:\n Py_XDECREF(router_route);\n Py_XDECREF(api_capsule);\n return m;\n}\n" }, { "path": "src/japronto/router/cmatcher.h", "content": "#pragma once\n\n#include \n#include \n\n#include \"match_dict.h\"\n\ntypedef struct {\n PyObject* route;\n PyObject* handler;\n bool coro_func;\n bool simple;\n size_t pattern_len;\n size_t methods_len;\n size_t placeholder_cnt;\n char buffer[];\n} MatcherEntry;\n\n\ntypedef struct _Matcher Matcher;\n\n\ntypedef struct {\n MatcherEntry* (*Matcher_match_request)\n (Matcher* matcher, PyObject* request,\n MatchDictEntry** match_dict_entries, size_t* match_dict_length);\n} Matcher_CAPI;\n" }, { "path": "src/japronto/router/cmatcher_ext.py", "content": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.router.cmatcher',\n sources=['cmatcher.c', 'match_dict.c', '../capsule.c'],\n include_dirs=['.', '../request', '..', '../response'])\n" }, { "path": "src/japronto/router/match_dict.c", "content": "#include \n\n#include \"match_dict.h\"\n\n\nPyObject*\nMatchDict_entries_to_dict(MatchDictEntry* entries, size_t length)\n{\n PyObject* match_dict = NULL;\n if(!(match_dict = PyDict_New()))\n goto error;\n\n for(MatchDictEntry* entry = entries; entry < entries + length; entry++) {\n PyObject* key = NULL;\n PyObject* value = NULL;\n\n if(!(key = PyUnicode_FromStringAndSize(entry->key, entry->key_length)))\n goto loop_error;\n\n if(!(value = PyUnicode_FromStringAndSize(entry->value, entry->value_length)))\n goto loop_error;\n\n if(PyDict_SetItem(match_dict, key, value) == -1)\n goto loop_error;\n\n goto loop_finally;\n\n loop_error:\n Py_XDECREF(match_dict);\n match_dict = NULL;\n\n loop_finally:\n Py_XDECREF(key);\n Py_XDECREF(value);\n if(!match_dict)\n goto error;\n }\n\n goto finally;\n\n error:\n Py_XDECREF(match_dict);\n match_dict = NULL;\n\n finally:\n return match_dict;\n}\n" }, { "path": "src/japronto/router/match_dict.h", "content": "#pragma once\n\n#include \n\ntypedef struct {\n char* key;\n size_t key_length;\n char* value;\n size_t value_length;\n} MatchDictEntry;\n\n\nPyObject*\nMatchDict_entries_to_dict(MatchDictEntry* entries, size_t length);\n" }, { "path": "src/japronto/router/matcher.py", "content": "class Matcher:\n def __init__(self, routes):\n self._routes = routes\n\n def match_request(self, request):\n for route in self._routes:\n match_dict = {}\n rest = request.path\n\n value = True\n for typ, data in route.segments:\n if typ == 'exact':\n if not rest.startswith(data):\n break\n\n rest = rest[len(data):]\n elif typ == 'placeholder':\n value, slash, rest = rest.partition('/')\n if not value:\n break\n match_dict[data] = value\n rest = slash + rest\n else:\n assert 0, 'Unknown type'\n\n if rest:\n continue\n\n if not value:\n continue\n\n if len(match_dict) != route.placeholder_cnt:\n continue\n\n if route.methods and request.method not in route.methods:\n continue\n\n return route, match_dict\n" }, { "path": "src/japronto/router/route.py", "content": "import asyncio\nfrom enum import IntEnum\nfrom struct import Struct\n\nfrom . import analyzer\n\n\nclass RouteNotFoundException(Exception):\n pass\n\n\nclass Route:\n def __init__(self, pattern, handler, methods):\n self.pattern = pattern\n self.handler = handler\n self.methods = methods\n self.segments = parse(pattern)\n self.placeholder_cnt = \\\n sum(1 for s in self.segments if s[0] == 'placeholder')\n\n def __repr__(self):\n return ''.format(\n self.pattern, self.methods, hex(id(self)))\n\n def describe(self):\n return self.pattern + (' ' if self.methods else '') + \\\n ' '.join(self.methods)\n\n def __eq__(self, other):\n return self.pattern == other.pattern and self.methods == other.methods\n\n\ndef parse(pattern):\n names = set()\n result = []\n\n rest = pattern\n while rest:\n exact = ''\n while rest:\n chunk, _, rest = rest.partition('{')\n exact += chunk\n if rest and rest[0] == '{':\n exact += '{{'\n rest = rest[1:]\n else:\n break\n\n if exact:\n exact = exact.replace('{{', '{').replace('}}', '}')\n result.append(('exact', exact))\n if not rest:\n break\n\n name, _, rest = rest.partition('}')\n if not _:\n raise ValueError('Unbalanced \"{\" in pattern')\n if rest and rest[0] != '/':\n raise ValueError(\n '\"}\" must be followed by \"/\" or appear at the end')\n if name in names:\n raise ValueError('Duplicate name \"{}\" in pattern'.format(name))\n names.add(name)\n result.append(('placeholder', name))\n\n return result\n\n\nclass SegmentType(IntEnum):\n EXACT = 0\n PLACEHOLDER = 1\n\n\n\"\"\"\ntypedef struct {\n PyObject* route;\n PyObject* handler;\n bool coro_func;\n bool simple;\n size_t pattern_len;\n size_t methods_len;\n size_t placeholder_cnt;\n char buffer[];\n} MatcherEntry;\n\"\"\"\nMatcherEntry = Struct('PP??NNN')\n\n\"\"\"\ntypedef enum {\n SEGMENT_EXACT,\n SEGMENT_PLACEHOLDER\n} SegmentType;\n\n\ntypedef struct {\n size_t data_length;\n char data[];\n} ExactSegment;\n\n\ntypedef struct {\n size_t name_length;\n char name[];\n} PlaceholderSegment;\n\n\ntypedef struct {\n SegmentType type;\n\n union {\n ExactSegment exact;\n PlaceholderSegment placeholder;\n };\n} Segment;\n\"\"\"\nExactSegment = Struct('iN')\nPlaceholderSegment = Struct('iN')\nSegment = Struct('iN')\n\n\ndef roundto8(v):\n return (v + 7) & ~7\n\n\ndef padto8(data):\n \"\"\"Pads data to the multiplies of 8 bytes.\n\n This makes x86_64 faster and prevents\n undefined behavior on other platforms\"\"\"\n length = len(data)\n return data + b'\\xdb' * (roundto8(length) - length)\n\n\nretain_handlers = set()\n\n\ndef compile(route):\n pattern_buf = b''\n for segment in route.segments:\n typ = getattr(SegmentType, segment[0].upper())\n pattern_buf += Segment.pack(typ, len(segment[1].encode('utf-8'))) \\\n + padto8(segment[1].encode('utf-8'))\n methods_buf = ' '.join(route.methods).encode('ascii')\n methods_len = len(methods_buf)\n if methods_buf:\n methods_buf += b' '\n methods_len += 1\n methods_buf = padto8(methods_buf)\n\n handler = route.handler\n if asyncio.iscoroutinefunction(handler) \\\n and analyzer.is_pointless_coroutine(handler):\n handler = analyzer.coroutine_to_func(handler)\n # since we save id to handler in matcher entry and this is the only\n # reference before INCREF-ed in matcher we store it in set to prevent\n # destruction\n retain_handlers.add(handler)\n\n return MatcherEntry.pack(\n id(route), id(handler),\n asyncio.iscoroutinefunction(handler),\n analyzer.is_simple(handler),\n len(pattern_buf), methods_len, route.placeholder_cnt) \\\n + pattern_buf + methods_buf\n\n\ndef compile_all(routes):\n return b''.join(compile(r) for r in routes)\n" }, { "path": "src/japronto/router/test_analyzer.py", "content": "from collections import OrderedDict\n\nimport pytest\n\nfrom . import analyzer\n\n\nsimple_fixtures = OrderedDict([\n ('empty', ('def a(): pass', False)),\n ('arg', ('def a(b): return b', False)),\n ('simple', ('def a(c): return c.Response()', True)),\n ('body', ('def a(c): return c.Response(body=\"abc\")', True)),\n ('wrongattr', ('def a(d): return d.R()', False)),\n ('extracall', (\n '''\ndef a(b):\n d()\n return b.Response()\n\ndef d():\n pass\n ''', False)),\n ('expressions', (\n '''\ndef a(b):\n c = \"Hey!\"\n d = \"Dude\"\n return b.Response(json={c: d})\n ''', True))\n])\n\n\n@pytest.mark.parametrize(\n 'code,simple', simple_fixtures.values(), ids=list(simple_fixtures.keys()))\ndef test_is_simple(code, simple):\n module = compile(code, '?', 'exec')\n fun_code = module.co_consts[0]\n\n assert analyzer.is_simple(fun_code) == simple\n\n\npointless_fixtures = OrderedDict([\n ('empty', ('async def a(): pass', True)),\n ('simple', ('async def a(): return 1', True)),\n ('yieldfrom', ('def a(b): yield from b', False)),\n ('await', ('async def a(b): await b', False))\n])\n\n\n@pytest.mark.parametrize(\n 'code,pointless', pointless_fixtures.values(),\n ids=list(pointless_fixtures.keys()))\ndef test_is_pointless(code, pointless):\n module = compile(code, '?', 'exec')\n fun_code = module.co_consts[0]\n\n assert analyzer.is_pointless_coroutine(fun_code) == pointless\n" }, { "path": "src/japronto/router/test_matcher.py", "content": "from functools import partial\n\nimport pytest\n\nfrom . import Route\nfrom .matcher import Matcher\nfrom .cmatcher import Matcher as CMatcher\n\n\nclass FakeRequest:\n def __init__(self, method, path):\n self.method = method\n self.path = path\n\n @classmethod\n def from_str(cls, value):\n return cls(*value.split())\n\n\nclass TracingRoute(Route):\n cnt = 0\n\n def __new__(cls, *args, **kw):\n print('new', args)\n cls.cnt += 1\n return Route.__new__(cls)\n\n def __init__(self, pattern, methods):\n super().__init__(pattern, lambda x: None, methods=methods)\n\n def __del__(self):\n type(self).cnt -= 1\n print('del')\n\n\ndef route_from_str(value):\n pattern, *methods = value.split()\n if methods:\n methods = methods[0].split(',')\n\n return TracingRoute(pattern, methods=methods)\n\n\ndef parametrize_make_matcher():\n def make(cls):\n routes = [route_from_str(r) for r in [\n '/',\n '/test GET',\n '/hi/{there} POST,DELETE',\n '/{oh}/{dear} PATCH',\n '/lets PATCH'\n ]]\n\n return cls(routes)\n\n make_matcher = partial(make, Matcher)\n make_cmatcher = partial(make, CMatcher)\n\n return pytest.mark.parametrize(\n 'make_matcher', [make_matcher, make_cmatcher], ids=['py', 'c'])\n\n\ndef parametrize_request_route_and_dict(cases):\n return pytest.mark.parametrize(\n 'req,route,match_dict',\n ((FakeRequest.from_str(req), route_from_str(route), match_dict)\n for req, route, match_dict in cases),\n ids=[req + '-' + route for req, route, _ in cases])\n\n\n@parametrize_request_route_and_dict([\n ('GET /', '/', {}),\n ('POST /', '/', {}),\n ('GET /test', '/test GET', {}),\n ('DELETE /hi/jane', '/hi/{there} POST,DELETE', {'there': 'jane'}),\n ('PATCH /lets/go', '/{oh}/{dear} PATCH', {'oh': 'lets', 'dear': 'go'}),\n ('PATCH /lets', '/lets PATCH', {})\n])\n@parametrize_make_matcher()\ndef test_matcher(make_matcher, req, route, match_dict):\n cnt = TracingRoute.cnt\n\n matcher = make_matcher()\n assert matcher.match_request(req) == (route, match_dict)\n del matcher\n\n assert cnt == TracingRoute.cnt\n\n\ndef parametrize_request(requests):\n return pytest.mark.parametrize(\n 'req', (FakeRequest.from_str(r) for r in requests), ids=requests)\n\n\n@parametrize_request([\n 'POST /test',\n 'GET /test/',\n 'GET /hi/jane',\n 'POST /hi/jane/',\n 'POST /hi/',\n 'GET /abc',\n 'PATCH //dance'\n])\n@parametrize_make_matcher()\ndef test_matcher_not_found(make_matcher, req):\n cnt = TracingRoute.cnt\n\n matcher = make_matcher()\n assert matcher.match_request(req) is None\n del matcher\n\n assert cnt == TracingRoute.cnt\n" }, { "path": "src/japronto/router/test_route.py", "content": "import asyncio\nfrom collections import namedtuple\n\nimport pytest\n\nfrom .route import parse, MatcherEntry, Segment, SegmentType, Route, \\\n compile, roundto8\n\n\n@pytest.mark.parametrize('pattern,result', [\n ('/', [('exact', '/')]),\n ('/{{a}}', [('exact', '/{a}')]),\n ('{a}', [('placeholder', 'a')]),\n ('a/{a}', [('exact', 'a/'), ('placeholder', 'a')]),\n ('{a}/a', [('placeholder', 'a'), ('exact', '/a')]),\n ('{a}/{{a}}', [('placeholder', 'a'), ('exact', '/{a}')]),\n ('{a}/{b}', [('placeholder', 'a'), ('exact', '/'), ('placeholder', 'b')])\n])\ndef test_parse(pattern, result):\n assert parse(pattern) == result\n\n\n@pytest.mark.parametrize('pattern,error', [\n ('{a', 'Unbalanced'),\n ('{a}/{b', 'Unbalanced'),\n ('{a}a', 'followed by'),\n ('{a}/{a}', 'Duplicate')\n])\ndef test_parse_error(pattern, error):\n with pytest.raises(ValueError) as info:\n parse(pattern)\n assert error in info.value.args[0]\n\n\nDecodedRoute = namedtuple(\n 'DecodedRoute',\n 'route_id,handler_id,coro_func,simple,placeholder_cnt,segments,methods')\n\n\ndef decompile(buffer):\n route_id, handler_id, coro_func, simple, \\\n pattern_len, methods_len, placeholder_cnt \\\n = MatcherEntry.unpack_from(buffer, 0)\n offset = MatcherEntry.size\n pattern_offset_end = offset + roundto8(pattern_len)\n\n segments = []\n while offset < pattern_offset_end:\n typ, segment_length = Segment.unpack_from(buffer, offset)\n offset += Segment.size\n typ = SegmentType(typ).name.lower()\n data = buffer[offset:offset + segment_length].decode('utf-8')\n offset += roundto8(segment_length)\n\n segments.append((typ, data))\n\n methods = buffer[offset:offset + methods_len].strip().decode('ascii') \\\n .split()\n\n return DecodedRoute(\n route_id, handler_id, coro_func, simple,\n placeholder_cnt, segments, methods)\n\n\ndef handler():\n pass\n\n\nasync def coro():\n # needs to have await to prevent being promoted to function\n await asyncio.sleep(1)\n\n\n@pytest.mark.parametrize('route', [\n Route('/', handler, []),\n Route('/', coro, ['GET']),\n Route('/test/{hi}', handler, []),\n Route('/test/{hi}', coro, ['POST']),\n Route('/tést', coro, ['POST'])\n], ids=Route.describe)\ndef test_compile(route):\n decompiled = decompile(compile(route))\n\n assert decompiled.route_id == id(route)\n assert decompiled.handler_id == id(route.handler)\n assert decompiled.coro_func == asyncio.iscoroutinefunction(route.handler)\n assert not decompiled.simple\n assert decompiled.placeholder_cnt == route.placeholder_cnt\n assert decompiled.segments == route.segments\n assert decompiled.methods == route.methods\n" }, { "path": "src/japronto/runner.py", "content": "from argparse import ArgumentParser, SUPPRESS\nfrom importlib import import_module\nimport os\nimport sys\nimport runpy\n\nfrom .app import Application\n\ntry:\n ModuleNotFoundError\nexcept NameError:\n ModuleNotFoundError = ImportError\n\n\ndef get_parser():\n prog = 'python -m japronto' if sys.argv[0].endswith('__main__.py') \\\n else 'japronto'\n parser = ArgumentParser(prog=prog)\n parser.add_argument('--host', dest='host', type=str, default='0.0.0.0')\n parser.add_argument('--port', dest='port', type=int, default=8080)\n parser.add_argument('--worker-num', dest='worker_num', type=int, default=1)\n parser.add_argument(\n '--reload', dest='reload', action='store_const',\n const=True, default=False)\n\n parser.add_argument(\n '--reloader-pid', dest='reloader_pid', type=int, help=SUPPRESS)\n parser.add_argument(\n '--script', dest='script', action='store_const',\n const=True, default=False, help=SUPPRESS)\n\n parser.add_argument('application')\n\n return parser\n\n\ndef verify(args):\n if args.script:\n script = args.application\n\n if not os.path.exists(script):\n print(\"Script '{}' not found.\".format(script))\n\n return script\n else:\n try:\n module, attribute = args.application.rsplit('.', 1)\n except ValueError:\n print(\n \"Application specificer must contain at least one '.',\" +\n \"got '{}'.\".format(args.application))\n return False\n\n try:\n module = import_module(module)\n except ModuleNotFoundError as e:\n print(e.args[0] + ' on Python search path.')\n return False\n\n try:\n attribute = getattr(module, attribute)\n except AttributeError:\n print(\n \"Module '{}' does not have an attribute '{}'.\"\n .format(module.__name__, attribute))\n return False\n\n if not isinstance(attribute, Application):\n print(\"{} is not an instance of 'japronto.Application'.\")\n return False\n\n return attribute\n\n\ndef run(attribute, args):\n if args.script:\n runpy.run_path(attribute)\n else:\n attribute._run(\n host=args.host, port=args.port,\n worker_num=args.worker_num, reloader_pid=args.reloader_pid)\n" }, { "path": "src/picohttpparser/build", "content": "#!/bin/bash\n\nset -ex\n\ngcc -c picohttpparser.c -O3 -fPIC -msse4.2 -o ssepicohttpparser.o\ngcc -c picohttpparser.c -O3 -fPIC -o picohttpparser.o\ngcc -fPIC -shared -o libpicohttpparser.so picohttpparser.o ssepicohttpparser.o\n" }, { "path": "src/picohttpparser/picohttpparser.c", "content": "/*\n * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase,\n * Shigeo Mitsunari\n *\n * The software is licensed under either the MIT License (below) or the Perl\n * license.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy\n * of this software and associated documentation files (the \"Software\"), to\n * deal in the Software without restriction, including without limitation the\n * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or\n * sell copies of the Software, and to permit persons to whom the Software is\n * furnished to do so, subject to the following conditions:\n *\n * The above copyright notice and this permission notice shall be included in\n * all copies or substantial portions of the Software.\n *\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS\n * IN THE SOFTWARE.\n */\n\n#include \n#include \n#include \n#ifdef __SSE4_2__\n#ifdef _MSC_VER\n#include \n#else\n#include \n#endif\n#endif\n#include \"picohttpparser.h\"\n\n/* $Id: 3228bad61525d68b757d7542bb817ccba6d7bc7f $ */\n\n#if __GNUC__ >= 3\n#define likely(x) __builtin_expect(!!(x), 1)\n#define unlikely(x) __builtin_expect(!!(x), 0)\n#else\n#define likely(x) (x)\n#define unlikely(x) (x)\n#endif\n\n#ifdef _MSC_VER\n#define ALIGNED(n) _declspec(align(n))\n#else\n#define ALIGNED(n) __attribute__((aligned(n)))\n#endif\n\n#define IS_PRINTABLE_ASCII(c) ((unsigned char)(c)-040u < 0137u)\n\n#define CHECK_EOF() \\\n if (buf == buf_end) { \\\n *ret = -2; \\\n return NULL; \\\n }\n\n#define EXPECT_CHAR(ch) \\\n CHECK_EOF(); \\\n if (*buf++ != ch) { \\\n *ret = -1; \\\n return NULL; \\\n }\n\n#define ADVANCE_TOKEN(tok, toklen) \\\n do { \\\n const char *tok_start = buf; \\\n static const char ALIGNED(16) ranges2[] = \"\\000\\040\\177\\177\"; \\\n int found2; \\\n buf = findchar_fast(buf, buf_end, ranges2, sizeof(ranges2) - 1, &found2); \\\n if (!found2) { \\\n CHECK_EOF(); \\\n } \\\n while (1) { \\\n if (*buf == ' ') { \\\n break; \\\n } else if (unlikely(!IS_PRINTABLE_ASCII(*buf))) { \\\n if ((unsigned char)*buf < '\\040' || *buf == '\\177') { \\\n *ret = -1; \\\n return NULL; \\\n } \\\n } \\\n ++buf; \\\n CHECK_EOF(); \\\n } \\\n tok = tok_start; \\\n toklen = buf - tok_start; \\\n } while (0)\n\nstatic const char *token_char_map = \"\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n \"\\0\\1\\0\\1\\1\\1\\1\\1\\0\\0\\1\\1\\0\\1\\1\\0\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\0\\0\\0\\0\\0\\0\"\n \"\\0\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\0\\0\\0\\1\\1\"\n \"\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\1\\0\\1\\0\\1\\0\"\n \"\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n \"\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n \"\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n \"\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\";\n\nstatic const char *findchar_fast(const char *buf, const char *buf_end, const char *ranges, size_t ranges_size, int *found)\n{\n *found = 0;\n#if __SSE4_2__\n if (likely(buf_end - buf >= 16)) {\n __m128i ranges16 = _mm_loadu_si128((const __m128i *)ranges);\n\n size_t left = (buf_end - buf) & ~15;\n do {\n __m128i b16 = _mm_loadu_si128((void *)buf);\n int r = _mm_cmpestri(ranges16, ranges_size, b16, 16, _SIDD_LEAST_SIGNIFICANT | _SIDD_CMP_RANGES | _SIDD_UBYTE_OPS);\n if (unlikely(r != 16)) {\n buf += r;\n *found = 1;\n break;\n }\n buf += 16;\n left -= 16;\n } while (likely(left != 0));\n }\n#else\n /* suppress unused parameter warning */\n (void)buf_end;\n (void)ranges;\n (void)ranges_size;\n#endif\n return buf;\n}\n\nstatic const char *get_token_to_eol(const char *buf, const char *buf_end, const char **token, size_t *token_len, int *ret)\n{\n const char *token_start = buf;\n\n#ifdef __SSE4_2__\n static const char ranges1[] = \"\\0\\010\"\n /* allow HT */\n \"\\012\\037\"\n /* allow SP and up to but not including DEL */\n \"\\177\\177\"\n /* allow chars w. MSB set */\n ;\n int found;\n buf = findchar_fast(buf, buf_end, ranges1, sizeof(ranges1) - 1, &found);\n if (found)\n goto FOUND_CTL;\n#else\n /* find non-printable char within the next 8 bytes, this is the hottest code; manually inlined */\n while (likely(buf_end - buf >= 8)) {\n#define DOIT() \\\n do { \\\n if (unlikely(!IS_PRINTABLE_ASCII(*buf))) \\\n goto NonPrintable; \\\n ++buf; \\\n } while (0)\n DOIT();\n DOIT();\n DOIT();\n DOIT();\n DOIT();\n DOIT();\n DOIT();\n DOIT();\n#undef DOIT\n continue;\n NonPrintable:\n if ((likely((unsigned char)*buf < '\\040') && likely(*buf != '\\011')) || unlikely(*buf == '\\177')) {\n goto FOUND_CTL;\n }\n ++buf;\n }\n#endif\n for (;; ++buf) {\n CHECK_EOF();\n if (unlikely(!IS_PRINTABLE_ASCII(*buf))) {\n if ((likely((unsigned char)*buf < '\\040') && likely(*buf != '\\011')) || unlikely(*buf == '\\177')) {\n goto FOUND_CTL;\n }\n }\n }\nFOUND_CTL:\n if (likely(*buf == '\\015')) {\n ++buf;\n EXPECT_CHAR('\\012');\n *token_len = buf - 2 - token_start;\n } else if (*buf == '\\012') {\n *token_len = buf - token_start;\n ++buf;\n } else {\n *ret = -1;\n return NULL;\n }\n *token = token_start;\n\n return buf;\n}\n\nstatic const char *is_complete(const char *buf, const char *buf_end, size_t last_len, int *ret)\n{\n int ret_cnt = 0;\n buf = last_len < 3 ? buf : buf + last_len - 3;\n\n while (1) {\n CHECK_EOF();\n if (*buf == '\\015') {\n ++buf;\n CHECK_EOF();\n EXPECT_CHAR('\\012');\n ++ret_cnt;\n } else if (*buf == '\\012') {\n ++buf;\n ++ret_cnt;\n } else {\n ++buf;\n ret_cnt = 0;\n }\n if (ret_cnt == 2) {\n return buf;\n }\n }\n\n *ret = -2;\n return NULL;\n}\n\n/* *_buf is always within [buf, buf_end) upon success */\nstatic const char *parse_int(const char *buf, const char *buf_end, int *value, int *ret)\n{\n int v;\n CHECK_EOF();\n if (!('0' <= *buf && *buf <= '9')) {\n *ret = -1;\n return NULL;\n }\n v = 0;\n for (;; ++buf) {\n CHECK_EOF();\n if ('0' <= *buf && *buf <= '9') {\n v = v * 10 + *buf - '0';\n } else {\n break;\n }\n }\n\n *value = v;\n return buf;\n}\n\n/* returned pointer is always within [buf, buf_end), or null */\nstatic const char *parse_http_version(const char *buf, const char *buf_end, int *minor_version, int *ret)\n{\n EXPECT_CHAR('H');\n EXPECT_CHAR('T');\n EXPECT_CHAR('T');\n EXPECT_CHAR('P');\n EXPECT_CHAR('/');\n EXPECT_CHAR('1');\n EXPECT_CHAR('.');\n return parse_int(buf, buf_end, minor_version, ret);\n}\n\nstatic const char *parse_headers(const char *buf, const char *buf_end, struct phr_header *headers, size_t *num_headers,\n size_t max_headers, int *ret)\n{\n for (;; ++*num_headers) {\n CHECK_EOF();\n if (*buf == '\\015') {\n ++buf;\n EXPECT_CHAR('\\012');\n break;\n } else if (*buf == '\\012') {\n ++buf;\n break;\n }\n if (*num_headers == max_headers) {\n *ret = -1;\n return NULL;\n }\n if (!(*num_headers != 0 && (*buf == ' ' || *buf == '\\t'))) {\n /* parsing name, but do not discard SP before colon, see\n * http://www.mozilla.org/security/announce/2006/mfsa2006-33.html */\n headers[*num_headers].name = buf;\n static const char ranges1[] __attribute__((aligned(16))) = \"\\x00 \" /* control chars and up to SP */\n \"\\\"\\\"\" /* 0x22 */\n \"()\" /* 0x28,0x29 */\n \",,\" /* 0x2c */\n \"//\" /* 0x2f */\n \":@\" /* 0x3a-0x40 */\n \"[]\" /* 0x5b-0x5d */\n \"{\\377\"; /* 0x7b-0xff */\n int found;\n buf = findchar_fast(buf, buf_end, ranges1, sizeof(ranges1) - 1, &found);\n if (!found) {\n CHECK_EOF();\n }\n while (1) {\n if (*buf == ':') {\n break;\n } else if (!token_char_map[(unsigned char)*buf]) {\n *ret = -1;\n return NULL;\n }\n ++buf;\n CHECK_EOF();\n }\n if ((headers[*num_headers].name_len = buf - headers[*num_headers].name) == 0) {\n *ret = -1;\n return NULL;\n }\n ++buf;\n for (;; ++buf) {\n CHECK_EOF();\n if (!(*buf == ' ' || *buf == '\\t')) {\n break;\n }\n }\n } else {\n headers[*num_headers].name = NULL;\n headers[*num_headers].name_len = 0;\n }\n if ((buf = get_token_to_eol(buf, buf_end, &headers[*num_headers].value, &headers[*num_headers].value_len, ret)) == NULL) {\n return NULL;\n }\n }\n return buf;\n}\n\nstatic const char *parse_request(const char *buf, const char *buf_end, const char **method, size_t *method_len, const char **path,\n size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers,\n size_t max_headers, int *ret)\n{\n /* skip first empty line (some clients add CRLF after POST content) */\n CHECK_EOF();\n if (*buf == '\\015') {\n ++buf;\n EXPECT_CHAR('\\012');\n } else if (*buf == '\\012') {\n ++buf;\n }\n\n /* parse request line */\n ADVANCE_TOKEN(*method, *method_len);\n ++buf;\n ADVANCE_TOKEN(*path, *path_len);\n ++buf;\n if ((buf = parse_http_version(buf, buf_end, minor_version, ret)) == NULL) {\n return NULL;\n }\n if (*buf == '\\015') {\n ++buf;\n EXPECT_CHAR('\\012');\n } else if (*buf == '\\012') {\n ++buf;\n } else {\n *ret = -1;\n return NULL;\n }\n\n return parse_headers(buf, buf_end, headers, num_headers, max_headers, ret);\n}\n\n#ifdef __SSE4_2__\n#define PHR_PARSE_REQUEST phr_parse_request_sse42\n#else\n#define PHR_PARSE_REQUEST phr_parse_request\n#endif\n\nint PHR_PARSE_REQUEST(const char *buf_start, size_t len, const char **method, size_t *method_len, const char **path,\n size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len)\n{\n const char *buf = buf_start, *buf_end = buf_start + len;\n size_t max_headers = *num_headers;\n int r;\n\n *method = NULL;\n *method_len = 0;\n *path = NULL;\n *path_len = 0;\n *minor_version = -1;\n *num_headers = 0;\n\n /* if last_len != 0, check if the request is complete (a fast countermeasure\n against slowloris */\n if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) {\n return r;\n }\n\n if ((buf = parse_request(buf, buf_end, method, method_len, path, path_len, minor_version, headers, num_headers, max_headers,\n &r)) == NULL) {\n return r;\n }\n\n return (int)(buf - buf_start);\n}\n\n// squeaky_pl: we don't use this yet\n#if 0\nstatic const char *parse_response(const char *buf, const char *buf_end, int *minor_version, int *status, const char **msg,\n size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret)\n{\n /* parse \"HTTP/1.x\" */\n if ((buf = parse_http_version(buf, buf_end, minor_version, ret)) == NULL) {\n return NULL;\n }\n /* skip space */\n if (*buf++ != ' ') {\n *ret = -1;\n return NULL;\n }\n /* parse status code */\n if ((buf = parse_int(buf, buf_end, status, ret)) == NULL) {\n return NULL;\n }\n /* skip space */\n if (*buf++ != ' ') {\n *ret = -1;\n return NULL;\n }\n /* get message */\n if ((buf = get_token_to_eol(buf, buf_end, msg, msg_len, ret)) == NULL) {\n return NULL;\n }\n\n return parse_headers(buf, buf_end, headers, num_headers, max_headers, ret);\n}\n\nint phr_parse_response(const char *buf_start, size_t len, int *minor_version, int *status, const char **msg, size_t *msg_len,\n struct phr_header *headers, size_t *num_headers, size_t last_len)\n{\n const char *buf = buf_start, *buf_end = buf + len;\n size_t max_headers = *num_headers;\n int r;\n\n *minor_version = -1;\n *status = 0;\n *msg = NULL;\n *msg_len = 0;\n *num_headers = 0;\n\n /* if last_len != 0, check if the response is complete (a fast countermeasure\n against slowloris */\n if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) {\n return r;\n }\n\n if ((buf = parse_response(buf, buf_end, minor_version, status, msg, msg_len, headers, num_headers, max_headers, &r)) == NULL) {\n return r;\n }\n\n return (int)(buf - buf_start);\n}\n\nint phr_parse_headers(const char *buf_start, size_t len, struct phr_header *headers, size_t *num_headers, size_t last_len)\n{\n const char *buf = buf_start, *buf_end = buf + len;\n size_t max_headers = *num_headers;\n int r;\n\n *num_headers = 0;\n\n /* if last_len != 0, check if the response is complete (a fast countermeasure\n against slowloris */\n if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) {\n return r;\n }\n\n if ((buf = parse_headers(buf, buf_end, headers, num_headers, max_headers, &r)) == NULL) {\n return r;\n }\n\n return (int)(buf - buf_start);\n}\n#endif\n\nenum {\n CHUNKED_IN_CHUNK_SIZE,\n CHUNKED_IN_CHUNK_EXT,\n CHUNKED_IN_CHUNK_DATA,\n CHUNKED_IN_CHUNK_CRLF,\n CHUNKED_IN_TRAILERS_LINE_HEAD,\n CHUNKED_IN_TRAILERS_LINE_MIDDLE\n};\n\n#ifndef __SSE4_2__\nstatic int decode_hex(int ch)\n{\n if ('0' <= ch && ch <= '9') {\n return ch - '0';\n } else if ('A' <= ch && ch <= 'F') {\n return ch - 'A' + 0xa;\n } else if ('a' <= ch && ch <= 'f') {\n return ch - 'a' + 0xa;\n } else {\n return -1;\n }\n}\n\nssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *_bufsz)\n{\n size_t dst = 0, src = 0, bufsz = *_bufsz;\n ssize_t ret = -2; /* incomplete */\n\n while (1) {\n switch (decoder->_state) {\n case CHUNKED_IN_CHUNK_SIZE:\n for (;; ++src) {\n int v;\n if (src == bufsz)\n goto Exit;\n if ((v = decode_hex(buf[src])) == -1) {\n if (decoder->_hex_count == 0) {\n ret = -1;\n goto Exit;\n }\n break;\n }\n if (decoder->_hex_count == sizeof(size_t) * 2) {\n ret = -1;\n goto Exit;\n }\n decoder->bytes_left_in_chunk = decoder->bytes_left_in_chunk * 16 + v;\n ++decoder->_hex_count;\n }\n decoder->_hex_count = 0;\n decoder->_state = CHUNKED_IN_CHUNK_EXT;\n /* fallthru */\n case CHUNKED_IN_CHUNK_EXT:\n /* RFC 7230 A.2 \"Line folding in chunk extensions is disallowed\" */\n for (;; ++src) {\n if (src == bufsz)\n goto Exit;\n if (buf[src] == '\\012')\n break;\n }\n ++src;\n if (decoder->bytes_left_in_chunk == 0) {\n if (decoder->consume_trailer) {\n decoder->_state = CHUNKED_IN_TRAILERS_LINE_HEAD;\n break;\n } else {\n goto Complete;\n }\n }\n decoder->_state = CHUNKED_IN_CHUNK_DATA;\n /* fallthru */\n case CHUNKED_IN_CHUNK_DATA: {\n size_t avail = bufsz - src;\n if (avail < decoder->bytes_left_in_chunk) {\n if (dst != src)\n memmove(buf + dst, buf + src, avail);\n src += avail;\n dst += avail;\n decoder->bytes_left_in_chunk -= avail;\n goto Exit;\n }\n if (dst != src)\n memmove(buf + dst, buf + src, decoder->bytes_left_in_chunk);\n src += decoder->bytes_left_in_chunk;\n dst += decoder->bytes_left_in_chunk;\n decoder->bytes_left_in_chunk = 0;\n decoder->_state = CHUNKED_IN_CHUNK_CRLF;\n }\n /* fallthru */\n case CHUNKED_IN_CHUNK_CRLF:\n for (;; ++src) {\n if (src == bufsz)\n goto Exit;\n if (buf[src] != '\\015')\n break;\n }\n if (buf[src] != '\\012') {\n ret = -1;\n goto Exit;\n }\n ++src;\n decoder->_state = CHUNKED_IN_CHUNK_SIZE;\n break;\n case CHUNKED_IN_TRAILERS_LINE_HEAD:\n for (;; ++src) {\n if (src == bufsz)\n goto Exit;\n if (buf[src] != '\\015')\n break;\n }\n if (buf[src++] == '\\012')\n goto Complete;\n decoder->_state = CHUNKED_IN_TRAILERS_LINE_MIDDLE;\n /* fallthru */\n case CHUNKED_IN_TRAILERS_LINE_MIDDLE:\n for (;; ++src) {\n if (src == bufsz)\n goto Exit;\n if (buf[src] == '\\012')\n break;\n }\n ++src;\n decoder->_state = CHUNKED_IN_TRAILERS_LINE_HEAD;\n break;\n default:\n assert(!\"decoder is corrupt\");\n }\n }\n\nComplete:\n ret = bufsz - src;\nExit:\n if (dst != src)\n memmove(buf + dst, buf + src, bufsz - src);\n *_bufsz = dst;\n return ret;\n}\n#endif\n\n// squeaky_pl: we dont use this\n#if 0\nint phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder)\n{\n return decoder->_state == CHUNKED_IN_CHUNK_DATA;\n}\n#endif\n\n#undef CHECK_EOF\n#undef EXPECT_CHAR\n#undef ADVANCE_TOKEN\n" }, { "path": "src/picohttpparser/picohttpparser.h", "content": "/*\n * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase,\n * Shigeo Mitsunari\n *\n * The software is licensed under either the MIT License (below) or the Perl\n * license.\n *\n * Permission is hereby granted, free of charge, to any person obtaining a copy\n * of this software and associated documentation files (the \"Software\"), to\n * deal in the Software without restriction, including without limitation the\n * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or\n * sell copies of the Software, and to permit persons to whom the Software is\n * furnished to do so, subject to the following conditions:\n *\n * The above copyright notice and this permission notice shall be included in\n * all copies or substantial portions of the Software.\n *\n * THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS\n * IN THE SOFTWARE.\n */\n\n#ifndef picohttpparser_h\n#define picohttpparser_h\n\n#include \n\n#ifdef _MSC_VER\n#define ssize_t intptr_t\n#endif\n\n/* $Id: 67fd3ee74103ada60258d8a16e868f483abcca87 $ */\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\n/* contains name and value of a header (name == NULL if is a continuing line\n * of a multiline header */\nstruct phr_header {\n const char *name;\n size_t name_len;\n const char *value;\n size_t value_len;\n};\n\n/* returns number of bytes consumed if successful, -2 if request is partial,\n * -1 if failed */\nint phr_parse_request(const char *buf, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len,\n int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len);\n\nint phr_parse_request_sse42(const char *buf, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len,\n int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len);\n\n\n/* ditto */\n// squeaky_p: we don't use it yet\n#if 0\nint phr_parse_response(const char *_buf, size_t len, int *minor_version, int *status, const char **msg, size_t *msg_len,\n struct phr_header *headers, size_t *num_headers, size_t last_len);\n\n/* ditto */\nint phr_parse_headers(const char *buf, size_t len, struct phr_header *headers, size_t *num_headers, size_t last_len);\n#endif\n\n/* should be zero-filled before start */\nstruct phr_chunked_decoder {\n size_t bytes_left_in_chunk; /* number of bytes left in current chunk */\n char consume_trailer; /* if trailing headers should be consumed */\n char _hex_count;\n char _state;\n};\n\n/* the function rewrites the buffer given as (buf, bufsz) removing the chunked-\n * encoding headers. When the function returns without an error, bufsz is\n * updated to the length of the decoded data available. Applications should\n * repeatedly call the function while it returns -2 (incomplete) every time\n * supplying newly arrived data. If the end of the chunked-encoded data is\n * found, the function returns a non-negative number indicating the number of\n * octets left undecoded at the tail of the supplied buffer. Returns -1 on\n * error.\n */\nssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *bufsz);\n\n/* returns if the chunked decoder is in middle of chunked data */\nint phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder);\n\n#ifdef __cplusplus\n}\n#endif\n\n#endif\n" }, { "path": "tutorial/1_hello.md", "content": "# Getting Started\n\nMake sure you have both [pip](https://pip.pypa.io/en/stable/installing/) and at\nleast version 3.5 of Python before starting. On Linux or MacOS X you can install\ndirectly using pip. If you are on Windows you can still develop and use\nJapronto with [Docker](https://docs.docker.com/engine/installation/#/on-macos-and-windows).\n\nInstalling\n----------\n\nOn Linux and OSX install Japronto with `python3 -m pip install japronto`.\nOn Windows or if you simply prefer Docker pull Japronto image with `docker pull japronto/japronto`.\n\nCreating your Hello world app\n-----------------------------\n\nCopy and paste following code into a file named `hello.py`:\n\n ```python\n # examples/1_hello/hello.py\n from japronto import Application\n\n\n # Views handle logic, take request as a parameter and\n # returns Response object back to the client\n def hello(request):\n return request.Response(text='Hello world!')\n\n\n # The Application instance is a fundamental concept.\n # It is a parent to all the resources and all the settings\n # can be tweaked here.\n app = Application()\n\n # The Router instance lets you register your handlers and execute\n # them depending on the url path and methods\n app.router.add_route('/', hello)\n\n # Finally start our server and handle requests until termination is\n # requested. Enabling debug lets you see request logs and stack traces.\n app.run(debug=True)\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\nRun it\n------\n\nOn Linux and OSX run the server with just: `python3 hello.py`.\n\nIf using Docker run `docker run -p 8080:8080 -v $(pwd)/hello.py:/hello.py japronto/japronto --script /hello.py`. This will mount local `hello.py` into container as `/hello.py` which is later passed to Docker entry point.\n\nNow open the address `http://localhost:8080` in your web browser. You should see the message *Hello world!*.\n\nYou now have a working Japronto server!\n\n\n**Next:** [Asynchronous handlers](2_async.md)\n" }, { "path": "tutorial/2_async.md", "content": "# Asynchronous handlers\n\nWith Japronto you can freely combine synchronous and asynchronous handlers and\nfully take advantage of both ecosystems. Choose wisely when to use asynchronous\nprogramming. Unless you are connecting to third party APIs, want to run input-output tasks in the background, expect large\nlatency or do long-running blocking queries to your database you are probably\nbetter off programming synchronously.\n\n\n ```python\n # examples/2_async/async.py\n import asyncio\n from japronto import Application\n\n\n # This is a synchronous handler.\n def synchronous(request):\n return request.Response(text='I am synchronous!')\n\n\n # This is an asynchronous handler, it spends most of the time in the event loop.\n # It wakes up every second 1 to print and finally returns after 3 seconds.\n # This does let other handlers to be executed in the same processes while\n # from the point of view of the client it took 3 seconds to complete.\n async def asynchronous(request):\n for i in range(1, 4):\n await asyncio.sleep(1)\n print(i, 'seconds elapsed')\n\n return request.Response(text='3 seconds elapsed')\n\n\n app = Application()\n\n r = app.router\n r.add_route('/sync', synchronous)\n r.add_route('/async', asynchronous)\n\n app.run()\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\n\n**Next:** [Router](3_router.md)\n" }, { "path": "tutorial/3_router.md", "content": "# Router\n\nThe router is a subsystem responsible for directing incoming requests to\nparticular handlers based on some conditions, namely the URL path\nand HTTP method. It's available under `router` property of an `Application`\ninstance and presents `add_router` method which takes `path` pattern, `handler`\nand optionally one or more `method`s.\n\n\n ```python\n # examples/3_router/router.py\n from japronto import Application\n\n\n app = Application()\n r = app.router\n\n\n # Requests with the path set exactly to `/` and whatever method\n # will be directed here.\n def slash(request):\n return request.Response(text='Hello {} /!'.format(request.method))\n\n\n r.add_route('/', slash)\n\n\n # Requests with the path set exactly to '/love' and the method\n # set exactly to `GET` will be directed here.\n def get_love(request):\n return request.Response(text='Got some love')\n\n\n r.add_route('/love', get_love, 'GET')\n\n\n # Requests with the path set exactly to '/methods' and the method\n # set to `POST` or `DELETE` will be directed here.\n def methods(request):\n return request.Response(text=request.method)\n\n\n r.add_route('/methods', methods, methods=['POST', 'DELETE'])\n\n\n # Requests with the path starting with `/params/` segment and followed\n # by two additional segments will be directed here.\n # Values of the additional segments will be stored in side `request.match_dict`\n # dictionary with keys taken from {} placeholders. A request to `/params/1/2`\n # would leave `match_dict` set to `{'p1': 1, 'p2': '2'}`.\n def params(request):\n return request.Response(text=str(request.match_dict))\n\n\n r.add_route('/params/{p1}/{p2}', params)\n\n app.run()\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\n**Next:** [Request object](4_request.md)\n" }, { "path": "tutorial/4_request.md", "content": "# Request Object\n\nRequest represents an incoming HTTP request with a rich set of properties. They can be divided into\nthree categories: Request line and headers, message body and miscellaneous.\n\n ```python\n # examples/4_request/request.py\n from json import JSONDecodeError\n\n from japronto import Application\n\n\n # Request line and headers.\n # This represents the part of a request that comes before message body.\n # Given a HTTP 1.1 `GET` request to `/basic?a=1` this would yield\n # `method` set to `GET`, `path` set to `/basic`, `version` set to `1.1`\n # `query_string` set to `a=1` and `query` set to `{'a': '1'}`.\n # Additionally if headers are sent they will be present in `request.headers`\n # dictionary. The keys are normalized to standard `Camel-Cased` convention.\n def basic(request):\n text = \"\"\"Basic request properties:\n Method: {0.method}\n Path: {0.path}\n HTTP version: {0.version}\n Query string: {0.query_string}\n Query: {0.query}\"\"\".format(request)\n\n if request.headers:\n text += \"\\nHeaders:\\n\"\n for name, value in request.headers.items():\n text += \" {0}: {1}\\n\".format(name, value)\n\n return request.Response(text=text)\n\n\n # Message body\n # If there is a message body attached to a request (as in a case of `POST`)\n # method the following attributes can be used to examine it.\n # Given a `POST` request with body set to `b'J\\xc3\\xa1'`, `Content-Length` header set\n # to `3` and `Content-Type` header set to `text/plain; charset=utf-8` this\n # would yield `mime_type` set to `'text/plain'`, `encoding` set to `'utf-8'`,\n # `body` set to `b'J\\xc3\\xa1'` and `text` set to `'Já'`.\n # `form` and `files` attributes are dictionaries respectively used for HTML forms and\n # HTML file uploads. The `json` helper property will try to decode `body` as a\n # JSON document and give you resulting Python data type.\n def body(request):\n text = \"\"\"Body related properties:\n Mime type: {0.mime_type}\n Encoding: {0.encoding}\n Body: {0.body}\n Text: {0.text}\n Form parameters: {0.form}\n Files: {0.files}\n \"\"\".format(request)\n\n try:\n json = request.json\n except JSONDecodeError:\n pass\n else:\n text += \"\\nJSON:\\n\"\n text += str(json)\n\n return request.Response(text=text)\n\n\n # Miscellaneous\n # `route` will point to an instance of `Route` object representing\n # route chosen by router to handle this request. `hostname` and `port`\n # represent parsed `Host` header if any. `remote_addr` is the address of\n # a client or reverse proxy. If `keep_alive` is true the client requested to\n # keep connection open after the response is delivered. `match_dict` contains\n # route placeholder values as documented in `2_router.md`. `cookies` contains\n # a dictionary of HTTP cookies if any.\n def misc(request):\n text = \"\"\"Miscellaneous:\n Matched route: {0.route}\n Hostname: {0.hostname}\n Port: {0.port}\n Remote address: {0.remote_addr},\n HTTP Keep alive: {0.keep_alive}\n Match parameters: {0.match_dict}\n \"\"\".strip().format(request)\n\n if request.cookies:\n text += \"\\nCookies:\\n\"\n for name, value in request.cookies.items():\n text += \" {0}: {1}\\n\".format(name, value)\n\n return request.Response(text=text)\n\n\n app = Application()\n app.router.add_route('/basic', basic)\n app.router.add_route('/body', body)\n app.router.add_route('/misc', misc)\n app.run()\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\n\n**Next:** [Response object](5_response.md)\n" }, { "path": "tutorial/5_response.md", "content": "# Response object\n\nHandlers return Response instances to fulfill requests. They can contain status code, headers and almost always a body.\nAt the moment Response instances are immutable once created, this\nrestriction will be lifted in a next version.\n\n ```python\n # examples/5_response/response.py\n import random\n from http.cookies import SimpleCookie\n\n from japronto.app import Application\n\n\n # Providing just text argument yields a `text/plain` response\n # encoded with `utf8` codec (charset set accordingly)\n def text(request):\n return request.Response(text='Hello world!')\n\n\n # You can override encoding by providing `encoding` attribute.\n def encoding(request):\n return request.Response(text='Já pronto!', encoding='iso-8859-1')\n\n\n # You can also set a custom MIME type.\n def mime(request):\n return request.Response(\n mime_type=\"image/svg+xml\",\n text=\"\"\"\n \n \n \n \"\"\")\n\n\n # Or serve binary data. `Content-Type` set to `application/octet-stream`\n # automatically but you can always provide your own `mime_type`.\n def body(request):\n return request.Response(body=b'\\xde\\xad\\xbe\\xef')\n\n\n # There exist a shortcut `json` argument. This automatically encodes the\n # provided object as JSON and servers it with `Content-Type` set to\n # `application/json; charset=utf8`\n def json(request):\n return request.Response(json={'hello': 'world'})\n\n\n # You can change the default 200 status `code` for another\n def code(request):\n return request.Response(code=random.choice([200, 201, 400, 404, 500]))\n\n\n # And of course you can provide custom `headers`.\n def headers(request):\n return request.Response(\n text='headers',\n headers={'X-Header': 'Value',\n 'Refresh': '5; url=https://xkcd.com/353/'})\n\n\n # Or `cookies` by using Python standard library `http.cookies.SimpleCookie`.\n def cookies(request):\n cookies = SimpleCookie()\n cookies['hello'] = 'world'\n cookies['hello']['domain'] = 'localhost'\n cookies['hello']['path'] = '/'\n cookies['hello']['max-age'] = 3600\n cookies['city'] = 'São Paulo'\n\n return request.Response(text='cookies', cookies=cookies)\n\n\n app = Application()\n router = app.router\n router.add_route('/text', text)\n router.add_route('/encoding', encoding)\n router.add_route('/mime', mime)\n router.add_route('/body', body)\n router.add_route('/json', json)\n router.add_route('/code', code)\n router.add_route('/headers', headers)\n router.add_route('/cookies', cookies)\n app.run()\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\n\n**Next:** [Handling exceptions](6_exceptions.md)\n" }, { "path": "tutorial/6_exceptions.md", "content": "# Handling exceptions\n\nThere may be cases where you may want to respond with a custom response instead of a 500 Internal Server Error when an exception(s) is raised. Or you might want to override the default 404 handler. Enter exception handlers.\n\n ```python\n # examples/6_exceptions/exceptions.py\n from japronto import Application, RouteNotFoundException\n\n\n # Those are our custom exceptions we want to turn into 200 response.\n class KittyError(Exception):\n def __init__(self):\n self.greet = 'meow'\n\n\n class DoggieError(Exception):\n def __init__(self):\n self.greet = 'woof'\n\n\n # The two handlers below raise exceptions which will be turned\n # into 200 responses by the handlers registered later\n def cat(request):\n raise KittyError()\n\n\n def dog(request):\n raise DoggieError()\n\n\n # This handler raises ZeroDivisionError which doesnt have an error\n # handler registered so it will result in 500 Internal Server Error\n def unhandled(request):\n 1 / 0\n\n\n app = Application()\n\n r = app.router\n r.add_route('/cat', cat)\n r.add_route('/dog', dog)\n r.add_route('/unhandled', unhandled)\n\n\n # These two are handlers for `Kitty` and `DoggyError`s.\n def handle_cat(request, exception):\n return request.Response(text='Just a kitty, ' + exception.greet)\n\n\n def handle_dog(request, exception):\n return request.Response(text='Just a doggie, ' + exception.greet)\n\n\n # You can also override default 404 handler if you want\n def handle_not_found(request, exception):\n return request.Response(code=404, text=\"Are you lost, pal?\")\n\n\n # register all the error handlers so they are actually effective\n app.add_error_handler(KittyError, handle_cat)\n app.add_error_handler(DoggieError, handle_dog)\n app.add_error_handler(RouteNotFoundException, handle_not_found)\n\n app.run()\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\n**Next:** [Extending request](7_extend.md)\n" }, { "path": "tutorial/7_extend.md", "content": "# Extending Request object\n\nYou can register custom properties and methods on Request object. This is typically done to accommodate objects that\nare bound to request lifetime such as database connections, sessions or caching. This lets you easily access that logic in one place. There also exist a mechanism to execute code\non request completion by registering a callback in a view.\n\n ```python\n # examples/7_extend/extend.py\n from japronto import Application\n\n\n # This view accesses custom method host_startswith\n # and a custom property reversed_agent. Both are registered later.\n def extended_hello(request):\n if request.host_startswith('api.'):\n text = 'Hello ' + request.reversed_agent\n else:\n text = 'Hello stranger'\n\n return request.Response(text=text)\n\n\n # This view registers a callback, such callbacks are executed after handler\n # exit and the response is ready to be sent over the wire.\n def with_callback(request):\n def cb(r):\n print('Done!')\n\n request.add_done_callback(cb)\n\n return request.Response(text='cb')\n\n\n # This is a body for reversed_agent property\n def reversed_agent(request):\n return request.headers['User-Agent'][::-1]\n\n\n # This is a body for host_startswith method\n # Custom methods and properties always accept request\n # object.\n def host_startswith(request, prefix):\n return request.headers['Host'].startswith(prefix)\n\n app = Application()\n # Finally register out custom property and method\n # By default the names are taken from function names\n # unelss you provide `name` keyword parameter.\n app.extend_request(reversed_agent, property=True)\n app.extend_request(host_startswith)\n\n r = app.router\n r.add_route('/', extended_hello)\n r.add_route('/callback', with_callback)\n\n app.run()\n ```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n" }, { "path": "tutorial/8_template.md", "content": "# Responding with HTML\n\nServing HTML from japronto is as simple as adding a MIME type of `text/html` to the Response. Jinja2 templating can be leveraged as well, although in the meantime you will have to do the heavy lifting of rendering templates before sending in a response.\n\nCopy and paste following code into a file named `html.py`:\n\n```python\n# examples/8_template/template.py\nfrom japronto import Application\nfrom jinja2 import Template\n\n\n# A view can read HTML from a file\ndef index(request):\n with open('index.html') as html_file:\n return request.Response(text=html_file.read(), mime_type='text/html')\n\n\n# A view could also return a raw HTML string\ndef example(request):\n return request.Response(text='

Some HTML!

', mime_type='text/html')\n\n\n# A view could also return a rendered jinja2 template\ndef jinja(request):\n template = Template('

Hello {{ name }}!

')\n return request.Response(text=template.render(name='World'),\n mime_type='text/html')\n\n\n# Create the japronto application\napp = Application()\n\n# Add routes to the app\napp.router.add_route('/', index)\napp.router.add_route('/example', example)\napp.router.add_route('/jinja2', jinja)\n\n# Start the server\napp.run(debug=True)\n```\n\nThe source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).\n\n" } ]