Repository: squeaky-pl/japronto Branch: master Commit: e73b76ea6ee2 Files: 148 Total size: 337.7 KB Directory structure: gitextract_jn3kd_d7/ ├── .gitignore ├── .gitmodules ├── .travis.yml ├── CHANGELOG.md ├── LICENSE.txt ├── MANIFEST.in ├── README.md ├── benchmarks/ │ ├── aiohttp/ │ │ ├── micro.py │ │ └── requirements.txt │ ├── gevent/ │ │ ├── micro.py │ │ └── requirements.txt │ ├── golang/ │ │ ├── README.md │ │ └── micro.go │ ├── golang-fasthttp/ │ │ ├── README.md │ │ └── micro.go │ ├── japronto/ │ │ └── micro.py │ ├── meinheld/ │ │ ├── micro.py │ │ └── requirements.txt │ ├── nodejs/ │ │ └── micro.js │ ├── results.ods │ ├── sanic/ │ │ ├── micro.py │ │ └── requirements.txt │ └── tornado/ │ ├── micro.py │ └── requirements.txt ├── build.py ├── cases/ │ ├── __init__.py │ ├── base.toml │ └── websites.toml ├── conftest.py ├── do_wrk.py ├── examples/ │ ├── 1_hello/ │ │ └── hello.py │ ├── 2_async/ │ │ └── async.py │ ├── 3_router/ │ │ └── router.py │ ├── 4_request/ │ │ └── request.py │ ├── 5_response/ │ │ └── response.py │ ├── 6_exceptions/ │ │ └── exceptions.py │ ├── 7_extend/ │ │ └── extend.py │ ├── 8_template/ │ │ ├── index.html │ │ └── template.py │ └── todo_api/ │ ├── .gitignore │ └── todo_api.py ├── integration_tests/ │ ├── __init__.py │ ├── common.py │ ├── drain.py │ ├── dump.py │ ├── experiments.py │ ├── generators.py │ ├── longrun.py │ ├── noleak.py │ ├── reaper.py │ ├── strategies.py │ ├── test_drain.py │ ├── test_noleak.py │ ├── test_perror.py │ ├── test_reaper.py │ └── test_request.py ├── misc/ │ ├── __init__.py │ ├── bootstrap.sh │ ├── buggers.py │ ├── cleanup_script.py │ ├── client.py │ ├── collector.py │ ├── cpu.py │ ├── do_perf.py │ ├── docker/ │ │ └── Dockerfile │ ├── parts.py │ ├── perf.md │ ├── pipeline.lua │ ├── report.py │ ├── requirements-test.txt │ ├── requirements.txt │ ├── rpm-requirements.txt │ ├── runpytest.py │ ├── simple.py │ ├── suppr.txt │ └── travis/ │ ├── before_install.sh │ ├── install.sh │ └── script.sh ├── setup.py ├── src/ │ ├── japronto/ │ │ ├── __init__.py │ │ ├── __main__.py │ │ ├── app/ │ │ │ └── __init__.py │ │ ├── capsule.c │ │ ├── capsule.h │ │ ├── common.h │ │ ├── cpu_features.c │ │ ├── cpu_features.h │ │ ├── parser/ │ │ │ ├── .gitignore │ │ │ ├── __init__.py │ │ │ ├── build_libpicohttpparser.py │ │ │ ├── cffiparser.py │ │ │ ├── cparser.c │ │ │ ├── cparser.gcda │ │ │ ├── cparser.h │ │ │ ├── cparser_ext.py │ │ │ └── test_parser.py │ │ ├── pipeline/ │ │ │ ├── __init__.py │ │ │ ├── cpipeline.c │ │ │ ├── cpipeline.h │ │ │ ├── cpipeline_ext.py │ │ │ └── test_pipeline.py │ │ ├── protocol/ │ │ │ ├── __init__.py │ │ │ ├── cprotocol.c │ │ │ ├── cprotocol.h │ │ │ ├── cprotocol_ext.py │ │ │ ├── creaper.c │ │ │ ├── creaper_ext.py │ │ │ ├── generator.c │ │ │ ├── generator.h │ │ │ ├── generator_ext.py │ │ │ ├── handler.py │ │ │ ├── null.py │ │ │ └── tracing.py │ │ ├── reloader.py │ │ ├── request/ │ │ │ ├── __init__.py │ │ │ ├── crequest.c │ │ │ ├── crequest.h │ │ │ └── crequest_ext.py │ │ ├── response/ │ │ │ ├── __init__.py │ │ │ ├── cresponse.c │ │ │ ├── cresponse.h │ │ │ ├── cresponse_ext.py │ │ │ ├── py.py │ │ │ └── reasons.h │ │ ├── router/ │ │ │ ├── __init__.py │ │ │ ├── analyzer.py │ │ │ ├── cmatcher.c │ │ │ ├── cmatcher.h │ │ │ ├── cmatcher_ext.py │ │ │ ├── match_dict.c │ │ │ ├── match_dict.h │ │ │ ├── matcher.py │ │ │ ├── route.py │ │ │ ├── test_analyzer.py │ │ │ ├── test_matcher.py │ │ │ └── test_route.py │ │ └── runner.py │ └── picohttpparser/ │ ├── build │ ├── picohttpparser.c │ └── picohttpparser.h └── tutorial/ ├── 1_hello.md ├── 2_async.md ├── 3_router.md ├── 4_request.md ├── 5_response.md ├── 6_exceptions.md ├── 7_extend.md └── 8_template.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ __pycache__ .hypothesis .test coverage.info *.egg-info *.build.toml *.so *.o ================================================ FILE: .gitmodules ================================================ [submodule "misc/terryfy"] path = misc/terryfy url = https://github.com/MacPython/terryfy/ ================================================ FILE: .travis.yml ================================================ language: python sudo: required services: - docker env: global: - secure: "HO3zCuv0FtNFTQ7kkBpIqKYAZW8sYZPfc1ROk6+ChoxufXcu529CTKNAr3KklfZCbMHiZKc3W83N7x9B/L2rtSuBQvJPPgVtIlaVKRyWWnY4nqrpwKEoOLUd3RjpAMfCB09sXQ2aTfQV8Ds5Zk+cF7R2toI6s2s4vymXvCLvfugrtO4sd91frSDv/fzjEEKOIeey8KXtPAPPFv6v64OScksPt1oCsVOPDtkZ7q0KSIzS9JN6BvM9oafPt9MaFPH84ITtdPMTjgQOQ+YFe8YBwgjkV/cX9rNs+vzSP6Bm2NQ9/xxd8XTDj6ukuEYD5HQi26IS6ddRyVGsn6/WRZx6/kQboJKh5r5pa4OAHPWnPirRWPQW46HI2iknAGTFWh8ARX2R208mK1vbQ66J+9zQDnkjMXGgX67gWyWQWfxwVHFsPyQEiSHHh2vEBkhs1+tqtvp7Ktnc+uCxXn0v/Humu3OvFSBxSXfyjvE9uUOGyB2zDwqmxLQQ5ftKAcGfLOaSqauJ1vQy1CWc5bROCn8aoch5iRf/tcX85TUDirAgAp3OUdt3VwcRNY+Fci7IU50gn2rghJWFzB5Zz9p1ShnZxIaD5GEPE45ju4UIpwYbs8iSqh+/RS8sR2Ffzx4M+6QJjj1BJABdtVPS9Jn5OkbuSdBW0K+MuLtmtbg4WLXv6+E=" jobs: include: - python: 3.5 env: VERSION=3.5.3 - python: 3.6 env: VERSION=3.6.0 - python: 3.7 env: VERSION=3.7.1 - python: 3.8 env: VERSION=3.8.0 before_install: source misc/travis/before_install.sh install: source misc/travis/install.sh script: misc/travis/script.sh ================================================ FILE: CHANGELOG.md ================================================ 0.1.1 - Feb 9 2017 ------------------ - Native support for OSX - Support for older hardware without SSE4.2 - Better crash info with faulthandler ================================================ FILE: LICENSE.txt ================================================ Copyright (c) 2017 Paweł Piotr Przeradowski Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: MANIFEST.in ================================================ include README.md LICENSE.txt src/picohttpparser/picohttpparser.c src/picohttpparser/picohttpparser.h ================================================ FILE: README.md ================================================ # Japronto! [![irc: #japronto](https://img.shields.io/badge/irc-%23japronto-brightgreen.svg)](https://webchat.freenode.net/?channels=japronto) [![Gitter japronto/Lobby](https://badges.gitter.im/japronto/Lobby.svg)](https://gitter.im/japronto/Lobby) [![Build Status](https://travis-ci.org/squeaky-pl/japronto.svg?branch=master)](https://travis-ci.org/squeaky-pl/japronto) [![PyPI](https://img.shields.io/pypi/v/japronto.svg)](https://pypi.python.org/pypi/japronto) [![PyPI version](https://img.shields.io/pypi/pyversions/japronto.svg)](https://pypi.python.org/pypi/japronto/) __There is no new project development happening at the moment, but it's not abandoned either. Pull requests and new maintainers are welcome__. __If you are a novice Python programmer, you don't like plumbing yourself or you don't have basic understanding of C, this project is not probably what you are looking for__. Japronto (from Portuguese "já pronto" /ˈʒa pɾõtu/ meaning "already done") is a __screaming-fast__, __scalable__, __asynchronous__ Python 3.5+ HTTP __toolkit__ integrated with __pipelining HTTP server__ based on [uvloop](https://github.com/MagicStack/uvloop) and [picohttpparser](https://github.com/h2o/picohttpparser). It's targeted at speed enthusiasts, people who like plumbing and early adopters. You can read more in the [release announcement on medium](https://medium.com/@squeaky_pl/million-requests-per-second-with-python-95c137af319) Performance ----------- Here's a chart to help you imagine what kind of things you can do with Japronto: ![Requests per second](benchmarks/results.png) As user @heppu points out Go’s stdlib HTTP server can be 12% faster than the graph shows when written more carefully. Also there is the awesome fasthttp server for Go that apparently is only 18% slower than Japronto in this particular benchmark. Awesome! For details see https://github.com/squeaky-pl/japronto/pull/12 and https://github.com/squeaky-pl/japronto/pull/14. These results of a simple "Hello world" application were obtained on AWS c4.2xlarge instance. To be fair all the contestants (including Go) were running single worker process. Servers were load tested using [wrk](https://github.com/wg/wrk) with 1 thread, 100 connections and 24 simultaneous (pipelined) requests per connection (cumulative parallelism of 2400 requests). The source code for the benchmark can be found in [benchmarks](benchmarks) directory. The server is written in hand tweaked C trying to take advantage of modern CPUs. It relies on picohttpparser for header & chunked-encoding parsing while uvloop provides asynchronous I/O. It also tries to save up on system calls by combining writes together when possible. Early preview ------------- This is an early preview with alpha quality implementation. APIs are provisional meaning that they will change between versions and more testing is needed. Don't use it for anything serious for now and definitely don't use it in production. Please try it though and report back feedback. If you are shopping for your next project's framework I would recommend [Sanic](https://github.com/channelcat/sanic). At the moment the work is focused on CPython but I have PyPy on my radar, though I am not gonna look into it until PyPy reaches 3.5 compatibility somewhere later this year and most known JIT regressions are removed. Hello world ----------- Here is how a simple web application looks like in Japronto: ```python from japronto import Application def hello(request): return request.Response(text='Hello world!') app = Application() app.router.add_route('/', hello) app.run(debug=True) ``` Tutorial -------- 1. [Getting started](tutorial/1_hello.md) 2. [Asynchronous handlers](tutorial/2_async.md) 3. [Router](tutorial/3_router.md) 4. [Request object](tutorial/4_request.md) 5. [Response object](tutorial/5_response.md) 6. [Handling exceptions](tutorial/6_exceptions.md) 7. [Extending request](tutorial/7_extend.md) Features -------- - HTTP 1.x implementation with support for chunked uploads - Full support for HTTP pipelining - Keep-alive connections with configurable reaper - Support for synchronous and asynchronous views - Master-multiworker model based on forking - Support for code reloading on changes - Simple routing License ------- This software is distributed under [MIT License](https://en.wikipedia.org/wiki/MIT_License). This is a very permissive license that lets you use this software for any commercial and non-commercial work. Full text of the license is included in [LICENSE.txt](LICENSE.txt) file. The source distribution of this software includes a copy of picohttpparser which is distributed under MIT license as well. ================================================ FILE: benchmarks/aiohttp/micro.py ================================================ from aiohttp import web import asyncio import uvloop loop = uvloop.new_event_loop() asyncio.set_event_loop(loop) async def hello(request): return web.Response(text='Hello world!') app = web.Application(loop=loop) app.router.add_route('GET', '/', hello) web.run_app(app, port=8080, access_log=None) ================================================ FILE: benchmarks/aiohttp/requirements.txt ================================================ aiohttp==1.2.0 ================================================ FILE: benchmarks/gevent/micro.py ================================================ from gevent.pywsgi import WSGIServer def hello(environ, start_response): if(environ['PATH_INFO'] == '/' and environ['REQUEST_METHOD'] == 'GET'): status = '200 OK' text = "Hello world!" else: status = '404 Not Found' text = "Not Found" body = text.encode('utf-8') response_headers = [ ('Content-type', 'text/plain; charset=utf-8'), ('Content-Length', str(len(body)))] start_response(status, response_headers) return [body] WSGIServer(('0.0.0.0', 8080), hello, log=None).serve_forever() ================================================ FILE: benchmarks/gevent/requirements.txt ================================================ gevent==1.2.1 ================================================ FILE: benchmarks/golang/README.md ================================================ ``` go build . GOMAXPROCS=1 ./bin ``` ================================================ FILE: benchmarks/golang/micro.go ================================================ package main import "net/http" var ( helloResp = []byte("Hello world!") notFoundResp = []byte("Not Found") ) func hello(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/" { w.WriteHeader(http.StatusNotFound) w.Write(notFoundResp) return } w.Write(helloResp) } func main() { http.HandleFunc("/", hello) http.ListenAndServe("0.0.0.0:8080", nil) } ================================================ FILE: benchmarks/golang-fasthttp/README.md ================================================ ``` go build . GOMAXPROCS=1 ./bin ``` ================================================ FILE: benchmarks/golang-fasthttp/micro.go ================================================ package main import "github.com/valyala/fasthttp" func hello(ctx *fasthttp.RequestCtx) { if string(ctx.Path()) != "/" { ctx.SetStatusCode(404) ctx.WriteString("Not Found") return } ctx.WriteString("Hello world!") } func main() { fasthttp.ListenAndServe("0.0.0.0:8080", hello) } ================================================ FILE: benchmarks/japronto/micro.py ================================================ from japronto import Application def hello(request): return request.Response(text='Hello world!') app = Application() r = app.router r.add_route('/', hello, method='GET') app.run() ================================================ FILE: benchmarks/meinheld/micro.py ================================================ from meinheld import server def hello(environ, start_response): if(environ['PATH_INFO'] == '/' and environ['REQUEST_METHOD'] == 'GET'): status = '200 OK' text = "Hello world!" else: status = '404 Not Found' text = "Not Found" body = text.encode('utf-8') response_headers = [ ('Content-type', 'text/plain; charset=utf-8'), ('Content-Length', str(len(body)))] start_response(status, response_headers) return [body] server.listen(('0.0.0.0', 8080)) server.set_access_logger(None) server.set_keepalive(1) server.run(hello) ================================================ FILE: benchmarks/meinheld/requirements.txt ================================================ meinheld==0.6.1 ================================================ FILE: benchmarks/nodejs/micro.js ================================================ const http = require('http'); var srv = http.createServer( (req, res) => { res.sendDate = false; if(req.url == '/') { data = 'Hello world!' status = 200 } else { data = 'Not Found' status = 404 } res.writeHead(status, { 'Content-Type': 'text/plain; encoding=utf-8', 'Content-Length': data.length}); res.end(data); }); srv.listen(8080, '0.0.0.0'); ================================================ FILE: benchmarks/sanic/micro.py ================================================ from sanic import Sanic from sanic.response import text app = Sanic(__name__) @app.route("/") async def hello(request): return text("Hello world!") app.run(host="0.0.0.0", port=8080) ================================================ FILE: benchmarks/sanic/requirements.txt ================================================ sanic==0.2.0 ================================================ FILE: benchmarks/tornado/micro.py ================================================ from tornado import web from tornado.httputil import HTTPHeaders, responses from tornado.platform.asyncio import AsyncIOMainLoop import asyncio import uvloop loop = uvloop.new_event_loop() asyncio.set_event_loop(loop) AsyncIOMainLoop().install() class MainHandler(web.RequestHandler): def get(self): self.write('Hello world!') # skip calculating ETag, ~8% faster def set_etag_header(self): pass def check_etag_header(self): return False # torando sends Server and Date headers by default, ~4% faster def clear(self): self._headers = HTTPHeaders( {'Content-Type': 'text/plain; charset=utf-8'}) self._write_buffer = [] self._status_code = 200 self._reason = responses[200] app = web.Application([('/', MainHandler)]) app.listen(8080) loop.run_forever() ================================================ FILE: benchmarks/tornado/requirements.txt ================================================ tornado==4.4.2 ================================================ FILE: build.py ================================================ import argparse import distutils from distutils.command.build_ext import build_ext, CompileError from distutils.core import Distribution from glob import glob import os.path import shutil import sysconfig import os import sys import subprocess try: import pytoml except ImportError: pytoml = None import runpy SRC_LOCATION = 'src' sys.path.insert(0, SRC_LOCATION) class BuildSystem: def __init__(self, args, relative_source=False): self.args = args self.dest = self.args.dest self.relative_source = relative_source def get_extension_by_path(self, path): path = SRC_LOCATION + '/' + path result = runpy.run_path(path, {'system': self}) extension = result['get_extension']() base_path = os.path.dirname(path) def fix_path(p): if os.path.isabs(p): return p return os.path.abspath(os.path.join(base_path, p)) attrs = ['sources', 'include_dirs', 'library_dirs', 'runtime_library_dirs'] for attr in attrs: val = getattr(extension, attr) if not val: continue if attr == 'sources' and self.relative_source: val = [ (os.path.normpath(os.path.join(base_path, v)) if not v.startswith('src') else v) for v in val] elif attr == 'runtime_library_dirs' and self.relative_source: pass else: val = [fix_path(v) for v in val] if attr == 'runtime_library_dirs': setattr(extension, attr, None) attr = 'extra_link_args' val = ['-Wl,-rpath,' + v for v in val] val = (getattr(extension, attr) or []) + val setattr(extension, attr, val) return extension def discover_extensions(self): self.extensions = [] ext_files = glob(SRC_LOCATION + '/**/*_ext.py', recursive=True) ext_files = [os.path.relpath(p, SRC_LOCATION) for p in ext_files] self.extensions = [self.get_extension_by_path(f) for f in ext_files] return self.extensions def dest_folder(self, mod_name): return self.dest + '/' + '/'.join(mod_name.split('.')[:-1]) def build_toml(self, mod_name): return self.dest + '/' + '/'.join(mod_name.split('.')) + '.build.toml' def get_so(self, ext): return self.dest + '/' + '/'.join(ext.name.split('.')) + '.' + \ sysconfig.get_config_var('SOABI') + '.so' def flags_changed(self, ext): toml = self.build_toml(ext.name) if not os.path.exists(toml): return True with open(toml) as f: flags = pytoml.load(f) ext_flags = { "extra_compile_args": ext.extra_compile_args, "extra_link_args": ext.extra_link_args, "define_macros": dict(ext.define_macros), "sources": ext.sources} return flags != ext_flags def should_rebuild(self, ext): so = self.get_so(ext) if not os.path.exists(so): return True so_mtime = os.stat(so).st_mtime includes = get_includes(ext) input_mtimes = [os.stat(s).st_mtime for s in ext.sources + includes] if max(input_mtimes) > so_mtime: return True if self.flags_changed(ext): return True return False def prune(dest): paths = glob(os.path.join(dest, '.build/**/*.o'), recursive=True) paths.extend(glob(os.path.join(dest, '.build/**/*.so'), recursive=True)) for path in paths: os.remove(path) def profile_clean(): paths = glob('build/**/*.gcda', recursive=True) for path in paths: os.remove(path) def get_includes(ext): includes = [] include_base = SRC_LOCATION + '/' + '/'.join(ext.name.split('.')[:-1]) include_paths = [os.path.join(include_base, i) for i in ext.include_dirs] for source in ext.sources: with open(source) as f: for line in f: line = line.strip() if not line.startswith('#include'): continue header = line.split()[1][1:-1] for path in include_paths: if not os.path.exists(os.path.join(path, header)): continue includes.append(os.path.join(path, header)) break return includes def symlink_python_files(dest): if dest == SRC_LOCATION: return for parent, dirs, files in os.walk(SRC_LOCATION): if os.path.basename(parent) == '__pycache__': continue def _is_python_file(f): return f.endswith('.py') and not f.endswith('_ext.py') \ and not f.startswith('test_') files = [f for f in files if _is_python_file(f)] if not files: continue dest_parent = os.path.join(dest, *parent.split(os.sep)[1:]) os.makedirs(dest_parent, exist_ok=True) for file in files: dst = os.path.join(dest_parent, file) src = os.path.relpath(os.path.join(parent, file), dest_parent) if os.path.exists(dst): os.unlink(dst) os.symlink(src, dst) kits = { 'platform': [ 'japronto.request.crequest', 'japronto.protocol.cprotocol', 'japronto.protocol.creaper', 'japronto.router.cmatcher', 'japronto.response.cresponse'] } def get_parser(): argparser = argparse.ArgumentParser('build') argparser.add_argument( '-d', dest='debug', const=True, action='store_const', default=False) argparser.add_argument( '--sanitize', dest='sanitize', const=True, action='store_const', default=False) argparser.add_argument( '--profile-generate', dest='profile_generate', const=True, action='store_const', default=False) argparser.add_argument('--dest', dest='dest', default='src') argparser.add_argument( '--profile-use', dest='profile_use', const=True, action='store_const', default=False) argparser.add_argument( '-flto', dest='flto', const=True, action='store_const', default=False) argparser.add_argument( '--profile-clean', dest='profile_clean', const=True, action='store_const', default=False) argparser.add_argument( '--disable-reaper', dest='enable_reaper', const=False, action='store_const', default=True) argparser.add_argument( '--disable-response-cache', dest='enable_response_cache', const=False, action='store_const', default=True) argparser.add_argument( '--coverage', dest='coverage', const=True, action='store_const', default=False) argparser.add_argument('-O1', dest='optimization', const='1', action='store_const') argparser.add_argument('-O2', dest='optimization', const='2', action='store_const') argparser.add_argument('-O3', dest='optimization', const='3', action='store_const') argparser.add_argument('-Os', dest='optimization', const='s', action='store_const') argparser.add_argument('-native', dest='native', const=True, action='store_const', default=False) argparser.add_argument('--path', dest='path') argparser.add_argument('--extra-compile', dest='extra_compile', default='') argparser.add_argument('--kit', dest='kit') return argparser def get_platform(): argparser = get_parser() args = argparser.parse_args([]) system = BuildSystem(args, relative_source=True) ext_modules = system.discover_extensions() ext_modules = [e for e in ext_modules if e.name in kits['platform']] print({e.name: e.sources for e in ext_modules}) return ext_modules class custom_build_ext(build_ext): def build_extensions(self): if self.compiler.compiler_type == 'unix': for ext in self.extensions: if not ext.extra_compile_args: ext.extra_compiler_args = [] extra_compile_args = ['-std=c99', '-UNDEBUG', '-D_GNU_SOURCE'] if self.compiler.compiler_so[0].startswith('gcc') and sys.platform != 'darwin': extra_compile_args.append('-frecord-gcc-switches') ext.extra_compile_args.extend(extra_compile_args) compile_c( self.compiler, 'src/picohttpparser/picohttpparser.c', 'src/picohttpparser/ssepicohttpparser.o', options={'unix': ['-msse4.2']}) compile_c( self.compiler, 'src/picohttpparser/picohttpparser.c', 'src/picohttpparser/picohttpparser.o') build_ext.build_extensions(self) def compile_c(compiler, cfile, ofile, *, options=None): if not options: options = {} options = options.get(compiler.compiler_type, []) cmd = [*compiler.compiler_so, *options, '-c', '-o', ofile, cfile] print("building '{}'".format(ofile)) print(' '.join(cmd)) subprocess.check_call(cmd) def main(): argparser = get_parser() args = argparser.parse_args(sys.argv[1:]) if args.profile_clean: profile_clean() return distutils.log.set_verbosity(1) system = BuildSystem(args) if args.path: ext_modules = [system.get_extension_by_path(args.path)] else: ext_modules = system.discover_extensions() if args.kit: ext_modules = [e for e in ext_modules if e.name in kits[args.kit]] def add_args(arg_name, values, append=True): for ext_module in ext_modules: arg_value = getattr(ext_module, arg_name) or [] if append: arg_value.extend(values) else: newvalues = list(values) newvalues.extend(arg_value) arg_value = newvalues setattr(ext_module, arg_name, arg_value) def append_compile_args(*values): add_args('extra_compile_args', values) def append_link_args(*values): add_args('extra_link_args', values) def prepend_libraries(*values): add_args('libraries', values, append=False) if args.native: append_compile_args('-march=native') if args.optimization: append_compile_args('-O' + args.optimization) if args.debug: append_compile_args('-g3', '-O0', '-Wp,-U_FORTIFY_SOURCE') if args.sanitize: append_compile_args('-g3', '-fsanitize=address', '-fsanitize=undefined', '-fno-common', '-fno-omit-frame-pointer') prepend_libraries('asan', 'ubsan') if args.profile_generate: append_compile_args('--profile-generate') append_link_args('-lgcov') if args.profile_use: for ext_module in ext_modules: if ext_module.name in ('parser.cparser', 'pipeline.cpipeline'): continue ext_module.extra_compile_args.append('--profile-use') if args.flto: append_compile_args('-flto') append_link_args('-flto') if args.coverage: append_compile_args('--coverage') append_link_args('-lgcov') if args.extra_compile: append_compile_args(args.extra_compile) ext_modules = [e for e in ext_modules if system.should_rebuild(e)] if not ext_modules: return dist = Distribution(dict(ext_modules=ext_modules)) prune(args.dest) cmd = custom_build_ext(dist) cmd.build_lib = os.path.join(args.dest, '.build/lib') cmd.build_temp = os.path.join(args.dest, '.build/temp') cmd.finalize_options() try: cmd.run() except CompileError: sys.exit(1) symlink_python_files(args.dest) for ext_module in ext_modules: os.makedirs(system.dest_folder(ext_module.name), exist_ok=True) shutil.copy( cmd.get_ext_fullpath(ext_module.name), system.dest_folder(ext_module.name)) for ext_module in ext_modules: with open(system.build_toml(ext_module.name), 'w') as f: build_info = { 'extra_compile_args': ext_module.extra_compile_args, 'extra_link_args': ext_module.extra_link_args, 'define_macros': dict(ext_module.define_macros), 'sources': ext_module.sources } pytoml.dump(build_info, f) if __name__ == '__main__': main() ================================================ FILE: cases/__init__.py ================================================ from collections import namedtuple import glob import os.path import pytoml import pytest testcase_fields = 'data,method,path,version,headers,body,error,disconnect' HttpTestCase = namedtuple('HTTPTestCase', testcase_fields) def parse_casesel(suite, casesel): for casespec in casesel.split('+'): *transforms, case = casespec.split(':') if case.endswith('!'): transforms.append('!') case = case[:-1] case = suite[case] for transform in reversed(transforms): func, *args = transform.split() case = transorm_dict[func](case, *args) yield case def parametrize_cases(suite, *args): suite = suites[suite] cases_list = [ list(parse_casesel(suite, sel)) for sel in args] return pytest.mark.parametrize('cases', cases_list, ids=args) def load_casefile(path): result = {} with open(path) as casefile: cases = pytoml.load(casefile) for case_name, case_data in cases.items(): case_data['data'] = case_data['data'].encode('utf-8') case_data['body'] = case_data['body'].encode('utf-8') \ if 'body' in case_data else None case_data['disconnect'] = False case = HttpTestCase._make( case_data.get(f) for f in testcase_fields.split(',')) result[case_name] = case return result def load_cases(): cases = {} for filename in glob.glob('cases/*.toml'): suite_name, _ = os.path.splitext(os.path.basename(filename)) cases[suite_name] = load_casefile(filename) return cases def keep_alive(case): headers = case.headers.copy() headers['Connection'] = 'keep-alive' # if case.body is not None \ # and headers.get('Transfer-Encoding', 'identity') == 'identity': # headers['Content-Length'] = str(len(case.body)) return update_case(case, headers) def close(case): headers = case.headers.copy() headers['Connection'] = 'close' return update_case(case, headers) def should_keep_alive(case): return case.headers.get( 'Connection', 'close' if case.version == '1.0' else 'keep-alive') == 'keep-alive' def set_error(case, error): return update_case(case, error=error) def disconnect(case): return update_case(case, disconnect=True) def update_case(case, headers=False, error=False, disconnect=None): data = False if headers: data = bytearray() status = case.method + ' ' + case.path + ' HTTP/' + case.version + '\r\n' data += status.encode('ascii') for name, value in headers.items(): data += name.encode('ascii') + b': ' + value.encode('latin1') + b'\r\n' data += b'\r\n' if case.body: data += case.body headers = headers or case.headers data = data or case.data error = error or case.error disconnect = disconnect if disconnect is not None else case.disconnect return case._replace( headers=headers, error=error, disconnect=disconnect, data=bytes(data)) transorm_dict = { 'keep': keep_alive, 'close': close, 'e': set_error, '!': disconnect } suites = load_cases() globals().update(suites) ================================================ FILE: cases/base.toml ================================================ [10msg] data = """\ POST \ /wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg \ HTTP/1.0\r\n\ HOST: www.kittyhell.com\r\n\ User-Agent: Mozilla/5.0 \ (Macintosh; U; Intel Mac OS X 10.6; ja-JP-mac; rv:1.9.2.3) \ Gecko/20100401 Firefox/3.6.3 Pathtraq/0.9\r\n\ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n\ Accept-Language: ja,en-us;q=0.7,en;q=0.3\r\n\ Accept-Encoding: gzip,deflate\r\n\ Accept-Charset: Shift_JIS,utf-8;q=0.7,*;q=0.7\r\n\ Keep-Alive: 115\r\n\ Cookie: wp_ozh_wsa_visits=2; \ wp_ozh_wsa_visit_lasttime=xxxxxxxxxx; \ __utma=xxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.x; \ __utmz=xxxxxxxxx.xxxxxxxxxx.x.x.utmccn=(referral)|utmcsr=reader.livedoor.com|utmcct=/reader/|utmcmd=referral\r\n\ Content-Length: 11\r\n\ \r\n\ Hello there\ """ method = "POST" path = "/wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg" version = "1.0" body = "Hello there" [10msg.headers] Host = "www.kittyhell.com" User-Agent = """Mozilla/5.0 \ (Macintosh; U; Intel Mac OS X 10.6; ja-JP-mac; rv:1.9.2.3) \ Gecko/20100401 Firefox/3.6.3 Pathtraq/0.9""" Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" Accept-Language = "ja,en-us;q=0.7,en;q=0.3" Accept-Encoding = "gzip,deflate" Accept-Charset = "Shift_JIS,utf-8;q=0.7,*;q=0.7" Keep-Alive = "115" Cookie = """wp_ozh_wsa_visits=2; \ wp_ozh_wsa_visit_lasttime=xxxxxxxxxx; \ __utma=xxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.x; \ __utmz=xxxxxxxxx.xxxxxxxxxx.x.x.utmccn=(referral)|utmcsr=reader.livedoor.com|utmcct=/reader/|utmcmd=referral""" Content-Length = "11" [10get] data = "GET / HTTP/1.0\r\nHost: www.example.com\r\n\r\n" method = "GET" path = "/" version = "1.0" headers = { Host = "www.example.com" } [10malformed_headers1] data = "GET / HTTP 1.0" error = "malformed_headers" [10malformed_headers2] data = "GET / HTTP/2" error = "malformed_headers" [10incomplete_headers] data = "GET / HTTP/1.0\r\nH" error = "incomplete_headers" [11get] data = "GET /index.html HTTP/1.1\r\n\r\n" method = "GET" path = "/index.html" version = "1.1" headers = {} # body = null [11getmsg] data = """\ GET /body HTTP/1.1\r\n\ Content-Length: 12\r\n\ \r\n\ Hello World!\ """ method = "GET" path = "/body" version = "1.1" headers = { Content-Length = "12" } body = "Hello World!" [11msg] data = "POST /login HTTP/1.1\r\nContent-Length: 5\r\n\r\nHello" method = "POST" path = "/login" version = "1.1" headers = { Content-Length = "5" } body = "Hello" [11msgzero] data = "POST /zero HTTP/1.1\r\nContent-Length: 0\r\n\r\n" method = "POST" path = "/zero" version = "1.1" headers = { Content-Length = "0" } body = "" [11clincomplete_headers] data = """\ POST / HTTP/1.1\r\n\ Content-Length: 3\r\n\ I""" error = "incomplete_headers" [11clincomplete_body] data = "POST / HTTP/1.1\r\nContent-Length: 5\r\n\r\nI" method = "POST" path = "/" version = "1.1" headers = { Content-Length = "5" } error = "incomplete_body" [11clinvalid1] data = "POST / HTTP/1.1\r\nContent-Length: asd\r\n\r\n" method = "POST" path = "/" version = "1.1" headers = { Content-Length = "asd" } error = "invalid_headers" [11clinvalid2] data = "POST / HTTP/1.1\r\nContent-Length: +5\r\n\r\n" method = "POST" path = "/" version = "1.1" headers = { Content-Length = "+5" } error = "invalid_headers" [11clinvalid3] data = "POST / HTTP/1.1\r\nContent-Length: -5\r\n\r\n" method = "POST" path = "/" version = "1.1" headers = { Content-Length = "+5" } error = "invalid_headers" [11clinvalid4] data = "POST / HTTP/1.1\r\nContent-Length: 4f\r\n\r\n" method = "POST" path = "/" version = "1.1" headers = { Content-Length = "4f" } error = "invalid_headers" [11clinvalid5] data = "POST / HTTP/1.1\r\nContent-Length: \r\n\r\n" method = "POST" path = "/" version = "1.1" headers = { Content-Length = "" } error = "invalid_headers" [11chunked1] data = """\ POST /chunked HTTP/1.1\r\n\ Transfer-Encoding: chunked\r\n\ \r\n\ 4\r\n\ Wiki\r\n\ 5\r\n\ pedia\r\n\ E\r\n in\r\n\ \r\n\ chunks.\r\n\ 0\r\n\ \r\n\ """ method = "POST" path = "/chunked" version = "1.1" headers = { Transfer-Encoding = "chunked" } body = "Wikipedia in\r\n\r\nchunks." [11chunkedzero] data = """ PUT /zero HTTP/1.1\r\n\ Transfer-Encoding: chunked\r\n\ \r\n\ 0\r\n\ \r\n\ """ method = "PUT" path = "/zero" version = "1.1" headers = { Transfer-Encoding = "chunked" } body = "" [11chunked2] data = """\ POST /chunked HTTP/1.1\r\n\ Transfer-Encoding: chunked\r\n\ \r\n\ 1;token=123;x=3\r\n\ r\r\n\ 0\r\n\ \r\n\ """ method = "POST" path = "/chunked" version = "1.1" headers = { Transfer-Encoding = "chunked" } body = "r" [11chunked3] data = """\ POST / HTTP/1.1\r\n\ Transfer-Encoding: chunked\r\n\ \r\n\ 000002\r\n\ ab\r\n\ 0;q=1\r\n\ This: is trailer header\r\n\ \r\n\ """ method = "POST" path = "/" version = "1.1" headers = { Transfer-Encoding = "chunked" } body = "ab" [11chunkedincomplete_body] data = "POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n10\r\nasd" method = "POST" path = "/" version = "1.1" headers = { Transfer-Encoding = "chunked" } error = "incomplete_body" [11chunkedmalformed_body] data = "POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n1x\r\nhello" method = "POST" path = "/" version = "1.1" headers = { Transfer-Encoding = "chunked" } error = "malformed_body" ================================================ FILE: cases/websites.toml ================================================ [github] data = """\ GET / HTTP/1.1\r\n\ Host: github.com\r\n\ User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0\r\n\ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n\ Accept-Language: en-US,en;q=0.5\r\n\ Accept-Encoding: gzip, deflate, br\r\n\ Cookie: _octo=GHx; logged_in=yes; _ga=GAx; user_session=x; dotcom_user=x; __Host-user_session_same_site=x; _gh_sess=x; tz=America%2FSao_Paulo; _gat=x\r\n\ Connection: keep-alive\r\n\ Upgrade-Insecure-Requests: 1\r\n\ Pragma: no-cache\r\n\ Cache-Control: no-cache\r\n\ \r\n\ """ [google] data = """\ GET / HTTP/1.1\r\n\ Host: google.com\r\n\ Connection: keep-alive\r\n\ Upgrade-Insecure-Requests: 1\r\n\ User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36\r\n\ X-Client-Data: Qwerty\r\n\ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n\ Accept-Encoding: gzip, deflate, sdch\r\n\ Accept-Language: en-US,en;q=0.8,es;q=0.6\r\n\ Cookie: NID=x; SID=x; HSID=x; APISID=x\r\n\ \r\n\ """ [amazon] data = """\ GET / HTTP/1.1\r\n\ Host: www.amazon.com\r\n\ Connection: keep-alive\r\n\ Upgrade-Insecure-Requests: 1\r\n\ User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.50 Safari/537.36 OPR/41.0.2353.23 (Edition beta)\r\n\ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n\ Accept-Encoding: gzip, deflate, lzma, sdch, br\r\n\ Accept-Language: en-US,en;q=0.8\r\n\ \r\n\ """ [xkcd] data = """\ GET /comics/mushrooms.png HTTP/1.1\r\n\ Host: imgs.xkcd.com\r\n\ Connection: keep-alive\r\n\ Upgrade-Insecure-Requests: 1\r\n\ User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.50 Safari/537.36 OPR/41.0.2353.23 (Edition beta)\r\n\ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n\ Accept-Encoding: gzip, deflate, lzma, sdch\r\n\ Accept-Language: en-US,en;q=0.8\r\n\ \r\n\ """ [4chan] data = """\ GET /image/favicon.ico HTTP/1.1\r\n\ Host: s.4cdn.org\r\n\ Connection: keep-alive\r\n\ User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36\r\n\ Accept: */*\r\n\ Referer: http://www.4chan.org/\r\n\ Accept-Encoding: gzip, deflate, sdch\r\n\ Accept-Language: en-US,en;q=0.8,es;q=0.6\r\n\ \r\n\ """ ================================================ FILE: conftest.py ================================================ import subprocess import sys import os import shutil builds = [] coverages = set() def add_build(mark): global builds args, kwargs = list(mark.args), mark.kwargs.copy() kwargs.pop('coverage', None) cfg = args, kwargs if cfg not in builds: builds.append(cfg) def execute_builds(): common_options = ['--coverage', '-d', '--sanitize'] for args, kwargs in builds: build_options = args[:] build_options.extend(['--dest', kwargs.get('dest', '.test')]) if 'kit' not in kwargs: build_options.extend(['--kit', 'platform']) build_options.extend(common_options) print('Executing build', *build_options) subprocess.check_call([sys.executable, 'build.py', *build_options]) def add_coverage(mark): dest = mark.kwargs.get('dest', '.test') coverages.add(dest) def setup_coverage(): if coverages: print('Setting up C coverage for', *coverages) for dest in coverages: subprocess.check_call([ 'lcov', '--base-directory', '.', '--directory', dest + '/.build/temp', '--zerocounters', '-q']) def make_coverage(): for dest in coverages: try: os.unlink(dest + '/coverage.info') except FileNotFoundError: pass subprocess.check_call([ 'lcov', '--base-directory', '.', '--directory', dest + '/.build/temp', '-c', '-o', dest + '/coverage.info', '-q']) subprocess.check_call([ 'lcov', '--remove', dest + '/coverage.info', '/usr*', '-o', 'coverage.info', '-q']) try: shutil.rmtree(dest + '/coverage_report') except FileNotFoundError: pass subprocess.check_call([ 'genhtml', '-o', dest + '/coverage_report', dest + '/coverage.info', '-q' ]) print('C coverage report saved in', dest + '/coverage_report/index.html') def pytest_itemcollected(item): needs_build = item.get_closest_marker('needs_build') if needs_build: add_build(needs_build) if needs_build and needs_build.kwargs.get('coverage'): add_coverage(needs_build) def pytest_collection_modifyitems(config, items): execute_builds() setup_coverage() def pytest_unconfigure(): make_coverage() ================================================ FILE: do_wrk.py ================================================ import argparse import sys import asyncio as aio import os from asyncio.subprocess import PIPE, STDOUT import statistics import uvloop import psutil from misc import cpu from misc import buggers def run_wrk(loop, endpoint=None): endpoint = endpoint or 'http://localhost:8080' wrk_fut = aio.create_subprocess_exec( './wrk', '-t', '1', '-c', '100', '-d', '2', '-s', 'misc/pipeline.lua', endpoint, stdout=PIPE, stderr=STDOUT) wrk = loop.run_until_complete(wrk_fut) lines = [] while 1: line = loop.run_until_complete(wrk.stdout.readline()) if line: line = line.decode('utf-8') lines.append(line) if line.startswith('Requests/sec:'): rps = float(line.split()[-1]) else: break retcode = loop.run_until_complete(wrk.wait()) if retcode != 0: print('\r\n'.join(lines)) return rps def cpu_usage(p): return p.cpu_percent() + sum(c.cpu_percent() for c in p.children()) def connections(process): return len( set(c.fd for c in process.connections()) | set(c.fd for p in process.children() for c in p.connections())) def memory(p): return p.memory_percent('uss') \ + sum(c.memory_percent('uss') for c in p.children()) if __name__ == '__main__': buggers.silence() loop = uvloop.new_event_loop() argparser = argparse.ArgumentParser('do_wrk') argparser.add_argument('-s', dest='server', default='') argparser.add_argument('-e', dest='endpoint') argparser.add_argument('--pid', dest='pid', type=int) argparser.add_argument( '--no-cpu', dest='cpu_change', default=True, action='store_const', const=False) args = argparser.parse_args(sys.argv[1:]) if args.cpu_change: cpu.change('userspace', cpu.min_freq()) cpu.dump() aio.set_event_loop(loop) if not args.endpoint: os.putenv('PYTHONPATH', 'src') server_fut = aio.create_subprocess_exec( 'python', 'benchmarks/japronto/micro.py', *args.server.split()) server = loop.run_until_complete(server_fut) os.unsetenv('PYTHONPATH') if not args.endpoint: process = psutil.Process(server.pid) elif args.pid: process = psutil.Process(args.pid) else: process = None cpu_p = 100 while cpu_p > 5: cpu_p = psutil.cpu_percent(interval=1) print('CPU usage in 1 sec:', cpu_p) results = [] cpu_usages = [] process_cpu_usages = [] mem_usages = [] conn_cnt = [] if process: cpu_usage(process) for _ in range(10): results.append(run_wrk(loop, args.endpoint)) cpu_usages.append(psutil.cpu_percent()) if process: process_cpu_usages.append(cpu_usage(process)) conn_cnt.append(connections(process)) mem_usages.append(round(memory(process), 2)) print('.', end='') sys.stdout.flush() if not args.endpoint: server.terminate() loop.run_until_complete(server.wait()) if args.cpu_change: cpu.change('ondemand') print() print('RPS', results) print('Mem', mem_usages) print('Conn', conn_cnt) print('Server', process_cpu_usages) print('System', cpu_usages) median = statistics.median_grouped(results) stdev = round(statistics.stdev(results), 2) p = round((stdev / median) * 100, 2) print('median:', median, 'stdev:', stdev, '%', p) ================================================ FILE: examples/1_hello/hello.py ================================================ from japronto import Application # Views handle logic, take request as a parameter and # return the Response object back to the client def hello(request): return request.Response(text='Hello world!') # The Application instance is a fundamental concept. # It is a parent to all the resources and all the settings # can be tweaked there. app = Application() # The Router instance lets you register your handlers and execute # them depending on the url path and methods. app.router.add_route('/', hello) # Finally, start our server and handle requests until termination is # requested. Enabling debug lets you see request logs and stack traces. app.run(debug=True) ================================================ FILE: examples/2_async/async.py ================================================ import asyncio from japronto import Application # This is a synchronous handler. def synchronous(request): return request.Response(text='I am synchronous!') # This is an asynchronous handler. It spends most of the time in the event loop. # It wakes up every second 1 to print and finally returns after 3 seconds. # This lets other handlers execute in the same processes while # from the point of view of the client it took 3 seconds to complete. async def asynchronous(request): for i in range(1, 4): await asyncio.sleep(1) print(i, 'seconds elapsed') return request.Response(text='3 seconds elapsed') app = Application() r = app.router r.add_route('/sync', synchronous) r.add_route('/async', asynchronous) app.run() ================================================ FILE: examples/3_router/router.py ================================================ from japronto import Application app = Application() r = app.router # Requests with the path set exactly to `/` and whatever method # will be directed here. def slash(request): return request.Response(text='Hello {} /!'.format(request.method)) r.add_route('/', slash) # Requests with the path set exactly to '/love' and the method # set exactly to `GET` will be directed here. def get_love(request): return request.Response(text='Got some love') r.add_route('/love', get_love, 'GET') # Requests with the path set exactly to '/methods' and the method # set to `POST` or `DELETE` will be directed here. def methods(request): return request.Response(text=request.method) r.add_route('/methods', methods, methods=['POST', 'DELETE']) # Requests with the path starting with `/params/` segment and followed # by two additional segments will be directed here. # Values of the additional segments will be stored inside `request.match_dict` # dictionary with keys taken from {} placeholders. A request to `/params/1/2` # would leave `match_dict` set to `{'p1': 1, 'p2': '2'}`. def params(request): return request.Response(text=str(request.match_dict)) r.add_route('/params/{p1}/{p2}', params) app.run() ================================================ FILE: examples/4_request/request.py ================================================ from json import JSONDecodeError from japronto import Application # Request line and headers. # This represents the part of a request that comes before message body. # Given an HTTP 1.1 `GET` request to `/basic?a=1` this would yield # `method` set to `GET`, `path` set to `/basic`, `version` set to `1.1` # `query_string` set to `a=1` and `query` set to `{'a': '1'}`. # Additionally if headers are sent they will be present in `request.headers` # dictionary. The keys are normalized to standard `Camel-Cased` convention. def basic(request): text = """Basic request properties: Method: {0.method} Path: {0.path} HTTP version: {0.version} Query string: {0.query_string} Query: {0.query}""".format(request) if request.headers: text += "\nHeaders:\n" for name, value in request.headers.items(): text += " {0}: {1}\n".format(name, value) return request.Response(text=text) # Message body # If there is a message body attached to a request (as in a case of `POST`) # the following attributes can be used to examine it. # Given a `POST` request with body set to `b'J\xc3\xa1'`, `Content-Length` header set # to `3` and `Content-Type` header set to `text/plain; charset=utf-8` this # would yield `mime_type` set to `'text/plain'`, `encoding` set to `'utf-8'`, # `body` set to `b'J\xc3\xa1'` and `text` set to `'Já'`. # `form` and `files` attributes are dictionaries respectively used for HTML forms and # HTML file uploads. The `json` helper property will try to decode `body` as a # JSON document and give you resulting Python data type. def body(request): text = """Body related properties: Mime type: {0.mime_type} Encoding: {0.encoding} Body: {0.body} Text: {0.text} Form parameters: {0.form} Files: {0.files} """.format(request) try: json = request.json except JSONDecodeError: pass else: text += "\nJSON:\n" text += str(json) return request.Response(text=text) # Miscellaneous # `route` will point to an instance of `Route` object representing # route chosen by router to handle this request. `hostname` and `port` # represent parsed `Host` header if any. `remote_addr` is the address of # a client or reverse proxy. If `keep_alive` is true the client requested to # keep the connection open after the response is delivered. `match_dict` contains # route placeholder values as documented in `2_router.md`. `cookies` contains # a dictionary of HTTP cookies if any. def misc(request): text = """Miscellaneous: Matched route: {0.route} Hostname: {0.hostname} Port: {0.port} Remote address: {0.remote_addr}, HTTP Keep alive: {0.keep_alive} Match parameters: {0.match_dict} """.strip().format(request) if request.cookies: text += "\nCookies:\n" for name, value in request.cookies.items(): text += " {0}: {1}\n".format(name, value) return request.Response(text=text) app = Application() app.router.add_route('/basic', basic) app.router.add_route('/body', body) app.router.add_route('/misc', misc) app.run() ================================================ FILE: examples/5_response/response.py ================================================ import random from http.cookies import SimpleCookie from japronto.app import Application # Providing just a text argument yields a `text/plain` response # encoded with `utf8` codec (charset set accordingly) def text(request): return request.Response(text='Hello world!') # You can override encoding by providing the `encoding` attribute. def encoding(request): return request.Response(text='Já pronto!', encoding='iso-8859-1') # You can also set a custom MIME type. def mime(request): return request.Response( mime_type="image/svg+xml", text=""" """) # Or serve binary data. `Content-Type` is set to `application/octet-stream` # automatically but you can always provide your own `mime_type`. def body(request): return request.Response(body=b'\xde\xad\xbe\xef') # There exist a shortcut `json` argument. This automatically encodes the # provided object as JSON and servers it with `Content-Type` set to # `application/json; charset=utf8` def json(request): return request.Response(json={'hello': 'world'}) # You can change the default 200 status `code` for another def code(request): return request.Response(code=random.choice([200, 201, 400, 404, 500])) # And of course you can provide custom `headers`. def headers(request): return request.Response( text='headers', headers={'X-Header': 'Value', 'Refresh': '5; url=https://xkcd.com/353/'}) # Or `cookies` by using Python standard library `http.cookies.SimpleCookie`. def cookies(request): cookies = SimpleCookie() cookies['hello'] = 'world' cookies['hello']['domain'] = 'localhost' cookies['hello']['path'] = '/' cookies['hello']['max-age'] = 3600 cookies['city'] = 'São Paulo' return request.Response(text='cookies', cookies=cookies) app = Application() router = app.router router.add_route('/text', text) router.add_route('/encoding', encoding) router.add_route('/mime', mime) router.add_route('/body', body) router.add_route('/json', json) router.add_route('/code', code) router.add_route('/headers', headers) router.add_route('/cookies', cookies) app.run() ================================================ FILE: examples/6_exceptions/exceptions.py ================================================ from japronto import Application, RouteNotFoundException # These are our custom exceptions we want to turn into 200 response. class KittyError(Exception): def __init__(self): self.greet = 'meow' class DoggieError(Exception): def __init__(self): self.greet = 'woof' # The two handlers below raise exceptions which will be turned # into 200 responses by the handlers registered later def cat(request): raise KittyError() def dog(request): raise DoggieError() # This handler raises ZeroDivisionError which doesn't have an error # handler registered so it will result in 500 Internal Server Error def unhandled(request): 1 / 0 app = Application() r = app.router r.add_route('/cat', cat) r.add_route('/dog', dog) r.add_route('/unhandled', unhandled) # These two are handlers for `Kitty` and `DoggyError`s. def handle_cat(request, exception): return request.Response(text='Just a kitty, ' + exception.greet) def handle_dog(request, exception): return request.Response(text='Just a doggie, ' + exception.greet) # You can also override default 404 handler if you want def handle_not_found(request, exception): return request.Response(code=404, text="Are you lost, pal?") # register all the error handlers so they are actually effective app.add_error_handler(KittyError, handle_cat) app.add_error_handler(DoggieError, handle_dog) app.add_error_handler(RouteNotFoundException, handle_not_found) app.run() ================================================ FILE: examples/7_extend/extend.py ================================================ from japronto import Application # This view accesses custom method host_startswith # and a custom property reversed_agent. Both are registered later. def extended_hello(request): if request.host_startswith('api.'): text = 'Hello ' + request.reversed_agent else: text = 'Hello stranger' return request.Response(text=text) # This view registers a callback, such callbacks are executed after handler # exits and the response is ready to be sent over the wire. def with_callback(request): def cb(r): print('Done!') request.add_done_callback(cb) return request.Response(text='cb') # This is a body for reversed_agent property def reversed_agent(request): return request.headers['User-Agent'][::-1] # This is a body for host_startswith method # Custom methods and properties always accept request # object. def host_startswith(request, prefix): return request.headers['Host'].startswith(prefix) app = Application() # Finally register the custom property and method # By default the names are taken from function names # unelss you provide `name` keyword parameter. app.extend_request(reversed_agent, property=True) app.extend_request(host_startswith) r = app.router r.add_route('/', extended_hello) r.add_route('/callback', with_callback) app.run() ================================================ FILE: examples/8_template/index.html ================================================ japronto

Hello World!

Behold, the power of japronto!

================================================ FILE: examples/8_template/template.py ================================================ # examples/8_template/template.py from japronto import Application from jinja2 import Template # A view can read HTML from a file def index(request): with open('index.html') as html_file: return request.Response(text=html_file.read(), mime_type='text/html') # A view could also return a raw HTML string def example(request): return request.Response(text='

Some HTML!

', mime_type='text/html') template = Template('

Hello {{ name }}!

') # A view could also return a rendered jinja2 template def jinja(request): return request.Response(text=template.render(name='World'), mime_type='text/html') # Create the japronto application app = Application() # Add routes to the app app.router.add_route('/', index) app.router.add_route('/example', example) app.router.add_route('/jinja2', jinja) # Start the server app.run(debug=True) ================================================ FILE: examples/todo_api/.gitignore ================================================ todo.sqlite ================================================ FILE: examples/todo_api/todo_api.py ================================================ import os.path import sqlite3 from functools import partial from japronto import Application def add_todo(request): cur = request.cursor todo = request.json["todo"] cur.execute("""INSERT INTO todos (todo) VALUES (?)""", (todo,)) last_id = cur.lastrowid cur.connection.commit() return request.Response(json={"id": last_id, "todo": todo}) def list_todos(request): cur = request.cursor cur.execute("""SELECT id, todo FROM todos""") todos = [{"id": id, "todo": todo} for id, todo in cur] return request.Response(json={"results": todos}) def show_todo(request): cur = request.cursor id = int(request.match_dict['id']) cur.execute("""SELECT id, todo FROM todos WHERE id = ?""", (id,)) todo = cur.fetchone() if not todo: return request.Response(code=404, json={"error": "not found"}) todo = {"id": todo[0], "todo": todo[1]} return request.Response(json=todo) def delete_todo(request): cur = request.cursor id = int(request.match_dict['id']) cur.execute("""DELETE FROM todos WHERE id = ?""", (id,)) if not cur.rowcount: return request.Response(code=404, json={"error": "not found"}) cur.connection.commit() return request.Response(json={}) DB_FILE = os.path.abspath( os.path.join(os.path.dirname(__file__), 'todo.sqlite')) db_connect = partial(sqlite3.connect, DB_FILE) def maybe_create_schema(): db = db_connect() db.execute(""" CREATE TABLE IF NOT EXISTS todos (id INTEGER PRIMARY KEY, todo TEXT)""") db.close() maybe_create_schema() app = Application() def cursor(request): def done_cb(request): request.extra['conn'].close() if 'conn' not in request.extra: request.extra['conn'] = db_connect() request.add_done_callback(done_cb) return request.extra['conn'].cursor() app.extend_request(cursor, property=True) router = app.router router.add_route('/todos', list_todos, method='GET') router.add_route('/todos/{id}', show_todo, method='GET') router.add_route('/todos/{id}', delete_todo, method='DELETE') router.add_route('/todos', add_todo, method='POST') app.run() ================================================ FILE: integration_tests/__init__.py ================================================ ================================================ FILE: integration_tests/common.py ================================================ import os import subprocess import ctypes.util import sys import time import psutil def start_server(script, *, stdout=None, path=None, sanitize=True, wait=True, return_process=False, buffer=False): if not isinstance(script, list): script = [script] if path: os.putenv('PYTHONPATH', path) if sanitize: os.putenv('LD_PRELOAD', ctypes.util.find_library('asan')) os.putenv('LSAN_OPTIONS', 'suppressions=misc/suppr.txt') if not buffer: os.putenv('PYTHONUNBUFFERED', '1') server = subprocess.Popen([sys.executable, *script], stdout=stdout) if not buffer: os.unsetenv('PYTHONUNBUFFERED') if sanitize: os.unsetenv('LSAN_OPTIONS') os.unsetenv('LD_PRELOAD') if path: os.unsetenv('PYTHONPATH') process = psutil.Process(server.pid) if wait: # wait until the server socket is open while 1: assert server.poll() is None conn_num = len(process.connections()) for child in process.children(): conn_num += len(child.connections()) if conn_num: break time.sleep(.001) assert server.poll() is None if return_process: return server, process else: return server ================================================ FILE: integration_tests/drain.py ================================================ import asyncio from japronto.app import Application def slash(request): return request.Response() async def sleep(request): await asyncio.sleep(int(request.match_dict['sleep'])) return request.Response() app = Application() r = app.router r.add_route('/', slash) r.add_route('/sleep/{sleep}', sleep) if __name__ == '__main__': app.run() ================================================ FILE: integration_tests/dump.py ================================================ import base64 import asyncio from japronto.app import Application class ForcedException(Exception): pass def dump(request, exception=None): if not exception and 'Force-Raise' in request.headers: raise ForcedException(request.headers['Force-Raise']) body = request.body if body is not None: body = base64.b64encode(body).decode('ascii') result = { "method": request.method, "path": request.path, "query_string": request.query_string, "headers": request.headers, "match_dict": request.match_dict, "body": body, "route": request.route and request.route.pattern } if exception: result['exception'] = { "type": type(exception).__name__, "args": ", ".join(str(a) for a in exception.args) } return request.Response(code=500 if exception else 200, json=result) async def adump(request): sleep = float(request.query.get('sleep', 0)) await asyncio.sleep(sleep) return dump(request) app = Application() r = app.router r.add_route('/dump/{p1}/{p2}', dump) r.add_route('/dump1/{p1}/{p2}', dump) r.add_route('/dump2/{p1}/{p2}', dump) r.add_route('/async/dump/{p1}/{p2}', adump) r.add_route('/async/dump1/{p1}/{p2}', adump) r.add_route('/async/dump2/{p1}/{p2}', adump) app.add_error_handler(None, dump) if __name__ == '__main__': app.run() ================================================ FILE: integration_tests/experiments.py ================================================ import pytest @pytest.fixture(autouse=True) def my_fix(request): print('auto') pytest.set_trace() @pytest.fixture(params=[1, 2]) def size_k(request): return request.param # @pytest.fixture(autouse=True, scope='module') # def a(): # print('a') # @pytest.fixture(scope='function', params=[1,2]) # def fix(): # print('> fix') # yield 3 # print('< fix') # def test(my_fix): # print(test) def test1(size_k): print(test1) ================================================ FILE: integration_tests/generators.py ================================================ from integration_tests import strategies as st from hypothesis.strategies import SearchStrategy def generate_body(body, size_k): if size_k and body: if isinstance(body, list): length = sum(len(b) for b in body) else: length = len(body) body = body * ((size_k * 1024) // length + 1) return body def makeval(v, default_st, default=None): if isinstance(v, SearchStrategy): return v.example() if v is True: return default_st.example() if v is not None: return v return default def print_request(request): body = request['body'] if body: if isinstance(body, list): body = '{} chunks'.format(len(body)) else: if len(body) > 32: body = body[:32] + b'...' print(repr(request['method']), repr(request['path']), repr(request['query_string']), body) def generate_request(*, method=None, path=None, query_string=None, headers=None, body=None, size_k=None): request = {} request['method'] = makeval(method, st.method, 'GET') request['path'] = makeval(path, st.path, '/') request['query_string'] = makeval(query_string, st.query_string) request['headers'] = makeval(headers, st.headers) request['body'] = generate_body(makeval(body, st.body), size_k) return request def generate_combinations(reverse=False): props = ['method', 'path', 'query_string', 'headers', 'body'] sizes = [None, 8, 32, 64] if reverse: props = reversed(props) sizes = reversed(sizes) for prop in props: if prop == 'body': for size_k in sizes: yield {'body': True, 'size_k': size_k} else: yield {prop: True} def send_requests(conn, number, **kwargs): for _ in range(number): request = generate_request(**kwargs) print_request(request) conn.request(**request) conn.getresponse() ================================================ FILE: integration_tests/longrun.py ================================================ import subprocess import sys import signal import atexit import os import time sys.path.insert(0, '.') import integration_tests.common # noqa import integration_tests.generators # noqa from misc import client # noqa def setup(): subprocess.check_call([ sys.executable, 'build.py', '--dest', '.test/longrun', '--kit', 'platform', '--disable-response-cache']) os.putenv('MALLOC_TRIM_THRESHOLD_', '0') server = integration_tests.common.start_server( 'integration_tests/dump.py', path='.test/longrun', sanitize=False) os.unsetenv('MALLOC_TRIM_THRESHOLD_') os.makedirs('.collector', exist_ok=True) os.putenv('COLLECTOR_FILE', '.collector/{}.json'.format(server.pid)) collector = subprocess.Popen([ sys.executable, 'misc/collector.py', str(server.pid)]) os.unsetenv('COLLECTOR_FILE') def cleanup(*args): try: server.terminate() assert server.wait() == 0 finally: atexit.unregister(cleanup) atexit.register(cleanup) signal.signal(signal.SIGINT, cleanup) def run(): time.sleep(2) for reverse in [True, False]: for combination in integration_tests.generators.generate_combinations( reverse=reverse): conn = client.Connection('localhost:8080') time.sleep(2) integration_tests.generators.send_requests( conn, 200, **combination) time.sleep(2) conn.close() time.sleep(2) def main(): setup() run() if __name__ == '__main__': main() ================================================ FILE: integration_tests/noleak.py ================================================ import sys from japronto.app import Application prop = sys.argv[1] if prop == 'method': def noleak(request): return request.Response(text=request.method) elif prop == 'path': def noleak(request): return request.Response(text=request.path) elif prop == 'match_dict': def noleak(request): return request.Response(json=request.match_dict) elif prop == 'query_string': def noleak(request): return request.Response(text=request.query_string) elif prop == 'headers': def noleak(request): return request.Response(json=request.headers) elif prop == 'body': def noleak(request): return request.Response(body=request.body) elif prop == 'keep_alive': def noleak(request): return request.Response(text=str(request.keep_alive)) elif prop == 'route': def noleak(request): return request.Response(text=str(request.route)) app = Application() r = app.router r.add_route('/noleak/{p1}/{p2}', noleak) if __name__ == '__main__': app.run() ================================================ FILE: integration_tests/reaper.py ================================================ import sys from japronto.app import Application reaper_settings = { 'check_interval': int(sys.argv[1]), 'idle_timeout': int(sys.argv[2]) } app = Application(reaper_settings=reaper_settings) if __name__ == '__main__': app.run() ================================================ FILE: integration_tests/strategies.py ================================================ import string import re from hypothesis import strategies as st sampled_from = st.sampled_from fixed_dictionaries = st.fixed_dictionaries lists = st.lists builds = st.builds integers = st.integers _method_alphabet = ''.join(chr(x) for x in range(33, 256) if x != 127) method = st.text(_method_alphabet, min_size=1) _path_alphabet = st.characters( blacklist_characters='?', blacklist_categories=['Cs']) path = st.text(_path_alphabet).map(lambda x: '/' + x) _param_alphabet = st.characters( blacklist_characters='/?', blacklist_categories=['Cs']) param = st.text(_param_alphabet, min_size=1) query_string = st.one_of(st.text(), st.none()) _name_alphabet = string.digits + string.ascii_letters + '!#$%&\'*+-.^_`|~' _names = st.text(_name_alphabet, min_size=1).map(lambda x: 'X-' + x) _value_alphabet = ''.join(chr(x) for x in range(ord(' '), 256) if x != 127) _is_illegal_value = re.compile(r'\n(?![ \t])|\r(?![ \t\n])').search _values = st.text(_value_alphabet, min_size=1) \ .filter(lambda x: not _is_illegal_value(x)).map(lambda x: x.strip()) headers = st.lists(st.tuples(_names, _values), max_size=48) identity_body = st.one_of(st.binary(), st.none()) chunked_body = st.lists(st.binary(min_size=24)) body = st.one_of(st.binary(), st.none(), chunked_body) ================================================ FILE: integration_tests/test_drain.py ================================================ import pytest import subprocess import time from misc import client import integration_tests.common pytestmark = pytest.mark.needs_build @pytest.fixture(scope='function') def server(): return integration_tests.common.start_server( 'integration_tests/drain.py', stdout=subprocess.PIPE, path='.test') @pytest.fixture(scope='function') def server_terminate(server): def terminate(): server.terminate() assert server.wait() == 0 stdout = server.stdout.read() return [l.decode('utf-8').strip() for l in stdout.splitlines()] yield terminate server.terminate() @pytest.fixture(scope='function') def connect(): connections = [] def _connect(): conn = client.Connection('localhost:8080') conn.maybe_connect() connections.append(conn) return conn yield _connect for c in connections: c.close() def test_no_connections(server_terminate): lines = server_terminate() assert lines[-1] == 'Termination request received' @pytest.mark.parametrize('num', range(1, 5)) def test_unclosed_connections(num, connect, server_terminate): for _ in range(num): connect() lines = server_terminate() assert lines[-1] == '{} idle connections closed immediately'.format(num) @pytest.mark.parametrize('num', range(1, 5)) def test_closed_connections(num, connect, server_terminate): for _ in range(num): con = connect() con.close() lines = server_terminate() assert lines[-1] == 'Termination request received' @pytest.mark.parametrize('num', range(1, 5)) def test_unclosed_requests(num, connect, server_terminate): for _ in range(num): con = connect() con.putrequest('GET', '/') con.endheaders() lines = server_terminate() assert lines[-1] == '{} idle connections closed immediately'.format(num) @pytest.mark.parametrize('num', range(1, 5)) def test_closed_requests(num, connect, server_terminate): for _ in range(num): con = connect() con.putrequest('GET', '/') con.endheaders() con.getresponse() con.close() lines = server_terminate() assert lines[-1] == 'Termination request received' @pytest.mark.parametrize('num', range(1, 3)) def test_pipelined(num, connect, server_terminate): connections = [] for _ in range(num): con = connect() connections.append(con) con.putrequest('GET', '/sleep/1') con.endheaders() lines = server_terminate() assert '{} connections busy, read-end closed'.format(num) in lines assert not any(l.startswith('Forcefully killing') for l in lines) assert all(c.getresponse().status == 200 for c in connections) @pytest.mark.parametrize('num', range(1, 3)) def test_pipelined_timeout(num, connect, server_terminate): connections = [] for _ in range(num): con = connect() connections.append(con) con.putrequest('GET', '/sleep/10') con.endheaders() lines = server_terminate() assert '{} connections busy, read-end closed'.format(num) in lines assert 'Forcefully killing {} connections'.format(num) in lines assert all(c.getresponse().status == 503 for c in connections) def test_refuse(connect, server): con = connect() con.putrequest('GET', '/sleep/10') con.endheaders() server.terminate() # give time for the signal to propagate time.sleep(1) with pytest.raises(ConnectionRefusedError): con = connect() assert server.wait() == 0 ================================================ FILE: integration_tests/test_noleak.py ================================================ import pytest from misc import client import integration_tests.common pytestmark = pytest.mark.needs_build( '--extra-compile=-DPROTOCOL_TRACK_REFCNT=1', dest='.test/noleak') @pytest.fixture(scope='function') def server(request): arg = request.node.get_marker('arg').args[0] server = integration_tests.common.start_server([ 'integration_tests/noleak.py', arg], path='.test/noleak') yield server server.terminate() server.wait() == 0 @pytest.fixture(scope='function') def connection(server): conn = client.Connection('localhost:8080') yield conn conn.close() @pytest.mark.arg('method') def test_method(connection): methods = ['GET', 'POST', 'PATCH', 'DELETE', 'PUT'] for method in methods: connection.putrequest(method, '/noleak/1/2') connection.endheaders() response = connection.getresponse() assert response.status == 200 @pytest.mark.arg('path') def test_path(connection): paths = ['/noleak/1/2', '/noleak/3/4', '/noleak/5/4', '/noleak/6/7'] for path in paths: connection.putrequest('GET', path) connection.endheaders() response = connection.getresponse() assert response.status == 200 @pytest.mark.arg('match_dict') def test_match_dict(connection): paths = ['/noleak/1/2', '/noleak/3/4', '/noleak/5/4', '/noleak/6/7'] for path in paths: connection.putrequest('GET', path) connection.endheaders() response = connection.getresponse() assert response.status == 200 @pytest.mark.arg('query_string') def test_query_string(connection): query_strings = ['?', None, '?a', None, '?', None, '?b', None, '?'] for query_string in query_strings: connection.putrequest('GET', '/noleak/1/2', query_string) connection.endheaders() response = connection.getresponse() assert response.status == 200 @pytest.mark.arg('headers') def test_headers(connection): header_list = [{}, {"X-a": "b"}, {}, {"X-b": "c"}, {}, {"X-c": "d"}] for headers in header_list: connection.putrequest('GET', '/noleak/1/2') for name, value in headers.items(): connection.putheader(name, value) connection.endheaders() response = connection.getresponse() assert response.status == 200 @pytest.mark.arg('body') def test_body(connection): bodies = [None, b'a', None, b'b', None, b'c', None, b'd', None, b'e'] for body in bodies: connection.putrequest('GET', '/noleak/1/2') if body: connection.putheader('Content-Length', str(len(body))) connection.endheaders(body) response = connection.getresponse() assert response.status == 200 @pytest.mark.arg('keep_alive') def test_keep_alive(request): keep_alives = [True, False, True, True, False, False, True, False] still_open = False for keep_alive in keep_alives: if not still_open: connection_gen = connection(request.getfixturevalue('server')) conn = next(connection_gen) still_open = keep_alive conn.putrequest('GET', '/noleak/1/2') if not keep_alive: conn.putheader('Connection', 'close') conn.endheaders() response = conn.getresponse() assert response.status == 200 @pytest.mark.arg('route') def test_route(connection): for _ in range(7): connection.putrequest('GET', '/noleak/1/2') connection.endheaders() response = connection.getresponse() assert response.status == 200 ================================================ FILE: integration_tests/test_perror.py ================================================ import pytest from hypothesis import given, strategies as st, settings, Verbosity import subprocess import queue import threading from functools import partial from misc import client import integration_tests.common pytestmark = pytest.mark.needs_build @pytest.fixture(autouse=True, scope='module') def server(): server = integration_tests.common.start_server( ['-u', 'integration_tests/dump.py'], stdout=subprocess.PIPE, path='.test') accepting = server.stdout.readline().decode('utf-8') assert accepting.startswith('Accepting connections ') yield server server.terminate() server.wait() == 0 @pytest.fixture def line_getter(server): q = queue.Queue() def enqueue_output(): q.put(server.stdout.readline().strip().decode('utf-8')) class LineGetter: def start(self): self.thread = threading.Thread(target=enqueue_output) self.thread.start() def wait(self): self.thread.join() return q.get() return LineGetter() @pytest.fixture() def connect(request): return partial(client.Connection, 'localhost:8080') full_request_line = 'GET /asd?qwe HTTP/1.1' def make_truncated_request_line(cut): return full_request_line[:-cut] st_request_cut = st.integers(min_value=1, max_value=len(full_request_line) - 1) st_request_line = st.builds(make_truncated_request_line, st_request_cut) @given(request_line=st_request_line) @settings(verbosity=Verbosity.verbose, max_examples=20) def test_truncated_request_line(line_getter, connect, request_line): connection = connect() line_getter.start() connection.putline(request_line) assert line_getter.wait() == 'malformed_headers' response = connection.getresponse() assert response.status == 400 assert response.text == 'malformed_headers' @given(request_line=st_request_line) @settings(verbosity=Verbosity.verbose, max_examples=20) def test_truncated_request_line_disconnect(line_getter, connect, request_line): connection = connect() line_getter.start() connection.putclose(request_line) assert line_getter.wait() == 'incomplete_headers' full_header = 'X-Header: asd' def make_truncated_header(cut): return full_header[:-cut] st_header_cut = st.integers(min_value=5, max_value=len(full_header) - 1) st_header_line = st.builds(make_truncated_header, st_header_cut) @given(header_line=st_header_line) @settings(verbosity=Verbosity.verbose, max_examples=20) def test_truncated_header(line_getter, connect, header_line): connection = connect() line_getter.start() connection.putline(full_request_line) connection.putline(header_line) connection.putline() assert line_getter.wait() == 'malformed_headers' response = connection.getresponse() assert response.status == 400 assert response.text == 'malformed_headers' @given(header_line=st_header_line) @settings(verbosity=Verbosity.verbose, max_examples=20) def test_truncated_header_disconnect(line_getter, connect, header_line): connection = connect() line_getter.start() connection.putline(full_request_line) connection.putclose(header_line) assert line_getter.wait() == 'incomplete_headers' @pytest.mark.parametrize('value', [ '', '+5', '-5', '0x12', '12a' ]) def test_invalid_content_length(line_getter, connect, value): connection = connect() line_getter.start() connection.putline(full_request_line) connection.putheader('Content-Length', value) connection.putline() assert line_getter.wait() == 'invalid_headers' response = connection.getresponse() assert response.status == 400 assert response.text == 'invalid_headers' ================================================ FILE: integration_tests/test_reaper.py ================================================ from functools import partial import time import pytest from misc import client import integration_tests.common pytestmark = pytest.mark.needs_build @pytest.fixture(scope='function', params=[2, 3, 4]) def get_connections_and_wait(request): server, process = integration_tests.common.start_server([ 'integration_tests/reaper.py', '1', str(request.param)], path='.test', return_process=True) def connection_num(): return len( set(c.fd for c in process.connections()) | set(c.fd for p in process.children() for c in p.connections())) yield connection_num, partial(time.sleep, request.param) server.terminate() assert server.wait() == 0 def test_empty(get_connections_and_wait): get_connections, wait = get_connections_and_wait conn = client.Connection('localhost:8080') assert get_connections() == 1 conn.maybe_connect() time.sleep(.1) assert get_connections() == 2 wait() assert get_connections() == 1 def test_request(get_connections_and_wait): get_connections, wait = get_connections_and_wait conn = client.Connection('localhost:8080') assert get_connections() == 1 conn.putrequest('GET', '/') conn.endheaders() assert get_connections() == 2 wait() time.sleep(1) assert get_connections() == 1 ================================================ FILE: integration_tests/test_request.py ================================================ import pytest import json import base64 from functools import partial from hypothesis import given, settings, Verbosity, HealthCheck import integration_tests.common from integration_tests import strategies as st from misc import client pytestmark = pytest.mark.needs_build(coverage=True) @pytest.fixture(autouse=True, scope='module') def server(): server = integration_tests.common.start_server( 'integration_tests/dump.py', path='.test') yield server server.terminate() server.wait() == 0 @pytest.fixture(params=['example', 'test']) def connect(request): if request.param == 'example': yield partial(client.Connection, 'localhost:8080') elif request.param == 'test': connection = client.Connection('localhost:8080') close = connection.close connection.close = lambda: None yield lambda: connection close() @pytest.fixture(params=['', '/async'], ids=['sync', 'async']) def prefix(request): return request.param @given(method=st.method) @settings(verbosity=Verbosity.verbose) def test_method(prefix, connect, method): connection = connect() connection.request(method, prefix + '/dump/1/2') response = connection.getresponse() json_body = json.loads(response.body) assert response.status == 200 assert json_body['method'] == method connection.close() st_route_prefix = st.sampled_from(['/dump/', '/dump1/', '/dump2/']) @given(route_prefix=st_route_prefix) @settings(verbosity=Verbosity.verbose) def test_route(prefix, connect, route_prefix): connection = connect() connection.request('GET', prefix + route_prefix + '1/2') response = connection.getresponse() json_body = json.loads(response.body) assert response.status == 200 assert json_body['route'].startswith(prefix + route_prefix) connection.close() @given(param1=st.param, param2=st.param) @settings(verbosity=Verbosity.verbose) def test_match_dict(prefix, connect, param1, param2): connection = connect() connection.request('GET', prefix + '/dump/{}/{}'.format(param1, param2)) response = connection.getresponse() json_body = json.loads(response.body) assert response.status == 200 assert json_body['match_dict'] == {'p1': param1, 'p2': param2} connection.close() @given(query_string=st.query_string) @settings(verbosity=Verbosity.verbose) def test_query_string(prefix, connect, query_string): connection = connect() connection.request('GET', prefix + '/dump/1/2', query_string) response = connection.getresponse() json_body = json.loads(response.body) assert response.status == 200 assert json_body['query_string'] == query_string connection.close() @given(headers=st.headers) @settings( verbosity=Verbosity.verbose, suppress_health_check=[HealthCheck.too_slow] ) def test_headers(prefix, connect, headers): connection = connect() connection.request('GET', prefix + '/dump/1/2', headers=headers) response = connection.getresponse() json_body = json.loads(response.body) assert response.status == 200 headers = {k.title(): v for k, v in headers} assert json_body['headers'] == headers connection.close() st_errors = st.sampled_from(['not-found', 'forced-1', 'forced-2']) @given(error=st_errors) @settings( verbosity=Verbosity.verbose, suppress_health_check=[HealthCheck.too_slow] ) def test_error(prefix, connect, error): connection = connect() connection.putrequest( 'GET', prefix + '/not-found' if error == 'not-found' else '/dump/1/2') if error != 'not-found': connection.putheader('Force-Raise', error) connection.endheaders() response = connection.getresponse() assert response.status == 500 json_body = json.loads(response.body) assert json_body['exception']['type'] == \ 'RouteNotFoundException' if error == 'not-found' else 'ForcedException' assert json_body['exception']['args'] == \ '' if error == 'not-found' else error @given(body=st.identity_body) @settings(verbosity=Verbosity.verbose) @pytest.mark.parametrize( 'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k']) def test_body(prefix, connect, size_k, body): if size_k and body: body = body * ((size_k * 1024) // len(body) + 1) connection = connect() connection.request('GET', prefix + '/dump/1/2', body=body) response = connection.getresponse() assert response.status == 200 json_body = json.loads(response.body) if body is not None: assert base64.b64decode(json_body['body']) == body else: assert json_body['body'] == body connection.close() @given(body=st.chunked_body) @settings(verbosity=Verbosity.verbose) @pytest.mark.parametrize( 'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k']) def test_chunked(prefix, connect, size_k, body): length = sum(len(b) for b in body) if size_k and length: body = body * ((size_k * 1024) // length + 1) connection = connect() connection.request('POST', prefix + '/dump/1/2', body=body) response = connection.getresponse() assert response.status == 200 json_body = json.loads(response.body) assert base64.b64decode(json_body['body']) == b''.join(body) connection.close() st_errors = st.sampled_from([None, None, None, 'not-found', 'forced-1']) @given( method=st.method, error=st_errors, route_prefix=st_route_prefix, param1=st.param, param2=st.param, query_string=st.query_string, headers=st.headers, body=st.identity_body ) @settings( verbosity=Verbosity.verbose, suppress_health_check=[HealthCheck.too_slow] ) @pytest.mark.parametrize( 'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k']) def test_all(prefix, connect, size_k, method, error, route_prefix, param1, param2, query_string, headers, body): connection = connect() if size_k and body: body = body * ((size_k * 1024) // len(body) + 1) url = prefix + ('/not-found' if error == 'not-found' else '') \ + route_prefix + '{}/{}'.format(param1, param2) connection.putrequest(method, url, query_string) for name, value in headers: connection.putheader(name, value) if body is not None: headers.append(('Content-Length', str(len(body)))) connection.putheader('Content-Length', str(len(body))) if error == 'forced-1': headers.append(('Force-Raise', 'forced-1')) connection.putheader('Force-Raise', 'forced-1') connection.endheaders(body) response = connection.getresponse() assert response.status == 500 if error else 200 json_body = json.loads(response.body) assert json_body['method'] == method if error != 'not-found': assert json_body['route'].startswith(prefix + route_prefix) else: assert json_body['route'] is None assert json_body['match_dict'] == \ {} if error == 'not-found' else {'p1': param1, 'p2': param2} assert json_body['query_string'] == query_string headers = {k.title(): v for k, v in headers} assert json_body['headers'] == headers if body is not None: assert base64.b64decode(json_body['body']) == body else: assert json_body['body'] is None if error: assert json_body['exception']['type'] == \ 'RouteNotFoundException' if error == 'not-found' \ else 'ForcedException' assert json_body['exception']['args'] == \ '' if error == 'not-found' else error else: assert 'exception' not in json_body connection.close() st_request = st.fixed_dictionaries({ 'method': st.method, 'error': st_errors, 'route_prefix': st_route_prefix, 'param1': st.param, 'param2': st.param, 'query_string': st.query_string, 'headers': st.headers, 'body': st.identity_body }) st_requests = st.lists(st_request, min_size=2) @given(requests=st_requests) @settings( verbosity=Verbosity.verbose, suppress_health_check=[HealthCheck.too_slow]) def test_pipeline(requests): connection = client.Connection('localhost:8080') for request in requests: connection.putrequest( request['method'], request['route_prefix'] + ('/not-found' if request['error'] == 'not-found' else '') + '{param1}/{param2}'.format_map(request), request['query_string']) for name, value in request['headers']: connection.putheader(name, value) if request['body'] is not None: body_len = str(len(request['body'])) request['headers'].append(('Content-Length', body_len)) connection.putheader('Content-Length', body_len) if request['error'] == 'forced-1': request['headers'].append(('Force-Raise', 'forced-1')) connection.putheader('Force-Raise', 'forced-1') connection.endheaders(request['body']) for request in requests: response = connection.getresponse() assert response.status == 500 if request['error'] else 200 json_body = response.json assert json_body['method'] == request['method'] if request['error'] != 'not-found': assert json_body['route'].startswith(request['route_prefix']) else: assert json_body['route'] is None assert json_body['match_dict'] == \ {} if request['error'] == 'not-found' else \ {'p1': request['param1'], 'p2': request['param2']} assert json_body['query_string'] == request['query_string'] assert json_body['headers'] == \ {k.title(): v for k, v in request['headers']} if request['body'] is not None: assert base64.b64decode(json_body['body']) == request['body'] else: assert json_body['body'] is None if request['error']: assert json_body['exception']['type'] == \ 'RouteNotFoundException' if request['error'] == 'not-found' \ else 'ForcedException' assert json_body['exception']['args'] == \ '' if request['error'] == 'not-found' else request['error'] else: assert 'exception' not in json_body connection.close() def format_sleep_qs(val): return 'sleep=' + str(val / 100) st_sleep = st.builds(format_sleep_qs, st.integers(min_value=0, max_value=10)) st_prefix = st.sampled_from(['/dump', '/async/dump']) st_async_request = st.fixed_dictionaries({ 'query_string': st_sleep, 'prefix': st_prefix, 'error': st_errors }) st_async_requests = st.lists(st_async_request, min_size=2, max_size=5) \ .filter(lambda rs: any(r['prefix'].startswith('/async') for r in rs)) @given(requests=st_async_requests) @settings(verbosity=Verbosity.verbose) def test_async_pipeline(requests): connection = client.Connection('localhost:8080') for request in requests: connection.putrequest( 'GET', request['prefix'] + ('/not-found' if request['error'] == 'not-found' else '') + '/1/2', request['query_string']) if request['error'] == 'forced-1': connection.putheader('Force-Raise', 'forced-1') connection.endheaders() for request in requests: response = connection.getresponse() assert response.status == 500 if request['error'] else 200 json_body = response.json assert json_body['query_string'] == request['query_string'] connection.close() ================================================ FILE: misc/__init__.py ================================================ ================================================ FILE: misc/bootstrap.sh ================================================ #!/bin/bash cd ~ sudo apt-get update sudo apt-get install -y python3 git libbz2-dev libz-dev libsqlite3-dev libssl-dev gcc make libffi-dev lcov git clone https://github.com/yyuu/pyenv .pyenv .pyenv/bin/pyenv install -v 3.6.0 wget https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz tar xvfz virtualenv-15.1.0.tar.gz python3 virtualenv-15.1.0/virtualenv.py -p .pyenv/versions/3.6.0/bin/python japronto-env git clone https://github.com/squeaky-pl/japronto cd japronto/src/picohttpparser ./build cd - cd japronto ../japronto-env/bin/pip install -r requirements.txt ../japronto-env/bin/python build.py --kit=platform cd - git clone https://github.com/wg/wrk cd wrk make cd - cp wrk/wrk japronto ================================================ FILE: misc/buggers.py ================================================ import atexit import psutil noisy = ['atom', 'chrome', 'firefox', 'dropbox', 'opera', 'spotify', 'gnome-documents'] def silence(): for proc in psutil.process_iter(): if proc.name() in noisy: proc.suspend() def noise(): for proc in psutil.process_iter(): if proc.name() in noisy: proc.resume() atexit.register(noise) ================================================ FILE: misc/cleanup_script.py ================================================ import sys def main(): fp = open(sys.argv[1]) for line in fp: line = line.rstrip() if line.startswith('\t'): rest = line[18:] name_addr, _, rest = rest.partition(' ') name, _, addr = name_addr.partition('+') line = line[:18] + name + ' ' + rest print(line) fp.close() if __name__ == '__main__': main() ================================================ FILE: misc/client.py ================================================ import socket import urllib.parse import json def readline(sock): line = b'' while not line.endswith(b'\r\n'): line += sock.recv(1) return line def readexact(sock, size): data = b'' while size: chunk = sock.recv(size) data += chunk size -= len(chunk) return data class Response: def __init__(self, sock): self.sock = sock self.read_status_line() self.read_headers() self.read_body() def read_status_line(self): status_line = b'' while not status_line: status_line = readline(self.sock).strip() _, self.status, self.reason = status_line.split(None, 2) self.status = int(self.status) def read_headers(self): self.headers = {} while 1: line = readline(self.sock).strip() if not line: break name, value = line.split(b':') name = name.strip().decode('ascii').title() value = value.strip().decode('latin1') self.headers[name] = value @property def encoding(self): content_type = self.headers.get('Content-Type') if not content_type: return 'latin1' _, *rest = [v.split('=') for v in content_type.split(';')] rest = {k.strip(): v.strip() for k, v in rest} return rest.get('charset', 'iso-8859-1') def read_body(self): self.body = readexact(self.sock, int(self.headers['Content-Length'])) self.text = self.body.decode(self.encoding) @property def json(self): return json.loads(self.text) class Connection: def __init__(self, addr): self.addr = addr self.sock = None def maybe_connect(self): if self.sock: return self.sock addr = self.addr.split(':') addr[1] = int(addr[1]) addr = tuple(addr) self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.sock.connect(addr) return self.sock def putline(self, line=None): line = line or b'' sock = self.maybe_connect() if not isinstance(line, bytes): line = str(line).encode('latin1') sock.sendall(line + b'\r\n') def putclose(self, data): sock = self.maybe_connect() if not isinstance(data, bytes): data = str(data).encode('latin1') sock.sendall(data) self.close() def putrequest(self, method, path, query_string=None): url = urllib.parse.quote(path) if query_string is not None: url += '?' + urllib.parse.quote(query_string) request_line = "{method} {url} HTTP/1.1" \ .format(method=method, url=url) self.putline(request_line) def request(self, method, path, query_string=None, headers=None, body=None): self.putrequest(method, path, query_string) headers = headers or [] for name, value in headers: self.putheader(name, value) if body is not None: if isinstance(body, list): self.putheader('Transfer-Encoding', 'chunked') else: self.putheader('Content-Length', str(len(body))) self.endheaders(body) def putheader(self, name, value): header_line = name + ': ' + value self.putline(header_line) def endheaders(self, body=None): self.putline() if body is not None: sock = self.maybe_connect() if isinstance(body, list): for chunk in chunked_encoder(body): sock.sendall(chunk) else: sock.sendall(body) def getresponse(self): return Response(self.sock) def close(self): self.sock.close() def chunked_encoder(data): for chunk in data: if not chunk: continue yield '{:X}\r\n'.format(len(chunk)).encode('ascii') yield chunk + b'\r\n' yield b'0\r\n\r\n' ================================================ FILE: misc/collector.py ================================================ import psutil import sys import time import os import json from functools import partial def get_connections(process): return len( set(c.fd for c in process.connections()) | set(c.fd for p in process.children() for c in p.connections())) def get_memory(p): return p.memory_full_info().uss \ + sum(c.memory_full_info().uss for c in p.children()) def sample_process(pid): process = psutil.Process(pid) samples = [] while 1: try: uss = get_memory(process) conn = get_connections(process) except (psutil.NoSuchProcess, psutil.AccessDenied): break samples.append({ 't': time.monotonic(), 'uss': uss, 'conn': conn, 'type': 'proc'}) time.sleep(.5) return samples def main(): pid = int(sys.argv[1]) samples = sample_process(pid) with open(os.environ['COLLECTOR_FILE'], 'a') as fp: for sample in samples: fp.write(json.dumps(sample) + '\n') print('Collector info written to', os.environ['COLLECTOR_FILE']) if __name__ == '__main__': main() ================================================ FILE: misc/cpu.py ================================================ """ CPU file """ # module imports import subprocess # cpu location CPU_PREFIX = '/sys/devices/system/cpu/' def save(): """ save function """ results = {} cpu_number = 0 while True: try: _file = open( CPU_PREFIX + 'cpu{}/cpufreq/scaling_governor'.format(cpu_number)) except: break governor = _file.read().strip() results.setdefault(cpu_number, {})['governor'] = governor _file.close() try: _file = open( CPU_PREFIX + 'cpu{}/cpufreq/scaling_cur_freq'.format(cpu_number)) except: break results[cpu_number]['freq'] = _file.read().strip() _file.close() cpu_number += 1 return results def change(governor, freq=None): """ change function """ cpu_number = 0 while True: try: subprocess.check_output([ "sudo", "bash", "-c", "echo {governor} > {CPU_PREFIX}cpu{cpu_number}/cpufreq/scaling_governor" .format(governor=governor, CPU_PREFIX=CPU_PREFIX, cpu_number=cpu_number)], stderr=subprocess.STDOUT) except: break if freq: subprocess.check_output([ "sudo", "bash", "-c", "echo {freq} > {CPU_PREFIX}cpu{cpu_number}/cpufreq/scaling_setspeed" .format(freq=freq, CPU_PREFIX=CPU_PREFIX, cpu_number=cpu_number)], stderr=subprocess.STDOUT) cpu_number += 1 def available_freq(): """ function for checking available frequency """ _file = open(CPU_PREFIX + 'cpu0/cpufreq/scaling_available_frequencies') freq = [int(_file) for _file in _file.read().strip().split()] _file.close() return freq def min_freq(): """ function for returning minimum available frequency """ return min(available_freq()) def max_freq(): """ function for returning maximum avaliable frequency """ return max(available_freq()) def dump(): """ dump function """ try: sensors = subprocess.check_output('sensors').decode('utf-8') except (FileNotFoundError, subprocess.CalledProcessError): print("Couldn't read CPU temp") else: cores = [] for line in sensors.splitlines(): if line.startswith('Core '): core, rest = line.split(':') temp = rest.strip().split()[0] cores.append((core, temp)) for core, temp in cores: print(core + ':', temp) cpu_number = 0 while True: try: _file = open( CPU_PREFIX + 'cpu{}/cpufreq/scaling_governor'.format(cpu_number)) except: break print('Core ' + str(cpu_number) + ':', _file.read().strip(), end=', ') _file.close() try: _file = open( CPU_PREFIX + 'cpu{}/cpufreq/scaling_cur_freq'.format(cpu_number)) except: break freq = round(int(_file.read()) / 10 ** 6, 2) print(freq, 'GHz') cpu_number += 1 ================================================ FILE: misc/do_perf.py ================================================ import subprocess import os import sys import argparse import parsers import cases import buggers import cpu def get_http10long(): return cases.base['10long'].data def get_websites(size=2 ** 18): data = b'' while len(data) < size: for c in cases.websites.values(): data += c.data return data if __name__ == '__main__': print('pid', os.getpid()) cpu.dump() buggers.silence() argparser = argparse.ArgumentParser(description='do_perf') argparser.add_argument( '-p', '--parsers', dest='parsers', default='cffi,cext') argparser.add_argument( '-b', '--benchmarks', dest='benchmarks', default='http10long,websites,websitesn') result = argparser.parse_args(sys.argv[1:]) parsers = result.parsers.split(',') benchmarks = result.benchmarks.split(',') one_shot = [b for b in benchmarks if b in ['http10long', 'websites']] multi_shot = [b for b in benchmarks if b in ['websitesn']] setup = """ import parsers import do_perf parser, _ = parsers.make_{}(parsers.NullProtocol) data = do_perf.get_{}() """ loop = """ parser.feed(data) parser.feed_disconnect() """ for dataset in one_shot: for parser in parsers: print('-- {} {} --'.format(dataset, parser)) subprocess.check_call([ 'python', '-m', 'perf', 'timeit', '-s', setup.format(parser, dataset), loop]) print() setup += """ import parts p = parts.make_parts(data, parts.fancy_series(1450)) """ loop = """ for i in p: parser.feed(i) parser.feed_disconnect() """ if multi_shot: for parser in parsers: print('-- website parts {} --'.format(parser)) subprocess.check_call([ 'python', '-m', 'perf', 'timeit', '-s', setup.format(parser, 'websites'), loop]) print() ================================================ FILE: misc/docker/Dockerfile ================================================ FROM python:3.6.0-slim RUN pip3 install japronto ENV PYTHONUNBUFFERED=1 ENTRYPOINT ["japronto"] ================================================ FILE: misc/parts.py ================================================ from functools import partial import types import math def make_parts(value, get_size, dir=1): parts = [] left = bytearray(value) while left: if isinstance(get_size, types.GeneratorType): size = next(get_size) else: size = get_size if dir == 1: parts.append(bytes(left[:size])) left = left[size:] else: parts.append(bytes(left[-size:])) left = left[:-size] return parts if dir == 1 else list(reversed(parts)) def one_part(value): return [value] def geometric_series(): s = 2 while 1: yield s s *= 2 def fancy_series(minimum=2): x = 0 while 1: yield int(minimum + abs(math.sin(x / 3)) * 64) x += 1 ================================================ FILE: misc/perf.md ================================================ Capturing performance data with perf ==================================== For best results the C source should be built with `-g -O0`. Run the server, record performance events with `-F` frequency `997 Hz`, `-a` all processes and `-g` take stack info. ``` python examples/simple/simple.py -p c & \ perf record -F 997 -a -g -- sleep 59 & \ sleep 1 && ./wrk -c 100 -t 1 -d 60 http://localhost:8080 & ``` Write data to a text file filtering for pid `2836` (should be server process) ``` perf script --pid 2836 > out.perf ``` Remove address info to make graphs easier to read ``` python cleanup_script.py out.perf > out2.perf ``` Visualize data with a flame graph ================================= ``` ./stackcollapse-perf.pl out2.perf > out.folded ./flamegraph.pl out.folded > test.svg ``` ================================================ FILE: misc/pipeline.lua ================================================ -- example script demonstrating HTTP pipelining init = function(args) local r = {} r[1] = wrk.format(nil, "/") r[2] = wrk.format(nil, "/") r[3] = wrk.format(nil, "/") r[4] = wrk.format(nil, "/") r[5] = wrk.format(nil, "/") r[6] = wrk.format(nil, "/") r[7] = wrk.format(nil, "/") r[8] = wrk.format(nil, "/") r[9] = wrk.format(nil, "/") r[10] = wrk.format(nil, "/") r[11] = wrk.format(nil, "/") r[12] = wrk.format(nil, "/") r[13] = wrk.format(nil, "/") r[14] = wrk.format(nil, "/") r[15] = wrk.format(nil, "/") r[16] = wrk.format(nil, "/") r[17] = wrk.format(nil, "/") r[18] = wrk.format(nil, "/") r[19] = wrk.format(nil, "/") r[20] = wrk.format(nil, "/") r[21] = wrk.format(nil, "/") r[22] = wrk.format(nil, "/") r[23] = wrk.format(nil, "/") r[24] = wrk.format(nil, "/") req = table.concat(r) end request = function() return req end ================================================ FILE: misc/report.py ================================================ import matplotlib.pyplot as plt import sys import os import json def report(samples, pid): plt.figure(figsize=(25, 10)) x = [s['t'] for s in samples if s['type'] == 'proc'] lines = [s for s in samples if s['type'] == 'event'] # minuss = min(s['uss'] for s in samples if s['type'] == 'proc') ussplot = plt.subplot(211) ussplot.set_title('uss') ussplot.plot( x, [s['uss'] for s in samples if s['type'] == 'proc'], '.') for l in lines: # ussplot.text(l['t'], minuss, l['event'], horizontalalignment='right', # rotation=-90, rotation_mode='anchor') ussplot.axvline(l['t']) connplot = plt.subplot(212) connplot.set_title('conn') connplot.plot( x, [s['conn'] for s in samples if s['type'] == 'proc'], '.') os.makedirs('.reports', exist_ok=True) path = '.reports/{}.png'.format(pid) plt.savefig(path) return path def load(filepath): samples = [] with open(filepath) as fp: for line in fp: line = line.strip() samples.append(json.loads(line)) return samples def order(samples): return sorted(samples, key=lambda x: x['t']) def normalize_time(samples): if not samples: return [] base_time = samples[0]['t'] return [{**s, 't': s['t'] - base_time} for s in samples] def main(): samples = load(sys.argv[1]) pid, _ = os.path.splitext(os.path.basename(sys.argv[1])) samples = order(samples) samples = normalize_time(samples) report(samples, pid) if __name__ == '__main__': main() ================================================ FILE: misc/requirements-test.txt ================================================ -r requirements.txt hypothesis==3.80.0 psutil==5.6.6 pytest==3.9.2 pytoml==0.1.20 perf==1.5.1 cffi==1.11.5 ================================================ FILE: misc/requirements.txt ================================================ uvloop>=0.11.3 ================================================ FILE: misc/rpm-requirements.txt ================================================ libasan lcov libubsan ================================================ FILE: misc/runpytest.py ================================================ from pytest import main main() ================================================ FILE: misc/simple.py ================================================ import asyncio import argparse import os.path import sys import socket sys.path.insert(0, os.path.abspath(os.path.dirname(__file__) + '/../../src')) import japronto.protocol.handler # noqa from japronto.router.cmatcher import Matcher # noqa from japronto.router import Router # noqa from japronto.app import Application # noqa def slash(request): return request.Response(text='Hello slash!') def hello(request): return request.Response(text='Hello hello!') async def sleep(request): await asyncio.sleep(3) return request.Response(text='I am sleepy') async def loop(request): i = 0 while i < 10: await asyncio.sleep(1) print(i) i += 1 return request.Response(text='Loop finished') def dump(request): sock = request.transport.get_extra_info('socket') no_delay = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY) text = """ Method: {0.method} Path: {0.path} Version: {0.version} Headers: {0.headers} Match: {0.match_dict} Body: {0.body} QS: {0.query_string} query: {0.query} mime_type: {0.mime_type} encoding: {0.encoding} form: {0.form} keep_alive: {0.keep_alive} no_delay: {1} route: {0.route} """.strip().format(request, no_delay) return request.Response(text=text, headers={'X-Version': '123'}) app = Application() r = app.router r.add_route('/', slash) r.add_route('/hello', hello) r.add_route('/dump/{this}/{that}', dump) r.add_route('/sleep/{pinch}', sleep) r.add_route('/loop', loop) if __name__ == '__main__': argparser = argparse.ArgumentParser('server') argparser.add_argument( '-p', dest='flavor', default='block') args = argparser.parse_args(sys.argv[1:]) app.run(protocol_factory=japronto.protocol.handler.make_class(args.flavor)) ================================================ FILE: misc/suppr.txt ================================================ # This is a known leak. # Python leaks a little, we use malloc directly leak:PyMem_RawMalloc leak:_PyObject_GC_Resize leak:PyThread_allocate_lock leak:resize_compact ================================================ FILE: misc/travis/before_install.sh ================================================ #!/bin/bash export JAPR_MSG=`git show -s --format=%B | xargs` export JAPR_WHEEL=`[[ $JAPR_MSG == *"[travis-wheel]"* ]] && echo 1 || echo 0` export JAPR_OS=`uname` if [[ $VERSION == "3.5."* ]]; then export PYTHON_TAG=cp35-cp35m elif [[ $VERSION == "3.6."* ]]; then export PYTHON_TAG=cp36-cp36m elif [[ $VERSION == "3.7."* ]]; then export PYTHON_TAG=cp37-cp37m elif [[ $VERSION == "3.8."* ]]; then export PYTHON_TAG=cp38-cp38m fi env | grep "^JAPR_" ================================================ FILE: misc/travis/install.sh ================================================ #!/bin/bash set -ex if [[ $JAPR_OS == "Darwin" ]]; then /usr/bin/clang --version source misc/terryfy/travis_tools.sh source misc/terryfy/library_installers.sh clean_builds get_python_environment macpython $VERSION venv fi if [[ $JAPR_WHEEL == "1" ]]; then pip install twine if [[ $JAPR_OS == "Linux" ]]; then docker info docker pull quay.io/pypa/manylinux1_x86_64 fi fi ================================================ FILE: misc/travis/script.sh ================================================ #!/bin/bash set -ex if [[ $JAPR_WHEEL == "1" ]]; then if [[ $JAPR_OS == "Linux" ]]; then docker run --rm -u `id -u` -w /io -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /opt/python/$PYTHON_TAG/bin/python setup.py bdist_wheel docker run --rm -u `id -u` -w /io -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 auditwheel repair dist/*-$PYTHON_TAG-linux_x86_64.whl rm -r dist/* cp wheelhouse/*.whl dist fi if [[ $JAPR_OS == "Darwin" ]]; then python setup.py bdist_wheel fi ls -lha dist unzip -l dist/*.whl twine upload -u squeaky dist/*.whl fi ================================================ FILE: setup.py ================================================ """ Japronto """ import codecs import os import re from setuptools import setup, find_packages import build with codecs.open(os.path.join(os.path.abspath(os.path.dirname( __file__)), 'src', 'japronto', '__init__.py'), 'r', 'latin1') as fp: try: version = re.findall(r"^__version__ = '([^']+)'\r?$", fp.read(), re.M)[0] except IndexError: raise RuntimeError('Unable to determine version.') setup( name='japronto', version=version, url='http://github.com/squeaky-pl/japronto/', license='MIT', author='Paweł Piotr Przeradowski', author_email='przeradowski@gmail.com', description='A HTTP application toolkit and server bundle ' + 'based on uvloop and picohttpparser', package_dir={'': 'src'}, packages=find_packages('src'), keywords=['web', 'asyncio'], platforms='x86_64 Linux and MacOS X', install_requires=[ 'uvloop>=0.11.3', ], entry_points=""" [console_scripts] japronto = japronto.__main__:main """, classifiers=[ 'Development Status :: 2 - Pre-Alpha', 'Intended Audience :: Developers', 'Environment :: Web Environment', 'License :: OSI Approved :: MIT License', 'Operating System :: MacOS :: MacOS X', 'Operating System :: POSIX :: Linux', 'Programming Language :: C', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: Implementation :: CPython', 'Topic :: Internet :: WWW/HTTP' ], zip_safe=False, include_package_data=True, package_data={'picohttpparser': ['*.so']}, ext_modules=build.get_platform(), cmdclass={'build_ext': build.custom_build_ext} ) ================================================ FILE: src/japronto/__init__.py ================================================ from .app import Application # noqa from .router import RouteNotFoundException # noqa __version__ = '0.1.2' ================================================ FILE: src/japronto/__main__.py ================================================ import sys import os from .runner import get_parser, verify, run def main(): parser = get_parser() args = parser.parse_args() if not args.script: os.putenv('_JAPR_IGNORE_RUN', '1') if args.reload: os.execv( sys.executable, [sys.executable, '-m', 'japronto.reloader', *sys.argv[1:]]) if not args.script: os.environ['_JAPR_IGNORE_RUN'] = '1' attribute = verify(args) if not attribute: return 1 run(attribute, args) sys.exit(main()) ================================================ FILE: src/japronto/app/__init__.py ================================================ import signal import asyncio import traceback import socket import os import sys import multiprocessing import faulthandler import uvloop from japronto.router import Router, RouteNotFoundException from japronto.protocol.cprotocol import Protocol from japronto.protocol.creaper import Reaper signames = { int(v): v.name for k, v in signal.__dict__.items() if isinstance(v, signal.Signals)} class Application: def __init__(self, *, reaper_settings=None, log_request=None, protocol_factory=None, debug=False): self._router = None self._loop = None self._connections = set() self._reaper_settings = reaper_settings or {} self._error_handlers = [] self._log_request = log_request self._request_extensions = {} self._protocol_factory = protocol_factory or Protocol self._debug = debug @property def loop(self): if not self._loop: self._loop = uvloop.new_event_loop() return self._loop @property def router(self): if not self._router: self._router = Router() return self._router def __finalize(self): self.loop self.router self._reaper = Reaper(self, **self._reaper_settings) self._matcher = self._router.get_matcher() def protocol_error_handler(self, error): print(error) error = error.encode('utf-8') response = [ 'HTTP/1.0 400 Bad Request\r\n', 'Content-Type: text/plain; charset=utf-8\r\n', 'Content-Length: {}\r\n\r\n'.format(len(error))] return ''.join(response).encode('utf-8') + error def default_request_logger(self, request): print(request.remote_addr, request.method, request.path) def add_error_handler(self, typ, handler): self._error_handlers.append((typ, handler)) def default_error_handler(self, request, exception): if isinstance(exception, RouteNotFoundException): return request.Response(code=404, text='Not Found') if isinstance(exception, asyncio.CancelledError): return request.Response(code=503, text='Service unavailable') tb = traceback.format_exception( None, exception, exception.__traceback__) tb = ''.join(tb) print(tb, file=sys.stderr, end='') return request.Response( code=500, text=tb if self._debug else 'Internal Server Error') def error_handler(self, request, exception): for typ, handler in self._error_handlers: if typ is not None and not isinstance(exception, typ): continue try: return handler(request, exception) except: print('-- Exception in error_handler occured:') traceback.print_exc() print('-- while handling:') traceback.print_exception(None, exception, exception.__traceback__) return request.Response( code=500, text='Internal Server Error') return self.default_error_handler(request, exception) def _get_idle_and_busy_connections(self): # FIXME if there is buffered data in gather the connections should be # considered busy, now it's idle return \ [c for c in self._connections if c.pipeline_empty], \ [c for c in self._connections if not c.pipeline_empty] async def drain(self): # TODO idle connections will close connection with half-read requests idle, busy = self._get_idle_and_busy_connections() for c in idle: c.transport.close() # for c in busy_connections: # need to implement something that makes protocol.on_data # start rejecting incoming data # this closes transport unfortunately # sock = c.transport.get_extra_info('socket') # sock.shutdown(socket.SHUT_RD) if idle or busy: print('Draining connections...') else: return if idle: print('{} idle connections closed immediately'.format(len(idle))) if busy: print('{} connections busy, read-end closed'.format(len(busy))) for x in range(5, 0, -1): await asyncio.sleep(1) idle, busy = self._get_idle_and_busy_connections() for c in idle: c.transport.close() if not busy: break else: print( "{} seconds remaining, {} connections still busy" .format(x, len(busy))) _, busy = self._get_idle_and_busy_connections() if busy: print('Forcefully killing {} connections'.format(len(busy))) for c in busy: c.pipeline_cancel() def extend_request(self, handler, *, name=None, property=False): if not name: name = handler.__name__ self._request_extensions[name] = (handler, property) def serve(self, *, sock, host, port, reloader_pid): faulthandler.enable() self.__finalize() loop = self.loop asyncio.set_event_loop(loop) server_coro = loop.create_server( lambda: self._protocol_factory(self), sock=sock) server = loop.run_until_complete(server_coro) loop.add_signal_handler(signal.SIGTERM, loop.stop) loop.add_signal_handler(signal.SIGINT, loop.stop) if reloader_pid: from japronto.reloader import ChangeDetector detector = ChangeDetector(loop) detector.start() print('Accepting connections on http://{}:{}'.format(host, port)) try: loop.run_forever() finally: server.close() loop.run_until_complete(server.wait_closed()) loop.run_until_complete(self.drain()) self._reaper.stop() loop.close() # break reference and cleanup matcher buffer del self._matcher def _run(self, *, host, port, worker_num=None, reloader_pid=None, debug=None): self._debug = debug or self._debug if self._debug and not self._log_request: self._log_request = self._debug sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.bind((host, port)) os.set_inheritable(sock.fileno(), True) workers = set() terminating = False def stop(sig, frame): nonlocal terminating if reloader_pid and sig == signal.SIGHUP: print('Reload request received') elif not terminating: terminating = True print('Termination request received') for worker in workers: worker.terminate() signal.signal(signal.SIGINT, stop) signal.signal(signal.SIGTERM, stop) signal.signal(signal.SIGHUP, stop) for _ in range(worker_num or 1): worker = multiprocessing.Process( target=self.serve, kwargs=dict(sock=sock, host=host, port=port, reloader_pid=reloader_pid)) worker.daemon = True worker.start() workers.add(worker) # prevent further operations on socket in parent sock.close() for worker in workers: worker.join() if worker.exitcode > 0: print('Worker exited with code {}'.format(worker.exitcode)) elif worker.exitcode < 0: try: signame = signames[-worker.exitcode] except KeyError: print( 'Worker crashed with unknown code {}!' .format(worker.exitcode)) else: print('Worker crashed on signal {}!'.format(signame)) def run(self, host='0.0.0.0', port=8080, *, worker_num=None, reload=False, debug=False): if os.environ.get('_JAPR_IGNORE_RUN'): return reloader_pid = None if reload: if '_JAPR_RELOADER' not in os.environ: from japronto.reloader import exec_reloader exec_reloader(host=host, port=port, worker_num=worker_num) else: reloader_pid = int(os.environ['_JAPR_RELOADER']) self._run( host=host, port=port, worker_num=worker_num, reloader_pid=reloader_pid, debug=debug) ================================================ FILE: src/japronto/capsule.c ================================================ #include void* get_ptr_from_mod(const char* module_name, const char* attr_name, const char* capsule_name) { void* ptr; PyObject* module = NULL; PyObject* capsule = NULL; module = PyImport_ImportModule(module_name); if(!module) goto error; capsule = PyObject_GetAttrString(module, attr_name); if(!capsule) goto error; ptr = PyCapsule_GetPointer(capsule, capsule_name); if(!ptr) goto error; goto finally; error: ptr = NULL; finally: Py_XDECREF(capsule); Py_XDECREF(module); return ptr; } PyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name, const char* capsule_name) { PyObject* capsule = NULL; capsule = PyCapsule_New(ptr, capsule_name, NULL); if(!capsule) goto error; if(PyModule_AddObject(m, attr_name, capsule) == -1) goto error; Py_INCREF(capsule); goto finally; error: Py_XDECREF(capsule); capsule = NULL; finally: return capsule; } ================================================ FILE: src/japronto/capsule.h ================================================ #pragma once void* get_ptr_from_mod(const char* module_name, const char* attr_name, const char* capsule_name); PyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name, const char* capsule_name); #define import_capi(module_name) \ get_ptr_from_mod(module_name, "_capi", module_name "._capi") #define export_capi(m, module_name, capi) \ put_ptr_in_mod(m, capi, "_capi", module_name "._capi") ================================================ FILE: src/japronto/common.h ================================================ #pragma once typedef enum { KEEP_ALIVE_UNSET, KEEP_ALIVE_TRUE, KEEP_ALIVE_FALSE } KEEP_ALIVE; ================================================ FILE: src/japronto/cpu_features.c ================================================ #include #include #include "cpu_features.h" int supports_x86_sse42(void) { #if defined(__clang__) unsigned int eax = 0, ebx = 0, ecx = 0, edx = 0; __get_cpuid(1, &eax, &ebx, &ecx, &edx); return ecx & bit_SSE42; #else __builtin_cpu_init(); return __builtin_cpu_supports("sse4.2"); #endif } ================================================ FILE: src/japronto/cpu_features.h ================================================ #pragma once int supports_x86_sse42(void); ================================================ FILE: src/japronto/parser/.gitignore ================================================ libpicohttpparser.c ================================================ FILE: src/japronto/parser/__init__.py ================================================ header_errors = [ 'malformed_headers', 'incomplete_headers', 'invalid_headers', 'excessive_data'] body_errors = ['malformed_body', 'incomplete_body'] ================================================ FILE: src/japronto/parser/build_libpicohttpparser.py ================================================ import distutils.log distutils.log.set_verbosity(distutils.log.DEBUG) import os.path import cffi ffibuilder = cffi.FFI() shared_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../picohttpparser')) print(shared_path) ffibuilder.set_source("libpicohttpparser", """ #include "picohttpparser.h" """, libraries=['picohttpparser'], include_dirs=[shared_path], library_dirs=[shared_path], extra_link_args=['-Wl,-rpath=' + shared_path]) # extra_objects=[os.path.join(shared_path, 'picohttpparser.o')], # or a list of libraries to link with # (more arguments like setup.py's Extension class: # include_dirs=[..], extra_objects=[..], and so on) ffibuilder.cdef(""" struct phr_header { const char *name; size_t name_len; const char *value; size_t value_len; }; struct phr_chunked_decoder { size_t bytes_left_in_chunk; /* number of bytes left in current chunk */ char consume_trailer; /* if trailing headers should be consumed */ char _hex_count; char _state; }; int phr_parse_request(const char *buf, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len); ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *bufsz); """) if __name__ == "__main__": ffibuilder.compile(verbose=True) ================================================ FILE: src/japronto/parser/cffiparser.py ================================================ from parser.libpicohttpparser import ffi, lib class HttpRequestParser(object): def __init__(self, on_headers, on_body, on_error): self.on_headers = on_headers self.on_body = on_body self.on_error = on_error self.c_method = ffi.new('char **') self.method_len = ffi.new('size_t *') self.c_path = ffi.new('char **') self.path_len = ffi.new('size_t *') self.minor_version = ffi.new('int *') self.c_headers = ffi.new('struct phr_header[10]') self.num_headers = ffi.new('size_t *') self.chunked_offset = ffi.new('size_t*') self._reset_state(True) def _reset_state(self, disconnect=False): self.state = 'headers' self.transfer = None self.content_length = None self.chunked_decoder = None self.chunked_offset[0] = 0 if disconnect: self.connection = None self.buffer = bytearray() def _parse_headers(self): if self.connection == 'close': self.on_error('excessive_data') self._reset_state(True) return -1 self.num_headers[0] = 10 # FIXME: More than 10 headers result = lib.phr_parse_request( ffi.from_buffer(self.buffer), len(self.buffer), self.c_method, self.method_len, self.c_path, self.path_len, self.minor_version, self.c_headers, self.num_headers, 0) if result == -2: return result elif result == -1: self.on_error('malformed_headers') self._reset_state(True) return result else: self._reset_state() method = ffi.cast( 'char[{}]'.format(self.method_len[0]), self.c_method[0]) path = ffi.cast( 'char[{}]'.format(self.path_len[0]), self.c_path[0]) headers = ffi.cast( "struct phr_header[{}]".format(self.num_headers[0]), self.c_headers) if ffi.buffer(method)[:] in (b'GET', b'DELETE', b'HEAD'): self.no_semantics = True if self.minor_version[0] == 0: self.connection = 'close' else: self.connection = 'keep-alive' for header in headers: header_name = ffi.string(header.name, header.name_len).title() # maybe len + strcasecmp C style is faster? if header_name == b'Transfer-Encoding': self.transfer = ffi.string( header.value, header.value_len).decode('ascii') # FIXME comma separated and invalid values elif header_name == b'Connection': self.connection = ffi.string( header.value, header.value_len).decode('ascii') # FIXME other options for Connection like updgrade elif header_name == b'Content-Length': content_length_error = False if not header.value_len: content_length_error = True if not content_length_error: content_length = ffi.buffer(header.value, header.value_len) if not content_length_error and content_length[0] in b'+-': content_length_error = True if not content_length_error: try: self.content_length = int(content_length[:]) except ValueError: content_length_error = True if content_length_error: self.on_error('invalid_headers') self._reset_state(True) return -1 self.on_headers(method, path, self.minor_version[0], headers) self.buffer = self.buffer[result:] return result def _parse_body(self): if self.content_length is None and self.transfer is None: self.on_body(None) return 0 elif self.content_length == 0: self.on_body(ffi.from_buffer(b"")) return 0 elif self.content_length is not None: if self.content_length > len(self.buffer): return -2 body = memoryview(self.buffer)[:self.content_length] self.on_body(ffi.from_buffer(body)) self.buffer = self.buffer[self.content_length:] result = self.content_length return result elif self.transfer == 'chunked': if not self.chunked_decoder: self.chunked_decoder = ffi.new('struct phr_chunked_decoder*') self.chunked_decoder.consume_trailer = b'\x01' chunked_offset_start = self.chunked_offset[0] self.chunked_offset[0] = len(self.buffer) - self.chunked_offset[0] result = lib.phr_decode_chunked( self.chunked_decoder, ffi.from_buffer(self.buffer) + chunked_offset_start, self.chunked_offset) self.chunked_offset[0] = self.chunked_offset[0] \ + chunked_offset_start if result == -2: self.buffer = self.buffer[:self.chunked_offset[0]] return result elif result == -1: self.on_error('malformed_body') self._reset_state(True) return result body = memoryview(self.buffer)[:self.chunked_offset[0]] self.on_body(ffi.from_buffer(body)) self.buffer = self.buffer[ self.chunked_offset[0]:self.chunked_offset[0] + result] self._reset_state() return result def feed(self, data): self.buffer += data while self.buffer: if self.state == 'headers': result = self._parse_headers() if result <= 0: return None self.state = 'body' if self.state == 'body': result = self._parse_body() if result < 0: return None self.state = 'headers' def feed_disconnect(self): if self.state == 'headers' and self.buffer: self.on_error('incomplete_headers') elif self.state == 'body': self.on_error('incomplete_body') self._reset_state(True) ================================================ FILE: src/japronto/parser/cparser.c ================================================ #include #include #include "cparser.h" #include "cpu_features.h" #ifndef PARSER_STANDALONE #include "cprotocol.h" #endif static PyObject* malformed_headers; static PyObject* malformed_body; static PyObject* incomplete_headers; static PyObject* invalid_headers; static PyObject* incomplete_body; static PyObject* excessive_data; //static PyObject* empty_body; const char zero_body[] = ""; /*static PyObject* GET; static PyObject* POST; static PyObject* DELETE; static PyObject* HEAD; static PyObject* Host; static PyObject* User_Agent; static PyObject* Accept; static PyObject* Accept_Language; static PyObject* Accept_Encoding; static PyObject* Accept_Charset; static PyObject* Connection; static PyObject* Cookie; static PyObject* Content_Length; static PyObject* Transfer_Encoding; static PyObject* val_close; static PyObject* keep_alive;*/ static unsigned long const CONTENT_LENGTH_UNSET = ULONG_MAX; static void _reset_state(Parser* self, bool disconnect) { self->state = PARSER_HEADERS; self->transfer = PARSER_TRANSFER_UNSET; self->content_length = CONTENT_LENGTH_UNSET; memset(&self->chunked_decoder, 0, sizeof(struct phr_chunked_decoder)); self->chunked_decoder.consume_trailer = 1; self->chunked_offset = 0; if(disconnect) { self->connection = PARSER_CONNECTION_UNSET; self->buffer_start = 0; self->buffer_end = 0; } } #ifdef PARSER_STANDALONE static PyObject * Parser_new(PyTypeObject *type, PyObject *args, PyObject *kwds) #else void Parser_new(Parser* self) #endif { #ifdef PARSER_STANDALONE Parser *self = NULL; self = (Parser *)type->tp_alloc(type, 0); if (!self) goto finally; self->on_headers = NULL; self->on_body = NULL; self->on_error = NULL; #endif #ifdef PARSER_STANDALONE finally: return (PyObject *)self; #endif } #ifdef PARSER_STANDALONE static int Parser_init(Parser *self, PyObject *args, PyObject *kwds) #else int Parser_init(Parser* self, void* protocol) #endif { #ifdef PARSER_STANDALONE #ifdef DEBUG_PRINT printf("__init__\n"); #endif // FIXME: __init__ can be called many times // FIXME: check argument types int result = PyArg_ParseTuple( args, "OOO", &self->on_headers, &self->on_body, &self->on_error); if(!result) return -1; Py_INCREF(self->on_headers); Py_INCREF(self->on_body); Py_INCREF(self->on_error); #else self->protocol = protocol; #endif _reset_state(self, true); self->buffer_capacity = PARSER_INITIAL_BUFFER_SIZE; self->buffer = self->inline_buffer; return 0; } #ifdef PARSER_STANDALONE static void Parser_dealloc(Parser* self) #else void Parser_dealloc(Parser* self) #endif { #ifdef PARSER_STANDALONE #ifdef DEBUG_PRINT printf("__del__\n"); #endif #endif if(self->buffer != self->inline_buffer) free(self->buffer); #ifdef PARSER_STANDALONE Py_XDECREF(self->on_error); Py_XDECREF(self->on_body); Py_XDECREF(self->on_headers); Py_TYPE(self)->tp_free((PyObject*)self); #endif } static int (*_phr_parse_request)( const char *, size_t, const char **, size_t *, const char **, size_t *, int *, struct phr_header *, size_t *, size_t); static int _parse_headers(Parser* self) { #ifdef PARSER_STANDALONE PyObject* method_view = NULL; PyObject* path_view = NULL; PyObject* minor_version_long = NULL; PyObject* headers_view = NULL; #endif PyObject* error; int result = -1; if(self->connection == PARSER_CLOSE) { error = excessive_data; goto on_error; } char* method; size_t method_len; char* path; size_t path_len; int minor_version; struct phr_header headers[50]; size_t num_headers = sizeof(headers) / sizeof(struct phr_header); result = _phr_parse_request( self->buffer + self->buffer_start, self->buffer_end - self->buffer_start, (const char**)&method, &method_len, (const char**)&path, &path_len, &minor_version, headers, &num_headers, 0); // FIXME: More than 10 headers #ifdef DEBUG_PRINT printf("result: %d\n", result); #endif if(result == -2) goto finally; if(result == -1) { error = malformed_headers; goto on_error; } if(minor_version == 0) { self->connection = PARSER_CLOSE; } else { self->connection = PARSER_KEEP_ALIVE; } #define header_name_equal(val) \ header->name_len == strlen(val) && strncasecmp(header->name, val, header->name_len) == 0 #define header_value_equal(val) \ header->value_len == strlen(val) && strncasecmp(header->value, val, header->value_len) == 0 /*#define cmp_and_set_header_name(name, val) \ if(header_name_equal(val)) { \ py_header_name = name; \ Py_INCREF(name); \ } #define cmp_and_set_header_value(name, val) \ if(header_value_equal(val)) { \ py_header_value = name; \ Py_INCREF(name); \ }*/ for(struct phr_header* header = headers; header < headers + num_headers; header++) { // TODO: common names and values static /*PyObject* py_header_name = NULL; PyObject* py_header_value = NULL;*/ if(header_name_equal("Transfer-Encoding")) { if(header_value_equal("chunked")) self->transfer = PARSER_CHUNKED; else if(header_value_equal("identity")) self->transfer = PARSER_IDENTITY; else /*TODO: handle incorrept values for protocol version, also comma sep*/; /*py_header_name = Transfer_Encoding; Py_INCREF(Transfer_Encoding);*/ } else if(header_name_equal("Content-Length")) { if(!header->value_len) { error = invalid_headers; goto on_error; } if(*header->value == '+' || *header->value == '-') { error = invalid_headers; goto on_error; } char * endptr = (char *)header->value + header->value_len; self->content_length = strtol(header->value, &endptr, 10); // FIXME: overflow? if(endptr != (char*)header->value + header->value_len) { error = invalid_headers; goto on_error; } } else if(header_name_equal("Connection")) { if(header_value_equal("close")) self->connection = PARSER_CLOSE; else if(header_value_equal("keep-alive")) self->connection = PARSER_KEEP_ALIVE; else /* FIXME: on_error*/; /*py_header_name = Content_Length; Py_INCREF(Content_Length);*/ } /*else cmp_and_set_header_name(Host, "Host") else cmp_and_set_header_name(User_Agent, "User-Agent") else cmp_and_set_header_name(Accept, "Accept") else cmp_and_set_header_name(Accept_Language, "Accept-Language") else cmp_and_set_header_name(Accept_Encoding, "Accept-Encoding") else cmp_and_set_header_name(Accept_Charset, "Accept-Charset") else cmp_and_set_header_name(Connection, "Connection") else cmp_and_set_header_name(Cookie, "Cookie") else { bool prev_alpha = false; for(char* c = (char*)header.name; c < header.name + header.name_len; c++) { if(*c >= 'A' && *c <= 'Z') { if(prev_alpha) *c |= 0x20; prev_alpha = true; } else if (*c >= 'a' && *c <= 'z') prev_alpha = true; else prev_alpha = false; } // FIXME this should accept only ascii py_header_name = PyUnicode_FromStringAndSize( header.name, header.name_len); if(!py_header_name) { result = -3; goto finally_loop; } } if(py_header_name == Connection) { cmp_and_set_header_value(keep_alive, "keep-alive") else cmp_and_set_header_value(val_close, "close") else FIXME: invalid Connection value; } else { // FIXME: this can return NULL on codec error py_header_value = PyUnicode_DecodeLatin1( header.value, header.value_len, NULL); if(!py_header_value) { result = -3; goto finally_loop; } } if(PyDict_SetItem(py_headers, py_header_name, py_header_value) == -1) result = -3; #ifdef DEBUG_PRINT PyObject_Print(py_header_name, stdout, 0); printf(": "); PyObject_Print(py_header_value, stdout, 0); printf("\n"); #endif finally_loop: Py_XDECREF(py_header_value); Py_XDECREF(py_header_name); if(result == -3) goto finally;*/ } #ifdef DEBUG_PRINT if(self->content_length != CONTENT_LENGTH_UNSET) printf("self->content_length: %ld\n", self->content_length); if(self->transfer == PARSER_IDENTITY) printf("self->transfer: identity\n"); else if(self->transfer == PARSER_CHUNKED) printf("self->transfer: chunked\n"); #endif #ifdef PARSER_STANDALONE method_view = PyMemoryView_FromMemory(method, method_len, PyBUF_READ); path_view = PyMemoryView_FromMemory(path, path_len, PyBUF_READ); minor_version_long = PyLong_FromLong(minor_version); headers_view = PyMemoryView_FromMemory((char*)headers, sizeof(struct phr_header) * num_headers, PyBUF_READ); // FIXME the functions above can fail PyObject* on_headers_result = PyObject_CallFunctionObjArgs( self->on_headers, method_view, path_view, minor_version_long, headers_view, NULL); if(!on_headers_result) goto error; Py_DECREF(on_headers_result); #else if(!Protocol_on_headers( self->protocol, method, method_len, path, path_len, minor_version, headers, num_headers)) goto error; #endif self->buffer_start += (size_t)result; goto finally; #ifdef PARSER_STANDALONE PyObject* on_error_result; on_error: on_error_result = PyObject_CallFunctionObjArgs( self->on_error, error, NULL); if(!on_error_result) goto error; Py_DECREF(on_error_result); #else on_error: if(!Protocol_on_error(self->protocol, error)) goto error; #endif _reset_state(self, true); result = -1; goto finally; error: result = -3; finally: #ifdef PARSER_STANDALONE Py_XDECREF(headers_view); Py_XDECREF(minor_version_long); Py_XDECREF(path_view); Py_XDECREF(method_view); #endif return result; } static int _parse_body(Parser* self) { #ifdef PARSER_STANDALONE PyObject* body_view = NULL; #endif char* body = NULL; size_t body_len = 0; int result = -2; if(self->content_length == CONTENT_LENGTH_UNSET && self->transfer == PARSER_TRANSFER_UNSET) { result = 0; goto on_body; } if(self->content_length == 0) { body = (char*)zero_body; result = 0; goto on_body; } if(self->content_length != CONTENT_LENGTH_UNSET) { if(self->content_length > self->buffer_end - self->buffer_start) { result = -2; goto finally; } body = self->buffer + self->buffer_start; body_len = self->content_length; self->buffer_start += self->content_length; // TODO result = self->content_length (long) result = 1; goto on_body; } if(self->transfer == PARSER_CHUNKED) { size_t chunked_offset_start = self->chunked_offset; self->chunked_offset = self->buffer_end - self->buffer_start - self->chunked_offset; result = phr_decode_chunked( &self->chunked_decoder, self->buffer + self->buffer_start + chunked_offset_start, &self->chunked_offset); self->chunked_offset = self->chunked_offset + chunked_offset_start; if(result == -2) { self->buffer_end = self->buffer_start + self->chunked_offset; goto finally; } if(result == -1) goto on_error; body = self->buffer + self->buffer_start; body_len = self->chunked_offset; self->buffer_start += self->chunked_offset; self->buffer_end = self->buffer_start + (size_t)result; goto on_body; } goto finally; on_body: if(body) { #if 0 if(PyObject_SetAttrString(self->request, "body", body) == -1) { result = -3; goto finally; } #else /*((Request*)(self->request))->body = body; Py_INCREF(body);*/ #endif #ifdef DEBUG_PRINT printf("body: "); PyObject_Print(body, stdout, 0); printf("\n"); #endif } #ifdef PARSER_STANDALONE if(body) { body_view = PyMemoryView_FromMemory(body, body_len, PyBUF_READ); if(!body_view) goto error; } else { body_view = Py_None; Py_INCREF(body_view); } PyObject* on_body_result = PyObject_CallFunctionObjArgs( self->on_body, body_view, NULL); if(!on_body_result) goto error; Py_DECREF(on_body_result); #else if(!Protocol_on_body(self->protocol, body, body_len, self->buffer_end - self->buffer_start)) goto error; #endif _reset_state(self, false); goto finally; #ifdef PARSER_STANDALONE PyObject* on_error_result; on_error: on_error_result = PyObject_CallFunctionObjArgs( self->on_error, malformed_body, NULL); if(!on_error_result) goto error; Py_DECREF(on_error_result); #else on_error: if(!Protocol_on_error(self->protocol, malformed_body)) goto error; #endif _reset_state(self, true); result = -1; goto finally; error: result = -3; finally: #ifdef PARSER_STANDALONE Py_XDECREF(body_view); #endif return result; } #ifdef PARSER_STANDALONE static PyObject * Parser_feed(Parser* self, PyObject *args) #else Parser* Parser_feed(Parser* self, PyObject* py_data) #endif { char* data; int iresult = 0; #ifdef PARSER_STANDALONE PyObject* result = Py_None; // FIXME: can be called without __init__ #ifdef DEBUG_PRINT printf("feed\n"); #endif int data_len; if(!PyArg_ParseTuple(args, "y#", &data, &data_len)) goto error; #else Parser* result = self; Py_ssize_t data_len; if(PyBytes_AsStringAndSize(py_data, &data, &data_len) == -1) goto error; #endif if(self->buffer_start == self->buffer_end) { self->buffer_start = 0; self->buffer_end = 0; } else if((size_t)data_len > self->buffer_capacity - self->buffer_end) { memmove(self->buffer, self->buffer + self->buffer_start, self->buffer_end - self->buffer_start); self->buffer_end -= self->buffer_start; self->buffer_start = 0; } if((size_t)data_len > self->buffer_capacity - (self->buffer_end - self->buffer_start)) { self->buffer_capacity = MAX( self->buffer_capacity * 2, self->buffer_end - self->buffer_start + data_len); if(self->buffer == self->inline_buffer) { self->buffer = malloc(self->buffer_capacity); memcpy(self->buffer + self->buffer_start, self->inline_buffer + self->buffer_start, self->buffer_end - self->buffer_start); } else self->buffer = realloc(self->buffer, self->buffer_capacity); if(!self->buffer) goto error; } memcpy(self->buffer + self->buffer_end, data, (size_t)data_len); self->buffer_end += (size_t)data_len; while(self->buffer_start != self->buffer_end) { if(self->state == PARSER_HEADERS) { iresult = _parse_headers(self); if(iresult == -3) goto error; if(iresult <= 0) break; self->state = PARSER_BODY; } if(self->state == PARSER_BODY) { iresult = _parse_body(self); if(iresult == -3) goto error; if(iresult < 0) break; self->state = PARSER_HEADERS; } } #ifndef PARSER_STANDALONE if(iresult == -2) Protocol_on_incomplete(self->protocol); #endif goto finally; error: result = NULL; finally: #ifdef PARSER_STANDALONE if(result) Py_INCREF(result); #endif return result; } #ifdef PARSER_STANDALONE static PyObject * Parser_feed_disconnect(Parser* self) #else Parser* Parser_feed_disconnect(Parser* self) #endif { // FIXME: can be called without __init__ #ifdef DEBUG_PRINT printf("feed_disconnect\n"); #endif PyObject* error; if(self->state == PARSER_HEADERS && self->buffer_start != self->buffer_end) { error = incomplete_headers; goto on_error; } if(self->state == PARSER_BODY) { error = incomplete_body; goto on_error; } goto finally; #ifdef PARSER_STANDALONE PyObject* on_error_result; on_error: on_error_result = PyObject_CallFunctionObjArgs( self->on_error, error, NULL); if(!on_error_result) return NULL; /*FIXME maybe leak */ Py_DECREF(on_error_result); #else on_error: if(!Protocol_on_error(self->protocol, error)) { return NULL; /*FIXME maybe leak */ } #endif finally: _reset_state(self, true); #ifdef PARSER_STANDALONE Py_RETURN_NONE; #else return self; #endif } #ifdef PARSER_STANDALONE static PyObject * Parser_dump_buffer(Parser* self) { // printf("buffer: "); PyObject_Print(self->buffer, stdout, 0); printf("\n"); Py_RETURN_NONE; } static PyMethodDef Parser_methods[] = { {"feed", (PyCFunction)Parser_feed, METH_VARARGS, "feed"}, {"feed_disconnect", (PyCFunction)Parser_feed_disconnect, METH_NOARGS, "feed_disconnect" }, { "_dump_buffer", (PyCFunction)Parser_dump_buffer, METH_NOARGS, "_dump_buffer" }, {NULL} /* Sentinel */ }; static PyTypeObject ParserType = { PyVarObject_HEAD_INIT(NULL, 0) "cparser.HttpRequestParser", /* tp_name */ sizeof(Parser), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Parser_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "HttpRequestParser", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Parser_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Parser_init, /* tp_init */ 0, /* tp_alloc */ Parser_new, /* tp_new */ }; static PyModuleDef cparser = { PyModuleDef_HEAD_INIT, "cparser", "cparser", -1, NULL, NULL, NULL, NULL, NULL }; #endif #ifdef PARSER_STANDALONE PyMODINIT_FUNC PyInit_cparser(void) #else int cparser_init(void) #endif { if(supports_x86_sse42()) { _phr_parse_request = phr_parse_request_sse42; } else { printf("Warning: Host CPU doesn't support SSE 4.2, selecting slower implementation\n"); _phr_parse_request = phr_parse_request; } malformed_headers = NULL; invalid_headers = NULL; malformed_body = NULL; incomplete_headers = NULL; incomplete_body = NULL; excessive_data = NULL; /*empty_body = NULL; GET = NULL; POST = NULL; DELETE = NULL; HEAD = NULL; Host = NULL; User_Agent = NULL; Accept = NULL; Accept_Language = NULL; Accept_Encoding = NULL; Accept_Charset = NULL; Connection = NULL; Cookie = NULL; Content_Length = NULL; Transfer_Encoding = NULL; val_close = NULL; keep_alive = NULL;*/ #ifdef PARSER_STANDALONE PyObject* m = NULL; #else int m = 0; #endif #ifdef PARSER_STANDALONE if (PyType_Ready(&ParserType) < 0) goto error; m = PyModule_Create(&cparser); if (!m) goto error; #endif #define alloc_static(name) \ name = PyUnicode_FromString(#name); \ if(!name) \ goto error; #define alloc_static2(name, val) \ name = PyUnicode_FromString(val); \ if(!name) \ goto error; alloc_static(malformed_headers) alloc_static(malformed_body) alloc_static(incomplete_headers) alloc_static(invalid_headers) alloc_static(incomplete_body) alloc_static(excessive_data) /*empty_body = PyBytes_FromString(""); if(!empty_body) goto error; alloc_static(GET) alloc_static(POST) alloc_static(DELETE) alloc_static(HEAD) alloc_static(Host) alloc_static2(User_Agent, "User-Agent") alloc_static(Accept) alloc_static2(Accept_Language, "Accept-Language") alloc_static2(Accept_Encoding, "Accept-Encoding") alloc_static2(Accept_Charset, "Accept-Charset") alloc_static(Connection) alloc_static(Cookie) alloc_static2(Content_Length, "Content-Length") alloc_static2(Transfer_Encoding, "Transfer-Encoding") alloc_static2(val_close, "close") alloc_static2(keep_alive, "keep-alive")*/ #undef alloc_static #undef alloc_static2 #ifdef PARSER_STANDALONE Py_INCREF(&ParserType); PyModule_AddObject( m, "HttpRequestParser", (PyObject *)&ParserType); #endif goto finally; error: /*Py_XDECREF(keep_alive); Py_XDECREF(val_close); Py_XDECREF(Transfer_Encoding); Py_XDECREF(Content_Length); Py_XDECREF(Cookie); Py_XDECREF(Connection); Py_XDECREF(Accept_Charset); Py_XDECREF(Accept_Encoding); Py_XDECREF(Accept_Language); Py_XDECREF(Accept); Py_XDECREF(User_Agent); Py_XDECREF(Host); Py_XDECREF(HEAD); Py_XDECREF(DELETE); Py_XDECREF(POST); Py_XDECREF(GET); Py_XDECREF(empty_body);*/ Py_XDECREF(incomplete_body); Py_XDECREF(invalid_headers); Py_XDECREF(incomplete_headers); Py_XDECREF(malformed_body); Py_XDECREF(malformed_headers); #ifndef PARSER_STANDALONE m = -1; #endif finally: return m; } ================================================ FILE: src/japronto/parser/cparser.h ================================================ #pragma once #include #include #include "picohttpparser.h" enum Parser_state { PARSER_HEADERS, PARSER_BODY }; enum Parser_transfer { PARSER_TRANSFER_UNSET, PARSER_IDENTITY, PARSER_CHUNKED }; enum Parser_connection { PARSER_CONNECTION_UNSET, PARSER_CLOSE, PARSER_KEEP_ALIVE }; #define PARSER_INITIAL_BUFFER_SIZE 4096 typedef struct { #ifdef PARSER_STANDALONE PyObject_HEAD #endif enum Parser_state state; enum Parser_transfer transfer; enum Parser_connection connection; unsigned long content_length; struct phr_chunked_decoder chunked_decoder; size_t chunked_offset; char* buffer; size_t buffer_start; size_t buffer_end; size_t buffer_capacity; char inline_buffer[PARSER_INITIAL_BUFFER_SIZE]; #ifdef PARSER_STANDALONE PyObject* on_headers; PyObject* on_body; PyObject* on_error; #else void* protocol; #endif } Parser; #ifndef PARSER_STANDALONE void Parser_new(Parser* self); int Parser_init(Parser* self, void* protocol); void Parser_dealloc(Parser* self); Parser* Parser_feed(Parser* self, PyObject* py_data); Parser* Parser_feed_disconnect(Parser* self); int cparser_init(void); #endif ================================================ FILE: src/japronto/parser/cparser_ext.py ================================================ from distutils.core import Extension def get_extension(): return Extension( 'japronto.parser.cparser', sources=['cparser.c', '../cpu_features.c'], include_dirs=['../../picohttpparser', '..'], extra_objects=[ 'src/picohttpparser/picohttpparser.o', 'src/picohttpparser/ssepicohttpparser.o'], define_macros=[('PARSER_STANDALONE', 1)]) ================================================ FILE: src/japronto/parser/test_parser.py ================================================ from functools import partial from itertools import zip_longest import pytest from cases import parametrize_cases from parts import one_part, make_parts, geometric_series, fancy_series from protocol.tracing import CTracingProtocol, CffiTracingProtocol from parser import cffiparser, header_errors, body_errors try: from parser import cparser except ImportError: cparser = None if cparser: def make_c(protocol_factory=CTracingProtocol): protocol = protocol_factory() parser = cparser.HttpRequestParser( protocol.on_headers, protocol.on_body, protocol.on_error) return parser, protocol def make_cffi(protocol_factory=CffiTracingProtocol): protocol = protocol_factory() parser = cffiparser.HttpRequestParser( protocol.on_headers, protocol.on_body, protocol.on_error) return parser, protocol @pytest.mark.parametrize('data,get_size,dir,parts', [ (b'abcde', 2, 1, [b'ab', b'cd', b'e']), (b'abcde', 2, -1, [b'a', b'bc', b'de']), (b'aaBBBBccccCCCCd', geometric_series(), 1, [b'aa', b'BBBB', b'ccccCCCC', b'd']), (b'dCCCCccccBBBBaa', geometric_series(), -1, [b'd', b'CCCCcccc', b'BBBB', b'aa']) ]) def test_make_parts(data, get_size, dir, parts): assert make_parts(data, get_size, dir) == parts def parametrize_make_parser(): ids = [] factories = [] if 'make_c' in globals(): factories.append(make_c) ids.append('c') factories.append(make_cffi) ids.append('cffi') return pytest.mark.parametrize('make_parser', factories, ids=ids) def parametrize_do_parts(): funcs = [ one_part, partial(make_parts, get_size=15), partial(make_parts, get_size=geometric_series()), partial(make_parts, get_size=geometric_series(), dir=-1), partial(make_parts, get_size=fancy_series()) ] ids = ['one', 'const', 'geom', 'invgeom', 'fancy'] return pytest.mark.parametrize('do_parts', funcs, ids=ids) _begin = object() _end = object() @parametrize_do_parts() @parametrize_cases( 'base', '10msg', '10msg!', '10get', '10get!', 'keep:10msg+10get', 'keep:10get+10msg', '10malformed_headers1', '10malformed_headers2', '10incomplete_headers!', 'keep:10msg+10malformed_headers2', 'keep:10msg+10incomplete_headers!', 'keep:10get+10malformed_headers1', 'keep:10get+10malformed_headers2', '10msg!+10get!', '10get!+10msg!', '10msg!+keep:10get+keep:10msg+10get', '10msg+e excessive_data:10get', '10get+e excessive_data:10msg') @parametrize_make_parser() def test_http10(make_parser, do_parts, cases): parser, protocol = make_parser() def flush(): nonlocal data if not data: return parts = do_parts(data) for part in parts: parser.feed(part) if protocol.error: break data = b'' data = b'' for case in cases: data += case.data if case.disconnect: flush() parser.feed_disconnect() flush() header_count = 0 error_count = 0 body_count = 0 for case, request in zip_longest(cases, protocol.requests): if case.error: assert protocol.error == case.error if case.error in header_errors: error_count += 1 break header_count += 1 assert request.method == case.method assert request.path == case.path assert request.version == case.version assert request.headers == case.headers if case.error in body_errors: error_count += 1 break body_count += 1 assert request.body == case.body assert protocol.on_headers_call_count == header_count assert protocol.on_error_call_count == error_count assert protocol.on_body_call_count == body_count @parametrize_make_parser() def test_empty(make_parser): parser, protocol = make_parser() parser.feed_disconnect() parser.feed(b'') parser.feed(b'') parser.feed_disconnect() parser.feed_disconnect() parser.feed(b'') assert not protocol.on_headers_call_count assert not protocol.on_error_call_count assert not protocol.on_body_call_count @parametrize_do_parts() @parametrize_cases( 'base', '11get', '11getmsg', '11msg', '11msgzero', 'close:11get', 'close:11msg', '11get!', '11getmsg!', '11msg!', 'close:11msgzero!', '11msg+close:11msg', '11msg+11msg', 'close:11msg!+11msg', 'close:11msg!+close:11msg', '11msg!+close:11msg', '11msg!+11msg', '11get+close:11msg', '11msg+11get', '11getmsg+11get', '11get+close:11msg!', '11msg!+11get', '11getmsg!+11get!', '11msg+11msg+close:11msg', '11msg+11msg+11msg', '11msg+11msgzero+11msg', '11msgzero+11msg+11msgzero', '11msg+11get+11msgzero', '11msgzero+11msgzero', '11get+11getmsg+11get', 'close:11msg+e excessive_data:11msg', 'close:11msg+e excessive_data:close:11msg', 'close:11msg+e excessive_data:close:11msg+11msg', '11msg+close:11msgzero+e excessive_data:11get', '11clincomplete_headers!', '11clincomplete_body!', '11clinvalid1', '11clinvalid2', '11clinvalid3', '11clinvalid4', '11clinvalid5', '11msg+11clincomplete_headers!', 'close:11msg!+11clincomplete_body!', '11msgzero+11clincomplete_headers!', '11msgzero+11clincomplete_body!', 'close:11msg!+11msg+11clincomplete_body!', '11get+11clincomplete_body!', '11getmsg+11clincomplete_headers!' ) @parametrize_make_parser() def test_http11(make_parser, do_parts, cases): parser, protocol = make_parser() def flush(): nonlocal data if not data: return parts = do_parts(data) for part in parts: parser.feed(part) if protocol.error: break data = b'' data = b'' for case in cases: data += case.data if case.disconnect: flush() parser.feed_disconnect() flush() header_count = 0 error_count = 0 body_count = 0 for case, request in zip_longest(cases, protocol.requests): if case.error: assert protocol.error == case.error if case.error in header_errors: error_count += 1 break header_count += 1 assert request.method == case.method assert request.path == case.path assert request.version == case.version assert request.headers == case.headers if case.error in body_errors: error_count += 1 break body_count += 1 assert request.body == case.body assert protocol.on_headers_call_count == header_count assert protocol.on_error_call_count == error_count assert protocol.on_body_call_count == body_count @parametrize_do_parts() @parametrize_cases( 'base', '11chunked1', '11chunked2', '11chunked3', '11chunkedzero', '11chunked1+11chunked1', '11chunked1+11chunked2', '11chunked2+11chunked1', '11chunked2+11chunked3', '11chunked1+11chunked2+11chunked3', '11chunked3+11chunked2+11chunked1', '11chunked3+11chunked3+11chunked3', '11chunkedincomplete_body!', '11chunkedmalformed_body', '11chunked1+11chunkedincomplete_body!', '11chunked1+11chunkedmalformed_body', '11chunked2+11chunkedincomplete_body!', '11chunked2+11chunkedmalformed_body', '11chunked2+11chunked2+11chunkedincomplete_body!', '11chunked3+11chunked1+11chunkedmalformed_body' ) @parametrize_make_parser() def test_http11_chunked(make_parser, do_parts, cases): parser, protocol = make_parser() def flush(): nonlocal data if not data: return parts = do_parts(data) for part in parts: parser.feed(part) if protocol.error: break data = b'' data = b'' for case in cases: data += case.data if case.disconnect: flush() parser.feed_disconnect() flush() header_count = 0 error_count = 0 body_count = 0 for case, request in zip_longest(cases, protocol.requests): if case.error: assert protocol.error == case.error if case.error in header_errors: error_count += 1 break header_count += 1 assert request.method == case.method assert request.path == case.path assert request.version == case.version assert request.headers == case.headers if case.error in body_errors: error_count += 1 break body_count += 1 assert request.body == case.body assert protocol.on_headers_call_count == header_count assert protocol.on_error_call_count == error_count assert protocol.on_body_call_count == body_count @parametrize_do_parts() @parametrize_cases( 'base', '11chunked1+11msgzero', '11msg+11chunked2', '11chunked2+close:11msg', '11msgzero+11chunked3', 'close:11msg+e excessive_data:11chunked1+11chunked3', '11chunked3+11msg+close:11msg', '11chunked3+11chunked3+close:11msg' ) @parametrize_make_parser() def test_http11_mixed(make_parser, do_parts, cases): parser, protocol = make_parser() def flush(): nonlocal data if not data: return parts = do_parts(data) for part in parts: parser.feed(part) if protocol.error: break data = b'' data = b'' for case in cases: data += case.data if case.disconnect: flush() parser.feed_disconnect() flush() header_count = 0 error_count = 0 body_count = 0 for case, request in zip_longest(cases, protocol.requests): if case.error: assert protocol.error == case.error if case.error in header_errors: error_count += 1 break header_count += 1 assert request.method == case.method assert request.path == case.path assert request.version == case.version assert request.headers == case.headers if case.error in body_errors: error_count += 1 break body_count += 1 assert request.body == case.body assert protocol.on_headers_call_count == header_count assert protocol.on_error_call_count == error_count assert protocol.on_body_call_count == body_count ================================================ FILE: src/japronto/pipeline/__init__.py ================================================ class Pipeline: def __init__(self, ready): self._queue = [] self._ready = ready @property def empty(self): return not self._queue def queue(self, task): print("queued") self._queue.append(task) task.add_done_callback(self._task_done) def _task_done(self, task): print('Done', task.result()) pop_idx = 0 for task in self._queue: if not task.done(): break self.write(task) pop_idx += 1 if pop_idx: self._queue[:pop_idx] = [] def write(self, task): self._ready(task) print('Written', task.result()) if __name__ == '__main__': import asyncio async def coro(sleep): await asyncio.sleep(sleep) return sleep from uvloop import new_event_loop loop = new_event_loop() asyncio.set_event_loop(loop) pipeline = Pipeline() def queue(x): t = loop.create_task(coro(x)) pipeline.queue(t) loop.call_later(2, lambda: queue(2)) loop.call_later(12, lambda: queue(2)) queue(1) queue(10) queue(5) queue(1) loop.run_forever() ================================================ FILE: src/japronto/pipeline/cpipeline.c ================================================ #include #include "structmember.h" #include "cpipeline.h" static PyTypeObject PipelineType; #ifdef PIPELINE_OPAQUE static PyObject* Pipeline_new(PyTypeObject* type, PyObject* args, PyObject* kw) #else PyObject* Pipeline_new(Pipeline* self) #endif { #ifdef PIPELINE_OPAQUE Pipeline* self = NULL; self = (Pipeline*)type->tp_alloc(type, 0); if(!self) goto finally; #else ((PyObject*)self)->ob_refcnt = 1; ((PyObject*)self)->ob_type = &PipelineType; #endif self->ready = NULL; self->task_done = NULL; #ifdef PIPELINE_OPAQUE finally: #endif return (PyObject*)self; } #ifdef PIPELINE_OPAQUE static void #else void #endif Pipeline_dealloc(Pipeline* self) { #ifdef PIPELINE_OPAQUE Py_XDECREF(self->ready); #endif Py_XDECREF(self->task_done); #ifdef PIPELINE_OPAQUE Py_TYPE(self)->tp_free((PyObject*)self); #endif } #ifdef PIPELINE_OPAQUE static int Pipeline_init(Pipeline* self, PyObject *args, PyObject* kw) #else int Pipeline_init(Pipeline* self, void* (*ready)(PipelineEntry, PyObject*), PyObject* protocol) #endif { int result = 0; #ifdef PIPELINE_OPAQUE if(!PyArg_ParseTuple(args, "O", &self->ready)) goto error; Py_INCREF(self->ready); #else self->ready = ready; self->protocol = protocol; #endif if(!(self->task_done = PyObject_GetAttrString((PyObject*)self, "_task_done"))) goto error; self->queue_start = 0; self->queue_end = 0; goto finally; error: result = -1; finally: return result; } static PyObject* Pipeline__task_done(Pipeline* self, PyObject* task) { PyObject* result = Py_True; PipelineEntry *queue_entry; for(queue_entry = self->queue + self->queue_start; queue_entry < self->queue + self->queue_end; queue_entry++) { PyObject* done = NULL; PyObject* done_result = NULL; result = Py_True; if(PipelineEntry_is_task(*queue_entry)) { task = PipelineEntry_get_task(*queue_entry); if(!(done = PyObject_GetAttrString(task, "done"))) goto loop_error; if(!(done_result = PyObject_CallFunctionObjArgs(done, NULL))) goto loop_error; if(done_result == Py_False) { result = Py_False; goto loop_finally; } } #ifdef PIPELINE_OPAQUE PyObject* tmp; if(!(tmp = PyObject_CallFunctionObjArgs(self->ready, *queue_entry, NULL))) goto loop_error; Py_DECREF(tmp); #else if(!self->ready(*queue_entry, self->protocol)) goto loop_error; #endif PipelineEntry_DECREF(*queue_entry); goto loop_finally; loop_error: result = NULL; loop_finally: Py_XDECREF(done_result); Py_XDECREF(done); if(!result) goto error; if(result == Py_False) break; } self->queue_start = queue_entry - self->queue; #ifndef PIPELINE_OPAQUE if(PIPELINE_EMPTY(self)) // we became empty so release protocol Py_DECREF(self->protocol); #endif goto finally; error: result = NULL; finally: Py_XINCREF(result); return result; } #ifdef PIPELINE_OPAQUE static PyObject* #else PyObject* #endif Pipeline_queue(Pipeline* self, PipelineEntry entry) { PyObject* result = Py_None; PyObject* add_done_callback = NULL; if(PIPELINE_EMPTY(self)) { self->queue_start = self->queue_end = 0; #ifndef PIPELINE_OPAQUE // we will become non empty so hold a reference to protocol Py_INCREF(self->protocol); #endif } assert(self->queue_end < sizeof(self->queue) / sizeof(self->queue[0])); PipelineEntry* queue_entry = self->queue + self->queue_end; *queue_entry = entry; PipelineEntry_INCREF(*queue_entry); self->queue_end++; if(PipelineEntry_is_task(entry)) { PyObject* task = PipelineEntry_get_task(entry); if(!(add_done_callback = PyObject_GetAttrString(task, "add_done_callback"))) goto error; PyObject* tmp; if(!(tmp = PyObject_CallFunctionObjArgs(add_done_callback, self->task_done, NULL))) goto error; Py_DECREF(tmp); } goto finally; error: result = NULL; finally: Py_XDECREF(add_done_callback); #ifdef PIPELINE_OPAQUE Py_XINCREF(result); #endif return result; } #ifndef PIPELINE_OPAQUE void* Pipeline_cancel(Pipeline* self) { void* result = self; PipelineEntry *queue_entry; for(queue_entry = self->queue + self->queue_start; queue_entry < self->queue + self->queue_end; queue_entry++) { if(!PipelineEntry_is_task(*queue_entry)) continue; PyObject* task = PipelineEntry_get_task(*queue_entry); PyObject* cancel = NULL; if(!(cancel = PyObject_GetAttrString(task, "cancel"))) goto loop_error; PyObject* tmp; if(!(tmp = PyObject_CallFunctionObjArgs(cancel, NULL))) goto loop_error; Py_DECREF(tmp); goto loop_finally; loop_error: result = NULL; loop_finally: Py_XDECREF(cancel); if(!result) break; } return result; } #endif #ifdef PIPELINE_OPAQUE static PyObject* Pipeline_get_empty(Pipeline* self, void* closure) { PyObject* result = PIPELINE_EMPTY(self) ? Py_True : Py_False; Py_INCREF(result); return result; } #endif static PyMethodDef Pipeline_methods[] = { #ifdef PIPELINE_OPAQUE {"queue", (PyCFunction)Pipeline_queue, METH_O, ""}, #endif {"_task_done", (PyCFunction)Pipeline__task_done, METH_O, ""}, {NULL} }; #ifdef PIPELINE_OPAQUE static PyGetSetDef Pipeline_getset[] = { {"empty", (getter)Pipeline_get_empty, NULL, "", NULL}, {NULL} }; #endif static PyTypeObject PipelineType = { PyVarObject_HEAD_INIT(NULL, 0) "cpipeline.Pipeline", /* tp_name */ sizeof(Pipeline), /* tp_basicsize */ 0, /* tp_itemsize */ #ifdef PIPELINE_OPAQUE (destructor)Pipeline_dealloc, /* tp_dealloc */ #else 0, /* tp_dealloc */ #endif 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Pipeline", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Pipeline_methods, /* tp_methods */ 0, /* tp_members */ #ifdef PIPELINE_OPAQUE Pipeline_getset, /* tp_getset */ #else 0, /* tp_getset */ #endif 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ #ifdef PIPELINE_OPAQUE (initproc)Pipeline_init, /* tp_init */ #else 0, /* tp_init */ #endif 0, /* tp_alloc */ #ifdef PIPELINE_OPAQUE Pipeline_new, /* tp_new */ #else 0, /* tp_new */ #endif }; #ifdef PIPELINE_OPAQUE static PyModuleDef cpipeline = { PyModuleDef_HEAD_INIT, "cpipeline", "cpipeline", -1, NULL, NULL, NULL, NULL, NULL }; #endif #ifdef PIPELINE_OPAQUE PyMODINIT_FUNC PyInit_cpipeline(void) #else void* cpipeline_init(void) #endif { #ifdef PIPELINE_OPAQUE PyObject* m = NULL; #else void* m = &PipelineType; #endif if(PyType_Ready(&PipelineType) < 0) goto error; #ifdef PIPELINE_OPAQUE if(!(m = PyModule_Create(&cpipeline))) goto error; Py_INCREF(&PipelineType); PyModule_AddObject(m, "Pipeline", (PyObject*)&PipelineType); #endif goto finally; error: m = NULL; finally: return m; } ================================================ FILE: src/japronto/pipeline/cpipeline.h ================================================ #pragma once #include #include #ifdef PIPELINE_PAIR typedef struct { bool is_task; PyObject* request; PyObject* task; } PipelineEntry; static inline bool PipelineEntry_is_task(PipelineEntry entry) { return entry.is_task; } static inline void PipelineEntry_DECREF(PipelineEntry entry) { Py_DECREF(entry.request); // if not real task this was response, // that was inside request that was already freed above if(entry.is_task) Py_XDECREF(entry.task); } static inline void PipelineEntry_INCREF(PipelineEntry entry) { Py_INCREF(entry.request); Py_XINCREF(entry.task); } static inline PyObject* PipelineEntry_get_task(PipelineEntry entry) { return entry.task; } #else typedef PyObject* PipelineEntry; static inline bool PipelineEntry_is_task(PipelineEntry entry) { return true; } static inline void PipelineEntry_DECREF(PipelineEntry entry) { Py_DECREF(entry); } static inline void PipelineEntry_INCREF(PipelineEntry entry) { Py_INCREF(entry); } static inline PyObject* PipelineEntry_get_task(PipelineEntry entry) { return entry; } #endif typedef struct { PyObject_HEAD #ifdef PIPELINE_OPAQUE PyObject* ready; #else void* (*ready)(PipelineEntry, PyObject*); PyObject* protocol; #endif PyObject* task_done; PipelineEntry queue[10]; size_t queue_start; size_t queue_end; } Pipeline; #define PIPELINE_EMPTY(p) ((p)->queue_start == (p)->queue_end) #ifndef PIPELINE_OPAQUE PyObject* Pipeline_new(Pipeline* self); void Pipeline_dealloc(Pipeline* self); int Pipeline_init(Pipeline* self, void* (*ready)(PipelineEntry, PyObject*), PyObject* protocol); PyObject* Pipeline_queue(Pipeline* self, PipelineEntry entry); void* Pipeline_cancel(Pipeline* self); void* cpipeline_init(void); #endif ================================================ FILE: src/japronto/pipeline/cpipeline_ext.py ================================================ from distutils.core import Extension def get_extension(): return Extension( 'japronto.pipeline.cpipeline', sources=['cpipeline.c'], include_dirs=[], libraries=[], library_dirs=[], extra_link_args=[], define_macros=[('PIPELINE_OPAQUE', 1)]) ================================================ FILE: src/japronto/pipeline/test_pipeline.py ================================================ import asyncio import gc import sys from collections import namedtuple from functools import partial import pytest import uvloop from japronto.pipeline import Pipeline from japronto.pipeline.cpipeline import Pipeline as CPipeline Example = namedtuple('Example', 'value,delay') class FakeLoop: def call_soon(self, callback, val): callback(val) def get_debug(self): return False def create_future(self): return asyncio.Future(loop=self) class FakeFuture: cnt = 0 def __new__(cls): print('new') cls.cnt += 1 return object.__new__(cls) def __del__(self): type(self).cnt -= 1 print('del') def __init__(self): self.callbacks = [] def add_done_callback(self, cb): self.callbacks.append(cb) def done(self): return hasattr(self, '_result') def result(self): return self._result def set_result(self, result): self._result = result for cb in self.callbacks: cb(self) self.callbacks = [] def parametrize_make_pipeline(): def make_pipeline(cls): results = [] def append(task): results.append(task.result()) return cls(append), results return pytest.mark.parametrize( 'make_pipeline', [partial(make_pipeline, CPipeline), partial(make_pipeline, Pipeline)], ids=['c', 'py']) def parametrize_case(examples): cases = [parse_case(i) for i in examples] return pytest.mark.parametrize('case', cases, ids=examples) def parse_example(e, accum): value, delay = map(int, e.split('@')) if '@' in e else (int(e), 0) return Example(value, delay + accum) def parse_case(case): results = [] delay = 0 for c in case.split('+'): e = parse_example(c, delay) results.append(e) delay = e.delay return results def create_futures(resolves, case): futures = [None] * len(case) case = case[:] for c in sorted(case): idx = case.index(c) futures[idx] = resolves[idx]() case[idx] = None return tuple(futures) @parametrize_case([ '1', '1+5', '5+1', '1+5+10', '10+5+1', '5+1+10', '10+1+5', '1+10+5+1', '1+1+10+5', '10+5+1+1', '1+1+5+10' ]) @parametrize_make_pipeline() def test_fake_future(make_pipeline, case): pipeline, results = make_pipeline() def queue(x): fut = FakeFuture() pipeline.queue(fut) def resolve(): fut.set_result(x) return fut return resolve resolves = tuple(queue(v) for v in case) futures = create_futures(resolves, case) assert pipeline.empty del resolves # this loop is not pythonic on purpose # carefully don't create extra references for i in range(len(futures)): print(sys.getrefcount(futures[i])) del i assert results == case gc.collect() del futures gc.set_debug(gc.DEBUG_LEAK) gc.collect() print(gc.garbage) gc.set_debug(0) assert FakeFuture.cnt == 0 def parametrize_loop(): return pytest.mark.parametrize( 'loop', [uvloop.new_event_loop(), asyncio.new_event_loop()], ids=['uv', 'aio']) @parametrize_case([ '1', '1@1', '1+2', '2+1', '2+1@1', '1@1+2', '1+2+3', '3+2+1', '2+1+3', '3+1+2', '1+3+2+1', '1+3+2+1+1@1', '1+1+3+2', '3+2+1+1', '1+1+2+3' ]) @parametrize_make_pipeline() @parametrize_loop() def test_real_task(loop, make_pipeline, case): DIVISOR = 1000 pipeline, results = make_pipeline() async def coro(example): await asyncio.sleep(example.value / DIVISOR, loop=loop) return example def queue(x): task = loop.create_task(coro(x)) pipeline.queue(task) for v in case: if v.delay: loop.call_later(v.delay / DIVISOR, partial(queue, v)) else: queue(v) duration = max((e.value + e.delay) / DIVISOR for e in case) loop.run_until_complete(asyncio.sleep(duration, loop=loop)) # timing issue, wait a little bit more so we collect all the results if len(results) < len(case): loop.run_until_complete(asyncio.sleep(10 / DIVISOR, loop=loop)) assert pipeline.empty assert results == case ================================================ FILE: src/japronto/protocol/__init__.py ================================================ ================================================ FILE: src/japronto/protocol/cprotocol.c ================================================ #include #include "cprotocol.h" #include "cmatcher.h" #include "crequest.h" #include "cresponse.h" #include "capsule.h" #include "match_dict.h" #ifdef PARSER_STANDALONE static PyObject* Parser; #endif static PyObject* PyRequest; static PyObject* RouteNotFoundException; static Request_CAPI* request_capi; static Matcher_CAPI* matcher_capi; static Response_CAPI* response_capi; static PyObject * Protocol_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { Protocol* self = NULL; self = (Protocol*)type->tp_alloc(type, 0); if(!self) goto finally; #ifdef PARSER_STANDALONE self->feed = NULL; self->feed_disconnect = NULL; #else Parser_new(&self->parser); #endif Pipeline_new(&self->pipeline); Request_new(request_capi->RequestType, &self->static_request); self->app = NULL; self->matcher = NULL; self->error_handler = NULL; self->transport = NULL; self->write = NULL; self->create_task = NULL; self->request_logger = NULL; self->gather.prev_buffer = NULL; finally: return (PyObject*)self; } static void Protocol_dealloc(Protocol* self) { Py_XDECREF(self->gather.prev_buffer); Py_XDECREF(self->request_logger); Py_XDECREF(self->create_task); Py_XDECREF(self->write); Py_XDECREF(self->writelines); Py_XDECREF(self->transport); Py_XDECREF(self->error_handler); Py_XDECREF(self->matcher); Py_XDECREF(self->app); Request_dealloc(&self->static_request); Pipeline_dealloc(&self->pipeline); #ifdef PARSER_STANDALONE Py_XDECREF(self->feed_disconnect); Py_XDECREF(self->feed); #else Parser_dealloc(&self->parser); #endif Py_TYPE(self)->tp_free((PyObject*)self); } static void* Protocol_pipeline_ready(PipelineEntry entry, PyObject* protocol); static int Protocol_init(Protocol* self, PyObject *args, PyObject *kw) { int result = 0; PyObject* loop = NULL; PyObject* log_request = NULL; #ifdef PARSER_STANDALONE PyObject* parser = NULL; PyObject* on_headers = PyObject_GetAttrString((PyObject*)self, "on_headers"); if(!on_headers) // FIXME leak goto error; PyObject* on_body = PyObject_GetAttrString((PyObject*)self, "on_body"); if(!on_body) // FIXME leak goto error; PyObject* on_error = PyObject_GetAttrString((PyObject*)self, "on_error"); if(!on_error) // FIXME leak goto error; parser = PyObject_CallFunctionObjArgs( Parser, on_headers, on_body, on_error, NULL); if(!parser) goto error; self->feed = PyObject_GetAttrString(parser, "feed"); if(!self->feed) goto error; self->feed_disconnect = PyObject_GetAttrString(parser, "feed_disconnect"); if(!self->feed_disconnect) goto error; #else if(Parser_init(&self->parser, self) == -1) goto error; #endif if(Pipeline_init(&self->pipeline, Protocol_pipeline_ready, (PyObject*)self) == -1) goto error; if(!PyArg_ParseTuple(args, "O", &self->app)) goto error; Py_INCREF(self->app); self->matcher = PyObject_GetAttrString(self->app, "_matcher"); if(!self->matcher) goto error; self->error_handler = PyObject_GetAttrString(self->app, "error_handler"); if(!self->error_handler) goto error; loop = PyObject_GetAttrString(self->app, "_loop"); if(!loop) goto error; self->create_task = PyObject_GetAttrString(loop, "create_task"); if(!self->create_task) goto error; if(!(log_request = PyObject_GetAttrString(self->app, "_log_request"))) goto error; if(log_request == Py_True) { if(!(self->request_logger = PyObject_GetAttrString(self->app, "default_request_logger"))) goto error; } self->gather.responses_end = 0; self->gather.len = 0; goto finally; error: result = -1; finally: Py_XDECREF(log_request); Py_XDECREF(loop); #ifdef PARSER_STANDALONE Py_XDECREF(parser); #endif return result; } static PyObject* Protocol_connection_made(Protocol* self, PyObject* transport) { #ifdef PROTOCOL_TRACK_REFCNT printf("made: %ld, %ld, %ld, ", (size_t)Py_REFCNT(Py_None), (size_t)Py_REFCNT(Py_True), (size_t)Py_REFCNT(Py_False)); self->none_cnt = Py_REFCNT(Py_None); self->true_cnt = Py_REFCNT(Py_True); self->false_cnt = Py_REFCNT(Py_False); #endif PyObject* connections = NULL; self->transport = transport; Py_INCREF(self->transport); if(!(self->write = PyObject_GetAttrString(transport, "write"))) goto error; if(!(self->writelines = PyObject_GetAttrString(transport, "writelines"))) goto error; if(!(connections = PyObject_GetAttrString(self->app, "_connections"))) goto error; #ifdef REAPER_ENABLED self->idle_time = 0; self->read_ops = 0; self->last_read_ops = 0; #endif if(PySet_Add(connections, (PyObject*)self) == -1) goto error; self->closed = false; goto finally; error: return NULL; finally: Py_XDECREF(connections); Py_RETURN_NONE; } static void* Protocol_close(Protocol* self) { void* result = self; PyObject* close = NULL; close = PyObject_GetAttrString(self->transport, "close"); if(!close) goto error; PyObject* tmp = PyObject_CallFunctionObjArgs(close, NULL); if(!tmp) goto error; Py_DECREF(tmp); goto finally; error: result = NULL; finally: Py_XDECREF(close); return result; } static PyObject* Protocol_connection_lost(Protocol* self, PyObject* args) { self->closed = true; PyObject* connections = NULL; PyObject* result = Py_None; #ifdef PARSER_STANDALONE PyObject* result = PyObject_CallFunctionObjArgs( self->feed_disconnect, NULL); if(!result) goto error; Py_DECREF(result); // FIXME: result can leak #else if(!Parser_feed_disconnect(&self->parser)) goto error; #endif if(!(connections = PyObject_GetAttrString(self->app, "_connections"))) goto error; if(PySet_Discard(connections, (PyObject*)self) == -1) goto error; if(!Pipeline_cancel(&self->pipeline)) goto error; #ifdef PROTOCOL_TRACK_REFCNT printf("lost: %ld, %ld, %ld\n", (size_t)Py_REFCNT(Py_None), (size_t)Py_REFCNT(Py_True), (size_t)Py_REFCNT(Py_False)); assert(Py_REFCNT(Py_None) == self->none_cnt); assert(Py_REFCNT(Py_True) == self->true_cnt); assert(Py_REFCNT(Py_False) >= self->false_cnt); #endif goto finally; error: result = NULL; finally: Py_XDECREF(connections); Py_XINCREF(result); return result; } static PyObject* Protocol_data_received(Protocol* self, PyObject* data) { #ifdef REAPER_ENABLED self->read_ops++; #endif #ifdef PARSER_STANDALONE PyObject* result = PyObject_CallFunctionObjArgs( self->feed, data, NULL); if(!result) goto error; Py_DECREF(result); #else if(!Parser_feed(&self->parser, data)) goto error; #endif goto finally; error: return NULL; finally: Py_RETURN_NONE; } static inline PyObject* Gather_flush(Gather* gather); #ifndef PARSER_STANDALONE Protocol* Protocol_on_incomplete(Protocol* self) { Gather* gather = &self->gather; PyObject* gather_buffer = NULL; if(!gather->len) goto finally; if(!(gather_buffer = Gather_flush(gather))) goto error; PyObject* tmp; if(!(tmp = PyObject_CallFunctionObjArgs(self->write, gather_buffer, NULL))) goto error; Py_DECREF(tmp); goto finally; error: self = NULL; finally: Py_XDECREF(gather_buffer); return self; } #endif #ifdef PARSER_STANDALONE static PyObject* Protocol_on_headers(Protocol* self, PyObject *args) { Py_RETURN_NONE; } #else Protocol* Protocol_on_headers(Protocol* self, char* method, size_t method_len, char* path, size_t path_len, int minor_version, void* headers, size_t num_headers) { Protocol* result = self; Request_dealloc(&self->static_request); Request_new(request_capi->RequestType, &self->static_request); request_capi->Request_from_raw( &self->static_request, method, method_len, path, path_len, minor_version, headers, num_headers); goto finally; finally: return result; } #endif #define Protocol_catch_exception(request) \ { \ PyObject* etype; \ PyObject* evalue; \ PyObject* etraceback; \ \ PyErr_Fetch(&etype, &evalue, &etraceback); \ PyErr_NormalizeException(&etype, &evalue, &etraceback); \ if(etraceback) { \ PyException_SetTraceback(evalue, etraceback); \ Py_DECREF(etraceback); \ } \ Py_DECREF(etype); \ \ ((Request*)request)->exception = evalue; \ } static inline PyBytesObject* Bytes_FromSize(size_t size) { PyBytesObject* result; if(!(result = malloc(sizeof(PyBytesObject) + GATHER_MAX_LEN))) return (PyBytesObject*)PyErr_NoMemory(); result->ob_base.ob_base.ob_refcnt = 1; result->ob_base.ob_base.ob_type = &PyBytes_Type; result->ob_base.ob_size = (Py_ssize_t)size; return result; } static inline PyObject* Gather_flush(Gather* gather) { PyBytesObject* gather_buffer = NULL; if(gather->responses_end == 1) { gather_buffer = (PyBytesObject*)gather->responses[0]; goto reset; } if(gather->prev_buffer) { if(Py_REFCNT(gather->prev_buffer) == 1) { gather_buffer = gather->prev_buffer; Py_SIZE(gather_buffer) = (ssize_t)gather->len; } else { Py_DECREF(gather->prev_buffer); gather->prev_buffer = NULL; } } if(!gather_buffer && !(gather_buffer = Bytes_FromSize(gather->len))) goto error; size_t gather_offset = 0; for(size_t i = 0; i < gather->responses_end; i++) { PyObject* item = gather->responses[i]; memcpy( gather_buffer->ob_sval + gather_offset, PyBytes_AS_STRING(item), Py_SIZE(item)); gather_offset += Py_SIZE(item); Py_DECREF(item); } gather->prev_buffer = gather_buffer; reset: gather->responses_end = 0; gather->len = 0; goto finally; error: return NULL; finally: if(gather_buffer == gather->prev_buffer) Py_INCREF(gather_buffer); return (PyObject*)gather_buffer; } static inline Protocol* Protocol_write_response_or_err(Protocol* self, PyObject* request, Response* response) { Protocol* result = self; PyObject* response_bytes = NULL; PyObject* error_result = NULL; PyObject* gather_buffer = NULL; if(response && Py_TYPE(response) != response_capi->ResponseType) { PyErr_SetString(PyExc_ValueError, "View did not return Response instance"); Protocol_catch_exception(request); response = NULL; } if(!response) { error_result = PyObject_CallFunctionObjArgs( self->error_handler, request, ((Request*)request)->exception, NULL); if(!error_result) goto error; ((Request*)request)->simple = false; if(!Protocol_write_response_or_err(self, request, (Response*)error_result)) goto error; goto finally; } if(!(response_bytes = response_capi->Response_render(response, ((Request*)request)->simple))) goto error; PyObject* tmp; PyObject* done_callbacks = ((Request*)request)->done_callbacks; for(Py_ssize_t i = 0; done_callbacks && i < PyList_GET_SIZE(done_callbacks); i++) { PyObject* callback = PyList_GET_ITEM(done_callbacks, i); if(!(tmp = PyObject_CallFunctionObjArgs(callback, request, NULL))) goto error; Py_DECREF(tmp); } Gather* gather = &self->gather; if(!gather->enabled) goto maybe_flush; if(gather->responses_end == GATHER_MAX_RESP) goto maybe_flush; if(gather->len + Py_SIZE(response_bytes) > GATHER_MAX_LEN) goto maybe_flush; gather->responses[gather->responses_end] = response_bytes; gather->responses_end++; gather->len += Py_SIZE(response_bytes); response_bytes = NULL; goto dont_flush; maybe_flush: if(!gather->len) goto dont_flush; if(!(gather_buffer = Gather_flush(gather))) goto error; if(!(tmp = PyObject_CallFunctionObjArgs(self->write, gather_buffer, NULL))) goto error; Py_DECREF(tmp); dont_flush: if(response_bytes) { if(!(tmp = PyObject_CallFunctionObjArgs(self->write, response_bytes, NULL))) goto error; Py_DECREF(tmp); } if(self->request_logger) { if(!(tmp = PyObject_CallFunctionObjArgs(self->request_logger, request, NULL))) goto error; Py_DECREF(tmp); } if(response->keep_alive == KEEP_ALIVE_FALSE) { if(!Protocol_close(self)) goto error; } goto finally; error: result = NULL; finally: Py_XDECREF(gather_buffer); Py_XDECREF(error_result); Py_XDECREF(response_bytes); return result; } static void* Protocol_pipeline_ready(PipelineEntry entry, PyObject* protocol) { Protocol* self = (Protocol*)protocol; PyObject* get_result = NULL; PyObject* response = NULL; PyObject* request = entry.request; PyObject* task = entry.task; if(PipelineEntry_is_task(entry)) { if(!(get_result = PyObject_GetAttrString(task, "result"))) goto error; if(!(response = PyObject_CallFunctionObjArgs(get_result, NULL))) Protocol_catch_exception(request); } else { response = task; } if(!self->closed) { if(!Protocol_write_response_or_err(self, request, (Response*)response)) goto error; } else { // TODO: Send that to protocol_error printf("Connection closed, response dropped\n"); } // important: this breaks a cycle in case of an exception Py_CLEAR(((Request*)request)->exception); goto finally; error: self = NULL; finally: if(PipelineEntry_is_task(entry)) Py_XDECREF(response); Py_XDECREF(get_result); return self; } static inline Protocol* Protocol_handle_coro(Protocol* self, PyObject* request, PyObject* coro) { Protocol* result = self; PyObject* task = NULL; if(!(task = PyObject_CallFunctionObjArgs(self->create_task, coro, NULL))) goto error; if(!Pipeline_queue(&self->pipeline, (PipelineEntry){true, request, task})) goto error; goto finally; error: result = NULL; finally: Py_XDECREF(task); return result; } #ifdef PARSER_STANDALONE static PyObject* Protocol_on_body(Protocol* self, PyObject *args) #else Protocol* Protocol_on_body(Protocol* self, char* body, size_t body_len, size_t tail_len) #endif { #ifdef PARSER_STANDALONE PyObject* result = Py_None; #else Protocol* result = self; #endif PyObject* request = NULL; PyObject* handler_result = NULL; MatchDictEntry* entries; MatcherEntry* matcher_entry; size_t entries_length; #ifdef PARSER_STANDALONE /* PyObject* request; if(!PyArg_ParseTuple(args, "O", &request)) goto error; */ // FIXME implement body setting #endif matcher_entry = matcher_capi->Matcher_match_request( (Matcher*)self->matcher, (PyObject*)&self->static_request, &entries, &entries_length); request_capi->Request_set_match_dict_entries( &self->static_request, entries, entries_length); request_capi->Request_set_body( &self->static_request, body, body_len); self->static_request.simple = matcher_entry && matcher_entry->simple; request = (PyObject*)&self->static_request; if((matcher_entry && matcher_entry->coro_func) || !PIPELINE_EMPTY(&self->pipeline)) { self->gather.enabled = false; if(!(request = request_capi->Request_clone(&self->static_request))) goto error; } else // TODO: should be tweaked to minimal request length self->gather.enabled = tail_len > 0; ((Request*)request)->transport = self->transport; Py_INCREF(self->transport); ((Request*)request)->app = self->app; Py_INCREF(self->app); ((Request*)request)->matcher_entry = matcher_entry; if(!matcher_entry) { if(!(((Request*)request)->exception = PyObject_CallFunctionObjArgs( RouteNotFoundException, NULL))) goto error; goto queue_or_write; } if(!(handler_result = PyObject_CallFunctionObjArgs( matcher_entry->handler, request, NULL))) { Protocol_catch_exception(request); goto queue_or_write; } if(matcher_entry->coro_func) { if(!Protocol_handle_coro(self, request, handler_result)) goto error; goto finally; } queue_or_write: if(!PIPELINE_EMPTY(&self->pipeline)) { if(!Pipeline_queue(&self->pipeline, (PipelineEntry){false, request, handler_result})) goto error; goto finally; } if(!Protocol_write_response_or_err( self, (PyObject*)&self->static_request, (Response*)handler_result)) goto error; goto finally; error: result = NULL; finally: if(request != (PyObject*)&self->static_request) Py_XDECREF(request); Py_XDECREF(handler_result); #ifdef PARSER_STANDALONE if(result) Py_INCREF(result); #endif return result; } #ifdef PARSER_STANDALONE static PyObject* Protocol_on_error(Protocol* self, PyObject *args) { Py_RETURN_NONE; } #else Protocol* Protocol_on_error(Protocol* self, PyObject* error) { PyObject* protocol_error_handler = NULL; PyObject* response = NULL; if(!(protocol_error_handler = PyObject_GetAttrString(self->app, "protocol_error_handler"))) goto error; if(!(response = PyObject_CallFunctionObjArgs(protocol_error_handler, error, NULL))) goto error; PyObject* tmp; if(!(tmp = PyObject_CallFunctionObjArgs(self->write, response, NULL))) goto error; Py_DECREF(tmp); if(!Protocol_close(self)) goto error; goto finally; error: self = NULL; finally: Py_XDECREF(response); Py_XDECREF(protocol_error_handler); return self; } #endif static PyObject* Protocol_pipeline_cancel(Protocol* self) { if(!Pipeline_cancel(&self->pipeline)) return NULL; Py_RETURN_NONE; } static PyMethodDef Protocol_methods[] = { {"connection_made", (PyCFunction)Protocol_connection_made, METH_O, ""}, {"connection_lost", (PyCFunction)Protocol_connection_lost, METH_VARARGS, ""}, {"data_received", (PyCFunction)Protocol_data_received, METH_O, ""}, {"pipeline_cancel", (PyCFunction)Protocol_pipeline_cancel, METH_NOARGS, ""}, #ifdef PARSER_STANDALONE {"on_headers", (PyCFunction)Protocol_on_headers, METH_VARARGS, ""}, {"on_body", (PyCFunction)Protocol_on_body, METH_VARARGS, ""}, {"on_error", (PyCFunction)Protocol_on_error, METH_VARARGS, ""}, #endif {NULL} }; static PyObject* Protocol_get_pipeline_empty(Protocol* self) { if(PIPELINE_EMPTY(&self->pipeline)) Py_RETURN_TRUE; Py_RETURN_FALSE; } static PyObject* Protocol_get_transport(Protocol* self) { Py_INCREF(self->transport); return self->transport; } static PyGetSetDef Protocol_getset[] = { {"pipeline_empty", (getter)Protocol_get_pipeline_empty, NULL, "", NULL}, {"transport", (getter)Protocol_get_transport, NULL, "", NULL}, {NULL} }; static PyTypeObject ProtocolType = { PyVarObject_HEAD_INIT(NULL, 0) "cprotocol.Protocol", /* tp_name */ sizeof(Protocol), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Protocol_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Protocol", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Protocol_methods, /* tp_methods */ 0, /* tp_members */ Protocol_getset, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Protocol_init, /* tp_init */ 0, /* tp_alloc */ Protocol_new, /* tp_new */ }; static PyModuleDef cprotocol = { PyModuleDef_HEAD_INIT, "cprotocol", "cprotocol", -1, NULL, NULL, NULL, NULL, NULL }; PyMODINIT_FUNC PyInit_cprotocol(void) { PyObject* m = NULL; #ifdef PARSER_STANDALONE PyObject* cparser = NULL; Parser = NULL; #endif PyObject* api_capsule = NULL; PyObject* crequest = NULL; PyObject* route = NULL; if (PyType_Ready(&ProtocolType) < 0) goto error; m = PyModule_Create(&cprotocol); if(!m) goto error; #ifdef PARSER_STANDALONE cparser = PyImport_ImportModule("japronto.parser.cparser"); if(!cparser) goto error; Parser = PyObject_GetAttrString(cparser, "HttpRequestParser"); if(!Parser) goto error; #else if(cparser_init() == -1) goto error; #endif if(!cpipeline_init()) goto error; if(!crequest_init()) goto error; crequest = PyImport_ImportModule("japronto.request.crequest"); if(!crequest) goto error; PyRequest = PyObject_GetAttrString(crequest, "Request"); if(!PyRequest) goto error; if(!(route = PyImport_ImportModule("japronto.router.route"))) goto error; if(!(RouteNotFoundException = PyObject_GetAttrString( route, "RouteNotFoundException"))) goto error; request_capi = import_capi("japronto.request.crequest"); if(!request_capi) goto error; matcher_capi = import_capi("japronto.router.cmatcher"); if(!matcher_capi) goto error; response_capi = import_capi("japronto.response.cresponse"); if(!response_capi) goto error; Py_INCREF(&ProtocolType); PyModule_AddObject(m, "Protocol", (PyObject*)&ProtocolType); static Protocol_CAPI capi = { Protocol_close }; api_capsule = export_capi(m, "japronto.protocol.cprotocol", &capi); if(!api_capsule) goto error; goto finally; error: Py_XDECREF(PyRequest); #ifdef PARSER_STANDALONE Py_XDECREF(Parser); #endif m = NULL; finally: Py_XDECREF(api_capsule); Py_XDECREF(crequest); Py_XDECREF(route); #ifdef PARSER_STANDALONE Py_XDECREF(cparser); #endif return m; } ================================================ FILE: src/japronto/protocol/cprotocol.h ================================================ #pragma once #ifndef PARSER_STANDALONE #include "cparser.h" #endif #include "cpipeline.h" #include "crequest.h" #include #define GATHER_MAX_RESP 24 typedef struct { PyObject* responses[GATHER_MAX_RESP]; size_t responses_end; size_t len; PyBytesObject* prev_buffer; bool enabled; } Gather; typedef struct { PyObject_HEAD #ifdef PARSER_STANDALONE PyObject* feed; PyObject* feed_disconnect; #else Parser parser; #endif Request static_request; Pipeline pipeline; #ifdef REAPER_ENABLED unsigned long idle_time; unsigned long read_ops; unsigned long last_read_ops; #endif PyObject* app; PyObject* matcher; PyObject* error_handler; PyObject* transport; PyObject* write; PyObject* writelines; PyObject* create_task; PyObject* request_logger; #ifdef PROTOCOL_TRACK_REFCNT Py_ssize_t none_cnt; Py_ssize_t true_cnt; Py_ssize_t false_cnt; #endif bool closed; Gather gather; } Protocol; #define GATHER_MAX_LEN (4096 - sizeof(PyBytesObject)) #ifndef PARSER_STANDALONE Protocol* Protocol_on_incomplete(Protocol* self); Protocol* Protocol_on_headers(Protocol*, char* method, size_t method_len, char* path, size_t path_len, int minor_version, void* headers, size_t num_headers); Protocol* Protocol_on_body(Protocol*, char* body, size_t body_len, size_t tail_len); Protocol* Protocol_on_error(Protocol*, PyObject*); #endif typedef struct { void* (*Protocol_close) (Protocol* self); } Protocol_CAPI; ================================================ FILE: src/japronto/protocol/cprotocol_ext.py ================================================ from distutils.core import Extension def get_extension(): cparser = system.get_extension_by_path('japronto/parser/cparser_ext.py') cpipeline = system.get_extension_by_path( 'japronto/pipeline/cpipeline_ext.py') define_macros = [('PIPELINE_PAIR', 1)] if system.args.enable_reaper: define_macros.append(('REAPER_ENABLED', 1)) return Extension( 'japronto.protocol.cprotocol', sources=[ 'cprotocol.c', '../capsule.c', '../request/crequest.c', '../response/cresponse.c', *cparser.sources, *cpipeline.sources], include_dirs=[ '.', '..', '../parser', '../pipeline', '../router', '../request', '../response', *cparser.include_dirs], extra_objects=cparser.extra_objects, define_macros=define_macros) ================================================ FILE: src/japronto/protocol/creaper.c ================================================ #include #include "cprotocol.h" #include "capsule.h" typedef struct { PyObject_HEAD PyObject* connections; PyObject* call_later; PyObject* check_idle; PyObject* check_idle_handle; PyObject* check_interval; unsigned long idle_timeout; } Reaper; #ifdef REAPER_DEBUG_PRINT #define debug_print(format, ...) printf("reaper: " format "\n", __VA_ARGS__) #else #define debug_print(format, ...) #endif static Protocol_CAPI* protocol_capi; const long DEFAULT_CHECK_INTERVAL = 10; const unsigned long DEFAULT_IDLE_TIMEOUT = 60; static PyObject* default_check_interval; static PyObject* Reaper_new(PyTypeObject* type, PyObject* args, PyObject* kwds) { Reaper* self = NULL; self = (Reaper*)type->tp_alloc(type, 0); if(!self) goto finally; self->connections = NULL; self->call_later = NULL; self->check_idle = NULL; self->check_idle_handle = NULL; self->check_interval = NULL; finally: return (PyObject*)self; } static void Reaper_dealloc(Reaper* self) { Py_XDECREF(self->check_interval); Py_XDECREF(self->check_idle_handle); Py_XDECREF(self->check_idle); Py_XDECREF(self->call_later); Py_XDECREF(self->connections); Py_TYPE(self)->tp_free((PyObject*)self); } #ifdef REAPER_ENABLED static inline void* Reaper_schedule_check_idle(Reaper* self) { Py_XDECREF(self->check_idle_handle); self->check_idle_handle = PyObject_CallFunctionObjArgs( self->call_later, self->check_interval, self->check_idle, NULL); return self->check_idle_handle; } #endif static PyObject* Reaper_stop(Reaper* self) { #ifdef REAPER_ENABLED void* result = Py_None; PyObject* cancel = NULL; if(!(cancel = PyObject_GetAttrString(self->check_idle_handle, "cancel"))) goto error; PyObject* tmp; if(!(tmp = PyObject_CallFunctionObjArgs(cancel, NULL))) goto error; Py_DECREF(tmp); goto finally; error: result = NULL; finally: Py_XINCREF(result); Py_XDECREF(cancel); return result; #else Py_RETURN_NONE; #endif } static int Reaper_init(Reaper* self, PyObject* args, PyObject* kwds) { PyObject* loop = NULL; int result = 0; PyObject* app = NULL; PyObject* idle_timeout = NULL; static char* kwlist[] = {"app", "check_interval", "idle_timeout", NULL}; if (!PyArg_ParseTupleAndKeywords( args, kwds, "|OOO", kwlist, &app, &self->check_interval, &idle_timeout)) goto error; assert(app); if(!self->check_interval) self->check_interval = default_check_interval; Py_INCREF(self->check_interval); assert(PyLong_AsLong(self->check_interval) >= 0); if(!idle_timeout) self->idle_timeout = DEFAULT_IDLE_TIMEOUT; else self->idle_timeout = PyLong_AsLong(idle_timeout); assert(self->idle_timeout >= 0); debug_print("check_interval %ld", PyLong_AsLong(self->check_interval)); debug_print("idle_timeout %ld", self->idle_timeout); if(!(loop = PyObject_GetAttrString(app, "_loop"))) goto error; if(!(self->call_later = PyObject_GetAttrString(loop, "call_later"))) goto error; if(!(self->connections = PyObject_GetAttrString(app, "_connections"))) goto error; #ifdef REAPER_ENABLED if(!(self->check_idle = PyObject_GetAttrString((PyObject*)self, "_check_idle"))) goto error; if(!Reaper_schedule_check_idle(self)) goto error; #endif goto finally; error: result = -1; finally: Py_XDECREF(loop); return result; } #ifdef REAPER_ENABLED static PyObject* Reaper__check_idle(Reaper* self, PyObject* args) { PyObject* result = Py_None; PyObject* iterator = NULL; Protocol* conn = NULL; if(!(iterator = PyObject_GetIter(self->connections))) goto error; unsigned long check_interval = PyLong_AsLong(self->check_interval); while((conn = (Protocol*)PyIter_Next(iterator))) { debug_print( "conn %p, idle_time %ld, read_ops %ld, last_read_ops %ld", conn, conn->idle_time, conn->read_ops, conn->last_read_ops); if(conn->read_ops == conn->last_read_ops) { conn->idle_time += check_interval; if(conn->idle_time >= self->idle_timeout) { if(!protocol_capi->Protocol_close(conn)) goto error; } } else { conn->idle_time = 0; conn->last_read_ops = conn->read_ops; } Py_DECREF(conn); } if(!Reaper_schedule_check_idle(self)) goto error; goto finally; error: result = NULL; finally: Py_XDECREF(conn); Py_XDECREF(iterator); Py_XINCREF(result); return result; } #endif static PyMethodDef Reaper_methods[] = { #ifdef REAPER_ENABLED {"_check_idle", (PyCFunction)Reaper__check_idle, METH_NOARGS, ""}, #endif {"stop", (PyCFunction)Reaper_stop, METH_NOARGS, ""}, {NULL} }; static PyTypeObject ReaperType = { PyVarObject_HEAD_INIT(NULL, 0) "creaper.Reaper", /* tp_name */ sizeof(Reaper), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Reaper_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Reaper", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Reaper_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Reaper_init, /* tp_init */ 0, /* tp_alloc */ Reaper_new, /* tp_new */ }; static PyModuleDef creaper = { PyModuleDef_HEAD_INIT, "creaper", "creaper", -1, NULL, NULL, NULL, NULL, NULL }; PyMODINIT_FUNC PyInit_creaper(void) { PyObject* m = NULL; default_check_interval = NULL; if (PyType_Ready(&ReaperType) < 0) goto error; m = PyModule_Create(&creaper); if(!m) goto error; Py_INCREF(&ReaperType); PyModule_AddObject(m, "Reaper", (PyObject*)&ReaperType); if(!(default_check_interval = PyLong_FromLong(DEFAULT_CHECK_INTERVAL))) goto error; protocol_capi = import_capi("japronto.protocol.cprotocol"); if(!protocol_capi) goto error; goto finally; error: Py_XDECREF(default_check_interval); m = NULL; finally: return m; } ================================================ FILE: src/japronto/protocol/creaper_ext.py ================================================ from distutils.core import Extension def get_extension(): define_macros = [('PIPELINE_PAIR', 1)] if system.args.enable_reaper: define_macros.append(('REAPER_ENABLED', 1)) return Extension( 'japronto.protocol.creaper', sources=['creaper.c', '../capsule.c'], include_dirs=[ '../parser', '../../picohttpparser', '../pipeline', '../request', '../router', '../response', '..'], define_macros=define_macros) ================================================ FILE: src/japronto/protocol/generator.c ================================================ #include #include "generator.h" typedef struct _Generator { PyObject_HEAD PyObject* object; } Generator; static PyTypeObject GeneratorType; #ifdef GENERATOR_OPAQUE static PyObject* Generator_new(PyTypeObject* type, PyObject* args, PyObject* kw) #else PyObject* Generator_new(void) #endif { Generator* self = NULL; #ifdef GENERATOR_OPAQUE self = (Generator*)type->tp_alloc(type, 0); #else self = (Generator*)GeneratorType.tp_alloc(&GeneratorType, 0); #endif if(!self) goto finally; self->object = NULL; finally: return (PyObject*)self; } #ifdef GENERATOR_OPAQUE static void #else void #endif Generator_dealloc(Generator* self) { Py_XDECREF(self->object); Py_TYPE(self)->tp_free((PyObject*)self); } #ifdef GENERATOR_OPAQUE static int Generator_init(Generator* self, PyObject *args, PyObject *kw) #else int Generator_init(Generator* self, PyObject* object) #endif { int result = 0; #ifdef GENERATOR_OPAQUE if(!PyArg_ParseTuple(args, "O", &self->object)) goto error; #else self->object = object; #endif Py_INCREF(self->object); goto finally; #ifdef GENERATOR_OPAQUE error: result = -1; #endif finally: return result; } static PyObject* Generator_next(Generator* self) { PyErr_SetObject(PyExc_StopIteration, self->object); return NULL; } static PyObject* Generator_send(Generator* self, PyObject* arg) { return Generator_next(self); } static PyMethodDef Generator_methods[] = { {"send", (PyCFunction)Generator_send, METH_O, ""}, {NULL} }; static PyTypeObject GeneratorType = { PyVarObject_HEAD_INIT(NULL, 0) "protocol.Generator", /* tp_name */ sizeof(Generator), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Generator_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Generator", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ PyObject_SelfIter, /* tp_iter */ (iternextfunc)Generator_next, /* tp_iternext */ Generator_methods, /* tp_methods */ #ifdef GENERATOR_OPAQUE 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Generator_init, /* tp_init */ 0, /* tp_alloc */ Generator_new, /* tp_new */ #endif }; #ifdef GENERATOR_OPAQUE static PyModuleDef generator = { PyModuleDef_HEAD_INIT, "generator", "generator", -1, NULL, NULL, NULL, NULL, NULL }; #endif #ifdef GENERATOR_OPAQUE PyMODINIT_FUNC PyInit_generator(void) #else void* generator_init(void) #endif { #ifdef GENERATOR_OPAQUE PyObject* m = NULL; #else void* m = &GeneratorType; #endif if(PyType_Ready(&GeneratorType) < 0) goto error; #ifdef GENERATOR_OPAQUE m = PyModule_Create(&generator); if(!m) goto error; Py_INCREF(&GeneratorType); PyModule_AddObject(m, "Generator", (PyObject*)&GeneratorType); #endif goto finally; error: m = NULL; finally: return m; } ================================================ FILE: src/japronto/protocol/generator.h ================================================ #pragma once #ifndef GENERATOR_OPAQUE struct _Generator; #define GENERATOR struct _Generator PyObject* Generator_new(void); void Generator_dealloc(struct _Generator* self); int Generator_init(struct _Generator* self, PyObject* object); void* generator_init(void); #endif ================================================ FILE: src/japronto/protocol/generator_ext.py ================================================ from distutils.core import Extension def get_extension(): return Extension( 'protocol.generator', sources=['generator.c'], include_dirs=[], define_macros=[('GENERATOR_OPAQUE', 1)]) ================================================ FILE: src/japronto/protocol/handler.py ================================================ import asyncio from asyncio.queues import Queue from japronto.response.cresponse import Response from japronto.protocol.cprotocol import Protocol as CProtocol static_response = b"""HTTP/1.1 200 OK\r Connection: keep-alive\r Content-Length: 12\r Content-Type: text/plain; encoding=utf-8\r \r Hello statc! """ def make_class(flavor): if flavor == 'c': return CProtocol from japronto.parser import cparser class HttpProtocol(asyncio.Protocol): def __init__(self, loop, handler): self.parser = cparser.HttpRequestParser( self.on_headers, self.on_body, self.on_error) self.loop = loop self.response = Response() if flavor == 'queue': def connection_made(self, transport): self.transport = transport self.queue = Queue(loop=self.loop) self.loop.create_task(handle_requests(self.queue, transport)) else: def connection_made(self, transport): self.transport = transport def connection_lost(self, exc): self.parser.feed_disconnect() def data_received(self, data): self.parser.feed(data) def on_headers(self, request): return if flavor == 'block': def on_body(self, request): handle_request_block(request, self.transport, self.response) elif flavor == 'dump': def on_body(self, request): handle_dump(request, self.transport, self.response) elif flavor == 'task': def on_body(self, request): self.loop.create_task(handle_request(request, self.transport)) elif flavor == 'queue': def on_body(self, request): self.queue.put_nowait(request) elif flavor == 'inline': def on_body(self, request): body = 'Hello inlin!' status_code = 200 mime_type = 'text/plain' encoding = 'utf-8' text = [b'HTTP/1.1 '] text.extend([str(status_code).encode(), b' OK\r\n']) text.append(b'Connection: keep-alive\r\n') text.append(b'Content-Length: ') text.extend([str(len(body)).encode(), b'\r\n']) text.extend([ b'Content-Type: ', mime_type.encode(), b'; encoding=', encoding.encode(), b'\r\n\r\n']) text.append(body.encode()) self.transport.write(b''.join(text)) elif flavor == 'static': def on_body(self, request): self.transport.write(static_response) def on_error(self, error): print(error) return HttpProtocol async def handle_requests(queue, transport): while 1: await queue.get() response = Response(text='Hello queue!') transport.write(response.render()) async def handle_request(request, transport): response = Response(text='Hello ttask!') transport.write(response.render()) def handle_request_block(request, transport, response): response.__init__(404, text='Hello block') transport.write(response.render()) def handle_dump(request, transport, response): text = request.path response.__init__(text=text) transport.write(response.render()) ================================================ FILE: src/japronto/protocol/null.py ================================================ class NullProtocol: def on_headers(self, *args): pass def on_body(self, body): pass def on_error(self, error): pass ================================================ FILE: src/japronto/protocol/tracing.py ================================================ from functools import partial from parser.libpicohttpparser import ffi from request import HttpRequest class TracingProtocol: def __init__(self, on_headers_adapter: callable, on_body_adapter: callable): self.requests = [] self.error = None self.on_headers_adapter = on_headers_adapter self.on_body_adapter = on_body_adapter self.on_headers_call_count = 0 self.on_body_call_count = 0 self.on_error_call_count = 0 def on_headers(self, *args): self.request = self.on_headers_adapter(*args) self.requests.append(self.request) self.on_headers_call_count += 1 def on_body(self, body): self.request.body = self.on_body_adapter(body) self.on_body_call_count += 1 def on_error(self, error: str): self.error = error self.on_error_call_count += 1 def _request_from_cprotocol(method: memoryview, path: memoryview, version: int, headers: memoryview): method = method.tobytes().decode('ascii') path = path.tobytes().decode('ascii') version = "1.0" if version == 0 else "1.1" headers_len = headers.nbytes // ffi.sizeof("struct phr_header") headers_cdata = ffi.from_buffer(headers) headers_cdata = ffi.cast( 'struct phr_header[{}]'.format(headers_len), headers_cdata) headers = _extract_headers(headers_cdata) return HttpRequest(method, path, version, headers) def _body_from_cprotocol(body: memoryview): return None if body is None else body.tobytes() def _request_from_cffiprotocol(method: "char[]", path: "char[]", version: int, headers: "struct phr_header[]"): method = ffi.buffer(method)[:].decode('ascii') path = ffi.buffer(path)[:].decode('ascii') version = "1.0" if version == 0 else "1.1" headers = _extract_headers(headers) return HttpRequest(method, path, version, headers) def _body_from_cffiprotocol(body: "char[]"): return None if body is None else ffi.buffer(body)[:] def _extract_headers(headers_cdata: "struct phr_header[]"): headers = {} for header in headers_cdata: name = ffi.string(header.name, header.name_len).decode('ascii').title() value = ffi.string(header.value, header.value_len).decode('latin1') headers[name] = value return headers CTracingProtocol = partial( TracingProtocol, on_headers_adapter=_request_from_cprotocol, on_body_adapter=_body_from_cprotocol) CffiTracingProtocol = partial( TracingProtocol, on_headers_adapter=_request_from_cffiprotocol, on_body_adapter=_body_from_cffiprotocol) ================================================ FILE: src/japronto/reloader.py ================================================ import sys import os.path import os import time import threading import signal def main(): import subprocess terminating = False def signal_received(sig, frame): nonlocal terminating if sig == signal.SIGHUP: child.send_signal(signal.SIGHUP) else: terminating = True child.terminate() signal.signal(signal.SIGINT, signal_received) signal.signal(signal.SIGTERM, signal_received) signal.signal(signal.SIGHUP, signal_received) os.putenv('_JAPR_RELOADER', str(os.getpid())) while not terminating: child = subprocess.Popen([ sys.executable, '-m', 'japronto', '--reloader-pid', str(os.getpid()), *(v for v in sys.argv[1:] if v != '--reload')]) child.wait() if child.returncode != 0: break def exec_reloader(*, host, port, worker_num): args = ['--host', host, '--port', str(port)] if worker_num: args.extend(['--worker-num', str(worker_num)]) os.execv( sys.executable, [sys.executable, '-m', 'japronto.reloader', *args, '--script', *sys.argv]) def change_detector(): previous_mtimes = {} while 1: changed = False current_mtimes = {} for name, module in list(sys.modules.items()): try: filename = module.__file__ except AttributeError: continue if not filename.endswith('.py'): continue mtime = os.path.getmtime(filename) previous_mtime = previous_mtimes.get(name) if previous_mtime and previous_mtime != mtime: changed = True current_mtimes[name] = mtime yield changed previous_mtimes = current_mtimes class ChangeDetector(threading.Thread): def __init__(self, loop): super().__init__(daemon=True) self.loop = loop def run(self): for changed in change_detector(): if changed: self.loop.call_soon_threadsafe(self.loop.stop) os.kill(os.getppid(), signal.SIGHUP) return time.sleep(.5) if __name__ == "__main__": main() ================================================ FILE: src/japronto/request/__init__.py ================================================ import urllib.parse from json import loads as json_loads import cgi import encodings.idna import collections from http.cookies import _unquote as unquote_cookie class HttpRequest(object): __slots__ = ('path', 'method', 'version', 'headers', 'body') def __init__(self, method, path, version, headers): self.path = path self.method = method self.version = version self.headers = headers self.body = None def dump_headers(self): print('path', self.path) print('method', self.method) print('version', self.version) for n, v in self.headers.items(): print(n, v) def __repr__(self): return '' \ .format(self, len(self.headers)) def memoize(func): def wrapper(request): ns = request.extra.setdefault('_japronto', {}) try: return ns[func.__name__] except KeyError: pass result = func(request) ns[func.__name__] = result return result return wrapper @memoize def text(request): if request.body is None: return None return request.body.decode(request.encoding or 'utf-8') @memoize def json(request): if request.body is None: return None return json_loads(request.text) @memoize def query(request): qs = request.query_string if not qs: return {} return dict(urllib.parse.parse_qsl(qs)) def remote_addr(request): return request.transport.get_extra_info('peername')[0] @memoize def parsed_content_type(request): content_type = request.headers.get('Content-Type') if not content_type: return None, {} return cgi.parse_header(content_type) def mime_type(request): return parsed_content_type(request)[0] def encoding(request): return parsed_content_type(request)[1].get('charset') @memoize def parsed_form_and_files(request): if request.mime_type == 'application/x-www-form-urlencoded': return dict(urllib.parse.parse_qsl(request.text)), None elif request.mime_type == 'multipart/form-data': boundary = parsed_content_type(request)[1]['boundary'].encode('utf-8') return parse_multipart_form(request.body, boundary) return None, None def form(request): return parsed_form_and_files(request)[0] def files(request): return parsed_form_and_files(request)[1] @memoize def hostname_and_port(request): host = request.headers.get('Host') if not host: return None, None hostname, *rest = host.split(':', 1) port = rest[0] if rest else None return encodings.idna.ToUnicode(hostname), int(port) def port(request): return hostname_and_port(request)[1] def hostname(request): return hostname_and_port(request)[0] def parse_cookie(cookie): """Parse a ``Cookie`` HTTP header into a dict of name/value pairs. This function attempts to mimic browser cookie parsing behavior; it specifically does not follow any of the cookie-related RFCs (because browsers don't either). The algorithm used is identical to that used by Django version 1.9.10. """ cookiedict = {} for chunk in cookie.split(str(';')): if str('=') in chunk: key, val = chunk.split(str('='), 1) else: # Assume an empty name per # https://bugzilla.mozilla.org/show_bug.cgi?id=169091 key, val = str(''), chunk key, val = key.strip(), val.strip() if key or val: # unquote using Python's algorithm. cookiedict[key] = unquote_cookie(val) return cookiedict @memoize def cookies(request): if 'Cookie' not in request.headers: return {} try: cookies = parse_cookie(request.headers['Cookie']) except Exception: return {} return {k: urllib.parse.unquote(v) for k, v in cookies.items()} File = collections.namedtuple('File', ['type', 'body', 'name']) def parse_multipart_form(body, boundary): files = {} fields = {} form_parts = body.split(boundary) for form_part in form_parts[1:-1]: file_name = None file_type = None field_name = None line_index = 2 line_end_index = 0 while not line_end_index == -1: line_end_index = form_part.find(b'\r\n', line_index) form_line = form_part[line_index:line_end_index].decode('utf-8') line_index = line_end_index + 2 if not form_line: break colon_index = form_line.index(':') form_header_field = form_line[0:colon_index] form_header_value, form_parameters = cgi.parse_header( form_line[colon_index + 2:]) if form_header_field == 'Content-Disposition': if 'filename' in form_parameters: file_name = form_parameters['filename'] field_name = form_parameters.get('name') elif form_header_field == 'Content-Type': file_type = form_header_value post_data = form_part[line_index:-4] if file_name or file_type: file = File(type=file_type, name=file_name, body=post_data) files[field_name] = file else: value = post_data.decode('utf-8') fields[field_name] = value return fields, files ================================================ FILE: src/japronto/request/crequest.c ================================================ #include #include #include #include #include "crequest.h" #include "cresponse.h" #ifdef REQUEST_OPAQUE #include "picohttpparser.h" #endif #include "capsule.h" static PyObject* PyResponse; static PyObject* partial; #ifdef REQUEST_OPAQUE static PyObject* HTTP10; static PyObject* HTTP11; static PyObject* request; #endif static Response_CAPI* response_capi; #ifdef REQUEST_OPAQUE static PyObject* Request_new(PyTypeObject *type, PyObject *args, PyObject *kwds) #else PyObject* Request_new(PyTypeObject* type, Request* self) #endif { #ifdef REQUEST_OPAQUE Request* self = NULL; self = (Request*)type->tp_alloc(type, 0); if(!self) goto finally; #else ((PyObject*)self)->ob_refcnt = 1; ((PyObject*)self)->ob_type = type; #endif self->response_called = false; self->matcher_entry = NULL; self->exception = NULL; self->app = NULL; self->transport = NULL; self->py_method = NULL; self->py_path = NULL; self->py_qs = NULL; self->py_headers = NULL; self->py_match_dict = NULL; self->py_body = NULL; self->extra = NULL; self->done_callbacks = NULL; Response_new(response_capi->ResponseType, &self->response); self->buffer = self->inline_buffer; self->buffer_len = REQUEST_INITIAL_BUFFER_LEN; #ifdef REQUEST_OPAQUE finally: #endif return (PyObject*)self; } #ifdef REQUEST_OPAQUE static void #else void #endif Request_dealloc(Request* self) { if(self->buffer != self->inline_buffer) free(self->buffer); Response_dealloc(&self->response); Py_XDECREF(self->app); Py_XDECREF(self->done_callbacks); Py_XDECREF(self->extra); Py_XDECREF(self->py_body); Py_XDECREF(self->py_match_dict); Py_XDECREF(self->py_headers); Py_XDECREF(self->py_qs); Py_XDECREF(self->py_path); Py_XDECREF(self->py_method); Py_XDECREF(self->transport); Py_XDECREF(self->exception); #ifdef REQUEST_OPAQUE Py_TYPE(self)->tp_free((PyObject*)self); #endif } #ifdef REQUEST_OPAQUE static int Request_init(Request* self, PyObject *args, PyObject* kw) #else int Request_init(Request* self) #endif { return 0; } #ifdef REQUEST_OPAQUE static PyTypeObject RequestType; static PyObject* Request_clone(Request* original) { Request* clone = NULL; if(!(clone = (Request*)Request_new(&RequestType, NULL, NULL))) goto error; if(Request_init(clone, NULL, NULL) == -1) goto error; const size_t offset = offsetof(Request, method); const size_t length = offsetof(Request, transport) - offset; memcpy((char*)clone + offset, (char*)original + offset, length); if(original->buffer == original->inline_buffer) { clone->buffer = clone->inline_buffer; ptrdiff_t shift = (char*)clone - (char*)original; clone->method += shift; clone->path += shift; clone->headers = (struct phr_header*)((char*)clone->headers + shift); for(struct phr_header* header = clone->headers; header < clone->headers + clone->num_headers; header++) { header->name += shift; header->value += shift; } clone->match_dict_entries = (MatchDictEntry*)((char*)clone->match_dict_entries + shift); for(MatchDictEntry* entry = clone->match_dict_entries; entry < clone->match_dict_entries + clone->match_dict_length; entry++) { // the keys didnt move, they reference immutable memory from the router entry->value += shift; } if(clone->body) // body can be NULL clone->body += shift; } else { // just steal the buffer since the original request will be destroyed anyway clone->buffer = original->buffer; original->buffer = original->inline_buffer; } goto finally; error: Py_XDECREF(clone); clone = NULL; finally: return (PyObject*)clone; } static KEEP_ALIVE _Request_get_keep_alive(Request* self); static PyObject* Request_Response(Request* self, PyObject *args, PyObject* kw) { if(self->response_called) { PyErr_SetString( PyExc_RuntimeError, "Request.Response can only be called once per request"); goto error; } self->response_called = true; Response* result = &self->response; if(response_capi->Response_init(result, args, kw) == -1) goto error; result->minor_version = self->minor_version; result->keep_alive = _Request_get_keep_alive(self); goto finally; error: result = NULL; finally: Py_XINCREF(result); return (PyObject*)result; } typedef enum { REQUEST_HEADERS, REQUEST_MATCH_DICT, REQUEST_BODY } RequestCopy; #define ROUNDTO8(v) (((v) + 7) & ~7) static inline char* bfrcpy(Request* self, const RequestCopy what) { size_t len; char* dst; char* old_buffer = self->buffer; size_t headers_len; size_t header_entries_len; if(self->num_headers) { struct phr_header* last_header = &self->headers[self->num_headers - 1]; headers_len = last_header->value + last_header->value_len - self->method; header_entries_len = sizeof(struct phr_header) * self->num_headers; } else { headers_len = self->path + self->path_len + self->qs_len - self->method; header_entries_len = 0; } switch(what) { case REQUEST_HEADERS: len = ROUNDTO8(headers_len) + header_entries_len; dst = self->buffer; break; case REQUEST_MATCH_DICT: len = sizeof(MatchDictEntry) * self->match_dict_length; dst = self->buffer + ROUNDTO8(headers_len + header_entries_len); break; case REQUEST_BODY: len = self->body_length; dst = (char*)self->match_dict_entries \ + sizeof(MatchDictEntry) * self->match_dict_length; break; default: assert(0); } if(dst + len > self->buffer + self->buffer_len) { self->buffer_len = MAX(self->buffer_len * 2, self->buffer_len + len); if(self->buffer == self->inline_buffer) { self->buffer = malloc(self->buffer_len); if(!self->buffer) assert(0); // TODO, propagate memory error memcpy(self->buffer, self->inline_buffer, REQUEST_INITIAL_BUFFER_LEN); } else { self->buffer = realloc(self->buffer, self->buffer_len); if(!self->buffer) assert(0); // TODO, propagate memory error } } ptrdiff_t buffer_shift = self->buffer - old_buffer; dst += buffer_shift; ptrdiff_t shift; if(what == REQUEST_HEADERS) { shift = dst - self->method; memcpy(dst, self->method, headers_len); self->method += shift; self->path += shift; memcpy(dst + ROUNDTO8(headers_len), (char*)self->headers, header_entries_len); self->headers = (struct phr_header*)((char*)dst + ROUNDTO8(headers_len)); for(struct phr_header* header = self->headers; header < self->headers + self->num_headers; header++) { header->name += shift; header->value += shift; } goto finally; } if(buffer_shift) { self->method += buffer_shift; self->path += buffer_shift; self->headers = (struct phr_header*)((char*)self->headers + buffer_shift); for(struct phr_header* header = self->headers; header < self->headers + self->num_headers; header++) { header->name += buffer_shift; header->value += buffer_shift; } } if(what == REQUEST_HEADERS) goto finally; if(what == REQUEST_MATCH_DICT) { shift = dst - (char*)self->match_dict_entries; memcpy(dst, (char*)self->match_dict_entries, len); self->match_dict_entries = (MatchDictEntry*)((char*)self->match_dict_entries + shift); /* match_dict_entires values don't need moving by shift because the block * they reference couldn't move (the previous call) */ } if(buffer_shift) { if(what != REQUEST_MATCH_DICT) self->match_dict_entries = (MatchDictEntry*)((char*)self->match_dict_entries + buffer_shift); for(MatchDictEntry* entry = self->match_dict_entries; entry < self->match_dict_entries + self->match_dict_length; entry++) { // the keys didnt move, they reference immutable memory from the router entry->value += buffer_shift; } } if(what == REQUEST_MATCH_DICT) goto finally; if(what == REQUEST_BODY) { shift = dst - self->body; memcpy(dst, self->body, len); self->body += shift; goto finally; } assert(0); finally: return self->buffer; } static void Request_from_raw(Request* self, char* method, size_t method_len, char* path, size_t path_len, int minor_version, struct phr_header* headers, size_t num_headers) { // fill self->method = method; self->method_len = method_len; self->path = path; self->path_decoded = false; self->path_len = path_len; self->qs_len = 0; self->qs_decoded = false; self->minor_version = minor_version; self->headers = headers; self->num_headers = num_headers; self->keep_alive = KEEP_ALIVE_UNSET; bfrcpy(self, REQUEST_HEADERS); } static void Request_set_match_dict_entries(Request* self, MatchDictEntry* entries, size_t length) { self->match_dict_entries = entries; self->match_dict_length = length; bfrcpy(self, REQUEST_MATCH_DICT); } static void Request_set_body(Request* self, char* body, size_t body_len) { if(!body) { self->body = NULL; return; } self->body = body; self->body_length = body_len; bfrcpy(self, REQUEST_BODY); } #define hex_to_dec(x) \ ((x <= '9' ? 0 : 9) + (x & 0x0f)) #define is_hex(x) ((x >= '0' && x <= '9') || (x >= 'A' && x <= 'F')) static inline size_t percent_decode(char* data, ssize_t length, size_t* shifted_bytes, const char* stopchr) { if(shifted_bytes) *shifted_bytes = 0; for(char* end = data + length; data < end; data++) { if(stopchr && *data == *stopchr) { length -= end - data; break; } if(end - data < 3) continue; if(*data == '%' && is_hex(*(data + 1)) && is_hex(*(data + 2))) { *data = (hex_to_dec(*(data + 1)) << 4) + hex_to_dec(*(data + 2)); length -= 2; if(shifted_bytes) *shifted_bytes += 2; memmove(data + 1, data + 3, end - (data + 3)); end -= 2; } } return length; } #undef hex_to_dec #undef is_hex char* Request_get_decoded_path(Request* self, size_t* path_len) { if(!self->path_decoded) { size_t shifted_bytes; const char stopchr = '?'; *path_len = percent_decode( self->path, self->path_len, &shifted_bytes, &stopchr); self->path_decoded = true; self->qs_len = self->path_len - *path_len - shifted_bytes; self->path_len = *path_len; } *path_len = self->path_len; return self->path; } static char* Request_get_decoded_qs(Request* self, size_t* qs_len) { if(!self->qs_len) { *qs_len = 0; return NULL; } char* qs = self->path + self->path_len; if(!self->qs_decoded) { self->qs_len = percent_decode(qs, self->qs_len, NULL, NULL); self->qs_decoded = true; } *qs_len = self->qs_len; return qs; } static inline void title_case(char* data, size_t len) { bool prev_alpha = false; for(char* c = data; c < data + len; c++) { if(*c >= 'A' && *c <= 'Z') { if(prev_alpha) *c ^= 0x20; prev_alpha = true; } else if (*c >= 'a' && *c <= 'z') { if(!prev_alpha) *c ^= 0x20; prev_alpha = true; } else prev_alpha = false; } } static inline PyObject* Request_decode_headers(Request* self) { PyObject* result = NULL; PyObject* headers = PyDict_New(); if(!headers) goto error; result = headers; for(struct phr_header* header = self->headers; header < self->headers + self->num_headers; header++) { PyObject* name = NULL; PyObject* value = NULL; title_case((char*)header->name, header->name_len); // TODO by inserting 0 byte we could call PyDict_SetItemString // FIXME check ASCII name = PyUnicode_FromStringAndSize(header->name, header->name_len); if(!name) goto loop_error; // FIXME this can fail on codec errors value = PyUnicode_DecodeLatin1(header->value, header->value_len, NULL); if(!value) goto loop_error; if(PyDict_SetItem(headers, name, value) == -1) goto loop_error; goto loop_finally; loop_error: result = NULL; loop_finally: Py_XDECREF(name); Py_XDECREF(value); if(!result) goto error; } goto finally; error: Py_XDECREF(headers); result = NULL; finally: return result; } static PyObject* Request_get_method(Request* self, void* closure) { if(!self->py_method) { self->py_method = PyUnicode_DecodeLatin1( REQUEST_METHOD(self), self->method_len, NULL); } Py_XINCREF(self->py_method); return self->py_method; } static PyObject* Request_get_path(Request* self, void* closure) { if(!self->py_path) { size_t path_len; char* path = Request_get_decoded_path(self, &path_len); self->py_path = PyUnicode_FromStringAndSize(path, path_len); } Py_XINCREF(self->py_path); return self->py_path; } static PyObject* Request_get_qs(Request* self, void* closure) { if(!self->py_qs) { size_t qs_len; char* qs = Request_get_decoded_qs(self, &qs_len); if(!qs) Py_RETURN_NONE; // skip the ? char self->py_qs = PyUnicode_FromStringAndSize(qs + 1, qs_len - 1); } Py_XINCREF(self->py_qs); return self->py_qs; } static PyObject* Request_get_version(Request* self, void* closure) { PyObject* result = self->minor_version ? HTTP11 : HTTP10; Py_INCREF(result); return result; } static PyObject* Request_get_headers(Request* self, void* closure) { if(!self->py_headers) self->py_headers = Request_decode_headers(self); Py_XINCREF(self->py_headers); return self->py_headers; } static PyObject* Request_get_match_dict(Request* self, void* closure) { if(!self->py_match_dict) self->py_match_dict = MatchDict_entries_to_dict( self->match_dict_entries, self->match_dict_length); Py_XINCREF(self->py_match_dict); return self->py_match_dict; } static PyObject* Request_get_body(Request* self, void* closure) { if(!self->body) Py_RETURN_NONE; if(!self->py_body) self->py_body = PyBytes_FromStringAndSize(self->body, self->body_length); Py_XINCREF(self->py_body); return self->py_body; } static PyObject* Request_get_transport(Request* self, void* closure) { Py_INCREF(self->transport); return self->transport; } static KEEP_ALIVE _Request_get_keep_alive(Request* self) { if(self->keep_alive == KEEP_ALIVE_UNSET) { struct phr_header* Connection = NULL; for(struct phr_header* header = self->headers; header < self->headers + self->num_headers; header++) { if(header->name_len == strlen("Connection") && strncasecmp(header->name, "Connection", header->name_len) == 0) { Connection = header; break; } } if(self->minor_version == 0) { // FIXME: this should check what's before and after if(Connection && memmem(Connection->value, Connection->value_len, "keep-alive", strlen("keep-alive"))) self->keep_alive = KEEP_ALIVE_TRUE; else self->keep_alive = KEEP_ALIVE_FALSE; } else { if(Connection && memmem(Connection->value, Connection->value_len, "close", strlen("close"))) self->keep_alive = KEEP_ALIVE_FALSE; else self->keep_alive = KEEP_ALIVE_TRUE; } } return self->keep_alive; } static PyObject* Request_get_keep_alive(Request* self, void* closure) { if(_Request_get_keep_alive(self) == KEEP_ALIVE_TRUE) Py_RETURN_TRUE; else Py_RETURN_FALSE; } static PyObject* Request_get_route(Request* self, void* closure) { if(!self->matcher_entry) Py_RETURN_NONE; Py_INCREF(self->matcher_entry->route); return self->matcher_entry->route; } static PyObject* Request_get_extra(Request* self, void* closure) { if(!self->extra) self->extra = PyDict_New(); Py_XINCREF(self->extra); return self->extra; } static PyObject* Request_get_app(Request* self, void* app) { Py_INCREF(self->app); return self->app; } static PyObject* Request_get_proxy(Request* self, char* attr) { PyObject* callable = NULL; PyObject* result = NULL; if(!(callable = PyObject_GetAttrString(request, attr))) goto error; if(!(result = PyObject_CallFunctionObjArgs(callable, self, NULL))) goto error; goto finally; error: result = NULL; finally: Py_XDECREF(callable); return result; } #define PROXY(attr) \ {#attr, (getter)Request_get_proxy, NULL, "", #attr} static PyGetSetDef Request_getset[] = { {"method", (getter)Request_get_method, NULL, "", NULL}, {"path", (getter)Request_get_path, NULL, "", NULL}, {"query_string", (getter)Request_get_qs, NULL, "", NULL}, {"version", (getter)Request_get_version, NULL, "", NULL}, {"headers", (getter)Request_get_headers, NULL, "", NULL}, {"match_dict", (getter)Request_get_match_dict, NULL, "", NULL}, {"body", (getter)Request_get_body, NULL, "", NULL}, {"transport", (getter)Request_get_transport, NULL, "", NULL}, {"keep_alive", (getter)Request_get_keep_alive, NULL, "", NULL}, {"route", (getter)Request_get_route, NULL, "", NULL}, {"extra", (getter)Request_get_extra, NULL, "", NULL}, {"app", (getter)Request_get_app, NULL, "", NULL}, PROXY(text), PROXY(json), PROXY(query), PROXY(mime_type), PROXY(encoding), PROXY(form), PROXY(files), PROXY(remote_addr), PROXY(hostname), PROXY(port), PROXY(cookies), {NULL} }; #undef PROXY static PyObject* Request_getattro(Request* self, PyObject* name) { PyObject* result; if((result = PyObject_GenericGetAttr((PyObject*)self, name))) return result; PyObject* extensions = NULL; if(!(extensions = PyObject_GetAttrString(self->app, "_request_extensions"))) goto error; PyObject* entry; if(!(entry = PyDict_GetItem(extensions, name))) goto error; else PyErr_Clear(); PyObject* handler; PyObject* property; if(!(handler = PyTuple_GetItem(entry, 0))) goto error; if(!(property = PyTuple_GetItem(entry, 1))) goto error; if(property == Py_True) { if(!(result = PyObject_CallFunctionObjArgs(handler, self, NULL))) goto error; } else { if(!(result = PyObject_CallFunctionObjArgs(partial, handler, self, NULL))) goto error; } error: Py_XDECREF(extensions); return result; } static PyObject* Request_add_done_callback(Request* self, PyObject* callback) { if(!self->done_callbacks) { if(!(self->done_callbacks = PyList_New(0))) goto error; } if(PyList_Append(self->done_callbacks, callback) == -1) goto error; Py_RETURN_NONE; error: return NULL; } static PyMethodDef Request_methods[] = { {"Response", (PyCFunction)Request_Response, METH_VARARGS | METH_KEYWORDS, ""}, {"add_done_callback", (PyCFunction)Request_add_done_callback, METH_O, ""}, {NULL} }; static PyTypeObject RequestType = { PyVarObject_HEAD_INIT(NULL, 0) "crequest.Request", /* tp_name */ sizeof(Request), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Request_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ (getattrofunc)Request_getattro, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Request", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Request_methods, /* tp_methods */ 0, /* tp_members */ Request_getset, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Request_init, /* tp_init */ 0, /* tp_alloc */ Request_new, /* tp_new */ }; static PyModuleDef crequest = { PyModuleDef_HEAD_INIT, "crequest", "crequest", -1, NULL, NULL, NULL, NULL, NULL }; #endif #ifdef REQUEST_OPAQUE PyMODINIT_FUNC PyInit_crequest(void) #else void* crequest_init(void) #endif { #ifdef REQUEST_OPAQUE PyObject* m = NULL; PyObject* api_capsule = NULL; HTTP10 = NULL; HTTP11 = NULL; #else void* m = (void*)1; #endif PyObject* cresponse = NULL; PyObject* functools = NULL; PyResponse = NULL; #ifdef REQUEST_OPAQUE if (PyType_Ready(&RequestType) < 0) goto error; #define alloc_static2(name, val) \ name = PyUnicode_FromString(val); \ if(!name) \ goto error; alloc_static2(HTTP10, "1.0") alloc_static2(HTTP11, "1.1") m = PyModule_Create(&crequest); if(!m) goto error; #endif cresponse = PyImport_ImportModule("japronto.response.cresponse"); if(!cresponse) goto error; if(!(functools = PyImport_ImportModule("functools"))) goto error; if(!(partial = PyObject_GetAttrString(functools, "partial"))) goto error; PyResponse = PyObject_GetAttrString(cresponse, "Response"); if(!PyResponse) goto error; #ifdef REQUEST_OPAQUE request = PyImport_ImportModule("japronto.request"); if(!request) goto error; Py_INCREF(&RequestType); PyModule_AddObject(m, "Request", (PyObject*)&RequestType); static Request_CAPI capi = { &RequestType, Request_clone, Request_from_raw, Request_get_decoded_path, Request_set_match_dict_entries, Request_set_body }; api_capsule = export_capi(m, "japronto.request.crequest", &capi); if(!api_capsule) goto error; #endif response_capi = import_capi("japronto.response.cresponse"); if(!response_capi) goto error; goto finally; error: Py_XDECREF(PyResponse); #ifdef REQUEST_OPAQUE Py_XDECREF(HTTP10); Py_XDECREF(HTTP11); #endif m = NULL; finally: Py_XDECREF(functools); Py_XDECREF(cresponse); #ifdef REQUEST_OPAQUE Py_XDECREF(api_capsule); #endif return m; } ================================================ FILE: src/japronto/request/crequest.h ================================================ #pragma once #include #include #include "cmatcher.h" #include "cresponse.h" #include "common.h" #define REQUEST_INITIAL_BUFFER_LEN 1024 typedef struct { PyObject_HEAD char* method; size_t method_len; char* path; bool path_decoded; size_t path_len; bool qs_decoded; size_t qs_len; int minor_version; struct phr_header* headers; size_t num_headers; MatchDictEntry* match_dict_entries; size_t match_dict_length; char* body; size_t body_length; char* buffer; size_t buffer_len; char inline_buffer[REQUEST_INITIAL_BUFFER_LEN]; KEEP_ALIVE keep_alive; bool simple; bool response_called; MatcherEntry* matcher_entry; PyObject* exception; PyObject* transport; PyObject* app; PyObject* py_method; PyObject* py_path; PyObject* py_qs; PyObject* py_headers; PyObject* py_match_dict; PyObject* py_body; PyObject* extra; PyObject* done_callbacks; Response response; } Request; #define REQUEST(r) \ ((Request*)r) #define REQUEST_METHOD(r) \ REQUEST(r)->buffer #define REQUEST_PATH(r) \ REQUEST(r)->path typedef struct { PyTypeObject* RequestType; PyObject* (*Request_clone) (Request* original); void (*Request_from_raw) (Request* self, char* method, size_t method_len, char* path, size_t path_len, int minor_version, struct phr_header* headers, size_t num_headers); char* (*Request_get_decoded_path) (Request* self, size_t* path_len); void (*Request_set_match_dict_entries) (Request* self, MatchDictEntry* entries, size_t length); void (*Request_set_body) (Request* self, char* body, size_t body_len); } Request_CAPI; #ifndef REQUEST_OPAQUE PyObject* Request_new(PyTypeObject* type, Request* self); void Request_dealloc(Request* self); int Request_init(Request* self); void* crequest_init(void); #endif ================================================ FILE: src/japronto/request/crequest_ext.py ================================================ from distutils.core import Extension def get_extension(): return Extension( 'japronto.request.crequest', sources=['crequest.c', '../response/cresponse.c', '../router/match_dict.c', '../capsule.c'], include_dirs=['../../picohttpparser', '..', '../response', '../router'], define_macros=[('REQUEST_OPAQUE', 1)]) ================================================ FILE: src/japronto/response/__init__.py ================================================ ================================================ FILE: src/japronto/response/cresponse.c ================================================ #include #include #include "cresponse.h" #include "capsule.h" #include "reasons.h" #ifdef RESPONSE_OPAQUE static PyObject* json_dumps; static const size_t reason_offset = 13; static const size_t minor_offset = 7; #endif static const char header[] = "HTTP/1.1 200 OK\r\n" "Content-Length: "; #ifdef RESPONSE_OPAQUE static PyObject * Response_new(PyTypeObject *type, PyObject *args, PyObject *kwds) #else PyObject* Response_new(PyTypeObject* type, Response* self) #endif { #ifdef RESPONSE_OPAQUE Response* self = NULL; self = (Response*)type->tp_alloc(type, 0); if(!self) goto finally; self->opaque = true; #else ((PyObject*)self)->ob_refcnt = 1; ((PyObject*)self)->ob_type = type; self->opaque = false; #endif self->code = NULL; self->mime_type = NULL; self->body = NULL; self->encoding = NULL; self->headers = NULL; self->cookies = NULL; self->buffer = self->inline_buffer; self->buffer_len = RESPONSE_INITIAL_BUFFER_LEN; memcpy(self->buffer, header, strlen(header)); #ifdef RESPONSE_OPAQUE finally: #endif return (PyObject*)self; } #ifdef RESPONSE_OPAQUE static void #else void #endif Response_dealloc(Response* self) { if(self->buffer != self->inline_buffer) free(self->buffer); Py_XDECREF(self->cookies); Py_XDECREF(self->headers); Py_XDECREF(self->encoding); Py_XDECREF(self->body); Py_XDECREF(self->mime_type); Py_XDECREF(self->code); #ifdef RESPONSE_OPAQUE if(self->opaque) Py_TYPE(self)->tp_free((PyObject*)self); #endif } #ifdef RESPONSE_OPAQUE static const size_t code_offset = 9; #define empty(v) (!v || v == Py_None) static PyObject* application_json; static PyObject* application_octet; int Response_init(Response* self, PyObject *args, PyObject *kw) { static char *kwlist[] = {"text", "code", "body", "json", "mime_type", "encoding", "headers", "cookies", NULL}; PyObject* code = NULL; PyObject* body = NULL; PyObject* text = NULL; PyObject* json = NULL; PyObject* mime_type = NULL; PyObject* encoding = NULL; PyObject* headers = NULL; PyObject* cookies = NULL; // FIXME: check argument types if (!PyArg_ParseTupleAndKeywords( args, kw, "|OOOOOOOO", kwlist, &text, &code, &body, &json, &mime_type, &encoding, &headers, &cookies)) goto error; if(!empty(code)) { self->code = code; Py_INCREF(self->code); } if(!empty(json)) { assert(empty(text) && empty(body)); if(!(text = PyObject_CallFunctionObjArgs(json_dumps, json, NULL))) goto error; } else if(!empty(text)) { Py_INCREF(text); } if(!empty(text)) { assert(empty(body)); if(!encoding) { if(!(self->body = PyUnicode_AsUTF8String(text))) goto error; } else { char* cencoding; if(!(cencoding = PyUnicode_AsUTF8(encoding))) goto error; if(!(self->body = PyUnicode_AsEncodedString(text, cencoding, NULL))) goto error; } Py_DECREF(text); } if(!empty(body)) { self->body = body; Py_INCREF(self->body); } if(!empty(mime_type)) { self->mime_type = mime_type; Py_INCREF(self->mime_type); } else { if(!empty(json)) { self->mime_type = application_json; Py_INCREF(self->mime_type); } else if(!empty(body)) { self->mime_type = application_octet; Py_INCREF(self->mime_type); } } if(!empty(encoding)) { self->encoding = encoding; Py_INCREF(self->encoding); } if(!empty(headers)) { self->headers = headers; Py_INCREF(self->headers); } if(!empty(cookies)) { self->cookies = cookies; Py_INCREF(self->cookies); } goto finally; error: return -1; finally: return 0; } static const char Content_Type[] = "Content-Type: "; static const char charset[] = "; charset="; static const char utf8[] = "utf-8"; static const char text_plain[] = "text/plain"; #define CRLF \ *(self->buffer + buffer_offset) = '\r'; \ buffer_offset++; \ *(self->buffer + buffer_offset) = '\n'; \ buffer_offset++; #define bfrcpy(data, len) \ if(buffer_offset + len > self->buffer_len) \ { \ self->buffer_len = MAX(self->buffer_len * 2, self->buffer_len + len); \ \ if(self->buffer == self->inline_buffer) \ { \ self->buffer = malloc(self->buffer_len); \ if(!self->buffer) \ assert(0); \ memcpy(self->buffer, self->inline_buffer, buffer_offset); \ } else { \ self->buffer = realloc(self->buffer, self->buffer_len); \ if(!self->buffer) \ assert(0); \ } \ } \ \ memcpy(self->buffer + buffer_offset, data, len); \ buffer_offset += len; #ifdef RESPONSE_CACHE typedef struct { PyObject* body; PyObject* response_bytes; } CacheEntry; #define CACHE_LEN 10 #define CACHE_CUTOFF 4096 typedef struct { size_t end; CacheEntry entries[CACHE_LEN]; } Cache; static Cache cache = {0}; #define Bytes_AS_STRING(op) ((PyBytesObject *)op)->ob_sval #define Response_cacheable(r, simple) \ simple && r->body && Py_SIZE(r->body) < CACHE_CUTOFF \ && !r->code && !r->headers && !r->cookies && !r->mime_type \ && !r->encoding && r->minor_version == 1 && r->keep_alive == KEEP_ALIVE_TRUE static inline PyObject* Response_from_cache(PyObject* body) { CacheEntry* cache_entry; for(cache_entry = cache.entries; cache_entry < cache.entries + cache.end; cache_entry++) { if(Py_SIZE(cache_entry->body) != Py_SIZE(body)) continue; if(memcmp(Bytes_AS_STRING(cache_entry->body), Bytes_AS_STRING(body), Py_SIZE(body)) != 0) continue; Py_INCREF(cache_entry->response_bytes); return cache_entry->response_bytes; } return NULL; } static inline void Response_cache(PyObject* body, PyObject* response_bytes) { if(cache.end == CACHE_LEN) return; CacheEntry* entry = cache.entries + cache.end; entry->body = body; entry->response_bytes = response_bytes; Py_INCREF(body); Py_INCREF(response_bytes); cache.end++; } #endif PyObject* Response_render(Response* self, bool simple) { PyObject* response_bytes = NULL; PyObject* cookies_str = NULL; PyObject* cookies_bytes = NULL; #ifdef RESPONSE_CACHE bool cacheable = Response_cacheable(self, simple); if(cacheable && (response_bytes = Response_from_cache(self->body))) return response_bytes; #endif size_t buffer_offset; Py_ssize_t body_len = 0; const char* body = NULL; *(self->buffer + minor_offset) = '0' + (char)self->minor_version; if(self->code) { unsigned long code = PyLong_AsUnsignedLong(self->code); if(code < 100 || code > 599) { PyErr_SetString(PyExc_ValueError, "Invalid status code"); goto error; } unsigned int status_category = code / 100 - 1; unsigned int status_rest = code % 100; const ReasonRange* reason_range = reason_ranges + status_category; if(status_rest > reason_range->maximum) { PyErr_SetString(PyExc_ValueError, "Invalid status code"); goto error; } /* TODO these are always 3 digit, maybe modulus would be faster */ snprintf(self->buffer + code_offset, 4, "%ld", code); *(self->buffer + code_offset + 3) = ' '; const char* reason = reason_range->reasons[status_rest]; size_t reason_len = strlen(reason); memcpy(self->buffer + reason_offset, reason, reason_len); buffer_offset = reason_offset + reason_len; CRLF memcpy(self->buffer + buffer_offset, "Content-Length: ", strlen("Content-Length: ")); buffer_offset += strlen("Content-Length: "); } else { memcpy(self->buffer + code_offset, "200", 3); buffer_offset = strlen(header); } if(self->body) { if(PyBytes_AsStringAndSize(self->body, (char**)&body, &body_len) == -1) goto error; int result = sprintf( self->buffer + buffer_offset, "%ld", (unsigned long)body_len); buffer_offset += result; } else { *(self->buffer + buffer_offset) = '0'; buffer_offset++; } CRLF if(self->minor_version == 1 && self->keep_alive == KEEP_ALIVE_FALSE) { memcpy( self->buffer + buffer_offset, "Connection: close\r\n", strlen("Connection: close\r\n")); buffer_offset += strlen("Connection: close\r\n"); } else if(self->minor_version == 0 && self->keep_alive == KEEP_ALIVE_TRUE) { memcpy( self->buffer + buffer_offset, "Connection: keep-alive\r\n", strlen("Connection: keep-aplive\r\n")); buffer_offset += strlen("Connection: keep-alive\r\n"); } // dont output Content-Type if there is no body if(!self->body) goto headers; memcpy(self->buffer + buffer_offset, Content_Type, strlen(Content_Type)); buffer_offset += strlen(Content_Type); Py_ssize_t mime_type_len = strlen(text_plain); const char* mime_type = text_plain; if(self->mime_type) { mime_type = PyUnicode_AsUTF8AndSize(self->mime_type, &mime_type_len); if(!mime_type) goto error; } memcpy(self->buffer + buffer_offset, mime_type, (size_t)mime_type_len); buffer_offset += mime_type_len; Py_ssize_t encoding_len = strlen(utf8); const char* encoding = utf8; if(self->encoding) { encoding = PyUnicode_AsUTF8AndSize(self->encoding, &encoding_len); if(!encoding) goto error; } #define text_or_json \ (mime_type_len >= 5 \ && (memcmp(mime_type, "text/", 5) == 0 \ || memcmp(mime_type + mime_type_len - 5, "/json", 5) == 0)) if(self->encoding || text_or_json) { memcpy(self->buffer + buffer_offset, charset, strlen(charset)); buffer_offset += strlen(charset); memcpy(self->buffer + buffer_offset, encoding, (size_t)encoding_len); buffer_offset += (size_t)encoding_len; } CRLF headers: if(!self->headers) goto empty_headers; Py_ssize_t headers_len; if((headers_len = PyDict_Size(self->headers)) < 0) goto error; if(!headers_len) goto empty_headers; PyObject *name, *value; Py_ssize_t pos = 0; while (PyDict_Next(self->headers, &pos, &name, &value)) { const char* cname; Py_ssize_t name_len; const char* cvalue; Py_ssize_t value_len; if(!(cname = PyUnicode_AsUTF8AndSize(name, &name_len))) goto error; memcpy(self->buffer + buffer_offset, cname, (size_t)name_len); buffer_offset += (size_t)name_len; *(self->buffer + buffer_offset) = ':'; buffer_offset++; *(self->buffer + buffer_offset) = ' '; buffer_offset++; if(!(cvalue = PyUnicode_AsUTF8AndSize(value, &value_len))) goto error; memcpy(self->buffer + buffer_offset, cvalue, (size_t)value_len); buffer_offset += (size_t)value_len; CRLF } empty_headers: if(!self->cookies) goto empty_cookies; Py_ssize_t cookies_len; if((cookies_len = PyObject_Size(self->cookies)) < 0) goto error; if(!cookies_len) goto empty_cookies; if(!(cookies_str = PyObject_Str(self->cookies))) goto error; if(!(cookies_bytes = PyUnicode_AsASCIIString(cookies_str))) goto error; char* ccookies; Py_ssize_t ccookies_len; if(PyBytes_AsStringAndSize(cookies_bytes, &ccookies, &ccookies_len) == -1) goto error; memcpy(self->buffer + buffer_offset, ccookies, (size_t)ccookies_len); buffer_offset += (size_t)ccookies_len; CRLF empty_cookies: CRLF if(body) { bfrcpy(body, (size_t)body_len) } #undef CRLF if(!(response_bytes = PyBytes_FromStringAndSize(self->buffer, buffer_offset))) goto error; #ifdef RESPONSE_CACHE if(cacheable) Response_cache(self->body, response_bytes); #endif goto finally; error: Py_XDECREF(response_bytes); response_bytes = NULL; finally: Py_XDECREF(cookies_str); Py_XDECREF(cookies_bytes); return response_bytes; } static PyMethodDef Response_methods[] = { //{"render", (PyCFunction)Response_render, METH_NOARGS, "render"}, {NULL} }; static PyTypeObject ResponseType = { PyVarObject_HEAD_INIT(NULL, 0) "cresponse.Response", /* tp_name */ sizeof(Response), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Response_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Response", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Response_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Response_init, /* tp_init */ 0, /* tp_alloc */ Response_new, /* tp_new */ }; static PyModuleDef cresponse = { PyModuleDef_HEAD_INIT, "cresponse", "cresponse", -1, NULL, NULL, NULL, NULL, NULL }; PyMODINIT_FUNC PyInit_cresponse(void) { PyObject* m = NULL; PyObject* api_capsule = NULL; PyObject* json = NULL; if (PyType_Ready(&ResponseType) < 0) goto error; m = PyModule_Create(&cresponse); if(!m) goto error; Py_INCREF(&ResponseType); PyModule_AddObject(m, "Response", (PyObject*)&ResponseType); if(!(json = PyImport_ImportModule("json"))) goto error; if(!(json_dumps = PyObject_GetAttrString(json, "dumps"))) goto error; if(!(application_json = PyUnicode_FromString("application/json"))) goto error; if(!(application_octet = PyUnicode_FromString("application/octet-stream"))) goto error; static Response_CAPI capi = { &ResponseType, Response_render, Response_init }; api_capsule = export_capi(m, "japronto.response.cresponse", &capi); if(!api_capsule) goto error; goto finally; error: m = NULL; finally: Py_XDECREF(json); Py_XDECREF(api_capsule); return m; } #endif ================================================ FILE: src/japronto/response/cresponse.h ================================================ #pragma once #include #include #include "common.h" #define RESPONSE_INITIAL_BUFFER_LEN 1024 typedef struct { PyObject_HEAD bool opaque; int minor_version; KEEP_ALIVE keep_alive; PyObject* code; PyObject* mime_type; PyObject* body; PyObject* encoding; PyObject* headers; PyObject* cookies; char* buffer; size_t buffer_len; char inline_buffer[RESPONSE_INITIAL_BUFFER_LEN]; } Response; typedef struct { PyTypeObject* ResponseType; PyObject* (*Response_render)(Response*, bool); int (*Response_init)(Response* self, PyObject *args, PyObject *kw); } Response_CAPI; #ifndef RESPONSE_OPAQUE PyObject* Response_new(PyTypeObject* type, Response* self); void Response_dealloc(Response* self); #endif ================================================ FILE: src/japronto/response/cresponse_ext.py ================================================ from distutils.core import Extension def get_extension(): define_macros = [('RESPONSE_OPAQUE', 1)] if system.args.enable_response_cache: define_macros.append(('RESPONSE_CACHE', 1)) return Extension( 'japronto.response.cresponse', sources=['cresponse.c', '../capsule.c'], include_dirs=['..'], define_macros=define_macros) ================================================ FILE: src/japronto/response/py.py ================================================ _responses = None def factory(status_code=200, text='', mime_type='text/plain', encoding='utf-8'): global _responses if _responses is None: _responses = [] for _ in range(100): _responses.append(Response()) response = _responses.pop() response.status_code = status_code response.mime_type = mime_type response.text = text response.encoding = encoding return response def dispose(response): _responses.append(response) class Response: __slots__ = ('status_code', 'mime_type', 'text', 'encoding') def __init__(self, status_code=200, text='', mime_type='text/plain', encoding='utf-8'): self.status_code = status_code self.mime_type = mime_type self.text = text self.encoding = encoding def render(self): data = ['HTTP/1.1 ', str(self.status_code), ' OK\r\n'] data.append('Connection: keep-alive\r\n') body = self.text.encode(self.encoding) data.extend([ 'Content-Type: ', self.mime_type, '; encoding=', self.encoding, '\r\n']) data.extend(['Content-Length: ', str(len(body)), '\r\n\r\n']) return ''.join(data).encode(self.encoding) + body ================================================ FILE: src/japronto/response/reasons.h ================================================ #pragma once static const char* reasons_1xx[] = { "Continue", //100 "Switching Protocols", //101 }; static const char* reasons_2xx[] = { "OK", //200 "Created", //201 "Accepted", //202 "Non-Authoritative Information", //203 "No Content", // 204 "Reset Content", //205 "Partial Content", //206 }; static const char* reasons_3xx[] = { "Multiple Choices", //300 "Moved Permanently", //301 "Found", //302 "See Other", //303, "Not Modified", //304 "Use Proxy", //305 "Proxy Switch", //306 "Temporary Redirect", //307 }; static const char* reasons_4xx[] = { "Bad Request", //400 "Unauthorized", //401 "Payment Required", //402 "Forbidden", //403 "Not Found", //404 "Method Not Allowed", //405 "Not acceptable", //406 "Proxy Authentication Required", //407 "Request Timeout", //408 "Conflict", //409 "Gone", //410 "Length Required", //411 "Precondition Failed", //412 "Request Entity Too Large", //413 "Request-URI Too Long", //414 "Unsupported Media Type", //415 "Requested Range Not Satisfiable", //416 "Expectation Failed", //417 }; static const char* reasons_5xx[] = { "Internal Server Error", //500 "Not Implemented", //501 "Bad Gateway", //502 "Service Unavailable", //503 "Gateway Timeout", //504 "HTTP Version Not Supported", //505 }; typedef struct { const char** reasons; size_t maximum; } ReasonRange; static const ReasonRange reason_ranges[] = { {reasons_1xx, 1}, {reasons_2xx, 6}, {reasons_3xx, 7}, {reasons_4xx, 17}, {reasons_5xx, 5} }; ================================================ FILE: src/japronto/router/__init__.py ================================================ from .route import Route, RouteNotFoundException from .cmatcher import Matcher class Router: def __init__(self, matcher_factory=Matcher): self._routes = [] self.matcher_factory = matcher_factory def add_route(self, pattern, handler, method=None, methods=None): assert not(method and methods), "Cannot use method and methods" if method: methods = [method] if not methods: methods = [] methods = {m.upper() for m in methods} route = Route(pattern, handler, methods) self._routes.append(route) return route def get_matcher(self): return self.matcher_factory(self._routes) ================================================ FILE: src/japronto/router/analyzer.py ================================================ import dis import functools import types FLAG_COROUTINE = 128 def is_simple(fun): """A heuristic to find out if a function is simple enough.""" seen_load_fast_0 = False seen_load_response = False seen_call_fun = False for instruction in dis.get_instructions(fun): if instruction.opname == 'LOAD_FAST' and instruction.arg == 0: seen_load_fast_0 = True continue if instruction.opname == 'LOAD_ATTR' \ and instruction.argval == 'Response': seen_load_response = True continue if instruction.opname.startswith('CALL_FUNCTION'): if seen_call_fun: return False seen_call_fun = True continue return seen_call_fun and seen_load_fast_0 and seen_load_response def is_pointless_coroutine(fun): for instruction in dis.get_instructions(fun): if instruction.opname in ('GET_AWAITABLE', 'YIELD_FROM'): return False return True def coroutine_to_func(f): # Based on http://stackoverflow.com/questions/13503079/ # how-to-create-a-copy-of-a-python-function oc = f.__code__ code = types.CodeType( oc.co_argcount, oc.co_kwonlyargcount, oc.co_nlocals, oc.co_stacksize, oc.co_flags & ~FLAG_COROUTINE, oc.co_code, oc.co_consts, oc.co_names, oc.co_varnames, oc.co_filename, oc.co_name, oc.co_firstlineno, oc.co_lnotab, oc.co_freevars, oc.co_cellvars) g = types.FunctionType( code, f.__globals__, name=f.__name__, argdefs=f.__defaults__, closure=f.__closure__) g = functools.update_wrapper(g, f) g.__kwdefaults__ = f.__kwdefaults__ return g ================================================ FILE: src/japronto/router/cmatcher.c ================================================ #include #include #include "cmatcher.h" #include "crequest.h" #include "capsule.h" struct _Matcher { PyObject_HEAD size_t buffer_len; char* buffer; }; typedef enum { SEGMENT_EXACT, SEGMENT_PLACEHOLDER } SegmentType; typedef struct { size_t data_length; char data[]; } ExactSegment; typedef struct { size_t name_length; char name[]; } PlaceholderSegment; typedef struct { SegmentType type; union { ExactSegment exact; PlaceholderSegment placeholder; }; } Segment; static MatchDictEntry _match_dict_entries[10]; static Request_CAPI* request_capi; static PyObject* compile_all; static PyObject * Matcher_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { Matcher* self = NULL; self = (Matcher*)type->tp_alloc(type, 0); if(!self) goto finally; self->buffer = NULL; self->buffer_len = 0; finally: return (PyObject*)self; } #define ROUNDTO8(v) (((v) + 7) & ~7) #define ENTRY_LOOP \ char* entry_end = self->buffer + self->buffer_len; \ for(MatcherEntry* entry = (MatcherEntry*)self->buffer; \ (char*)entry < entry_end; \ entry = (MatcherEntry*)((char*)entry + sizeof(MatcherEntry) + \ ROUNDTO8(entry->pattern_len) + ROUNDTO8(entry->methods_len))) #define SEGMENT_LOOP \ char* segments_end = entry->buffer + entry->pattern_len; \ for(Segment* segment = (Segment*)entry->buffer; \ (char*)segment < segments_end; \ segment = (Segment*)((char*)segment + sizeof(Segment) + \ ROUNDTO8(segment->type == SEGMENT_EXACT ? \ segment->exact.data_length : segment->placeholder.name_length))) static void Matcher_dealloc(Matcher* self) { if(self->buffer) { ENTRY_LOOP { Py_DECREF(entry->handler); Py_DECREF(entry->route); } free(self->buffer); } Py_TYPE(self)->tp_free((PyObject*)self); } static int Matcher_init(Matcher* self, PyObject *args, PyObject *kw) { int result = 0; PyObject* compiled = NULL; PyObject* routes; if(!PyArg_ParseTuple(args, "O", &routes)) goto error; if(!(compiled = PyObject_CallFunctionObjArgs(compile_all, routes, NULL))) goto error; char* compiled_buffer; if(PyBytes_AsStringAndSize(compiled, &compiled_buffer, (Py_ssize_t*)&self->buffer_len) == -1) goto error; if(!(self->buffer = malloc(self->buffer_len))) goto error; memcpy(self->buffer, compiled_buffer, self->buffer_len); ENTRY_LOOP { Py_INCREF(entry->handler); Py_INCREF(entry->route); } goto finally; error: result = -1; finally: Py_XDECREF(compiled); return result; } // borrows route and handler in matcher entry MatcherEntry* Matcher_match_request(Matcher* self, PyObject* request, MatchDictEntry** match_dict_entries, size_t* match_dict_length) { MatcherEntry* result = NULL; PyObject* path = NULL; PyObject* method = NULL; size_t method_len; char* method_str; size_t path_len; char* path_str; if(Py_TYPE(request) != request_capi->RequestType) { path = PyObject_GetAttrString(request, "path"); if(!path) goto error; path_str = PyUnicode_AsUTF8AndSize(path, (Py_ssize_t*)&path_len); if(!path_str) goto error; method = PyObject_GetAttrString(request, "method"); if(!method) goto error; method_str = PyUnicode_AsUTF8AndSize(method, (Py_ssize_t*)&method_len); if(!method_str) goto error; } else { method_len = REQUEST(request)->method_len; method_str = REQUEST_METHOD(request); path_str = request_capi->Request_get_decoded_path( REQUEST(request), &path_len); } ENTRY_LOOP { char* rest = path_str; size_t rest_len = path_len; MatchDictEntry* current_mde = _match_dict_entries; size_t value_length = 1; SEGMENT_LOOP { if(segment->type == SEGMENT_EXACT) { if(rest_len < segment->exact.data_length) break; if(memcmp(rest, segment->exact.data, segment->exact.data_length) != 0) break; rest += segment->exact.data_length; rest_len -= segment->exact.data_length; } else if(segment->type == SEGMENT_PLACEHOLDER) { assert(((size_t)(current_mde - _match_dict_entries)) < sizeof(_match_dict_entries) / sizeof(MatchDictEntry)); char* slash = memchr(rest, '/', rest_len); current_mde->value = rest; if(slash) { value_length = current_mde->value_length = slash - rest; rest_len -= current_mde->value_length; rest = slash; } else { value_length = current_mde->value_length = rest_len; rest_len = 0; } if(!value_length) break; current_mde->key = segment->placeholder.name; current_mde->key_length = segment->placeholder.name_length; current_mde++; } else { assert(0); } } if(rest_len) continue; if(!value_length) continue; if((size_t)(current_mde - _match_dict_entries) != entry->placeholder_cnt) continue; if(!entry->methods_len) goto loop_finally; char* method_found = memmem( entry->buffer + entry->pattern_len, entry->methods_len, method_str, (size_t)method_len); if(!method_found) continue; if(*(method_found + (size_t)method_len) != ' ') continue; loop_finally: result = entry; if(match_dict_entries) *match_dict_entries = _match_dict_entries; if(match_dict_length) *match_dict_length = current_mde - _match_dict_entries; goto finally; } if(match_dict_length) *match_dict_length = 0; goto finally; error: result = NULL; finally: if(Py_TYPE(request) != request_capi->RequestType) { Py_XDECREF(method); Py_XDECREF(path); } return result; } static PyObject* _Matcher_match_request(Matcher* self, PyObject* request) { MatcherEntry* matcher_entry; MatchDictEntry* entries; PyObject* route = NULL; size_t length; PyObject* match_dict = NULL; PyObject* route_dict = NULL; if(!(matcher_entry = Matcher_match_request( self, request, &entries, &length))) Py_RETURN_NONE; route = matcher_entry->route; if(!(match_dict = MatchDict_entries_to_dict(entries, length))) goto error; if(!(route_dict = PyTuple_New(2))) goto error; PyTuple_SET_ITEM(route_dict, 0, route); PyTuple_SET_ITEM(route_dict, 1, match_dict); goto finally; error: Py_XDECREF(match_dict); route = NULL; finally: if(route) Py_INCREF(route); return route_dict; } static PyMethodDef Matcher_methods[] = { {"match_request", (PyCFunction)_Matcher_match_request, METH_O, ""}, {NULL} }; static PyTypeObject MatcherType = { PyVarObject_HEAD_INIT(NULL, 0) "cmatcher.Matcher", /* tp_name */ sizeof(Matcher), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Matcher_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ "Matcher", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Matcher_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Matcher_init, /* tp_init */ 0, /* tp_alloc */ Matcher_new, /* tp_new */ }; static PyModuleDef cmatcher = { PyModuleDef_HEAD_INIT, "cmatcher", "cmatcher", -1, NULL, NULL, NULL, NULL, NULL }; PyMODINIT_FUNC PyInit_cmatcher(void) { PyObject* m = NULL; PyObject* api_capsule = NULL; PyObject* router_route = NULL; if (PyType_Ready(&MatcherType) < 0) goto error; m = PyModule_Create(&cmatcher); if(!m) goto error; request_capi = import_capi("japronto.request.crequest"); if(!request_capi) goto error; if(!(router_route = PyImport_ImportModule("japronto.router.route"))) goto error; if(!(compile_all = PyObject_GetAttrString(router_route, "compile_all"))) goto error; Py_INCREF(&MatcherType); PyModule_AddObject(m, "Matcher", (PyObject*)&MatcherType); static Matcher_CAPI capi = { Matcher_match_request }; api_capsule = export_capi(m, "japronto.router.cmatcher", &capi); if(!api_capsule) goto error; goto finally; error: m = NULL; finally: Py_XDECREF(router_route); Py_XDECREF(api_capsule); return m; } ================================================ FILE: src/japronto/router/cmatcher.h ================================================ #pragma once #include #include #include "match_dict.h" typedef struct { PyObject* route; PyObject* handler; bool coro_func; bool simple; size_t pattern_len; size_t methods_len; size_t placeholder_cnt; char buffer[]; } MatcherEntry; typedef struct _Matcher Matcher; typedef struct { MatcherEntry* (*Matcher_match_request) (Matcher* matcher, PyObject* request, MatchDictEntry** match_dict_entries, size_t* match_dict_length); } Matcher_CAPI; ================================================ FILE: src/japronto/router/cmatcher_ext.py ================================================ from distutils.core import Extension def get_extension(): return Extension( 'japronto.router.cmatcher', sources=['cmatcher.c', 'match_dict.c', '../capsule.c'], include_dirs=['.', '../request', '..', '../response']) ================================================ FILE: src/japronto/router/match_dict.c ================================================ #include #include "match_dict.h" PyObject* MatchDict_entries_to_dict(MatchDictEntry* entries, size_t length) { PyObject* match_dict = NULL; if(!(match_dict = PyDict_New())) goto error; for(MatchDictEntry* entry = entries; entry < entries + length; entry++) { PyObject* key = NULL; PyObject* value = NULL; if(!(key = PyUnicode_FromStringAndSize(entry->key, entry->key_length))) goto loop_error; if(!(value = PyUnicode_FromStringAndSize(entry->value, entry->value_length))) goto loop_error; if(PyDict_SetItem(match_dict, key, value) == -1) goto loop_error; goto loop_finally; loop_error: Py_XDECREF(match_dict); match_dict = NULL; loop_finally: Py_XDECREF(key); Py_XDECREF(value); if(!match_dict) goto error; } goto finally; error: Py_XDECREF(match_dict); match_dict = NULL; finally: return match_dict; } ================================================ FILE: src/japronto/router/match_dict.h ================================================ #pragma once #include typedef struct { char* key; size_t key_length; char* value; size_t value_length; } MatchDictEntry; PyObject* MatchDict_entries_to_dict(MatchDictEntry* entries, size_t length); ================================================ FILE: src/japronto/router/matcher.py ================================================ class Matcher: def __init__(self, routes): self._routes = routes def match_request(self, request): for route in self._routes: match_dict = {} rest = request.path value = True for typ, data in route.segments: if typ == 'exact': if not rest.startswith(data): break rest = rest[len(data):] elif typ == 'placeholder': value, slash, rest = rest.partition('/') if not value: break match_dict[data] = value rest = slash + rest else: assert 0, 'Unknown type' if rest: continue if not value: continue if len(match_dict) != route.placeholder_cnt: continue if route.methods and request.method not in route.methods: continue return route, match_dict ================================================ FILE: src/japronto/router/route.py ================================================ import asyncio from enum import IntEnum from struct import Struct from . import analyzer class RouteNotFoundException(Exception): pass class Route: def __init__(self, pattern, handler, methods): self.pattern = pattern self.handler = handler self.methods = methods self.segments = parse(pattern) self.placeholder_cnt = \ sum(1 for s in self.segments if s[0] == 'placeholder') def __repr__(self): return ''.format( self.pattern, self.methods, hex(id(self))) def describe(self): return self.pattern + (' ' if self.methods else '') + \ ' '.join(self.methods) def __eq__(self, other): return self.pattern == other.pattern and self.methods == other.methods def parse(pattern): names = set() result = [] rest = pattern while rest: exact = '' while rest: chunk, _, rest = rest.partition('{') exact += chunk if rest and rest[0] == '{': exact += '{{' rest = rest[1:] else: break if exact: exact = exact.replace('{{', '{').replace('}}', '}') result.append(('exact', exact)) if not rest: break name, _, rest = rest.partition('}') if not _: raise ValueError('Unbalanced "{" in pattern') if rest and rest[0] != '/': raise ValueError( '"}" must be followed by "/" or appear at the end') if name in names: raise ValueError('Duplicate name "{}" in pattern'.format(name)) names.add(name) result.append(('placeholder', name)) return result class SegmentType(IntEnum): EXACT = 0 PLACEHOLDER = 1 """ typedef struct { PyObject* route; PyObject* handler; bool coro_func; bool simple; size_t pattern_len; size_t methods_len; size_t placeholder_cnt; char buffer[]; } MatcherEntry; """ MatcherEntry = Struct('PP??NNN') """ typedef enum { SEGMENT_EXACT, SEGMENT_PLACEHOLDER } SegmentType; typedef struct { size_t data_length; char data[]; } ExactSegment; typedef struct { size_t name_length; char name[]; } PlaceholderSegment; typedef struct { SegmentType type; union { ExactSegment exact; PlaceholderSegment placeholder; }; } Segment; """ ExactSegment = Struct('iN') PlaceholderSegment = Struct('iN') Segment = Struct('iN') def roundto8(v): return (v + 7) & ~7 def padto8(data): """Pads data to the multiplies of 8 bytes. This makes x86_64 faster and prevents undefined behavior on other platforms""" length = len(data) return data + b'\xdb' * (roundto8(length) - length) retain_handlers = set() def compile(route): pattern_buf = b'' for segment in route.segments: typ = getattr(SegmentType, segment[0].upper()) pattern_buf += Segment.pack(typ, len(segment[1].encode('utf-8'))) \ + padto8(segment[1].encode('utf-8')) methods_buf = ' '.join(route.methods).encode('ascii') methods_len = len(methods_buf) if methods_buf: methods_buf += b' ' methods_len += 1 methods_buf = padto8(methods_buf) handler = route.handler if asyncio.iscoroutinefunction(handler) \ and analyzer.is_pointless_coroutine(handler): handler = analyzer.coroutine_to_func(handler) # since we save id to handler in matcher entry and this is the only # reference before INCREF-ed in matcher we store it in set to prevent # destruction retain_handlers.add(handler) return MatcherEntry.pack( id(route), id(handler), asyncio.iscoroutinefunction(handler), analyzer.is_simple(handler), len(pattern_buf), methods_len, route.placeholder_cnt) \ + pattern_buf + methods_buf def compile_all(routes): return b''.join(compile(r) for r in routes) ================================================ FILE: src/japronto/router/test_analyzer.py ================================================ from collections import OrderedDict import pytest from . import analyzer simple_fixtures = OrderedDict([ ('empty', ('def a(): pass', False)), ('arg', ('def a(b): return b', False)), ('simple', ('def a(c): return c.Response()', True)), ('body', ('def a(c): return c.Response(body="abc")', True)), ('wrongattr', ('def a(d): return d.R()', False)), ('extracall', ( ''' def a(b): d() return b.Response() def d(): pass ''', False)), ('expressions', ( ''' def a(b): c = "Hey!" d = "Dude" return b.Response(json={c: d}) ''', True)) ]) @pytest.mark.parametrize( 'code,simple', simple_fixtures.values(), ids=list(simple_fixtures.keys())) def test_is_simple(code, simple): module = compile(code, '?', 'exec') fun_code = module.co_consts[0] assert analyzer.is_simple(fun_code) == simple pointless_fixtures = OrderedDict([ ('empty', ('async def a(): pass', True)), ('simple', ('async def a(): return 1', True)), ('yieldfrom', ('def a(b): yield from b', False)), ('await', ('async def a(b): await b', False)) ]) @pytest.mark.parametrize( 'code,pointless', pointless_fixtures.values(), ids=list(pointless_fixtures.keys())) def test_is_pointless(code, pointless): module = compile(code, '?', 'exec') fun_code = module.co_consts[0] assert analyzer.is_pointless_coroutine(fun_code) == pointless ================================================ FILE: src/japronto/router/test_matcher.py ================================================ from functools import partial import pytest from . import Route from .matcher import Matcher from .cmatcher import Matcher as CMatcher class FakeRequest: def __init__(self, method, path): self.method = method self.path = path @classmethod def from_str(cls, value): return cls(*value.split()) class TracingRoute(Route): cnt = 0 def __new__(cls, *args, **kw): print('new', args) cls.cnt += 1 return Route.__new__(cls) def __init__(self, pattern, methods): super().__init__(pattern, lambda x: None, methods=methods) def __del__(self): type(self).cnt -= 1 print('del') def route_from_str(value): pattern, *methods = value.split() if methods: methods = methods[0].split(',') return TracingRoute(pattern, methods=methods) def parametrize_make_matcher(): def make(cls): routes = [route_from_str(r) for r in [ '/', '/test GET', '/hi/{there} POST,DELETE', '/{oh}/{dear} PATCH', '/lets PATCH' ]] return cls(routes) make_matcher = partial(make, Matcher) make_cmatcher = partial(make, CMatcher) return pytest.mark.parametrize( 'make_matcher', [make_matcher, make_cmatcher], ids=['py', 'c']) def parametrize_request_route_and_dict(cases): return pytest.mark.parametrize( 'req,route,match_dict', ((FakeRequest.from_str(req), route_from_str(route), match_dict) for req, route, match_dict in cases), ids=[req + '-' + route for req, route, _ in cases]) @parametrize_request_route_and_dict([ ('GET /', '/', {}), ('POST /', '/', {}), ('GET /test', '/test GET', {}), ('DELETE /hi/jane', '/hi/{there} POST,DELETE', {'there': 'jane'}), ('PATCH /lets/go', '/{oh}/{dear} PATCH', {'oh': 'lets', 'dear': 'go'}), ('PATCH /lets', '/lets PATCH', {}) ]) @parametrize_make_matcher() def test_matcher(make_matcher, req, route, match_dict): cnt = TracingRoute.cnt matcher = make_matcher() assert matcher.match_request(req) == (route, match_dict) del matcher assert cnt == TracingRoute.cnt def parametrize_request(requests): return pytest.mark.parametrize( 'req', (FakeRequest.from_str(r) for r in requests), ids=requests) @parametrize_request([ 'POST /test', 'GET /test/', 'GET /hi/jane', 'POST /hi/jane/', 'POST /hi/', 'GET /abc', 'PATCH //dance' ]) @parametrize_make_matcher() def test_matcher_not_found(make_matcher, req): cnt = TracingRoute.cnt matcher = make_matcher() assert matcher.match_request(req) is None del matcher assert cnt == TracingRoute.cnt ================================================ FILE: src/japronto/router/test_route.py ================================================ import asyncio from collections import namedtuple import pytest from .route import parse, MatcherEntry, Segment, SegmentType, Route, \ compile, roundto8 @pytest.mark.parametrize('pattern,result', [ ('/', [('exact', '/')]), ('/{{a}}', [('exact', '/{a}')]), ('{a}', [('placeholder', 'a')]), ('a/{a}', [('exact', 'a/'), ('placeholder', 'a')]), ('{a}/a', [('placeholder', 'a'), ('exact', '/a')]), ('{a}/{{a}}', [('placeholder', 'a'), ('exact', '/{a}')]), ('{a}/{b}', [('placeholder', 'a'), ('exact', '/'), ('placeholder', 'b')]) ]) def test_parse(pattern, result): assert parse(pattern) == result @pytest.mark.parametrize('pattern,error', [ ('{a', 'Unbalanced'), ('{a}/{b', 'Unbalanced'), ('{a}a', 'followed by'), ('{a}/{a}', 'Duplicate') ]) def test_parse_error(pattern, error): with pytest.raises(ValueError) as info: parse(pattern) assert error in info.value.args[0] DecodedRoute = namedtuple( 'DecodedRoute', 'route_id,handler_id,coro_func,simple,placeholder_cnt,segments,methods') def decompile(buffer): route_id, handler_id, coro_func, simple, \ pattern_len, methods_len, placeholder_cnt \ = MatcherEntry.unpack_from(buffer, 0) offset = MatcherEntry.size pattern_offset_end = offset + roundto8(pattern_len) segments = [] while offset < pattern_offset_end: typ, segment_length = Segment.unpack_from(buffer, offset) offset += Segment.size typ = SegmentType(typ).name.lower() data = buffer[offset:offset + segment_length].decode('utf-8') offset += roundto8(segment_length) segments.append((typ, data)) methods = buffer[offset:offset + methods_len].strip().decode('ascii') \ .split() return DecodedRoute( route_id, handler_id, coro_func, simple, placeholder_cnt, segments, methods) def handler(): pass async def coro(): # needs to have await to prevent being promoted to function await asyncio.sleep(1) @pytest.mark.parametrize('route', [ Route('/', handler, []), Route('/', coro, ['GET']), Route('/test/{hi}', handler, []), Route('/test/{hi}', coro, ['POST']), Route('/tést', coro, ['POST']) ], ids=Route.describe) def test_compile(route): decompiled = decompile(compile(route)) assert decompiled.route_id == id(route) assert decompiled.handler_id == id(route.handler) assert decompiled.coro_func == asyncio.iscoroutinefunction(route.handler) assert not decompiled.simple assert decompiled.placeholder_cnt == route.placeholder_cnt assert decompiled.segments == route.segments assert decompiled.methods == route.methods ================================================ FILE: src/japronto/runner.py ================================================ from argparse import ArgumentParser, SUPPRESS from importlib import import_module import os import sys import runpy from .app import Application try: ModuleNotFoundError except NameError: ModuleNotFoundError = ImportError def get_parser(): prog = 'python -m japronto' if sys.argv[0].endswith('__main__.py') \ else 'japronto' parser = ArgumentParser(prog=prog) parser.add_argument('--host', dest='host', type=str, default='0.0.0.0') parser.add_argument('--port', dest='port', type=int, default=8080) parser.add_argument('--worker-num', dest='worker_num', type=int, default=1) parser.add_argument( '--reload', dest='reload', action='store_const', const=True, default=False) parser.add_argument( '--reloader-pid', dest='reloader_pid', type=int, help=SUPPRESS) parser.add_argument( '--script', dest='script', action='store_const', const=True, default=False, help=SUPPRESS) parser.add_argument('application') return parser def verify(args): if args.script: script = args.application if not os.path.exists(script): print("Script '{}' not found.".format(script)) return script else: try: module, attribute = args.application.rsplit('.', 1) except ValueError: print( "Application specificer must contain at least one '.'," + "got '{}'.".format(args.application)) return False try: module = import_module(module) except ModuleNotFoundError as e: print(e.args[0] + ' on Python search path.') return False try: attribute = getattr(module, attribute) except AttributeError: print( "Module '{}' does not have an attribute '{}'." .format(module.__name__, attribute)) return False if not isinstance(attribute, Application): print("{} is not an instance of 'japronto.Application'.") return False return attribute def run(attribute, args): if args.script: runpy.run_path(attribute) else: attribute._run( host=args.host, port=args.port, worker_num=args.worker_num, reloader_pid=args.reloader_pid) ================================================ FILE: src/picohttpparser/build ================================================ #!/bin/bash set -ex gcc -c picohttpparser.c -O3 -fPIC -msse4.2 -o ssepicohttpparser.o gcc -c picohttpparser.c -O3 -fPIC -o picohttpparser.o gcc -fPIC -shared -o libpicohttpparser.so picohttpparser.o ssepicohttpparser.o ================================================ FILE: src/picohttpparser/picohttpparser.c ================================================ /* * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase, * Shigeo Mitsunari * * The software is licensed under either the MIT License (below) or the Perl * license. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to * deal in the Software without restriction, including without limitation the * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or * sell copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS * IN THE SOFTWARE. */ #include #include #include #ifdef __SSE4_2__ #ifdef _MSC_VER #include #else #include #endif #endif #include "picohttpparser.h" /* $Id: 3228bad61525d68b757d7542bb817ccba6d7bc7f $ */ #if __GNUC__ >= 3 #define likely(x) __builtin_expect(!!(x), 1) #define unlikely(x) __builtin_expect(!!(x), 0) #else #define likely(x) (x) #define unlikely(x) (x) #endif #ifdef _MSC_VER #define ALIGNED(n) _declspec(align(n)) #else #define ALIGNED(n) __attribute__((aligned(n))) #endif #define IS_PRINTABLE_ASCII(c) ((unsigned char)(c)-040u < 0137u) #define CHECK_EOF() \ if (buf == buf_end) { \ *ret = -2; \ return NULL; \ } #define EXPECT_CHAR(ch) \ CHECK_EOF(); \ if (*buf++ != ch) { \ *ret = -1; \ return NULL; \ } #define ADVANCE_TOKEN(tok, toklen) \ do { \ const char *tok_start = buf; \ static const char ALIGNED(16) ranges2[] = "\000\040\177\177"; \ int found2; \ buf = findchar_fast(buf, buf_end, ranges2, sizeof(ranges2) - 1, &found2); \ if (!found2) { \ CHECK_EOF(); \ } \ while (1) { \ if (*buf == ' ') { \ break; \ } else if (unlikely(!IS_PRINTABLE_ASCII(*buf))) { \ if ((unsigned char)*buf < '\040' || *buf == '\177') { \ *ret = -1; \ return NULL; \ } \ } \ ++buf; \ CHECK_EOF(); \ } \ tok = tok_start; \ toklen = buf - tok_start; \ } while (0) static const char *token_char_map = "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\1\0\1\1\1\1\1\0\0\1\1\0\1\1\0\1\1\1\1\1\1\1\1\1\1\0\0\0\0\0\0" "\0\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\0\0\0\1\1" "\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\0\1\0\1\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"; static const char *findchar_fast(const char *buf, const char *buf_end, const char *ranges, size_t ranges_size, int *found) { *found = 0; #if __SSE4_2__ if (likely(buf_end - buf >= 16)) { __m128i ranges16 = _mm_loadu_si128((const __m128i *)ranges); size_t left = (buf_end - buf) & ~15; do { __m128i b16 = _mm_loadu_si128((void *)buf); int r = _mm_cmpestri(ranges16, ranges_size, b16, 16, _SIDD_LEAST_SIGNIFICANT | _SIDD_CMP_RANGES | _SIDD_UBYTE_OPS); if (unlikely(r != 16)) { buf += r; *found = 1; break; } buf += 16; left -= 16; } while (likely(left != 0)); } #else /* suppress unused parameter warning */ (void)buf_end; (void)ranges; (void)ranges_size; #endif return buf; } static const char *get_token_to_eol(const char *buf, const char *buf_end, const char **token, size_t *token_len, int *ret) { const char *token_start = buf; #ifdef __SSE4_2__ static const char ranges1[] = "\0\010" /* allow HT */ "\012\037" /* allow SP and up to but not including DEL */ "\177\177" /* allow chars w. MSB set */ ; int found; buf = findchar_fast(buf, buf_end, ranges1, sizeof(ranges1) - 1, &found); if (found) goto FOUND_CTL; #else /* find non-printable char within the next 8 bytes, this is the hottest code; manually inlined */ while (likely(buf_end - buf >= 8)) { #define DOIT() \ do { \ if (unlikely(!IS_PRINTABLE_ASCII(*buf))) \ goto NonPrintable; \ ++buf; \ } while (0) DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); #undef DOIT continue; NonPrintable: if ((likely((unsigned char)*buf < '\040') && likely(*buf != '\011')) || unlikely(*buf == '\177')) { goto FOUND_CTL; } ++buf; } #endif for (;; ++buf) { CHECK_EOF(); if (unlikely(!IS_PRINTABLE_ASCII(*buf))) { if ((likely((unsigned char)*buf < '\040') && likely(*buf != '\011')) || unlikely(*buf == '\177')) { goto FOUND_CTL; } } } FOUND_CTL: if (likely(*buf == '\015')) { ++buf; EXPECT_CHAR('\012'); *token_len = buf - 2 - token_start; } else if (*buf == '\012') { *token_len = buf - token_start; ++buf; } else { *ret = -1; return NULL; } *token = token_start; return buf; } static const char *is_complete(const char *buf, const char *buf_end, size_t last_len, int *ret) { int ret_cnt = 0; buf = last_len < 3 ? buf : buf + last_len - 3; while (1) { CHECK_EOF(); if (*buf == '\015') { ++buf; CHECK_EOF(); EXPECT_CHAR('\012'); ++ret_cnt; } else if (*buf == '\012') { ++buf; ++ret_cnt; } else { ++buf; ret_cnt = 0; } if (ret_cnt == 2) { return buf; } } *ret = -2; return NULL; } /* *_buf is always within [buf, buf_end) upon success */ static const char *parse_int(const char *buf, const char *buf_end, int *value, int *ret) { int v; CHECK_EOF(); if (!('0' <= *buf && *buf <= '9')) { *ret = -1; return NULL; } v = 0; for (;; ++buf) { CHECK_EOF(); if ('0' <= *buf && *buf <= '9') { v = v * 10 + *buf - '0'; } else { break; } } *value = v; return buf; } /* returned pointer is always within [buf, buf_end), or null */ static const char *parse_http_version(const char *buf, const char *buf_end, int *minor_version, int *ret) { EXPECT_CHAR('H'); EXPECT_CHAR('T'); EXPECT_CHAR('T'); EXPECT_CHAR('P'); EXPECT_CHAR('/'); EXPECT_CHAR('1'); EXPECT_CHAR('.'); return parse_int(buf, buf_end, minor_version, ret); } static const char *parse_headers(const char *buf, const char *buf_end, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret) { for (;; ++*num_headers) { CHECK_EOF(); if (*buf == '\015') { ++buf; EXPECT_CHAR('\012'); break; } else if (*buf == '\012') { ++buf; break; } if (*num_headers == max_headers) { *ret = -1; return NULL; } if (!(*num_headers != 0 && (*buf == ' ' || *buf == '\t'))) { /* parsing name, but do not discard SP before colon, see * http://www.mozilla.org/security/announce/2006/mfsa2006-33.html */ headers[*num_headers].name = buf; static const char ranges1[] __attribute__((aligned(16))) = "\x00 " /* control chars and up to SP */ "\"\"" /* 0x22 */ "()" /* 0x28,0x29 */ ",," /* 0x2c */ "//" /* 0x2f */ ":@" /* 0x3a-0x40 */ "[]" /* 0x5b-0x5d */ "{\377"; /* 0x7b-0xff */ int found; buf = findchar_fast(buf, buf_end, ranges1, sizeof(ranges1) - 1, &found); if (!found) { CHECK_EOF(); } while (1) { if (*buf == ':') { break; } else if (!token_char_map[(unsigned char)*buf]) { *ret = -1; return NULL; } ++buf; CHECK_EOF(); } if ((headers[*num_headers].name_len = buf - headers[*num_headers].name) == 0) { *ret = -1; return NULL; } ++buf; for (;; ++buf) { CHECK_EOF(); if (!(*buf == ' ' || *buf == '\t')) { break; } } } else { headers[*num_headers].name = NULL; headers[*num_headers].name_len = 0; } if ((buf = get_token_to_eol(buf, buf_end, &headers[*num_headers].value, &headers[*num_headers].value_len, ret)) == NULL) { return NULL; } } return buf; } static const char *parse_request(const char *buf, const char *buf_end, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret) { /* skip first empty line (some clients add CRLF after POST content) */ CHECK_EOF(); if (*buf == '\015') { ++buf; EXPECT_CHAR('\012'); } else if (*buf == '\012') { ++buf; } /* parse request line */ ADVANCE_TOKEN(*method, *method_len); ++buf; ADVANCE_TOKEN(*path, *path_len); ++buf; if ((buf = parse_http_version(buf, buf_end, minor_version, ret)) == NULL) { return NULL; } if (*buf == '\015') { ++buf; EXPECT_CHAR('\012'); } else if (*buf == '\012') { ++buf; } else { *ret = -1; return NULL; } return parse_headers(buf, buf_end, headers, num_headers, max_headers, ret); } #ifdef __SSE4_2__ #define PHR_PARSE_REQUEST phr_parse_request_sse42 #else #define PHR_PARSE_REQUEST phr_parse_request #endif int PHR_PARSE_REQUEST(const char *buf_start, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len) { const char *buf = buf_start, *buf_end = buf_start + len; size_t max_headers = *num_headers; int r; *method = NULL; *method_len = 0; *path = NULL; *path_len = 0; *minor_version = -1; *num_headers = 0; /* if last_len != 0, check if the request is complete (a fast countermeasure against slowloris */ if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) { return r; } if ((buf = parse_request(buf, buf_end, method, method_len, path, path_len, minor_version, headers, num_headers, max_headers, &r)) == NULL) { return r; } return (int)(buf - buf_start); } // squeaky_pl: we don't use this yet #if 0 static const char *parse_response(const char *buf, const char *buf_end, int *minor_version, int *status, const char **msg, size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret) { /* parse "HTTP/1.x" */ if ((buf = parse_http_version(buf, buf_end, minor_version, ret)) == NULL) { return NULL; } /* skip space */ if (*buf++ != ' ') { *ret = -1; return NULL; } /* parse status code */ if ((buf = parse_int(buf, buf_end, status, ret)) == NULL) { return NULL; } /* skip space */ if (*buf++ != ' ') { *ret = -1; return NULL; } /* get message */ if ((buf = get_token_to_eol(buf, buf_end, msg, msg_len, ret)) == NULL) { return NULL; } return parse_headers(buf, buf_end, headers, num_headers, max_headers, ret); } int phr_parse_response(const char *buf_start, size_t len, int *minor_version, int *status, const char **msg, size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t last_len) { const char *buf = buf_start, *buf_end = buf + len; size_t max_headers = *num_headers; int r; *minor_version = -1; *status = 0; *msg = NULL; *msg_len = 0; *num_headers = 0; /* if last_len != 0, check if the response is complete (a fast countermeasure against slowloris */ if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) { return r; } if ((buf = parse_response(buf, buf_end, minor_version, status, msg, msg_len, headers, num_headers, max_headers, &r)) == NULL) { return r; } return (int)(buf - buf_start); } int phr_parse_headers(const char *buf_start, size_t len, struct phr_header *headers, size_t *num_headers, size_t last_len) { const char *buf = buf_start, *buf_end = buf + len; size_t max_headers = *num_headers; int r; *num_headers = 0; /* if last_len != 0, check if the response is complete (a fast countermeasure against slowloris */ if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) { return r; } if ((buf = parse_headers(buf, buf_end, headers, num_headers, max_headers, &r)) == NULL) { return r; } return (int)(buf - buf_start); } #endif enum { CHUNKED_IN_CHUNK_SIZE, CHUNKED_IN_CHUNK_EXT, CHUNKED_IN_CHUNK_DATA, CHUNKED_IN_CHUNK_CRLF, CHUNKED_IN_TRAILERS_LINE_HEAD, CHUNKED_IN_TRAILERS_LINE_MIDDLE }; #ifndef __SSE4_2__ static int decode_hex(int ch) { if ('0' <= ch && ch <= '9') { return ch - '0'; } else if ('A' <= ch && ch <= 'F') { return ch - 'A' + 0xa; } else if ('a' <= ch && ch <= 'f') { return ch - 'a' + 0xa; } else { return -1; } } ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *_bufsz) { size_t dst = 0, src = 0, bufsz = *_bufsz; ssize_t ret = -2; /* incomplete */ while (1) { switch (decoder->_state) { case CHUNKED_IN_CHUNK_SIZE: for (;; ++src) { int v; if (src == bufsz) goto Exit; if ((v = decode_hex(buf[src])) == -1) { if (decoder->_hex_count == 0) { ret = -1; goto Exit; } break; } if (decoder->_hex_count == sizeof(size_t) * 2) { ret = -1; goto Exit; } decoder->bytes_left_in_chunk = decoder->bytes_left_in_chunk * 16 + v; ++decoder->_hex_count; } decoder->_hex_count = 0; decoder->_state = CHUNKED_IN_CHUNK_EXT; /* fallthru */ case CHUNKED_IN_CHUNK_EXT: /* RFC 7230 A.2 "Line folding in chunk extensions is disallowed" */ for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] == '\012') break; } ++src; if (decoder->bytes_left_in_chunk == 0) { if (decoder->consume_trailer) { decoder->_state = CHUNKED_IN_TRAILERS_LINE_HEAD; break; } else { goto Complete; } } decoder->_state = CHUNKED_IN_CHUNK_DATA; /* fallthru */ case CHUNKED_IN_CHUNK_DATA: { size_t avail = bufsz - src; if (avail < decoder->bytes_left_in_chunk) { if (dst != src) memmove(buf + dst, buf + src, avail); src += avail; dst += avail; decoder->bytes_left_in_chunk -= avail; goto Exit; } if (dst != src) memmove(buf + dst, buf + src, decoder->bytes_left_in_chunk); src += decoder->bytes_left_in_chunk; dst += decoder->bytes_left_in_chunk; decoder->bytes_left_in_chunk = 0; decoder->_state = CHUNKED_IN_CHUNK_CRLF; } /* fallthru */ case CHUNKED_IN_CHUNK_CRLF: for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] != '\015') break; } if (buf[src] != '\012') { ret = -1; goto Exit; } ++src; decoder->_state = CHUNKED_IN_CHUNK_SIZE; break; case CHUNKED_IN_TRAILERS_LINE_HEAD: for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] != '\015') break; } if (buf[src++] == '\012') goto Complete; decoder->_state = CHUNKED_IN_TRAILERS_LINE_MIDDLE; /* fallthru */ case CHUNKED_IN_TRAILERS_LINE_MIDDLE: for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] == '\012') break; } ++src; decoder->_state = CHUNKED_IN_TRAILERS_LINE_HEAD; break; default: assert(!"decoder is corrupt"); } } Complete: ret = bufsz - src; Exit: if (dst != src) memmove(buf + dst, buf + src, bufsz - src); *_bufsz = dst; return ret; } #endif // squeaky_pl: we dont use this #if 0 int phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder) { return decoder->_state == CHUNKED_IN_CHUNK_DATA; } #endif #undef CHECK_EOF #undef EXPECT_CHAR #undef ADVANCE_TOKEN ================================================ FILE: src/picohttpparser/picohttpparser.h ================================================ /* * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase, * Shigeo Mitsunari * * The software is licensed under either the MIT License (below) or the Perl * license. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to * deal in the Software without restriction, including without limitation the * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or * sell copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS * IN THE SOFTWARE. */ #ifndef picohttpparser_h #define picohttpparser_h #include #ifdef _MSC_VER #define ssize_t intptr_t #endif /* $Id: 67fd3ee74103ada60258d8a16e868f483abcca87 $ */ #ifdef __cplusplus extern "C" { #endif /* contains name and value of a header (name == NULL if is a continuing line * of a multiline header */ struct phr_header { const char *name; size_t name_len; const char *value; size_t value_len; }; /* returns number of bytes consumed if successful, -2 if request is partial, * -1 if failed */ int phr_parse_request(const char *buf, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len); int phr_parse_request_sse42(const char *buf, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len); /* ditto */ // squeaky_p: we don't use it yet #if 0 int phr_parse_response(const char *_buf, size_t len, int *minor_version, int *status, const char **msg, size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t last_len); /* ditto */ int phr_parse_headers(const char *buf, size_t len, struct phr_header *headers, size_t *num_headers, size_t last_len); #endif /* should be zero-filled before start */ struct phr_chunked_decoder { size_t bytes_left_in_chunk; /* number of bytes left in current chunk */ char consume_trailer; /* if trailing headers should be consumed */ char _hex_count; char _state; }; /* the function rewrites the buffer given as (buf, bufsz) removing the chunked- * encoding headers. When the function returns without an error, bufsz is * updated to the length of the decoded data available. Applications should * repeatedly call the function while it returns -2 (incomplete) every time * supplying newly arrived data. If the end of the chunked-encoded data is * found, the function returns a non-negative number indicating the number of * octets left undecoded at the tail of the supplied buffer. Returns -1 on * error. */ ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *bufsz); /* returns if the chunked decoder is in middle of chunked data */ int phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder); #ifdef __cplusplus } #endif #endif ================================================ FILE: tutorial/1_hello.md ================================================ # Getting Started Make sure you have both [pip](https://pip.pypa.io/en/stable/installing/) and at least version 3.5 of Python before starting. On Linux or MacOS X you can install directly using pip. If you are on Windows you can still develop and use Japronto with [Docker](https://docs.docker.com/engine/installation/#/on-macos-and-windows). Installing ---------- On Linux and OSX install Japronto with `python3 -m pip install japronto`. On Windows or if you simply prefer Docker pull Japronto image with `docker pull japronto/japronto`. Creating your Hello world app ----------------------------- Copy and paste following code into a file named `hello.py`: ```python # examples/1_hello/hello.py from japronto import Application # Views handle logic, take request as a parameter and # returns Response object back to the client def hello(request): return request.Response(text='Hello world!') # The Application instance is a fundamental concept. # It is a parent to all the resources and all the settings # can be tweaked here. app = Application() # The Router instance lets you register your handlers and execute # them depending on the url path and methods app.router.add_route('/', hello) # Finally start our server and handle requests until termination is # requested. Enabling debug lets you see request logs and stack traces. app.run(debug=True) ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). Run it ------ On Linux and OSX run the server with just: `python3 hello.py`. If using Docker run `docker run -p 8080:8080 -v $(pwd)/hello.py:/hello.py japronto/japronto --script /hello.py`. This will mount local `hello.py` into container as `/hello.py` which is later passed to Docker entry point. Now open the address `http://localhost:8080` in your web browser. You should see the message *Hello world!*. You now have a working Japronto server! **Next:** [Asynchronous handlers](2_async.md) ================================================ FILE: tutorial/2_async.md ================================================ # Asynchronous handlers With Japronto you can freely combine synchronous and asynchronous handlers and fully take advantage of both ecosystems. Choose wisely when to use asynchronous programming. Unless you are connecting to third party APIs, want to run input-output tasks in the background, expect large latency or do long-running blocking queries to your database you are probably better off programming synchronously. ```python # examples/2_async/async.py import asyncio from japronto import Application # This is a synchronous handler. def synchronous(request): return request.Response(text='I am synchronous!') # This is an asynchronous handler, it spends most of the time in the event loop. # It wakes up every second 1 to print and finally returns after 3 seconds. # This does let other handlers to be executed in the same processes while # from the point of view of the client it took 3 seconds to complete. async def asynchronous(request): for i in range(1, 4): await asyncio.sleep(1) print(i, 'seconds elapsed') return request.Response(text='3 seconds elapsed') app = Application() r = app.router r.add_route('/sync', synchronous) r.add_route('/async', asynchronous) app.run() ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). **Next:** [Router](3_router.md) ================================================ FILE: tutorial/3_router.md ================================================ # Router The router is a subsystem responsible for directing incoming requests to particular handlers based on some conditions, namely the URL path and HTTP method. It's available under `router` property of an `Application` instance and presents `add_router` method which takes `path` pattern, `handler` and optionally one or more `method`s. ```python # examples/3_router/router.py from japronto import Application app = Application() r = app.router # Requests with the path set exactly to `/` and whatever method # will be directed here. def slash(request): return request.Response(text='Hello {} /!'.format(request.method)) r.add_route('/', slash) # Requests with the path set exactly to '/love' and the method # set exactly to `GET` will be directed here. def get_love(request): return request.Response(text='Got some love') r.add_route('/love', get_love, 'GET') # Requests with the path set exactly to '/methods' and the method # set to `POST` or `DELETE` will be directed here. def methods(request): return request.Response(text=request.method) r.add_route('/methods', methods, methods=['POST', 'DELETE']) # Requests with the path starting with `/params/` segment and followed # by two additional segments will be directed here. # Values of the additional segments will be stored in side `request.match_dict` # dictionary with keys taken from {} placeholders. A request to `/params/1/2` # would leave `match_dict` set to `{'p1': 1, 'p2': '2'}`. def params(request): return request.Response(text=str(request.match_dict)) r.add_route('/params/{p1}/{p2}', params) app.run() ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). **Next:** [Request object](4_request.md) ================================================ FILE: tutorial/4_request.md ================================================ # Request Object Request represents an incoming HTTP request with a rich set of properties. They can be divided into three categories: Request line and headers, message body and miscellaneous. ```python # examples/4_request/request.py from json import JSONDecodeError from japronto import Application # Request line and headers. # This represents the part of a request that comes before message body. # Given a HTTP 1.1 `GET` request to `/basic?a=1` this would yield # `method` set to `GET`, `path` set to `/basic`, `version` set to `1.1` # `query_string` set to `a=1` and `query` set to `{'a': '1'}`. # Additionally if headers are sent they will be present in `request.headers` # dictionary. The keys are normalized to standard `Camel-Cased` convention. def basic(request): text = """Basic request properties: Method: {0.method} Path: {0.path} HTTP version: {0.version} Query string: {0.query_string} Query: {0.query}""".format(request) if request.headers: text += "\nHeaders:\n" for name, value in request.headers.items(): text += " {0}: {1}\n".format(name, value) return request.Response(text=text) # Message body # If there is a message body attached to a request (as in a case of `POST`) # method the following attributes can be used to examine it. # Given a `POST` request with body set to `b'J\xc3\xa1'`, `Content-Length` header set # to `3` and `Content-Type` header set to `text/plain; charset=utf-8` this # would yield `mime_type` set to `'text/plain'`, `encoding` set to `'utf-8'`, # `body` set to `b'J\xc3\xa1'` and `text` set to `'Já'`. # `form` and `files` attributes are dictionaries respectively used for HTML forms and # HTML file uploads. The `json` helper property will try to decode `body` as a # JSON document and give you resulting Python data type. def body(request): text = """Body related properties: Mime type: {0.mime_type} Encoding: {0.encoding} Body: {0.body} Text: {0.text} Form parameters: {0.form} Files: {0.files} """.format(request) try: json = request.json except JSONDecodeError: pass else: text += "\nJSON:\n" text += str(json) return request.Response(text=text) # Miscellaneous # `route` will point to an instance of `Route` object representing # route chosen by router to handle this request. `hostname` and `port` # represent parsed `Host` header if any. `remote_addr` is the address of # a client or reverse proxy. If `keep_alive` is true the client requested to # keep connection open after the response is delivered. `match_dict` contains # route placeholder values as documented in `2_router.md`. `cookies` contains # a dictionary of HTTP cookies if any. def misc(request): text = """Miscellaneous: Matched route: {0.route} Hostname: {0.hostname} Port: {0.port} Remote address: {0.remote_addr}, HTTP Keep alive: {0.keep_alive} Match parameters: {0.match_dict} """.strip().format(request) if request.cookies: text += "\nCookies:\n" for name, value in request.cookies.items(): text += " {0}: {1}\n".format(name, value) return request.Response(text=text) app = Application() app.router.add_route('/basic', basic) app.router.add_route('/body', body) app.router.add_route('/misc', misc) app.run() ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). **Next:** [Response object](5_response.md) ================================================ FILE: tutorial/5_response.md ================================================ # Response object Handlers return Response instances to fulfill requests. They can contain status code, headers and almost always a body. At the moment Response instances are immutable once created, this restriction will be lifted in a next version. ```python # examples/5_response/response.py import random from http.cookies import SimpleCookie from japronto.app import Application # Providing just text argument yields a `text/plain` response # encoded with `utf8` codec (charset set accordingly) def text(request): return request.Response(text='Hello world!') # You can override encoding by providing `encoding` attribute. def encoding(request): return request.Response(text='Já pronto!', encoding='iso-8859-1') # You can also set a custom MIME type. def mime(request): return request.Response( mime_type="image/svg+xml", text=""" """) # Or serve binary data. `Content-Type` set to `application/octet-stream` # automatically but you can always provide your own `mime_type`. def body(request): return request.Response(body=b'\xde\xad\xbe\xef') # There exist a shortcut `json` argument. This automatically encodes the # provided object as JSON and servers it with `Content-Type` set to # `application/json; charset=utf8` def json(request): return request.Response(json={'hello': 'world'}) # You can change the default 200 status `code` for another def code(request): return request.Response(code=random.choice([200, 201, 400, 404, 500])) # And of course you can provide custom `headers`. def headers(request): return request.Response( text='headers', headers={'X-Header': 'Value', 'Refresh': '5; url=https://xkcd.com/353/'}) # Or `cookies` by using Python standard library `http.cookies.SimpleCookie`. def cookies(request): cookies = SimpleCookie() cookies['hello'] = 'world' cookies['hello']['domain'] = 'localhost' cookies['hello']['path'] = '/' cookies['hello']['max-age'] = 3600 cookies['city'] = 'São Paulo' return request.Response(text='cookies', cookies=cookies) app = Application() router = app.router router.add_route('/text', text) router.add_route('/encoding', encoding) router.add_route('/mime', mime) router.add_route('/body', body) router.add_route('/json', json) router.add_route('/code', code) router.add_route('/headers', headers) router.add_route('/cookies', cookies) app.run() ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). **Next:** [Handling exceptions](6_exceptions.md) ================================================ FILE: tutorial/6_exceptions.md ================================================ # Handling exceptions There may be cases where you may want to respond with a custom response instead of a 500 Internal Server Error when an exception(s) is raised. Or you might want to override the default 404 handler. Enter exception handlers. ```python # examples/6_exceptions/exceptions.py from japronto import Application, RouteNotFoundException # Those are our custom exceptions we want to turn into 200 response. class KittyError(Exception): def __init__(self): self.greet = 'meow' class DoggieError(Exception): def __init__(self): self.greet = 'woof' # The two handlers below raise exceptions which will be turned # into 200 responses by the handlers registered later def cat(request): raise KittyError() def dog(request): raise DoggieError() # This handler raises ZeroDivisionError which doesnt have an error # handler registered so it will result in 500 Internal Server Error def unhandled(request): 1 / 0 app = Application() r = app.router r.add_route('/cat', cat) r.add_route('/dog', dog) r.add_route('/unhandled', unhandled) # These two are handlers for `Kitty` and `DoggyError`s. def handle_cat(request, exception): return request.Response(text='Just a kitty, ' + exception.greet) def handle_dog(request, exception): return request.Response(text='Just a doggie, ' + exception.greet) # You can also override default 404 handler if you want def handle_not_found(request, exception): return request.Response(code=404, text="Are you lost, pal?") # register all the error handlers so they are actually effective app.add_error_handler(KittyError, handle_cat) app.add_error_handler(DoggieError, handle_dog) app.add_error_handler(RouteNotFoundException, handle_not_found) app.run() ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). **Next:** [Extending request](7_extend.md) ================================================ FILE: tutorial/7_extend.md ================================================ # Extending Request object You can register custom properties and methods on Request object. This is typically done to accommodate objects that are bound to request lifetime such as database connections, sessions or caching. This lets you easily access that logic in one place. There also exist a mechanism to execute code on request completion by registering a callback in a view. ```python # examples/7_extend/extend.py from japronto import Application # This view accesses custom method host_startswith # and a custom property reversed_agent. Both are registered later. def extended_hello(request): if request.host_startswith('api.'): text = 'Hello ' + request.reversed_agent else: text = 'Hello stranger' return request.Response(text=text) # This view registers a callback, such callbacks are executed after handler # exit and the response is ready to be sent over the wire. def with_callback(request): def cb(r): print('Done!') request.add_done_callback(cb) return request.Response(text='cb') # This is a body for reversed_agent property def reversed_agent(request): return request.headers['User-Agent'][::-1] # This is a body for host_startswith method # Custom methods and properties always accept request # object. def host_startswith(request, prefix): return request.headers['Host'].startswith(prefix) app = Application() # Finally register out custom property and method # By default the names are taken from function names # unelss you provide `name` keyword parameter. app.extend_request(reversed_agent, property=True) app.extend_request(host_startswith) r = app.router r.add_route('/', extended_hello) r.add_route('/callback', with_callback) app.run() ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples). ================================================ FILE: tutorial/8_template.md ================================================ # Responding with HTML Serving HTML from japronto is as simple as adding a MIME type of `text/html` to the Response. Jinja2 templating can be leveraged as well, although in the meantime you will have to do the heavy lifting of rendering templates before sending in a response. Copy and paste following code into a file named `html.py`: ```python # examples/8_template/template.py from japronto import Application from jinja2 import Template # A view can read HTML from a file def index(request): with open('index.html') as html_file: return request.Response(text=html_file.read(), mime_type='text/html') # A view could also return a raw HTML string def example(request): return request.Response(text='

Some HTML!

', mime_type='text/html') # A view could also return a rendered jinja2 template def jinja(request): template = Template('

Hello {{ name }}!

') return request.Response(text=template.render(name='World'), mime_type='text/html') # Create the japronto application app = Application() # Add routes to the app app.router.add_route('/', index) app.router.add_route('/example', example) app.router.add_route('/jinja2', jinja) # Start the server app.run(debug=True) ``` The source code for all the examples can be found in [examples directory](https://github.com/squeaky-pl/japronto/tree/master/examples).