Showing preview only (370K chars total). Download the full file or copy to clipboard to get everything.
Repository: squeaky-pl/japronto
Branch: master
Commit: e73b76ea6ee2
Files: 148
Total size: 337.7 KB
Directory structure:
gitextract_jn3kd_d7/
├── .gitignore
├── .gitmodules
├── .travis.yml
├── CHANGELOG.md
├── LICENSE.txt
├── MANIFEST.in
├── README.md
├── benchmarks/
│ ├── aiohttp/
│ │ ├── micro.py
│ │ └── requirements.txt
│ ├── gevent/
│ │ ├── micro.py
│ │ └── requirements.txt
│ ├── golang/
│ │ ├── README.md
│ │ └── micro.go
│ ├── golang-fasthttp/
│ │ ├── README.md
│ │ └── micro.go
│ ├── japronto/
│ │ └── micro.py
│ ├── meinheld/
│ │ ├── micro.py
│ │ └── requirements.txt
│ ├── nodejs/
│ │ └── micro.js
│ ├── results.ods
│ ├── sanic/
│ │ ├── micro.py
│ │ └── requirements.txt
│ └── tornado/
│ ├── micro.py
│ └── requirements.txt
├── build.py
├── cases/
│ ├── __init__.py
│ ├── base.toml
│ └── websites.toml
├── conftest.py
├── do_wrk.py
├── examples/
│ ├── 1_hello/
│ │ └── hello.py
│ ├── 2_async/
│ │ └── async.py
│ ├── 3_router/
│ │ └── router.py
│ ├── 4_request/
│ │ └── request.py
│ ├── 5_response/
│ │ └── response.py
│ ├── 6_exceptions/
│ │ └── exceptions.py
│ ├── 7_extend/
│ │ └── extend.py
│ ├── 8_template/
│ │ ├── index.html
│ │ └── template.py
│ └── todo_api/
│ ├── .gitignore
│ └── todo_api.py
├── integration_tests/
│ ├── __init__.py
│ ├── common.py
│ ├── drain.py
│ ├── dump.py
│ ├── experiments.py
│ ├── generators.py
│ ├── longrun.py
│ ├── noleak.py
│ ├── reaper.py
│ ├── strategies.py
│ ├── test_drain.py
│ ├── test_noleak.py
│ ├── test_perror.py
│ ├── test_reaper.py
│ └── test_request.py
├── misc/
│ ├── __init__.py
│ ├── bootstrap.sh
│ ├── buggers.py
│ ├── cleanup_script.py
│ ├── client.py
│ ├── collector.py
│ ├── cpu.py
│ ├── do_perf.py
│ ├── docker/
│ │ └── Dockerfile
│ ├── parts.py
│ ├── perf.md
│ ├── pipeline.lua
│ ├── report.py
│ ├── requirements-test.txt
│ ├── requirements.txt
│ ├── rpm-requirements.txt
│ ├── runpytest.py
│ ├── simple.py
│ ├── suppr.txt
│ └── travis/
│ ├── before_install.sh
│ ├── install.sh
│ └── script.sh
├── setup.py
├── src/
│ ├── japronto/
│ │ ├── __init__.py
│ │ ├── __main__.py
│ │ ├── app/
│ │ │ └── __init__.py
│ │ ├── capsule.c
│ │ ├── capsule.h
│ │ ├── common.h
│ │ ├── cpu_features.c
│ │ ├── cpu_features.h
│ │ ├── parser/
│ │ │ ├── .gitignore
│ │ │ ├── __init__.py
│ │ │ ├── build_libpicohttpparser.py
│ │ │ ├── cffiparser.py
│ │ │ ├── cparser.c
│ │ │ ├── cparser.gcda
│ │ │ ├── cparser.h
│ │ │ ├── cparser_ext.py
│ │ │ └── test_parser.py
│ │ ├── pipeline/
│ │ │ ├── __init__.py
│ │ │ ├── cpipeline.c
│ │ │ ├── cpipeline.h
│ │ │ ├── cpipeline_ext.py
│ │ │ └── test_pipeline.py
│ │ ├── protocol/
│ │ │ ├── __init__.py
│ │ │ ├── cprotocol.c
│ │ │ ├── cprotocol.h
│ │ │ ├── cprotocol_ext.py
│ │ │ ├── creaper.c
│ │ │ ├── creaper_ext.py
│ │ │ ├── generator.c
│ │ │ ├── generator.h
│ │ │ ├── generator_ext.py
│ │ │ ├── handler.py
│ │ │ ├── null.py
│ │ │ └── tracing.py
│ │ ├── reloader.py
│ │ ├── request/
│ │ │ ├── __init__.py
│ │ │ ├── crequest.c
│ │ │ ├── crequest.h
│ │ │ └── crequest_ext.py
│ │ ├── response/
│ │ │ ├── __init__.py
│ │ │ ├── cresponse.c
│ │ │ ├── cresponse.h
│ │ │ ├── cresponse_ext.py
│ │ │ ├── py.py
│ │ │ └── reasons.h
│ │ ├── router/
│ │ │ ├── __init__.py
│ │ │ ├── analyzer.py
│ │ │ ├── cmatcher.c
│ │ │ ├── cmatcher.h
│ │ │ ├── cmatcher_ext.py
│ │ │ ├── match_dict.c
│ │ │ ├── match_dict.h
│ │ │ ├── matcher.py
│ │ │ ├── route.py
│ │ │ ├── test_analyzer.py
│ │ │ ├── test_matcher.py
│ │ │ └── test_route.py
│ │ └── runner.py
│ └── picohttpparser/
│ ├── build
│ ├── picohttpparser.c
│ └── picohttpparser.h
└── tutorial/
├── 1_hello.md
├── 2_async.md
├── 3_router.md
├── 4_request.md
├── 5_response.md
├── 6_exceptions.md
├── 7_extend.md
└── 8_template.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
__pycache__
.hypothesis
.test
coverage.info
*.egg-info
*.build.toml
*.so
*.o
================================================
FILE: .gitmodules
================================================
[submodule "misc/terryfy"]
path = misc/terryfy
url = https://github.com/MacPython/terryfy/
================================================
FILE: .travis.yml
================================================
language: python
sudo: required
services:
- docker
env:
global:
- secure: "HO3zCuv0FtNFTQ7kkBpIqKYAZW8sYZPfc1ROk6+ChoxufXcu529CTKNAr3KklfZCbMHiZKc3W83N7x9B/L2rtSuBQvJPPgVtIlaVKRyWWnY4nqrpwKEoOLUd3RjpAMfCB09sXQ2aTfQV8Ds5Zk+cF7R2toI6s2s4vymXvCLvfugrtO4sd91frSDv/fzjEEKOIeey8KXtPAPPFv6v64OScksPt1oCsVOPDtkZ7q0KSIzS9JN6BvM9oafPt9MaFPH84ITtdPMTjgQOQ+YFe8YBwgjkV/cX9rNs+vzSP6Bm2NQ9/xxd8XTDj6ukuEYD5HQi26IS6ddRyVGsn6/WRZx6/kQboJKh5r5pa4OAHPWnPirRWPQW46HI2iknAGTFWh8ARX2R208mK1vbQ66J+9zQDnkjMXGgX67gWyWQWfxwVHFsPyQEiSHHh2vEBkhs1+tqtvp7Ktnc+uCxXn0v/Humu3OvFSBxSXfyjvE9uUOGyB2zDwqmxLQQ5ftKAcGfLOaSqauJ1vQy1CWc5bROCn8aoch5iRf/tcX85TUDirAgAp3OUdt3VwcRNY+Fci7IU50gn2rghJWFzB5Zz9p1ShnZxIaD5GEPE45ju4UIpwYbs8iSqh+/RS8sR2Ffzx4M+6QJjj1BJABdtVPS9Jn5OkbuSdBW0K+MuLtmtbg4WLXv6+E="
jobs:
include:
- python: 3.5
env: VERSION=3.5.3
- python: 3.6
env: VERSION=3.6.0
- python: 3.7
env: VERSION=3.7.1
- python: 3.8
env: VERSION=3.8.0
before_install: source misc/travis/before_install.sh
install: source misc/travis/install.sh
script: misc/travis/script.sh
================================================
FILE: CHANGELOG.md
================================================
0.1.1 - Feb 9 2017
------------------
- Native support for OSX
- Support for older hardware without SSE4.2
- Better crash info with faulthandler
================================================
FILE: LICENSE.txt
================================================
Copyright (c) 2017 Paweł Piotr Przeradowski
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: MANIFEST.in
================================================
include README.md LICENSE.txt src/picohttpparser/picohttpparser.c src/picohttpparser/picohttpparser.h
================================================
FILE: README.md
================================================
# Japronto!
[](https://webchat.freenode.net/?channels=japronto)
[](https://gitter.im/japronto/Lobby) [](https://travis-ci.org/squeaky-pl/japronto) [](https://pypi.python.org/pypi/japronto) [](https://pypi.python.org/pypi/japronto/)
__There is no new project development happening at the moment, but it's not abandoned either. Pull requests and new maintainers are welcome__.
__If you are a novice Python programmer, you don't like plumbing yourself or you don't have basic understanding of C, this project is not probably what you are looking for__.
Japronto (from Portuguese "já pronto" /ˈʒa pɾõtu/ meaning "already done") is a __screaming-fast__, __scalable__, __asynchronous__
Python 3.5+ HTTP __toolkit__ integrated with __pipelining HTTP server__ based on [uvloop](https://github.com/MagicStack/uvloop) and [picohttpparser](https://github.com/h2o/picohttpparser). It's targeted at speed enthusiasts, people who like
plumbing and early adopters.
You can read more in the [release announcement on medium](https://medium.com/@squeaky_pl/million-requests-per-second-with-python-95c137af319)
Performance
-----------
Here's a chart to help you imagine what kind of things you can do with Japronto:
As user @heppu points out Go’s stdlib HTTP server can be 12% faster than the graph shows when written more carefully. Also there is the awesome fasthttp server for Go that apparently is only 18% slower than Japronto in this particular benchmark. Awesome! For details see https://github.com/squeaky-pl/japronto/pull/12 and https://github.com/squeaky-pl/japronto/pull/14.
These results of a simple "Hello world" application were obtained on AWS c4.2xlarge instance. To be fair all the contestants (including Go) were running single worker process. Servers were load tested using [wrk](https://github.com/wg/wrk) with 1 thread, 100 connections and 24 simultaneous (pipelined) requests per connection (cumulative parallelism of 2400 requests).
The source code for the benchmark can be found in [benchmarks](benchmarks) directory.
The server is written in hand tweaked C trying to take advantage of modern CPUs. It relies on picohttpparser for header &
chunked-encoding parsing while uvloop provides asynchronous I/O. It also tries to save up on
system calls by combining writes together when possible.
Early preview
-------------
This is an early preview with alpha quality implementation. APIs are provisional meaning that they will change between versions and more testing is needed. Don't use it for anything serious for now and definitely don't use it in production. Please try it though and report back feedback. If you are shopping for your next project's framework I would recommend [Sanic](https://github.com/channelcat/sanic).
At the moment the work is focused on CPython but I have PyPy on my radar, though I am not gonna look into it until PyPy reaches 3.5 compatibility somewhere later this year and most known JIT regressions are removed.
Hello world
-----------
Here is how a simple web application looks like in Japronto:
```python
from japronto import Application
def hello(request):
return request.Response(text='Hello world!')
app = Application()
app.router.add_route('/', hello)
app.run(debug=True)
```
Tutorial
--------
1. [Getting started](tutorial/1_hello.md)
2. [Asynchronous handlers](tutorial/2_async.md)
3. [Router](tutorial/3_router.md)
4. [Request object](tutorial/4_request.md)
5. [Response object](tutorial/5_response.md)
6. [Handling exceptions](tutorial/6_exceptions.md)
7. [Extending request](tutorial/7_extend.md)
Features
--------
- HTTP 1.x implementation with support for chunked uploads
- Full support for HTTP pipelining
- Keep-alive connections with configurable reaper
- Support for synchronous and asynchronous views
- Master-multiworker model based on forking
- Support for code reloading on changes
- Simple routing
License
-------
This software is distributed under [MIT License](https://en.wikipedia.org/wiki/MIT_License). This is a very permissive license that lets you use this software for any
commercial and non-commercial work. Full text of the license is
included in [LICENSE.txt](LICENSE.txt) file.
The source distribution of this software includes a copy of picohttpparser which is distributed under MIT license as well.
================================================
FILE: benchmarks/aiohttp/micro.py
================================================
from aiohttp import web
import asyncio
import uvloop
loop = uvloop.new_event_loop()
asyncio.set_event_loop(loop)
async def hello(request):
return web.Response(text='Hello world!')
app = web.Application(loop=loop)
app.router.add_route('GET', '/', hello)
web.run_app(app, port=8080, access_log=None)
================================================
FILE: benchmarks/aiohttp/requirements.txt
================================================
aiohttp==1.2.0
================================================
FILE: benchmarks/gevent/micro.py
================================================
from gevent.pywsgi import WSGIServer
def hello(environ, start_response):
if(environ['PATH_INFO'] == '/' and environ['REQUEST_METHOD'] == 'GET'):
status = '200 OK'
text = "Hello world!"
else:
status = '404 Not Found'
text = "Not Found"
body = text.encode('utf-8')
response_headers = [
('Content-type', 'text/plain; charset=utf-8'),
('Content-Length', str(len(body)))]
start_response(status, response_headers)
return [body]
WSGIServer(('0.0.0.0', 8080), hello, log=None).serve_forever()
================================================
FILE: benchmarks/gevent/requirements.txt
================================================
gevent==1.2.1
================================================
FILE: benchmarks/golang/README.md
================================================
```
go build .
GOMAXPROCS=1 ./bin
```
================================================
FILE: benchmarks/golang/micro.go
================================================
package main
import "net/http"
var (
helloResp = []byte("Hello world!")
notFoundResp = []byte("Not Found")
)
func hello(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
w.WriteHeader(http.StatusNotFound)
w.Write(notFoundResp)
return
}
w.Write(helloResp)
}
func main() {
http.HandleFunc("/", hello)
http.ListenAndServe("0.0.0.0:8080", nil)
}
================================================
FILE: benchmarks/golang-fasthttp/README.md
================================================
```
go build .
GOMAXPROCS=1 ./bin
```
================================================
FILE: benchmarks/golang-fasthttp/micro.go
================================================
package main
import "github.com/valyala/fasthttp"
func hello(ctx *fasthttp.RequestCtx) {
if string(ctx.Path()) != "/" {
ctx.SetStatusCode(404)
ctx.WriteString("Not Found")
return
}
ctx.WriteString("Hello world!")
}
func main() {
fasthttp.ListenAndServe("0.0.0.0:8080", hello)
}
================================================
FILE: benchmarks/japronto/micro.py
================================================
from japronto import Application
def hello(request):
return request.Response(text='Hello world!')
app = Application()
r = app.router
r.add_route('/', hello, method='GET')
app.run()
================================================
FILE: benchmarks/meinheld/micro.py
================================================
from meinheld import server
def hello(environ, start_response):
if(environ['PATH_INFO'] == '/' and environ['REQUEST_METHOD'] == 'GET'):
status = '200 OK'
text = "Hello world!"
else:
status = '404 Not Found'
text = "Not Found"
body = text.encode('utf-8')
response_headers = [
('Content-type', 'text/plain; charset=utf-8'),
('Content-Length', str(len(body)))]
start_response(status, response_headers)
return [body]
server.listen(('0.0.0.0', 8080))
server.set_access_logger(None)
server.set_keepalive(1)
server.run(hello)
================================================
FILE: benchmarks/meinheld/requirements.txt
================================================
meinheld==0.6.1
================================================
FILE: benchmarks/nodejs/micro.js
================================================
const http = require('http');
var srv = http.createServer( (req, res) => {
res.sendDate = false;
if(req.url == '/') {
data = 'Hello world!'
status = 200
} else {
data = 'Not Found'
status = 404
}
res.writeHead(status, {
'Content-Type': 'text/plain; encoding=utf-8',
'Content-Length': data.length});
res.end(data);
});
srv.listen(8080, '0.0.0.0');
================================================
FILE: benchmarks/sanic/micro.py
================================================
from sanic import Sanic
from sanic.response import text
app = Sanic(__name__)
@app.route("/")
async def hello(request):
return text("Hello world!")
app.run(host="0.0.0.0", port=8080)
================================================
FILE: benchmarks/sanic/requirements.txt
================================================
sanic==0.2.0
================================================
FILE: benchmarks/tornado/micro.py
================================================
from tornado import web
from tornado.httputil import HTTPHeaders, responses
from tornado.platform.asyncio import AsyncIOMainLoop
import asyncio
import uvloop
loop = uvloop.new_event_loop()
asyncio.set_event_loop(loop)
AsyncIOMainLoop().install()
class MainHandler(web.RequestHandler):
def get(self):
self.write('Hello world!')
# skip calculating ETag, ~8% faster
def set_etag_header(self):
pass
def check_etag_header(self):
return False
# torando sends Server and Date headers by default, ~4% faster
def clear(self):
self._headers = HTTPHeaders(
{'Content-Type': 'text/plain; charset=utf-8'})
self._write_buffer = []
self._status_code = 200
self._reason = responses[200]
app = web.Application([('/', MainHandler)])
app.listen(8080)
loop.run_forever()
================================================
FILE: benchmarks/tornado/requirements.txt
================================================
tornado==4.4.2
================================================
FILE: build.py
================================================
import argparse
import distutils
from distutils.command.build_ext import build_ext, CompileError
from distutils.core import Distribution
from glob import glob
import os.path
import shutil
import sysconfig
import os
import sys
import subprocess
try:
import pytoml
except ImportError:
pytoml = None
import runpy
SRC_LOCATION = 'src'
sys.path.insert(0, SRC_LOCATION)
class BuildSystem:
def __init__(self, args, relative_source=False):
self.args = args
self.dest = self.args.dest
self.relative_source = relative_source
def get_extension_by_path(self, path):
path = SRC_LOCATION + '/' + path
result = runpy.run_path(path, {'system': self})
extension = result['get_extension']()
base_path = os.path.dirname(path)
def fix_path(p):
if os.path.isabs(p):
return p
return os.path.abspath(os.path.join(base_path, p))
attrs = ['sources', 'include_dirs', 'library_dirs',
'runtime_library_dirs']
for attr in attrs:
val = getattr(extension, attr)
if not val:
continue
if attr == 'sources' and self.relative_source:
val = [
(os.path.normpath(os.path.join(base_path, v))
if not v.startswith('src')
else v) for v in val]
elif attr == 'runtime_library_dirs' and self.relative_source:
pass
else:
val = [fix_path(v) for v in val]
if attr == 'runtime_library_dirs':
setattr(extension, attr, None)
attr = 'extra_link_args'
val = ['-Wl,-rpath,' + v for v in val]
val = (getattr(extension, attr) or []) + val
setattr(extension, attr, val)
return extension
def discover_extensions(self):
self.extensions = []
ext_files = glob(SRC_LOCATION + '/**/*_ext.py', recursive=True)
ext_files = [os.path.relpath(p, SRC_LOCATION) for p in ext_files]
self.extensions = [self.get_extension_by_path(f) for f in ext_files]
return self.extensions
def dest_folder(self, mod_name):
return self.dest + '/' + '/'.join(mod_name.split('.')[:-1])
def build_toml(self, mod_name):
return self.dest + '/' + '/'.join(mod_name.split('.')) + '.build.toml'
def get_so(self, ext):
return self.dest + '/' + '/'.join(ext.name.split('.')) + '.' + \
sysconfig.get_config_var('SOABI') + '.so'
def flags_changed(self, ext):
toml = self.build_toml(ext.name)
if not os.path.exists(toml):
return True
with open(toml) as f:
flags = pytoml.load(f)
ext_flags = {
"extra_compile_args": ext.extra_compile_args,
"extra_link_args": ext.extra_link_args,
"define_macros": dict(ext.define_macros),
"sources": ext.sources}
return flags != ext_flags
def should_rebuild(self, ext):
so = self.get_so(ext)
if not os.path.exists(so):
return True
so_mtime = os.stat(so).st_mtime
includes = get_includes(ext)
input_mtimes = [os.stat(s).st_mtime for s in ext.sources + includes]
if max(input_mtimes) > so_mtime:
return True
if self.flags_changed(ext):
return True
return False
def prune(dest):
paths = glob(os.path.join(dest, '.build/**/*.o'), recursive=True)
paths.extend(glob(os.path.join(dest, '.build/**/*.so'), recursive=True))
for path in paths:
os.remove(path)
def profile_clean():
paths = glob('build/**/*.gcda', recursive=True)
for path in paths:
os.remove(path)
def get_includes(ext):
includes = []
include_base = SRC_LOCATION + '/' + '/'.join(ext.name.split('.')[:-1])
include_paths = [os.path.join(include_base, i) for i in ext.include_dirs]
for source in ext.sources:
with open(source) as f:
for line in f:
line = line.strip()
if not line.startswith('#include'):
continue
header = line.split()[1][1:-1]
for path in include_paths:
if not os.path.exists(os.path.join(path, header)):
continue
includes.append(os.path.join(path, header))
break
return includes
def symlink_python_files(dest):
if dest == SRC_LOCATION:
return
for parent, dirs, files in os.walk(SRC_LOCATION):
if os.path.basename(parent) == '__pycache__':
continue
def _is_python_file(f):
return f.endswith('.py') and not f.endswith('_ext.py') \
and not f.startswith('test_')
files = [f for f in files if _is_python_file(f)]
if not files:
continue
dest_parent = os.path.join(dest, *parent.split(os.sep)[1:])
os.makedirs(dest_parent, exist_ok=True)
for file in files:
dst = os.path.join(dest_parent, file)
src = os.path.relpath(os.path.join(parent, file), dest_parent)
if os.path.exists(dst):
os.unlink(dst)
os.symlink(src, dst)
kits = {
'platform': [
'japronto.request.crequest', 'japronto.protocol.cprotocol',
'japronto.protocol.creaper', 'japronto.router.cmatcher',
'japronto.response.cresponse']
}
def get_parser():
argparser = argparse.ArgumentParser('build')
argparser.add_argument(
'-d', dest='debug', const=True, action='store_const', default=False)
argparser.add_argument(
'--sanitize', dest='sanitize', const=True, action='store_const',
default=False)
argparser.add_argument(
'--profile-generate', dest='profile_generate', const=True,
action='store_const', default=False)
argparser.add_argument('--dest', dest='dest', default='src')
argparser.add_argument(
'--profile-use', dest='profile_use', const=True,
action='store_const', default=False)
argparser.add_argument(
'-flto', dest='flto', const=True,
action='store_const', default=False)
argparser.add_argument(
'--profile-clean', dest='profile_clean', const=True,
action='store_const', default=False)
argparser.add_argument(
'--disable-reaper', dest='enable_reaper', const=False,
action='store_const', default=True)
argparser.add_argument(
'--disable-response-cache', dest='enable_response_cache', const=False,
action='store_const', default=True)
argparser.add_argument(
'--coverage', dest='coverage', const=True,
action='store_const', default=False)
argparser.add_argument('-O1', dest='optimization', const='1',
action='store_const')
argparser.add_argument('-O2', dest='optimization', const='2',
action='store_const')
argparser.add_argument('-O3', dest='optimization', const='3',
action='store_const')
argparser.add_argument('-Os', dest='optimization', const='s',
action='store_const')
argparser.add_argument('-native', dest='native', const=True,
action='store_const', default=False)
argparser.add_argument('--path', dest='path')
argparser.add_argument('--extra-compile', dest='extra_compile', default='')
argparser.add_argument('--kit', dest='kit')
return argparser
def get_platform():
argparser = get_parser()
args = argparser.parse_args([])
system = BuildSystem(args, relative_source=True)
ext_modules = system.discover_extensions()
ext_modules = [e for e in ext_modules if e.name in kits['platform']]
print({e.name: e.sources for e in ext_modules})
return ext_modules
class custom_build_ext(build_ext):
def build_extensions(self):
if self.compiler.compiler_type == 'unix':
for ext in self.extensions:
if not ext.extra_compile_args:
ext.extra_compiler_args = []
extra_compile_args = ['-std=c99', '-UNDEBUG', '-D_GNU_SOURCE']
if self.compiler.compiler_so[0].startswith('gcc') and sys.platform != 'darwin':
extra_compile_args.append('-frecord-gcc-switches')
ext.extra_compile_args.extend(extra_compile_args)
compile_c(
self.compiler,
'src/picohttpparser/picohttpparser.c',
'src/picohttpparser/ssepicohttpparser.o',
options={'unix': ['-msse4.2']})
compile_c(
self.compiler,
'src/picohttpparser/picohttpparser.c',
'src/picohttpparser/picohttpparser.o')
build_ext.build_extensions(self)
def compile_c(compiler, cfile, ofile, *, options=None):
if not options:
options = {}
options = options.get(compiler.compiler_type, [])
cmd = [*compiler.compiler_so, *options, '-c', '-o', ofile, cfile]
print("building '{}'".format(ofile))
print(' '.join(cmd))
subprocess.check_call(cmd)
def main():
argparser = get_parser()
args = argparser.parse_args(sys.argv[1:])
if args.profile_clean:
profile_clean()
return
distutils.log.set_verbosity(1)
system = BuildSystem(args)
if args.path:
ext_modules = [system.get_extension_by_path(args.path)]
else:
ext_modules = system.discover_extensions()
if args.kit:
ext_modules = [e for e in ext_modules if e.name in kits[args.kit]]
def add_args(arg_name, values, append=True):
for ext_module in ext_modules:
arg_value = getattr(ext_module, arg_name) or []
if append:
arg_value.extend(values)
else:
newvalues = list(values)
newvalues.extend(arg_value)
arg_value = newvalues
setattr(ext_module, arg_name, arg_value)
def append_compile_args(*values):
add_args('extra_compile_args', values)
def append_link_args(*values):
add_args('extra_link_args', values)
def prepend_libraries(*values):
add_args('libraries', values, append=False)
if args.native:
append_compile_args('-march=native')
if args.optimization:
append_compile_args('-O' + args.optimization)
if args.debug:
append_compile_args('-g3', '-O0', '-Wp,-U_FORTIFY_SOURCE')
if args.sanitize:
append_compile_args('-g3', '-fsanitize=address',
'-fsanitize=undefined', '-fno-common',
'-fno-omit-frame-pointer')
prepend_libraries('asan', 'ubsan')
if args.profile_generate:
append_compile_args('--profile-generate')
append_link_args('-lgcov')
if args.profile_use:
for ext_module in ext_modules:
if ext_module.name in ('parser.cparser', 'pipeline.cpipeline'):
continue
ext_module.extra_compile_args.append('--profile-use')
if args.flto:
append_compile_args('-flto')
append_link_args('-flto')
if args.coverage:
append_compile_args('--coverage')
append_link_args('-lgcov')
if args.extra_compile:
append_compile_args(args.extra_compile)
ext_modules = [e for e in ext_modules if system.should_rebuild(e)]
if not ext_modules:
return
dist = Distribution(dict(ext_modules=ext_modules))
prune(args.dest)
cmd = custom_build_ext(dist)
cmd.build_lib = os.path.join(args.dest, '.build/lib')
cmd.build_temp = os.path.join(args.dest, '.build/temp')
cmd.finalize_options()
try:
cmd.run()
except CompileError:
sys.exit(1)
symlink_python_files(args.dest)
for ext_module in ext_modules:
os.makedirs(system.dest_folder(ext_module.name), exist_ok=True)
shutil.copy(
cmd.get_ext_fullpath(ext_module.name),
system.dest_folder(ext_module.name))
for ext_module in ext_modules:
with open(system.build_toml(ext_module.name), 'w') as f:
build_info = {
'extra_compile_args': ext_module.extra_compile_args,
'extra_link_args': ext_module.extra_link_args,
'define_macros': dict(ext_module.define_macros),
'sources': ext_module.sources
}
pytoml.dump(build_info, f)
if __name__ == '__main__':
main()
================================================
FILE: cases/__init__.py
================================================
from collections import namedtuple
import glob
import os.path
import pytoml
import pytest
testcase_fields = 'data,method,path,version,headers,body,error,disconnect'
HttpTestCase = namedtuple('HTTPTestCase', testcase_fields)
def parse_casesel(suite, casesel):
for casespec in casesel.split('+'):
*transforms, case = casespec.split(':')
if case.endswith('!'):
transforms.append('!')
case = case[:-1]
case = suite[case]
for transform in reversed(transforms):
func, *args = transform.split()
case = transorm_dict[func](case, *args)
yield case
def parametrize_cases(suite, *args):
suite = suites[suite]
cases_list = [
list(parse_casesel(suite, sel)) for sel in args]
return pytest.mark.parametrize('cases', cases_list, ids=args)
def load_casefile(path):
result = {}
with open(path) as casefile:
cases = pytoml.load(casefile)
for case_name, case_data in cases.items():
case_data['data'] = case_data['data'].encode('utf-8')
case_data['body'] = case_data['body'].encode('utf-8') \
if 'body' in case_data else None
case_data['disconnect'] = False
case = HttpTestCase._make(
case_data.get(f) for f in testcase_fields.split(','))
result[case_name] = case
return result
def load_cases():
cases = {}
for filename in glob.glob('cases/*.toml'):
suite_name, _ = os.path.splitext(os.path.basename(filename))
cases[suite_name] = load_casefile(filename)
return cases
def keep_alive(case):
headers = case.headers.copy()
headers['Connection'] = 'keep-alive'
# if case.body is not None \
# and headers.get('Transfer-Encoding', 'identity') == 'identity':
# headers['Content-Length'] = str(len(case.body))
return update_case(case, headers)
def close(case):
headers = case.headers.copy()
headers['Connection'] = 'close'
return update_case(case, headers)
def should_keep_alive(case):
return case.headers.get(
'Connection',
'close' if case.version == '1.0' else 'keep-alive') == 'keep-alive'
def set_error(case, error):
return update_case(case, error=error)
def disconnect(case):
return update_case(case, disconnect=True)
def update_case(case, headers=False, error=False, disconnect=None):
data = False
if headers:
data = bytearray()
status = case.method + ' ' + case.path + ' HTTP/' + case.version + '\r\n'
data += status.encode('ascii')
for name, value in headers.items():
data += name.encode('ascii') + b': ' + value.encode('latin1') + b'\r\n'
data += b'\r\n'
if case.body:
data += case.body
headers = headers or case.headers
data = data or case.data
error = error or case.error
disconnect = disconnect if disconnect is not None else case.disconnect
return case._replace(
headers=headers, error=error, disconnect=disconnect,
data=bytes(data))
transorm_dict = {
'keep': keep_alive,
'close': close,
'e': set_error,
'!': disconnect
}
suites = load_cases()
globals().update(suites)
================================================
FILE: cases/base.toml
================================================
[10msg]
data = """\
POST \
/wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg \
HTTP/1.0\r\n\
HOST: www.kittyhell.com\r\n\
User-Agent: Mozilla/5.0 \
(Macintosh; U; Intel Mac OS X 10.6; ja-JP-mac; rv:1.9.2.3) \
Gecko/20100401 Firefox/3.6.3 Pathtraq/0.9\r\n\
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n\
Accept-Language: ja,en-us;q=0.7,en;q=0.3\r\n\
Accept-Encoding: gzip,deflate\r\n\
Accept-Charset: Shift_JIS,utf-8;q=0.7,*;q=0.7\r\n\
Keep-Alive: 115\r\n\
Cookie: wp_ozh_wsa_visits=2; \
wp_ozh_wsa_visit_lasttime=xxxxxxxxxx; \
__utma=xxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.x; \
__utmz=xxxxxxxxx.xxxxxxxxxx.x.x.utmccn=(referral)|utmcsr=reader.livedoor.com|utmcct=/reader/|utmcmd=referral\r\n\
Content-Length: 11\r\n\
\r\n\
Hello there\
"""
method = "POST"
path = "/wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg"
version = "1.0"
body = "Hello there"
[10msg.headers]
Host = "www.kittyhell.com"
User-Agent = """Mozilla/5.0 \
(Macintosh; U; Intel Mac OS X 10.6; ja-JP-mac; rv:1.9.2.3) \
Gecko/20100401 Firefox/3.6.3 Pathtraq/0.9"""
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Accept-Language = "ja,en-us;q=0.7,en;q=0.3"
Accept-Encoding = "gzip,deflate"
Accept-Charset = "Shift_JIS,utf-8;q=0.7,*;q=0.7"
Keep-Alive = "115"
Cookie = """wp_ozh_wsa_visits=2; \
wp_ozh_wsa_visit_lasttime=xxxxxxxxxx; \
__utma=xxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.x; \
__utmz=xxxxxxxxx.xxxxxxxxxx.x.x.utmccn=(referral)|utmcsr=reader.livedoor.com|utmcct=/reader/|utmcmd=referral"""
Content-Length = "11"
[10get]
data = "GET / HTTP/1.0\r\nHost: www.example.com\r\n\r\n"
method = "GET"
path = "/"
version = "1.0"
headers = { Host = "www.example.com" }
[10malformed_headers1]
data = "GET / HTTP 1.0"
error = "malformed_headers"
[10malformed_headers2]
data = "GET / HTTP/2"
error = "malformed_headers"
[10incomplete_headers]
data = "GET / HTTP/1.0\r\nH"
error = "incomplete_headers"
[11get]
data = "GET /index.html HTTP/1.1\r\n\r\n"
method = "GET"
path = "/index.html"
version = "1.1"
headers = {}
# body = null
[11getmsg]
data = """\
GET /body HTTP/1.1\r\n\
Content-Length: 12\r\n\
\r\n\
Hello World!\
"""
method = "GET"
path = "/body"
version = "1.1"
headers = { Content-Length = "12" }
body = "Hello World!"
[11msg]
data = "POST /login HTTP/1.1\r\nContent-Length: 5\r\n\r\nHello"
method = "POST"
path = "/login"
version = "1.1"
headers = { Content-Length = "5" }
body = "Hello"
[11msgzero]
data = "POST /zero HTTP/1.1\r\nContent-Length: 0\r\n\r\n"
method = "POST"
path = "/zero"
version = "1.1"
headers = { Content-Length = "0" }
body = ""
[11clincomplete_headers]
data = """\
POST / HTTP/1.1\r\n\
Content-Length: 3\r\n\
I"""
error = "incomplete_headers"
[11clincomplete_body]
data = "POST / HTTP/1.1\r\nContent-Length: 5\r\n\r\nI"
method = "POST"
path = "/"
version = "1.1"
headers = { Content-Length = "5" }
error = "incomplete_body"
[11clinvalid1]
data = "POST / HTTP/1.1\r\nContent-Length: asd\r\n\r\n"
method = "POST"
path = "/"
version = "1.1"
headers = { Content-Length = "asd" }
error = "invalid_headers"
[11clinvalid2]
data = "POST / HTTP/1.1\r\nContent-Length: +5\r\n\r\n"
method = "POST"
path = "/"
version = "1.1"
headers = { Content-Length = "+5" }
error = "invalid_headers"
[11clinvalid3]
data = "POST / HTTP/1.1\r\nContent-Length: -5\r\n\r\n"
method = "POST"
path = "/"
version = "1.1"
headers = { Content-Length = "+5" }
error = "invalid_headers"
[11clinvalid4]
data = "POST / HTTP/1.1\r\nContent-Length: 4f\r\n\r\n"
method = "POST"
path = "/"
version = "1.1"
headers = { Content-Length = "4f" }
error = "invalid_headers"
[11clinvalid5]
data = "POST / HTTP/1.1\r\nContent-Length: \r\n\r\n"
method = "POST"
path = "/"
version = "1.1"
headers = { Content-Length = "" }
error = "invalid_headers"
[11chunked1]
data = """\
POST /chunked HTTP/1.1\r\n\
Transfer-Encoding: chunked\r\n\
\r\n\
4\r\n\
Wiki\r\n\
5\r\n\
pedia\r\n\
E\r\n in\r\n\
\r\n\
chunks.\r\n\
0\r\n\
\r\n\
"""
method = "POST"
path = "/chunked"
version = "1.1"
headers = { Transfer-Encoding = "chunked" }
body = "Wikipedia in\r\n\r\nchunks."
[11chunkedzero]
data = """
PUT /zero HTTP/1.1\r\n\
Transfer-Encoding: chunked\r\n\
\r\n\
0\r\n\
\r\n\
"""
method = "PUT"
path = "/zero"
version = "1.1"
headers = { Transfer-Encoding = "chunked" }
body = ""
[11chunked2]
data = """\
POST /chunked HTTP/1.1\r\n\
Transfer-Encoding: chunked\r\n\
\r\n\
1;token=123;x=3\r\n\
r\r\n\
0\r\n\
\r\n\
"""
method = "POST"
path = "/chunked"
version = "1.1"
headers = { Transfer-Encoding = "chunked" }
body = "r"
[11chunked3]
data = """\
POST / HTTP/1.1\r\n\
Transfer-Encoding: chunked\r\n\
\r\n\
000002\r\n\
ab\r\n\
0;q=1\r\n\
This: is trailer header\r\n\
\r\n\
"""
method = "POST"
path = "/"
version = "1.1"
headers = { Transfer-Encoding = "chunked" }
body = "ab"
[11chunkedincomplete_body]
data = "POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n10\r\nasd"
method = "POST"
path = "/"
version = "1.1"
headers = { Transfer-Encoding = "chunked" }
error = "incomplete_body"
[11chunkedmalformed_body]
data = "POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n1x\r\nhello"
method = "POST"
path = "/"
version = "1.1"
headers = { Transfer-Encoding = "chunked" }
error = "malformed_body"
================================================
FILE: cases/websites.toml
================================================
[github]
data = """\
GET / HTTP/1.1\r\n\
Host: github.com\r\n\
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0\r\n\
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n\
Accept-Language: en-US,en;q=0.5\r\n\
Accept-Encoding: gzip, deflate, br\r\n\
Cookie: _octo=GHx; logged_in=yes; _ga=GAx; user_session=x; dotcom_user=x; __Host-user_session_same_site=x; _gh_sess=x; tz=America%2FSao_Paulo; _gat=x\r\n\
Connection: keep-alive\r\n\
Upgrade-Insecure-Requests: 1\r\n\
Pragma: no-cache\r\n\
Cache-Control: no-cache\r\n\
\r\n\
"""
[google]
data = """\
GET / HTTP/1.1\r\n\
Host: google.com\r\n\
Connection: keep-alive\r\n\
Upgrade-Insecure-Requests: 1\r\n\
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36\r\n\
X-Client-Data: Qwerty\r\n\
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n\
Accept-Encoding: gzip, deflate, sdch\r\n\
Accept-Language: en-US,en;q=0.8,es;q=0.6\r\n\
Cookie: NID=x; SID=x; HSID=x; APISID=x\r\n\
\r\n\
"""
[amazon]
data = """\
GET / HTTP/1.1\r\n\
Host: www.amazon.com\r\n\
Connection: keep-alive\r\n\
Upgrade-Insecure-Requests: 1\r\n\
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.50 Safari/537.36 OPR/41.0.2353.23 (Edition beta)\r\n\
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n\
Accept-Encoding: gzip, deflate, lzma, sdch, br\r\n\
Accept-Language: en-US,en;q=0.8\r\n\
\r\n\
"""
[xkcd]
data = """\
GET /comics/mushrooms.png HTTP/1.1\r\n\
Host: imgs.xkcd.com\r\n\
Connection: keep-alive\r\n\
Upgrade-Insecure-Requests: 1\r\n\
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.50 Safari/537.36 OPR/41.0.2353.23 (Edition beta)\r\n\
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n\
Accept-Encoding: gzip, deflate, lzma, sdch\r\n\
Accept-Language: en-US,en;q=0.8\r\n\
\r\n\
"""
[4chan]
data = """\
GET /image/favicon.ico HTTP/1.1\r\n\
Host: s.4cdn.org\r\n\
Connection: keep-alive\r\n\
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36\r\n\
Accept: */*\r\n\
Referer: http://www.4chan.org/\r\n\
Accept-Encoding: gzip, deflate, sdch\r\n\
Accept-Language: en-US,en;q=0.8,es;q=0.6\r\n\
\r\n\
"""
================================================
FILE: conftest.py
================================================
import subprocess
import sys
import os
import shutil
builds = []
coverages = set()
def add_build(mark):
global builds
args, kwargs = list(mark.args), mark.kwargs.copy()
kwargs.pop('coverage', None)
cfg = args, kwargs
if cfg not in builds:
builds.append(cfg)
def execute_builds():
common_options = ['--coverage', '-d', '--sanitize']
for args, kwargs in builds:
build_options = args[:]
build_options.extend(['--dest', kwargs.get('dest', '.test')])
if 'kit' not in kwargs:
build_options.extend(['--kit', 'platform'])
build_options.extend(common_options)
print('Executing build', *build_options)
subprocess.check_call([sys.executable, 'build.py', *build_options])
def add_coverage(mark):
dest = mark.kwargs.get('dest', '.test')
coverages.add(dest)
def setup_coverage():
if coverages:
print('Setting up C coverage for', *coverages)
for dest in coverages:
subprocess.check_call([
'lcov', '--base-directory', '.', '--directory',
dest + '/.build/temp', '--zerocounters', '-q'])
def make_coverage():
for dest in coverages:
try:
os.unlink(dest + '/coverage.info')
except FileNotFoundError:
pass
subprocess.check_call([
'lcov', '--base-directory', '.', '--directory',
dest + '/.build/temp', '-c', '-o', dest + '/coverage.info', '-q'])
subprocess.check_call([
'lcov', '--remove', dest + '/coverage.info',
'/usr*', '-o', 'coverage.info', '-q'])
try:
shutil.rmtree(dest + '/coverage_report')
except FileNotFoundError:
pass
subprocess.check_call([
'genhtml', '-o', dest + '/coverage_report',
dest + '/coverage.info', '-q'
])
print('C coverage report saved in',
dest + '/coverage_report/index.html')
def pytest_itemcollected(item):
needs_build = item.get_closest_marker('needs_build')
if needs_build:
add_build(needs_build)
if needs_build and needs_build.kwargs.get('coverage'):
add_coverage(needs_build)
def pytest_collection_modifyitems(config, items):
execute_builds()
setup_coverage()
def pytest_unconfigure():
make_coverage()
================================================
FILE: do_wrk.py
================================================
import argparse
import sys
import asyncio as aio
import os
from asyncio.subprocess import PIPE, STDOUT
import statistics
import uvloop
import psutil
from misc import cpu
from misc import buggers
def run_wrk(loop, endpoint=None):
endpoint = endpoint or 'http://localhost:8080'
wrk_fut = aio.create_subprocess_exec(
'./wrk', '-t', '1', '-c', '100', '-d', '2', '-s', 'misc/pipeline.lua',
endpoint, stdout=PIPE, stderr=STDOUT)
wrk = loop.run_until_complete(wrk_fut)
lines = []
while 1:
line = loop.run_until_complete(wrk.stdout.readline())
if line:
line = line.decode('utf-8')
lines.append(line)
if line.startswith('Requests/sec:'):
rps = float(line.split()[-1])
else:
break
retcode = loop.run_until_complete(wrk.wait())
if retcode != 0:
print('\r\n'.join(lines))
return rps
def cpu_usage(p):
return p.cpu_percent() + sum(c.cpu_percent() for c in p.children())
def connections(process):
return len(
set(c.fd for c in process.connections()) |
set(c.fd for p in process.children() for c in p.connections()))
def memory(p):
return p.memory_percent('uss') \
+ sum(c.memory_percent('uss') for c in p.children())
if __name__ == '__main__':
buggers.silence()
loop = uvloop.new_event_loop()
argparser = argparse.ArgumentParser('do_wrk')
argparser.add_argument('-s', dest='server', default='')
argparser.add_argument('-e', dest='endpoint')
argparser.add_argument('--pid', dest='pid', type=int)
argparser.add_argument(
'--no-cpu', dest='cpu_change', default=True,
action='store_const', const=False)
args = argparser.parse_args(sys.argv[1:])
if args.cpu_change:
cpu.change('userspace', cpu.min_freq())
cpu.dump()
aio.set_event_loop(loop)
if not args.endpoint:
os.putenv('PYTHONPATH', 'src')
server_fut = aio.create_subprocess_exec(
'python', 'benchmarks/japronto/micro.py', *args.server.split())
server = loop.run_until_complete(server_fut)
os.unsetenv('PYTHONPATH')
if not args.endpoint:
process = psutil.Process(server.pid)
elif args.pid:
process = psutil.Process(args.pid)
else:
process = None
cpu_p = 100
while cpu_p > 5:
cpu_p = psutil.cpu_percent(interval=1)
print('CPU usage in 1 sec:', cpu_p)
results = []
cpu_usages = []
process_cpu_usages = []
mem_usages = []
conn_cnt = []
if process:
cpu_usage(process)
for _ in range(10):
results.append(run_wrk(loop, args.endpoint))
cpu_usages.append(psutil.cpu_percent())
if process:
process_cpu_usages.append(cpu_usage(process))
conn_cnt.append(connections(process))
mem_usages.append(round(memory(process), 2))
print('.', end='')
sys.stdout.flush()
if not args.endpoint:
server.terminate()
loop.run_until_complete(server.wait())
if args.cpu_change:
cpu.change('ondemand')
print()
print('RPS', results)
print('Mem', mem_usages)
print('Conn', conn_cnt)
print('Server', process_cpu_usages)
print('System', cpu_usages)
median = statistics.median_grouped(results)
stdev = round(statistics.stdev(results), 2)
p = round((stdev / median) * 100, 2)
print('median:', median, 'stdev:', stdev, '%', p)
================================================
FILE: examples/1_hello/hello.py
================================================
from japronto import Application
# Views handle logic, take request as a parameter and
# return the Response object back to the client
def hello(request):
return request.Response(text='Hello world!')
# The Application instance is a fundamental concept.
# It is a parent to all the resources and all the settings
# can be tweaked there.
app = Application()
# The Router instance lets you register your handlers and execute
# them depending on the url path and methods.
app.router.add_route('/', hello)
# Finally, start our server and handle requests until termination is
# requested. Enabling debug lets you see request logs and stack traces.
app.run(debug=True)
================================================
FILE: examples/2_async/async.py
================================================
import asyncio
from japronto import Application
# This is a synchronous handler.
def synchronous(request):
return request.Response(text='I am synchronous!')
# This is an asynchronous handler. It spends most of the time in the event loop.
# It wakes up every second 1 to print and finally returns after 3 seconds.
# This lets other handlers execute in the same processes while
# from the point of view of the client it took 3 seconds to complete.
async def asynchronous(request):
for i in range(1, 4):
await asyncio.sleep(1)
print(i, 'seconds elapsed')
return request.Response(text='3 seconds elapsed')
app = Application()
r = app.router
r.add_route('/sync', synchronous)
r.add_route('/async', asynchronous)
app.run()
================================================
FILE: examples/3_router/router.py
================================================
from japronto import Application
app = Application()
r = app.router
# Requests with the path set exactly to `/` and whatever method
# will be directed here.
def slash(request):
return request.Response(text='Hello {} /!'.format(request.method))
r.add_route('/', slash)
# Requests with the path set exactly to '/love' and the method
# set exactly to `GET` will be directed here.
def get_love(request):
return request.Response(text='Got some love')
r.add_route('/love', get_love, 'GET')
# Requests with the path set exactly to '/methods' and the method
# set to `POST` or `DELETE` will be directed here.
def methods(request):
return request.Response(text=request.method)
r.add_route('/methods', methods, methods=['POST', 'DELETE'])
# Requests with the path starting with `/params/` segment and followed
# by two additional segments will be directed here.
# Values of the additional segments will be stored inside `request.match_dict`
# dictionary with keys taken from {} placeholders. A request to `/params/1/2`
# would leave `match_dict` set to `{'p1': 1, 'p2': '2'}`.
def params(request):
return request.Response(text=str(request.match_dict))
r.add_route('/params/{p1}/{p2}', params)
app.run()
================================================
FILE: examples/4_request/request.py
================================================
from json import JSONDecodeError
from japronto import Application
# Request line and headers.
# This represents the part of a request that comes before message body.
# Given an HTTP 1.1 `GET` request to `/basic?a=1` this would yield
# `method` set to `GET`, `path` set to `/basic`, `version` set to `1.1`
# `query_string` set to `a=1` and `query` set to `{'a': '1'}`.
# Additionally if headers are sent they will be present in `request.headers`
# dictionary. The keys are normalized to standard `Camel-Cased` convention.
def basic(request):
text = """Basic request properties:
Method: {0.method}
Path: {0.path}
HTTP version: {0.version}
Query string: {0.query_string}
Query: {0.query}""".format(request)
if request.headers:
text += "\nHeaders:\n"
for name, value in request.headers.items():
text += " {0}: {1}\n".format(name, value)
return request.Response(text=text)
# Message body
# If there is a message body attached to a request (as in a case of `POST`)
# the following attributes can be used to examine it.
# Given a `POST` request with body set to `b'J\xc3\xa1'`, `Content-Length` header set
# to `3` and `Content-Type` header set to `text/plain; charset=utf-8` this
# would yield `mime_type` set to `'text/plain'`, `encoding` set to `'utf-8'`,
# `body` set to `b'J\xc3\xa1'` and `text` set to `'Já'`.
# `form` and `files` attributes are dictionaries respectively used for HTML forms and
# HTML file uploads. The `json` helper property will try to decode `body` as a
# JSON document and give you resulting Python data type.
def body(request):
text = """Body related properties:
Mime type: {0.mime_type}
Encoding: {0.encoding}
Body: {0.body}
Text: {0.text}
Form parameters: {0.form}
Files: {0.files}
""".format(request)
try:
json = request.json
except JSONDecodeError:
pass
else:
text += "\nJSON:\n"
text += str(json)
return request.Response(text=text)
# Miscellaneous
# `route` will point to an instance of `Route` object representing
# route chosen by router to handle this request. `hostname` and `port`
# represent parsed `Host` header if any. `remote_addr` is the address of
# a client or reverse proxy. If `keep_alive` is true the client requested to
# keep the connection open after the response is delivered. `match_dict` contains
# route placeholder values as documented in `2_router.md`. `cookies` contains
# a dictionary of HTTP cookies if any.
def misc(request):
text = """Miscellaneous:
Matched route: {0.route}
Hostname: {0.hostname}
Port: {0.port}
Remote address: {0.remote_addr},
HTTP Keep alive: {0.keep_alive}
Match parameters: {0.match_dict}
""".strip().format(request)
if request.cookies:
text += "\nCookies:\n"
for name, value in request.cookies.items():
text += " {0}: {1}\n".format(name, value)
return request.Response(text=text)
app = Application()
app.router.add_route('/basic', basic)
app.router.add_route('/body', body)
app.router.add_route('/misc', misc)
app.run()
================================================
FILE: examples/5_response/response.py
================================================
import random
from http.cookies import SimpleCookie
from japronto.app import Application
# Providing just a text argument yields a `text/plain` response
# encoded with `utf8` codec (charset set accordingly)
def text(request):
return request.Response(text='Hello world!')
# You can override encoding by providing the `encoding` attribute.
def encoding(request):
return request.Response(text='Já pronto!', encoding='iso-8859-1')
# You can also set a custom MIME type.
def mime(request):
return request.Response(
mime_type="image/svg+xml",
text="""
<svg xmlns="http://www.w3.org/2000/svg" version="1.1">
<line x1="10" y1="10" x2="80" y2="80" stroke="blue" />
</svg>
""")
# Or serve binary data. `Content-Type` is set to `application/octet-stream`
# automatically but you can always provide your own `mime_type`.
def body(request):
return request.Response(body=b'\xde\xad\xbe\xef')
# There exist a shortcut `json` argument. This automatically encodes the
# provided object as JSON and servers it with `Content-Type` set to
# `application/json; charset=utf8`
def json(request):
return request.Response(json={'hello': 'world'})
# You can change the default 200 status `code` for another
def code(request):
return request.Response(code=random.choice([200, 201, 400, 404, 500]))
# And of course you can provide custom `headers`.
def headers(request):
return request.Response(
text='headers',
headers={'X-Header': 'Value',
'Refresh': '5; url=https://xkcd.com/353/'})
# Or `cookies` by using Python standard library `http.cookies.SimpleCookie`.
def cookies(request):
cookies = SimpleCookie()
cookies['hello'] = 'world'
cookies['hello']['domain'] = 'localhost'
cookies['hello']['path'] = '/'
cookies['hello']['max-age'] = 3600
cookies['city'] = 'São Paulo'
return request.Response(text='cookies', cookies=cookies)
app = Application()
router = app.router
router.add_route('/text', text)
router.add_route('/encoding', encoding)
router.add_route('/mime', mime)
router.add_route('/body', body)
router.add_route('/json', json)
router.add_route('/code', code)
router.add_route('/headers', headers)
router.add_route('/cookies', cookies)
app.run()
================================================
FILE: examples/6_exceptions/exceptions.py
================================================
from japronto import Application, RouteNotFoundException
# These are our custom exceptions we want to turn into 200 response.
class KittyError(Exception):
def __init__(self):
self.greet = 'meow'
class DoggieError(Exception):
def __init__(self):
self.greet = 'woof'
# The two handlers below raise exceptions which will be turned
# into 200 responses by the handlers registered later
def cat(request):
raise KittyError()
def dog(request):
raise DoggieError()
# This handler raises ZeroDivisionError which doesn't have an error
# handler registered so it will result in 500 Internal Server Error
def unhandled(request):
1 / 0
app = Application()
r = app.router
r.add_route('/cat', cat)
r.add_route('/dog', dog)
r.add_route('/unhandled', unhandled)
# These two are handlers for `Kitty` and `DoggyError`s.
def handle_cat(request, exception):
return request.Response(text='Just a kitty, ' + exception.greet)
def handle_dog(request, exception):
return request.Response(text='Just a doggie, ' + exception.greet)
# You can also override default 404 handler if you want
def handle_not_found(request, exception):
return request.Response(code=404, text="Are you lost, pal?")
# register all the error handlers so they are actually effective
app.add_error_handler(KittyError, handle_cat)
app.add_error_handler(DoggieError, handle_dog)
app.add_error_handler(RouteNotFoundException, handle_not_found)
app.run()
================================================
FILE: examples/7_extend/extend.py
================================================
from japronto import Application
# This view accesses custom method host_startswith
# and a custom property reversed_agent. Both are registered later.
def extended_hello(request):
if request.host_startswith('api.'):
text = 'Hello ' + request.reversed_agent
else:
text = 'Hello stranger'
return request.Response(text=text)
# This view registers a callback, such callbacks are executed after handler
# exits and the response is ready to be sent over the wire.
def with_callback(request):
def cb(r):
print('Done!')
request.add_done_callback(cb)
return request.Response(text='cb')
# This is a body for reversed_agent property
def reversed_agent(request):
return request.headers['User-Agent'][::-1]
# This is a body for host_startswith method
# Custom methods and properties always accept request
# object.
def host_startswith(request, prefix):
return request.headers['Host'].startswith(prefix)
app = Application()
# Finally register the custom property and method
# By default the names are taken from function names
# unelss you provide `name` keyword parameter.
app.extend_request(reversed_agent, property=True)
app.extend_request(host_startswith)
r = app.router
r.add_route('/', extended_hello)
r.add_route('/callback', with_callback)
app.run()
================================================
FILE: examples/8_template/index.html
================================================
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>japronto</title>
<meta name="description" content="japronto">
</head>
<body>
<h1>Hello World!</h1>
<p>Behold, the power of japronto!</p>
</body>
</html>
================================================
FILE: examples/8_template/template.py
================================================
# examples/8_template/template.py
from japronto import Application
from jinja2 import Template
# A view can read HTML from a file
def index(request):
with open('index.html') as html_file:
return request.Response(text=html_file.read(), mime_type='text/html')
# A view could also return a raw HTML string
def example(request):
return request.Response(text='<h1>Some HTML!</h1>', mime_type='text/html')
template = Template('<h1>Hello {{ name }}!</h1>')
# A view could also return a rendered jinja2 template
def jinja(request):
return request.Response(text=template.render(name='World'),
mime_type='text/html')
# Create the japronto application
app = Application()
# Add routes to the app
app.router.add_route('/', index)
app.router.add_route('/example', example)
app.router.add_route('/jinja2', jinja)
# Start the server
app.run(debug=True)
================================================
FILE: examples/todo_api/.gitignore
================================================
todo.sqlite
================================================
FILE: examples/todo_api/todo_api.py
================================================
import os.path
import sqlite3
from functools import partial
from japronto import Application
def add_todo(request):
cur = request.cursor
todo = request.json["todo"]
cur.execute("""INSERT INTO todos (todo) VALUES (?)""", (todo,))
last_id = cur.lastrowid
cur.connection.commit()
return request.Response(json={"id": last_id, "todo": todo})
def list_todos(request):
cur = request.cursor
cur.execute("""SELECT id, todo FROM todos""")
todos = [{"id": id, "todo": todo} for id, todo in cur]
return request.Response(json={"results": todos})
def show_todo(request):
cur = request.cursor
id = int(request.match_dict['id'])
cur.execute("""SELECT id, todo FROM todos WHERE id = ?""", (id,))
todo = cur.fetchone()
if not todo:
return request.Response(code=404, json={"error": "not found"})
todo = {"id": todo[0], "todo": todo[1]}
return request.Response(json=todo)
def delete_todo(request):
cur = request.cursor
id = int(request.match_dict['id'])
cur.execute("""DELETE FROM todos WHERE id = ?""", (id,))
if not cur.rowcount:
return request.Response(code=404, json={"error": "not found"})
cur.connection.commit()
return request.Response(json={})
DB_FILE = os.path.abspath(
os.path.join(os.path.dirname(__file__), 'todo.sqlite'))
db_connect = partial(sqlite3.connect, DB_FILE)
def maybe_create_schema():
db = db_connect()
db.execute("""
CREATE TABLE IF NOT EXISTS todos
(id INTEGER PRIMARY KEY, todo TEXT)""")
db.close()
maybe_create_schema()
app = Application()
def cursor(request):
def done_cb(request):
request.extra['conn'].close()
if 'conn' not in request.extra:
request.extra['conn'] = db_connect()
request.add_done_callback(done_cb)
return request.extra['conn'].cursor()
app.extend_request(cursor, property=True)
router = app.router
router.add_route('/todos', list_todos, method='GET')
router.add_route('/todos/{id}', show_todo, method='GET')
router.add_route('/todos/{id}', delete_todo, method='DELETE')
router.add_route('/todos', add_todo, method='POST')
app.run()
================================================
FILE: integration_tests/__init__.py
================================================
================================================
FILE: integration_tests/common.py
================================================
import os
import subprocess
import ctypes.util
import sys
import time
import psutil
def start_server(script, *, stdout=None, path=None, sanitize=True, wait=True,
return_process=False, buffer=False):
if not isinstance(script, list):
script = [script]
if path:
os.putenv('PYTHONPATH', path)
if sanitize:
os.putenv('LD_PRELOAD', ctypes.util.find_library('asan'))
os.putenv('LSAN_OPTIONS', 'suppressions=misc/suppr.txt')
if not buffer:
os.putenv('PYTHONUNBUFFERED', '1')
server = subprocess.Popen([sys.executable, *script], stdout=stdout)
if not buffer:
os.unsetenv('PYTHONUNBUFFERED')
if sanitize:
os.unsetenv('LSAN_OPTIONS')
os.unsetenv('LD_PRELOAD')
if path:
os.unsetenv('PYTHONPATH')
process = psutil.Process(server.pid)
if wait:
# wait until the server socket is open
while 1:
assert server.poll() is None
conn_num = len(process.connections())
for child in process.children():
conn_num += len(child.connections())
if conn_num:
break
time.sleep(.001)
assert server.poll() is None
if return_process:
return server, process
else:
return server
================================================
FILE: integration_tests/drain.py
================================================
import asyncio
from japronto.app import Application
def slash(request):
return request.Response()
async def sleep(request):
await asyncio.sleep(int(request.match_dict['sleep']))
return request.Response()
app = Application()
r = app.router
r.add_route('/', slash)
r.add_route('/sleep/{sleep}', sleep)
if __name__ == '__main__':
app.run()
================================================
FILE: integration_tests/dump.py
================================================
import base64
import asyncio
from japronto.app import Application
class ForcedException(Exception):
pass
def dump(request, exception=None):
if not exception and 'Force-Raise' in request.headers:
raise ForcedException(request.headers['Force-Raise'])
body = request.body
if body is not None:
body = base64.b64encode(body).decode('ascii')
result = {
"method": request.method,
"path": request.path,
"query_string": request.query_string,
"headers": request.headers,
"match_dict": request.match_dict,
"body": body,
"route": request.route and request.route.pattern
}
if exception:
result['exception'] = {
"type": type(exception).__name__,
"args": ", ".join(str(a) for a in exception.args)
}
return request.Response(code=500 if exception else 200, json=result)
async def adump(request):
sleep = float(request.query.get('sleep', 0))
await asyncio.sleep(sleep)
return dump(request)
app = Application()
r = app.router
r.add_route('/dump/{p1}/{p2}', dump)
r.add_route('/dump1/{p1}/{p2}', dump)
r.add_route('/dump2/{p1}/{p2}', dump)
r.add_route('/async/dump/{p1}/{p2}', adump)
r.add_route('/async/dump1/{p1}/{p2}', adump)
r.add_route('/async/dump2/{p1}/{p2}', adump)
app.add_error_handler(None, dump)
if __name__ == '__main__':
app.run()
================================================
FILE: integration_tests/experiments.py
================================================
import pytest
@pytest.fixture(autouse=True)
def my_fix(request):
print('auto')
pytest.set_trace()
@pytest.fixture(params=[1, 2])
def size_k(request):
return request.param
# @pytest.fixture(autouse=True, scope='module')
# def a():
# print('a')
# @pytest.fixture(scope='function', params=[1,2])
# def fix():
# print('> fix')
# yield 3
# print('< fix')
# def test(my_fix):
# print(test)
def test1(size_k):
print(test1)
================================================
FILE: integration_tests/generators.py
================================================
from integration_tests import strategies as st
from hypothesis.strategies import SearchStrategy
def generate_body(body, size_k):
if size_k and body:
if isinstance(body, list):
length = sum(len(b) for b in body)
else:
length = len(body)
body = body * ((size_k * 1024) // length + 1)
return body
def makeval(v, default_st, default=None):
if isinstance(v, SearchStrategy):
return v.example()
if v is True:
return default_st.example()
if v is not None:
return v
return default
def print_request(request):
body = request['body']
if body:
if isinstance(body, list):
body = '{} chunks'.format(len(body))
else:
if len(body) > 32:
body = body[:32] + b'...'
print(repr(request['method']), repr(request['path']),
repr(request['query_string']), body)
def generate_request(*, method=None, path=None, query_string=None,
headers=None, body=None, size_k=None):
request = {}
request['method'] = makeval(method, st.method, 'GET')
request['path'] = makeval(path, st.path, '/')
request['query_string'] = makeval(query_string, st.query_string)
request['headers'] = makeval(headers, st.headers)
request['body'] = generate_body(makeval(body, st.body), size_k)
return request
def generate_combinations(reverse=False):
props = ['method', 'path', 'query_string', 'headers', 'body']
sizes = [None, 8, 32, 64]
if reverse:
props = reversed(props)
sizes = reversed(sizes)
for prop in props:
if prop == 'body':
for size_k in sizes:
yield {'body': True, 'size_k': size_k}
else:
yield {prop: True}
def send_requests(conn, number, **kwargs):
for _ in range(number):
request = generate_request(**kwargs)
print_request(request)
conn.request(**request)
conn.getresponse()
================================================
FILE: integration_tests/longrun.py
================================================
import subprocess
import sys
import signal
import atexit
import os
import time
sys.path.insert(0, '.')
import integration_tests.common # noqa
import integration_tests.generators # noqa
from misc import client # noqa
def setup():
subprocess.check_call([
sys.executable, 'build.py', '--dest', '.test/longrun',
'--kit', 'platform', '--disable-response-cache'])
os.putenv('MALLOC_TRIM_THRESHOLD_', '0')
server = integration_tests.common.start_server(
'integration_tests/dump.py', path='.test/longrun', sanitize=False)
os.unsetenv('MALLOC_TRIM_THRESHOLD_')
os.makedirs('.collector', exist_ok=True)
os.putenv('COLLECTOR_FILE', '.collector/{}.json'.format(server.pid))
collector = subprocess.Popen([
sys.executable, 'misc/collector.py', str(server.pid)])
os.unsetenv('COLLECTOR_FILE')
def cleanup(*args):
try:
server.terminate()
assert server.wait() == 0
finally:
atexit.unregister(cleanup)
atexit.register(cleanup)
signal.signal(signal.SIGINT, cleanup)
def run():
time.sleep(2)
for reverse in [True, False]:
for combination in integration_tests.generators.generate_combinations(
reverse=reverse):
conn = client.Connection('localhost:8080')
time.sleep(2)
integration_tests.generators.send_requests(
conn, 200, **combination)
time.sleep(2)
conn.close()
time.sleep(2)
def main():
setup()
run()
if __name__ == '__main__':
main()
================================================
FILE: integration_tests/noleak.py
================================================
import sys
from japronto.app import Application
prop = sys.argv[1]
if prop == 'method':
def noleak(request):
return request.Response(text=request.method)
elif prop == 'path':
def noleak(request):
return request.Response(text=request.path)
elif prop == 'match_dict':
def noleak(request):
return request.Response(json=request.match_dict)
elif prop == 'query_string':
def noleak(request):
return request.Response(text=request.query_string)
elif prop == 'headers':
def noleak(request):
return request.Response(json=request.headers)
elif prop == 'body':
def noleak(request):
return request.Response(body=request.body)
elif prop == 'keep_alive':
def noleak(request):
return request.Response(text=str(request.keep_alive))
elif prop == 'route':
def noleak(request):
return request.Response(text=str(request.route))
app = Application()
r = app.router
r.add_route('/noleak/{p1}/{p2}', noleak)
if __name__ == '__main__':
app.run()
================================================
FILE: integration_tests/reaper.py
================================================
import sys
from japronto.app import Application
reaper_settings = {
'check_interval': int(sys.argv[1]),
'idle_timeout': int(sys.argv[2])
}
app = Application(reaper_settings=reaper_settings)
if __name__ == '__main__':
app.run()
================================================
FILE: integration_tests/strategies.py
================================================
import string
import re
from hypothesis import strategies as st
sampled_from = st.sampled_from
fixed_dictionaries = st.fixed_dictionaries
lists = st.lists
builds = st.builds
integers = st.integers
_method_alphabet = ''.join(chr(x) for x in range(33, 256) if x != 127)
method = st.text(_method_alphabet, min_size=1)
_path_alphabet = st.characters(
blacklist_characters='?', blacklist_categories=['Cs'])
path = st.text(_path_alphabet).map(lambda x: '/' + x)
_param_alphabet = st.characters(
blacklist_characters='/?', blacklist_categories=['Cs'])
param = st.text(_param_alphabet, min_size=1)
query_string = st.one_of(st.text(), st.none())
_name_alphabet = string.digits + string.ascii_letters + '!#$%&\'*+-.^_`|~'
_names = st.text(_name_alphabet, min_size=1).map(lambda x: 'X-' + x)
_value_alphabet = ''.join(chr(x) for x in range(ord(' '), 256) if x != 127)
_is_illegal_value = re.compile(r'\n(?![ \t])|\r(?![ \t\n])').search
_values = st.text(_value_alphabet, min_size=1) \
.filter(lambda x: not _is_illegal_value(x)).map(lambda x: x.strip())
headers = st.lists(st.tuples(_names, _values), max_size=48)
identity_body = st.one_of(st.binary(), st.none())
chunked_body = st.lists(st.binary(min_size=24))
body = st.one_of(st.binary(), st.none(), chunked_body)
================================================
FILE: integration_tests/test_drain.py
================================================
import pytest
import subprocess
import time
from misc import client
import integration_tests.common
pytestmark = pytest.mark.needs_build
@pytest.fixture(scope='function')
def server():
return integration_tests.common.start_server(
'integration_tests/drain.py', stdout=subprocess.PIPE, path='.test')
@pytest.fixture(scope='function')
def server_terminate(server):
def terminate():
server.terminate()
assert server.wait() == 0
stdout = server.stdout.read()
return [l.decode('utf-8').strip() for l in stdout.splitlines()]
yield terminate
server.terminate()
@pytest.fixture(scope='function')
def connect():
connections = []
def _connect():
conn = client.Connection('localhost:8080')
conn.maybe_connect()
connections.append(conn)
return conn
yield _connect
for c in connections:
c.close()
def test_no_connections(server_terminate):
lines = server_terminate()
assert lines[-1] == 'Termination request received'
@pytest.mark.parametrize('num', range(1, 5))
def test_unclosed_connections(num, connect, server_terminate):
for _ in range(num):
connect()
lines = server_terminate()
assert lines[-1] == '{} idle connections closed immediately'.format(num)
@pytest.mark.parametrize('num', range(1, 5))
def test_closed_connections(num, connect, server_terminate):
for _ in range(num):
con = connect()
con.close()
lines = server_terminate()
assert lines[-1] == 'Termination request received'
@pytest.mark.parametrize('num', range(1, 5))
def test_unclosed_requests(num, connect, server_terminate):
for _ in range(num):
con = connect()
con.putrequest('GET', '/')
con.endheaders()
lines = server_terminate()
assert lines[-1] == '{} idle connections closed immediately'.format(num)
@pytest.mark.parametrize('num', range(1, 5))
def test_closed_requests(num, connect, server_terminate):
for _ in range(num):
con = connect()
con.putrequest('GET', '/')
con.endheaders()
con.getresponse()
con.close()
lines = server_terminate()
assert lines[-1] == 'Termination request received'
@pytest.mark.parametrize('num', range(1, 3))
def test_pipelined(num, connect, server_terminate):
connections = []
for _ in range(num):
con = connect()
connections.append(con)
con.putrequest('GET', '/sleep/1')
con.endheaders()
lines = server_terminate()
assert '{} connections busy, read-end closed'.format(num) in lines
assert not any(l.startswith('Forcefully killing') for l in lines)
assert all(c.getresponse().status == 200 for c in connections)
@pytest.mark.parametrize('num', range(1, 3))
def test_pipelined_timeout(num, connect, server_terminate):
connections = []
for _ in range(num):
con = connect()
connections.append(con)
con.putrequest('GET', '/sleep/10')
con.endheaders()
lines = server_terminate()
assert '{} connections busy, read-end closed'.format(num) in lines
assert 'Forcefully killing {} connections'.format(num) in lines
assert all(c.getresponse().status == 503 for c in connections)
def test_refuse(connect, server):
con = connect()
con.putrequest('GET', '/sleep/10')
con.endheaders()
server.terminate()
# give time for the signal to propagate
time.sleep(1)
with pytest.raises(ConnectionRefusedError):
con = connect()
assert server.wait() == 0
================================================
FILE: integration_tests/test_noleak.py
================================================
import pytest
from misc import client
import integration_tests.common
pytestmark = pytest.mark.needs_build(
'--extra-compile=-DPROTOCOL_TRACK_REFCNT=1', dest='.test/noleak')
@pytest.fixture(scope='function')
def server(request):
arg = request.node.get_marker('arg').args[0]
server = integration_tests.common.start_server([
'integration_tests/noleak.py', arg], path='.test/noleak')
yield server
server.terminate()
server.wait() == 0
@pytest.fixture(scope='function')
def connection(server):
conn = client.Connection('localhost:8080')
yield conn
conn.close()
@pytest.mark.arg('method')
def test_method(connection):
methods = ['GET', 'POST', 'PATCH', 'DELETE', 'PUT']
for method in methods:
connection.putrequest(method, '/noleak/1/2')
connection.endheaders()
response = connection.getresponse()
assert response.status == 200
@pytest.mark.arg('path')
def test_path(connection):
paths = ['/noleak/1/2', '/noleak/3/4', '/noleak/5/4', '/noleak/6/7']
for path in paths:
connection.putrequest('GET', path)
connection.endheaders()
response = connection.getresponse()
assert response.status == 200
@pytest.mark.arg('match_dict')
def test_match_dict(connection):
paths = ['/noleak/1/2', '/noleak/3/4', '/noleak/5/4', '/noleak/6/7']
for path in paths:
connection.putrequest('GET', path)
connection.endheaders()
response = connection.getresponse()
assert response.status == 200
@pytest.mark.arg('query_string')
def test_query_string(connection):
query_strings = ['?', None, '?a', None, '?', None, '?b', None, '?']
for query_string in query_strings:
connection.putrequest('GET', '/noleak/1/2', query_string)
connection.endheaders()
response = connection.getresponse()
assert response.status == 200
@pytest.mark.arg('headers')
def test_headers(connection):
header_list = [{}, {"X-a": "b"}, {}, {"X-b": "c"}, {}, {"X-c": "d"}]
for headers in header_list:
connection.putrequest('GET', '/noleak/1/2')
for name, value in headers.items():
connection.putheader(name, value)
connection.endheaders()
response = connection.getresponse()
assert response.status == 200
@pytest.mark.arg('body')
def test_body(connection):
bodies = [None, b'a', None, b'b', None, b'c', None, b'd', None, b'e']
for body in bodies:
connection.putrequest('GET', '/noleak/1/2')
if body:
connection.putheader('Content-Length', str(len(body)))
connection.endheaders(body)
response = connection.getresponse()
assert response.status == 200
@pytest.mark.arg('keep_alive')
def test_keep_alive(request):
keep_alives = [True, False, True, True, False, False, True, False]
still_open = False
for keep_alive in keep_alives:
if not still_open:
connection_gen = connection(request.getfixturevalue('server'))
conn = next(connection_gen)
still_open = keep_alive
conn.putrequest('GET', '/noleak/1/2')
if not keep_alive:
conn.putheader('Connection', 'close')
conn.endheaders()
response = conn.getresponse()
assert response.status == 200
@pytest.mark.arg('route')
def test_route(connection):
for _ in range(7):
connection.putrequest('GET', '/noleak/1/2')
connection.endheaders()
response = connection.getresponse()
assert response.status == 200
================================================
FILE: integration_tests/test_perror.py
================================================
import pytest
from hypothesis import given, strategies as st, settings, Verbosity
import subprocess
import queue
import threading
from functools import partial
from misc import client
import integration_tests.common
pytestmark = pytest.mark.needs_build
@pytest.fixture(autouse=True, scope='module')
def server():
server = integration_tests.common.start_server(
['-u', 'integration_tests/dump.py'],
stdout=subprocess.PIPE, path='.test')
accepting = server.stdout.readline().decode('utf-8')
assert accepting.startswith('Accepting connections ')
yield server
server.terminate()
server.wait() == 0
@pytest.fixture
def line_getter(server):
q = queue.Queue()
def enqueue_output():
q.put(server.stdout.readline().strip().decode('utf-8'))
class LineGetter:
def start(self):
self.thread = threading.Thread(target=enqueue_output)
self.thread.start()
def wait(self):
self.thread.join()
return q.get()
return LineGetter()
@pytest.fixture()
def connect(request):
return partial(client.Connection, 'localhost:8080')
full_request_line = 'GET /asd?qwe HTTP/1.1'
def make_truncated_request_line(cut):
return full_request_line[:-cut]
st_request_cut = st.integers(min_value=1, max_value=len(full_request_line) - 1)
st_request_line = st.builds(make_truncated_request_line, st_request_cut)
@given(request_line=st_request_line)
@settings(verbosity=Verbosity.verbose, max_examples=20)
def test_truncated_request_line(line_getter, connect, request_line):
connection = connect()
line_getter.start()
connection.putline(request_line)
assert line_getter.wait() == 'malformed_headers'
response = connection.getresponse()
assert response.status == 400
assert response.text == 'malformed_headers'
@given(request_line=st_request_line)
@settings(verbosity=Verbosity.verbose, max_examples=20)
def test_truncated_request_line_disconnect(line_getter, connect, request_line):
connection = connect()
line_getter.start()
connection.putclose(request_line)
assert line_getter.wait() == 'incomplete_headers'
full_header = 'X-Header: asd'
def make_truncated_header(cut):
return full_header[:-cut]
st_header_cut = st.integers(min_value=5, max_value=len(full_header) - 1)
st_header_line = st.builds(make_truncated_header, st_header_cut)
@given(header_line=st_header_line)
@settings(verbosity=Verbosity.verbose, max_examples=20)
def test_truncated_header(line_getter, connect, header_line):
connection = connect()
line_getter.start()
connection.putline(full_request_line)
connection.putline(header_line)
connection.putline()
assert line_getter.wait() == 'malformed_headers'
response = connection.getresponse()
assert response.status == 400
assert response.text == 'malformed_headers'
@given(header_line=st_header_line)
@settings(verbosity=Verbosity.verbose, max_examples=20)
def test_truncated_header_disconnect(line_getter, connect, header_line):
connection = connect()
line_getter.start()
connection.putline(full_request_line)
connection.putclose(header_line)
assert line_getter.wait() == 'incomplete_headers'
@pytest.mark.parametrize('value', [
'',
'+5',
'-5',
'0x12',
'12a'
])
def test_invalid_content_length(line_getter, connect, value):
connection = connect()
line_getter.start()
connection.putline(full_request_line)
connection.putheader('Content-Length', value)
connection.putline()
assert line_getter.wait() == 'invalid_headers'
response = connection.getresponse()
assert response.status == 400
assert response.text == 'invalid_headers'
================================================
FILE: integration_tests/test_reaper.py
================================================
from functools import partial
import time
import pytest
from misc import client
import integration_tests.common
pytestmark = pytest.mark.needs_build
@pytest.fixture(scope='function', params=[2, 3, 4])
def get_connections_and_wait(request):
server, process = integration_tests.common.start_server([
'integration_tests/reaper.py', '1', str(request.param)], path='.test',
return_process=True)
def connection_num():
return len(
set(c.fd for c in process.connections()) |
set(c.fd for p in process.children() for c in p.connections()))
yield connection_num, partial(time.sleep, request.param)
server.terminate()
assert server.wait() == 0
def test_empty(get_connections_and_wait):
get_connections, wait = get_connections_and_wait
conn = client.Connection('localhost:8080')
assert get_connections() == 1
conn.maybe_connect()
time.sleep(.1)
assert get_connections() == 2
wait()
assert get_connections() == 1
def test_request(get_connections_and_wait):
get_connections, wait = get_connections_and_wait
conn = client.Connection('localhost:8080')
assert get_connections() == 1
conn.putrequest('GET', '/')
conn.endheaders()
assert get_connections() == 2
wait()
time.sleep(1)
assert get_connections() == 1
================================================
FILE: integration_tests/test_request.py
================================================
import pytest
import json
import base64
from functools import partial
from hypothesis import given, settings, Verbosity, HealthCheck
import integration_tests.common
from integration_tests import strategies as st
from misc import client
pytestmark = pytest.mark.needs_build(coverage=True)
@pytest.fixture(autouse=True, scope='module')
def server():
server = integration_tests.common.start_server(
'integration_tests/dump.py', path='.test')
yield server
server.terminate()
server.wait() == 0
@pytest.fixture(params=['example', 'test'])
def connect(request):
if request.param == 'example':
yield partial(client.Connection, 'localhost:8080')
elif request.param == 'test':
connection = client.Connection('localhost:8080')
close = connection.close
connection.close = lambda: None
yield lambda: connection
close()
@pytest.fixture(params=['', '/async'], ids=['sync', 'async'])
def prefix(request):
return request.param
@given(method=st.method)
@settings(verbosity=Verbosity.verbose)
def test_method(prefix, connect, method):
connection = connect()
connection.request(method, prefix + '/dump/1/2')
response = connection.getresponse()
json_body = json.loads(response.body)
assert response.status == 200
assert json_body['method'] == method
connection.close()
st_route_prefix = st.sampled_from(['/dump/', '/dump1/', '/dump2/'])
@given(route_prefix=st_route_prefix)
@settings(verbosity=Verbosity.verbose)
def test_route(prefix, connect, route_prefix):
connection = connect()
connection.request('GET', prefix + route_prefix + '1/2')
response = connection.getresponse()
json_body = json.loads(response.body)
assert response.status == 200
assert json_body['route'].startswith(prefix + route_prefix)
connection.close()
@given(param1=st.param, param2=st.param)
@settings(verbosity=Verbosity.verbose)
def test_match_dict(prefix, connect, param1, param2):
connection = connect()
connection.request('GET', prefix + '/dump/{}/{}'.format(param1, param2))
response = connection.getresponse()
json_body = json.loads(response.body)
assert response.status == 200
assert json_body['match_dict'] == {'p1': param1, 'p2': param2}
connection.close()
@given(query_string=st.query_string)
@settings(verbosity=Verbosity.verbose)
def test_query_string(prefix, connect, query_string):
connection = connect()
connection.request('GET', prefix + '/dump/1/2', query_string)
response = connection.getresponse()
json_body = json.loads(response.body)
assert response.status == 200
assert json_body['query_string'] == query_string
connection.close()
@given(headers=st.headers)
@settings(
verbosity=Verbosity.verbose,
suppress_health_check=[HealthCheck.too_slow]
)
def test_headers(prefix, connect, headers):
connection = connect()
connection.request('GET', prefix + '/dump/1/2', headers=headers)
response = connection.getresponse()
json_body = json.loads(response.body)
assert response.status == 200
headers = {k.title(): v for k, v in headers}
assert json_body['headers'] == headers
connection.close()
st_errors = st.sampled_from(['not-found', 'forced-1', 'forced-2'])
@given(error=st_errors)
@settings(
verbosity=Verbosity.verbose,
suppress_health_check=[HealthCheck.too_slow]
)
def test_error(prefix, connect, error):
connection = connect()
connection.putrequest(
'GET', prefix + '/not-found' if error == 'not-found' else '/dump/1/2')
if error != 'not-found':
connection.putheader('Force-Raise', error)
connection.endheaders()
response = connection.getresponse()
assert response.status == 500
json_body = json.loads(response.body)
assert json_body['exception']['type'] == \
'RouteNotFoundException' if error == 'not-found' else 'ForcedException'
assert json_body['exception']['args'] == \
'' if error == 'not-found' else error
@given(body=st.identity_body)
@settings(verbosity=Verbosity.verbose)
@pytest.mark.parametrize(
'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k'])
def test_body(prefix, connect, size_k, body):
if size_k and body:
body = body * ((size_k * 1024) // len(body) + 1)
connection = connect()
connection.request('GET', prefix + '/dump/1/2', body=body)
response = connection.getresponse()
assert response.status == 200
json_body = json.loads(response.body)
if body is not None:
assert base64.b64decode(json_body['body']) == body
else:
assert json_body['body'] == body
connection.close()
@given(body=st.chunked_body)
@settings(verbosity=Verbosity.verbose)
@pytest.mark.parametrize(
'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k'])
def test_chunked(prefix, connect, size_k, body):
length = sum(len(b) for b in body)
if size_k and length:
body = body * ((size_k * 1024) // length + 1)
connection = connect()
connection.request('POST', prefix + '/dump/1/2', body=body)
response = connection.getresponse()
assert response.status == 200
json_body = json.loads(response.body)
assert base64.b64decode(json_body['body']) == b''.join(body)
connection.close()
st_errors = st.sampled_from([None, None, None, 'not-found', 'forced-1'])
@given(
method=st.method,
error=st_errors,
route_prefix=st_route_prefix,
param1=st.param, param2=st.param,
query_string=st.query_string,
headers=st.headers,
body=st.identity_body
)
@settings(
verbosity=Verbosity.verbose,
suppress_health_check=[HealthCheck.too_slow]
)
@pytest.mark.parametrize(
'size_k', [0, 1, 2, 4, 8], ids=['small', '1k', '2k', '4k', '8k'])
def test_all(prefix, connect, size_k, method, error, route_prefix,
param1, param2, query_string, headers, body):
connection = connect()
if size_k and body:
body = body * ((size_k * 1024) // len(body) + 1)
url = prefix + ('/not-found' if error == 'not-found' else '') \
+ route_prefix + '{}/{}'.format(param1, param2)
connection.putrequest(method, url, query_string)
for name, value in headers:
connection.putheader(name, value)
if body is not None:
headers.append(('Content-Length', str(len(body))))
connection.putheader('Content-Length', str(len(body)))
if error == 'forced-1':
headers.append(('Force-Raise', 'forced-1'))
connection.putheader('Force-Raise', 'forced-1')
connection.endheaders(body)
response = connection.getresponse()
assert response.status == 500 if error else 200
json_body = json.loads(response.body)
assert json_body['method'] == method
if error != 'not-found':
assert json_body['route'].startswith(prefix + route_prefix)
else:
assert json_body['route'] is None
assert json_body['match_dict'] == \
{} if error == 'not-found' else {'p1': param1, 'p2': param2}
assert json_body['query_string'] == query_string
headers = {k.title(): v for k, v in headers}
assert json_body['headers'] == headers
if body is not None:
assert base64.b64decode(json_body['body']) == body
else:
assert json_body['body'] is None
if error:
assert json_body['exception']['type'] == \
'RouteNotFoundException' if error == 'not-found' \
else 'ForcedException'
assert json_body['exception']['args'] == \
'' if error == 'not-found' else error
else:
assert 'exception' not in json_body
connection.close()
st_request = st.fixed_dictionaries({
'method': st.method,
'error': st_errors,
'route_prefix': st_route_prefix,
'param1': st.param,
'param2': st.param,
'query_string': st.query_string,
'headers': st.headers,
'body': st.identity_body
})
st_requests = st.lists(st_request, min_size=2)
@given(requests=st_requests)
@settings(
verbosity=Verbosity.verbose,
suppress_health_check=[HealthCheck.too_slow])
def test_pipeline(requests):
connection = client.Connection('localhost:8080')
for request in requests:
connection.putrequest(
request['method'], request['route_prefix'] +
('/not-found' if request['error'] == 'not-found' else '') +
'{param1}/{param2}'.format_map(request),
request['query_string'])
for name, value in request['headers']:
connection.putheader(name, value)
if request['body'] is not None:
body_len = str(len(request['body']))
request['headers'].append(('Content-Length', body_len))
connection.putheader('Content-Length', body_len)
if request['error'] == 'forced-1':
request['headers'].append(('Force-Raise', 'forced-1'))
connection.putheader('Force-Raise', 'forced-1')
connection.endheaders(request['body'])
for request in requests:
response = connection.getresponse()
assert response.status == 500 if request['error'] else 200
json_body = response.json
assert json_body['method'] == request['method']
if request['error'] != 'not-found':
assert json_body['route'].startswith(request['route_prefix'])
else:
assert json_body['route'] is None
assert json_body['match_dict'] == \
{} if request['error'] == 'not-found' else \
{'p1': request['param1'], 'p2': request['param2']}
assert json_body['query_string'] == request['query_string']
assert json_body['headers'] == \
{k.title(): v for k, v in request['headers']}
if request['body'] is not None:
assert base64.b64decode(json_body['body']) == request['body']
else:
assert json_body['body'] is None
if request['error']:
assert json_body['exception']['type'] == \
'RouteNotFoundException' if request['error'] == 'not-found' \
else 'ForcedException'
assert json_body['exception']['args'] == \
'' if request['error'] == 'not-found' else request['error']
else:
assert 'exception' not in json_body
connection.close()
def format_sleep_qs(val):
return 'sleep=' + str(val / 100)
st_sleep = st.builds(format_sleep_qs, st.integers(min_value=0, max_value=10))
st_prefix = st.sampled_from(['/dump', '/async/dump'])
st_async_request = st.fixed_dictionaries({
'query_string': st_sleep,
'prefix': st_prefix,
'error': st_errors
})
st_async_requests = st.lists(st_async_request, min_size=2, max_size=5) \
.filter(lambda rs: any(r['prefix'].startswith('/async') for r in rs))
@given(requests=st_async_requests)
@settings(verbosity=Verbosity.verbose)
def test_async_pipeline(requests):
connection = client.Connection('localhost:8080')
for request in requests:
connection.putrequest(
'GET', request['prefix'] +
('/not-found' if request['error'] == 'not-found' else '') +
'/1/2', request['query_string'])
if request['error'] == 'forced-1':
connection.putheader('Force-Raise', 'forced-1')
connection.endheaders()
for request in requests:
response = connection.getresponse()
assert response.status == 500 if request['error'] else 200
json_body = response.json
assert json_body['query_string'] == request['query_string']
connection.close()
================================================
FILE: misc/__init__.py
================================================
================================================
FILE: misc/bootstrap.sh
================================================
#!/bin/bash
cd ~
sudo apt-get update
sudo apt-get install -y python3 git libbz2-dev libz-dev libsqlite3-dev libssl-dev gcc make libffi-dev lcov
git clone https://github.com/yyuu/pyenv .pyenv
.pyenv/bin/pyenv install -v 3.6.0
wget https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz
tar xvfz virtualenv-15.1.0.tar.gz
python3 virtualenv-15.1.0/virtualenv.py -p .pyenv/versions/3.6.0/bin/python japronto-env
git clone https://github.com/squeaky-pl/japronto
cd japronto/src/picohttpparser
./build
cd -
cd japronto
../japronto-env/bin/pip install -r requirements.txt
../japronto-env/bin/python build.py --kit=platform
cd -
git clone https://github.com/wg/wrk
cd wrk
make
cd -
cp wrk/wrk japronto
================================================
FILE: misc/buggers.py
================================================
import atexit
import psutil
noisy = ['atom', 'chrome', 'firefox', 'dropbox', 'opera', 'spotify',
'gnome-documents']
def silence():
for proc in psutil.process_iter():
if proc.name() in noisy:
proc.suspend()
def noise():
for proc in psutil.process_iter():
if proc.name() in noisy:
proc.resume()
atexit.register(noise)
================================================
FILE: misc/cleanup_script.py
================================================
import sys
def main():
fp = open(sys.argv[1])
for line in fp:
line = line.rstrip()
if line.startswith('\t'):
rest = line[18:]
name_addr, _, rest = rest.partition(' ')
name, _, addr = name_addr.partition('+')
line = line[:18] + name + ' ' + rest
print(line)
fp.close()
if __name__ == '__main__':
main()
================================================
FILE: misc/client.py
================================================
import socket
import urllib.parse
import json
def readline(sock):
line = b''
while not line.endswith(b'\r\n'):
line += sock.recv(1)
return line
def readexact(sock, size):
data = b''
while size:
chunk = sock.recv(size)
data += chunk
size -= len(chunk)
return data
class Response:
def __init__(self, sock):
self.sock = sock
self.read_status_line()
self.read_headers()
self.read_body()
def read_status_line(self):
status_line = b''
while not status_line:
status_line = readline(self.sock).strip()
_, self.status, self.reason = status_line.split(None, 2)
self.status = int(self.status)
def read_headers(self):
self.headers = {}
while 1:
line = readline(self.sock).strip()
if not line:
break
name, value = line.split(b':')
name = name.strip().decode('ascii').title()
value = value.strip().decode('latin1')
self.headers[name] = value
@property
def encoding(self):
content_type = self.headers.get('Content-Type')
if not content_type:
return 'latin1'
_, *rest = [v.split('=') for v in content_type.split(';')]
rest = {k.strip(): v.strip() for k, v in rest}
return rest.get('charset', 'iso-8859-1')
def read_body(self):
self.body = readexact(self.sock, int(self.headers['Content-Length']))
self.text = self.body.decode(self.encoding)
@property
def json(self):
return json.loads(self.text)
class Connection:
def __init__(self, addr):
self.addr = addr
self.sock = None
def maybe_connect(self):
if self.sock:
return self.sock
addr = self.addr.split(':')
addr[1] = int(addr[1])
addr = tuple(addr)
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(addr)
return self.sock
def putline(self, line=None):
line = line or b''
sock = self.maybe_connect()
if not isinstance(line, bytes):
line = str(line).encode('latin1')
sock.sendall(line + b'\r\n')
def putclose(self, data):
sock = self.maybe_connect()
if not isinstance(data, bytes):
data = str(data).encode('latin1')
sock.sendall(data)
self.close()
def putrequest(self, method, path, query_string=None):
url = urllib.parse.quote(path)
if query_string is not None:
url += '?' + urllib.parse.quote(query_string)
request_line = "{method} {url} HTTP/1.1" \
.format(method=method, url=url)
self.putline(request_line)
def request(self, method, path, query_string=None, headers=None,
body=None):
self.putrequest(method, path, query_string)
headers = headers or []
for name, value in headers:
self.putheader(name, value)
if body is not None:
if isinstance(body, list):
self.putheader('Transfer-Encoding', 'chunked')
else:
self.putheader('Content-Length', str(len(body)))
self.endheaders(body)
def putheader(self, name, value):
header_line = name + ': ' + value
self.putline(header_line)
def endheaders(self, body=None):
self.putline()
if body is not None:
sock = self.maybe_connect()
if isinstance(body, list):
for chunk in chunked_encoder(body):
sock.sendall(chunk)
else:
sock.sendall(body)
def getresponse(self):
return Response(self.sock)
def close(self):
self.sock.close()
def chunked_encoder(data):
for chunk in data:
if not chunk:
continue
yield '{:X}\r\n'.format(len(chunk)).encode('ascii')
yield chunk + b'\r\n'
yield b'0\r\n\r\n'
================================================
FILE: misc/collector.py
================================================
import psutil
import sys
import time
import os
import json
from functools import partial
def get_connections(process):
return len(
set(c.fd for c in process.connections()) |
set(c.fd for p in process.children() for c in p.connections()))
def get_memory(p):
return p.memory_full_info().uss \
+ sum(c.memory_full_info().uss for c in p.children())
def sample_process(pid):
process = psutil.Process(pid)
samples = []
while 1:
try:
uss = get_memory(process)
conn = get_connections(process)
except (psutil.NoSuchProcess, psutil.AccessDenied):
break
samples.append({
't': time.monotonic(),
'uss': uss, 'conn': conn, 'type': 'proc'})
time.sleep(.5)
return samples
def main():
pid = int(sys.argv[1])
samples = sample_process(pid)
with open(os.environ['COLLECTOR_FILE'], 'a') as fp:
for sample in samples:
fp.write(json.dumps(sample) + '\n')
print('Collector info written to', os.environ['COLLECTOR_FILE'])
if __name__ == '__main__':
main()
================================================
FILE: misc/cpu.py
================================================
"""
CPU file
"""
# module imports
import subprocess
# cpu location
CPU_PREFIX = '/sys/devices/system/cpu/'
def save():
"""
save function
"""
results = {}
cpu_number = 0
while True:
try:
_file = open(
CPU_PREFIX + 'cpu{}/cpufreq/scaling_governor'.format(cpu_number))
except:
break
governor = _file.read().strip()
results.setdefault(cpu_number, {})['governor'] = governor
_file.close()
try:
_file = open(
CPU_PREFIX + 'cpu{}/cpufreq/scaling_cur_freq'.format(cpu_number))
except:
break
results[cpu_number]['freq'] = _file.read().strip()
_file.close()
cpu_number += 1
return results
def change(governor, freq=None):
"""
change function
"""
cpu_number = 0
while True:
try:
subprocess.check_output([
"sudo", "bash", "-c",
"echo {governor} > {CPU_PREFIX}cpu{cpu_number}/cpufreq/scaling_governor"
.format(governor=governor,
CPU_PREFIX=CPU_PREFIX,
cpu_number=cpu_number)],
stderr=subprocess.STDOUT)
except:
break
if freq:
subprocess.check_output([
"sudo", "bash", "-c",
"echo {freq} > {CPU_PREFIX}cpu{cpu_number}/cpufreq/scaling_setspeed"
.format(freq=freq,
CPU_PREFIX=CPU_PREFIX,
cpu_number=cpu_number)],
stderr=subprocess.STDOUT)
cpu_number += 1
def available_freq():
"""
function for checking available frequency
"""
_file = open(CPU_PREFIX + 'cpu0/cpufreq/scaling_available_frequencies')
freq = [int(_file) for _file in _file.read().strip().split()]
_file.close()
return freq
def min_freq():
"""
function for returning minimum available frequency
"""
return min(available_freq())
def max_freq():
"""
function for returning maximum avaliable frequency
"""
return max(available_freq())
def dump():
"""
dump function
"""
try:
sensors = subprocess.check_output('sensors').decode('utf-8')
except (FileNotFoundError, subprocess.CalledProcessError):
print("Couldn't read CPU temp")
else:
cores = []
for line in sensors.splitlines():
if line.startswith('Core '):
core, rest = line.split(':')
temp = rest.strip().split()[0]
cores.append((core, temp))
for core, temp in cores:
print(core + ':', temp)
cpu_number = 0
while True:
try:
_file = open(
CPU_PREFIX + 'cpu{}/cpufreq/scaling_governor'.format(cpu_number))
except:
break
print('Core ' + str(cpu_number) + ':', _file.read().strip(), end=', ')
_file.close()
try:
_file = open(
CPU_PREFIX + 'cpu{}/cpufreq/scaling_cur_freq'.format(cpu_number))
except:
break
freq = round(int(_file.read()) / 10 ** 6, 2)
print(freq, 'GHz')
cpu_number += 1
================================================
FILE: misc/do_perf.py
================================================
import subprocess
import os
import sys
import argparse
import parsers
import cases
import buggers
import cpu
def get_http10long():
return cases.base['10long'].data
def get_websites(size=2 ** 18):
data = b''
while len(data) < size:
for c in cases.websites.values():
data += c.data
return data
if __name__ == '__main__':
print('pid', os.getpid())
cpu.dump()
buggers.silence()
argparser = argparse.ArgumentParser(description='do_perf')
argparser.add_argument(
'-p', '--parsers', dest='parsers', default='cffi,cext')
argparser.add_argument(
'-b', '--benchmarks', dest='benchmarks',
default='http10long,websites,websitesn')
result = argparser.parse_args(sys.argv[1:])
parsers = result.parsers.split(',')
benchmarks = result.benchmarks.split(',')
one_shot = [b for b in benchmarks if b in ['http10long', 'websites']]
multi_shot = [b for b in benchmarks if b in ['websitesn']]
setup = """
import parsers
import do_perf
parser, _ = parsers.make_{}(parsers.NullProtocol)
data = do_perf.get_{}()
"""
loop = """
parser.feed(data)
parser.feed_disconnect()
"""
for dataset in one_shot:
for parser in parsers:
print('-- {} {} --'.format(dataset, parser))
subprocess.check_call([
'python', '-m', 'perf', 'timeit', '-s',
setup.format(parser, dataset), loop])
print()
setup += """
import parts
p = parts.make_parts(data, parts.fancy_series(1450))
"""
loop = """
for i in p:
parser.feed(i)
parser.feed_disconnect()
"""
if multi_shot:
for parser in parsers:
print('-- website parts {} --'.format(parser))
subprocess.check_call([
'python', '-m', 'perf', 'timeit', '-s',
setup.format(parser, 'websites'), loop])
print()
================================================
FILE: misc/docker/Dockerfile
================================================
FROM python:3.6.0-slim
RUN pip3 install japronto
ENV PYTHONUNBUFFERED=1
ENTRYPOINT ["japronto"]
================================================
FILE: misc/parts.py
================================================
from functools import partial
import types
import math
def make_parts(value, get_size, dir=1):
parts = []
left = bytearray(value)
while left:
if isinstance(get_size, types.GeneratorType):
size = next(get_size)
else:
size = get_size
if dir == 1:
parts.append(bytes(left[:size]))
left = left[size:]
else:
parts.append(bytes(left[-size:]))
left = left[:-size]
return parts if dir == 1 else list(reversed(parts))
def one_part(value):
return [value]
def geometric_series():
s = 2
while 1:
yield s
s *= 2
def fancy_series(minimum=2):
x = 0
while 1:
yield int(minimum + abs(math.sin(x / 3)) * 64)
x += 1
================================================
FILE: misc/perf.md
================================================
Capturing performance data with perf
====================================
For best results the C source should be built with `-g -O0`.
Run the server, record performance events with `-F` frequency `997 Hz`, `-a` all processes and `-g` take stack info.
```
python examples/simple/simple.py -p c & \
perf record -F 997 -a -g -- sleep 59 & \
sleep 1 && ./wrk -c 100 -t 1 -d 60 http://localhost:8080 &
```
Write data to a text file filtering for pid `2836` (should be server process)
```
perf script --pid 2836 > out.perf
```
Remove address info to make graphs easier to read
```
python cleanup_script.py out.perf > out2.perf
```
Visualize data with a flame graph
=================================
```
./stackcollapse-perf.pl out2.perf > out.folded
./flamegraph.pl out.folded > test.svg
```
================================================
FILE: misc/pipeline.lua
================================================
-- example script demonstrating HTTP pipelining
init = function(args)
local r = {}
r[1] = wrk.format(nil, "/")
r[2] = wrk.format(nil, "/")
r[3] = wrk.format(nil, "/")
r[4] = wrk.format(nil, "/")
r[5] = wrk.format(nil, "/")
r[6] = wrk.format(nil, "/")
r[7] = wrk.format(nil, "/")
r[8] = wrk.format(nil, "/")
r[9] = wrk.format(nil, "/")
r[10] = wrk.format(nil, "/")
r[11] = wrk.format(nil, "/")
r[12] = wrk.format(nil, "/")
r[13] = wrk.format(nil, "/")
r[14] = wrk.format(nil, "/")
r[15] = wrk.format(nil, "/")
r[16] = wrk.format(nil, "/")
r[17] = wrk.format(nil, "/")
r[18] = wrk.format(nil, "/")
r[19] = wrk.format(nil, "/")
r[20] = wrk.format(nil, "/")
r[21] = wrk.format(nil, "/")
r[22] = wrk.format(nil, "/")
r[23] = wrk.format(nil, "/")
r[24] = wrk.format(nil, "/")
req = table.concat(r)
end
request = function()
return req
end
================================================
FILE: misc/report.py
================================================
import matplotlib.pyplot as plt
import sys
import os
import json
def report(samples, pid):
plt.figure(figsize=(25, 10))
x = [s['t'] for s in samples if s['type'] == 'proc']
lines = [s for s in samples if s['type'] == 'event']
# minuss = min(s['uss'] for s in samples if s['type'] == 'proc')
ussplot = plt.subplot(211)
ussplot.set_title('uss')
ussplot.plot(
x, [s['uss'] for s in samples if s['type'] == 'proc'], '.')
for l in lines:
# ussplot.text(l['t'], minuss, l['event'], horizontalalignment='right',
# rotation=-90, rotation_mode='anchor')
ussplot.axvline(l['t'])
connplot = plt.subplot(212)
connplot.set_title('conn')
connplot.plot(
x, [s['conn'] for s in samples if s['type'] == 'proc'], '.')
os.makedirs('.reports', exist_ok=True)
path = '.reports/{}.png'.format(pid)
plt.savefig(path)
return path
def load(filepath):
samples = []
with open(filepath) as fp:
for line in fp:
line = line.strip()
samples.append(json.loads(line))
return samples
def order(samples):
return sorted(samples, key=lambda x: x['t'])
def normalize_time(samples):
if not samples:
return []
base_time = samples[0]['t']
return [{**s, 't': s['t'] - base_time} for s in samples]
def main():
samples = load(sys.argv[1])
pid, _ = os.path.splitext(os.path.basename(sys.argv[1]))
samples = order(samples)
samples = normalize_time(samples)
report(samples, pid)
if __name__ == '__main__':
main()
================================================
FILE: misc/requirements-test.txt
================================================
-r requirements.txt
hypothesis==3.80.0
psutil==5.6.6
pytest==3.9.2
pytoml==0.1.20
perf==1.5.1
cffi==1.11.5
================================================
FILE: misc/requirements.txt
================================================
uvloop>=0.11.3
================================================
FILE: misc/rpm-requirements.txt
================================================
libasan
lcov
libubsan
================================================
FILE: misc/runpytest.py
================================================
from pytest import main
main()
================================================
FILE: misc/simple.py
================================================
import asyncio
import argparse
import os.path
import sys
import socket
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__) + '/../../src'))
import japronto.protocol.handler # noqa
from japronto.router.cmatcher import Matcher # noqa
from japronto.router import Router # noqa
from japronto.app import Application # noqa
def slash(request):
return request.Response(text='Hello slash!')
def hello(request):
return request.Response(text='Hello hello!')
async def sleep(request):
await asyncio.sleep(3)
return request.Response(text='I am sleepy')
async def loop(request):
i = 0
while i < 10:
await asyncio.sleep(1)
print(i)
i += 1
return request.Response(text='Loop finished')
def dump(request):
sock = request.transport.get_extra_info('socket')
no_delay = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY)
text = """
Method: {0.method}
Path: {0.path}
Version: {0.version}
Headers: {0.headers}
Match: {0.match_dict}
Body: {0.body}
QS: {0.query_string}
query: {0.query}
mime_type: {0.mime_type}
encoding: {0.encoding}
form: {0.form}
keep_alive: {0.keep_alive}
no_delay: {1}
route: {0.route}
""".strip().format(request, no_delay)
return request.Response(text=text, headers={'X-Version': '123'})
app = Application()
r = app.router
r.add_route('/', slash)
r.add_route('/hello', hello)
r.add_route('/dump/{this}/{that}', dump)
r.add_route('/sleep/{pinch}', sleep)
r.add_route('/loop', loop)
if __name__ == '__main__':
argparser = argparse.ArgumentParser('server')
argparser.add_argument(
'-p', dest='flavor', default='block')
args = argparser.parse_args(sys.argv[1:])
app.run(protocol_factory=japronto.protocol.handler.make_class(args.flavor))
================================================
FILE: misc/suppr.txt
================================================
# This is a known leak.
# Python leaks a little, we use malloc directly
leak:PyMem_RawMalloc
leak:_PyObject_GC_Resize
leak:PyThread_allocate_lock
leak:resize_compact
================================================
FILE: misc/travis/before_install.sh
================================================
#!/bin/bash
export JAPR_MSG=`git show -s --format=%B | xargs`
export JAPR_WHEEL=`[[ $JAPR_MSG == *"[travis-wheel]"* ]] && echo 1 || echo 0`
export JAPR_OS=`uname`
if [[ $VERSION == "3.5."* ]]; then
export PYTHON_TAG=cp35-cp35m
elif [[ $VERSION == "3.6."* ]]; then
export PYTHON_TAG=cp36-cp36m
elif [[ $VERSION == "3.7."* ]]; then
export PYTHON_TAG=cp37-cp37m
elif [[ $VERSION == "3.8."* ]]; then
export PYTHON_TAG=cp38-cp38m
fi
env | grep "^JAPR_"
================================================
FILE: misc/travis/install.sh
================================================
#!/bin/bash
set -ex
if [[ $JAPR_OS == "Darwin" ]]; then
/usr/bin/clang --version
source misc/terryfy/travis_tools.sh
source misc/terryfy/library_installers.sh
clean_builds
get_python_environment macpython $VERSION venv
fi
if [[ $JAPR_WHEEL == "1" ]]; then
pip install twine
if [[ $JAPR_OS == "Linux" ]]; then
docker info
docker pull quay.io/pypa/manylinux1_x86_64
fi
fi
================================================
FILE: misc/travis/script.sh
================================================
#!/bin/bash
set -ex
if [[ $JAPR_WHEEL == "1" ]]; then
if [[ $JAPR_OS == "Linux" ]]; then
docker run --rm -u `id -u` -w /io -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /opt/python/$PYTHON_TAG/bin/python setup.py bdist_wheel
docker run --rm -u `id -u` -w /io -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 auditwheel repair dist/*-$PYTHON_TAG-linux_x86_64.whl
rm -r dist/*
cp wheelhouse/*.whl dist
fi
if [[ $JAPR_OS == "Darwin" ]]; then
python setup.py bdist_wheel
fi
ls -lha dist
unzip -l dist/*.whl
twine upload -u squeaky dist/*.whl
fi
================================================
FILE: setup.py
================================================
"""
Japronto
"""
import codecs
import os
import re
from setuptools import setup, find_packages
import build
with codecs.open(os.path.join(os.path.abspath(os.path.dirname(
__file__)), 'src', 'japronto', '__init__.py'), 'r', 'latin1') as fp:
try:
version = re.findall(r"^__version__ = '([^']+)'\r?$",
fp.read(), re.M)[0]
except IndexError:
raise RuntimeError('Unable to determine version.')
setup(
name='japronto',
version=version,
url='http://github.com/squeaky-pl/japronto/',
license='MIT',
author='Paweł Piotr Przeradowski',
author_email='przeradowski@gmail.com',
description='A HTTP application toolkit and server bundle ' +
'based on uvloop and picohttpparser',
package_dir={'': 'src'},
packages=find_packages('src'),
keywords=['web', 'asyncio'],
platforms='x86_64 Linux and MacOS X',
install_requires=[
'uvloop>=0.11.3',
],
entry_points="""
[console_scripts]
japronto = japronto.__main__:main
""",
classifiers=[
'Development Status :: 2 - Pre-Alpha',
'Intended Audience :: Developers',
'Environment :: Web Environment',
'License :: OSI Approved :: MIT License',
'Operating System :: MacOS :: MacOS X',
'Operating System :: POSIX :: Linux',
'Programming Language :: C',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: Implementation :: CPython',
'Topic :: Internet :: WWW/HTTP'
],
zip_safe=False,
include_package_data=True,
package_data={'picohttpparser': ['*.so']},
ext_modules=build.get_platform(),
cmdclass={'build_ext': build.custom_build_ext}
)
================================================
FILE: src/japronto/__init__.py
================================================
from .app import Application # noqa
from .router import RouteNotFoundException # noqa
__version__ = '0.1.2'
================================================
FILE: src/japronto/__main__.py
================================================
import sys
import os
from .runner import get_parser, verify, run
def main():
parser = get_parser()
args = parser.parse_args()
if not args.script:
os.putenv('_JAPR_IGNORE_RUN', '1')
if args.reload:
os.execv(
sys.executable,
[sys.executable, '-m', 'japronto.reloader', *sys.argv[1:]])
if not args.script:
os.environ['_JAPR_IGNORE_RUN'] = '1'
attribute = verify(args)
if not attribute:
return 1
run(attribute, args)
sys.exit(main())
================================================
FILE: src/japronto/app/__init__.py
================================================
import signal
import asyncio
import traceback
import socket
import os
import sys
import multiprocessing
import faulthandler
import uvloop
from japronto.router import Router, RouteNotFoundException
from japronto.protocol.cprotocol import Protocol
from japronto.protocol.creaper import Reaper
signames = {
int(v): v.name for k, v in signal.__dict__.items()
if isinstance(v, signal.Signals)}
class Application:
def __init__(self, *, reaper_settings=None, log_request=None,
protocol_factory=None, debug=False):
self._router = None
self._loop = None
self._connections = set()
self._reaper_settings = reaper_settings or {}
self._error_handlers = []
self._log_request = log_request
self._request_extensions = {}
self._protocol_factory = protocol_factory or Protocol
self._debug = debug
@property
def loop(self):
if not self._loop:
self._loop = uvloop.new_event_loop()
return self._loop
@property
def router(self):
if not self._router:
self._router = Router()
return self._router
def __finalize(self):
self.loop
self.router
self._reaper = Reaper(self, **self._reaper_settings)
self._matcher = self._router.get_matcher()
def protocol_error_handler(self, error):
print(error)
error = error.encode('utf-8')
response = [
'HTTP/1.0 400 Bad Request\r\n',
'Content-Type: text/plain; charset=utf-8\r\n',
'Content-Length: {}\r\n\r\n'.format(len(error))]
return ''.join(response).encode('utf-8') + error
def default_request_logger(self, request):
print(request.remote_addr, request.method, request.path)
def add_error_handler(self, typ, handler):
self._error_handlers.append((typ, handler))
def default_error_handler(self, request, exception):
if isinstance(exception, RouteNotFoundException):
return request.Response(code=404, text='Not Found')
if isinstance(exception, asyncio.CancelledError):
return request.Response(code=503, text='Service unavailable')
tb = traceback.format_exception(
None, exception, exception.__traceback__)
tb = ''.join(tb)
print(tb, file=sys.stderr, end='')
return request.Response(
code=500,
text=tb if self._debug else 'Internal Server Error')
def error_handler(self, request, exception):
for typ, handler in self._error_handlers:
if typ is not None and not isinstance(exception, typ):
continue
try:
return handler(request, exception)
except:
print('-- Exception in error_handler occured:')
traceback.print_exc()
print('-- while handling:')
traceback.print_exception(None, exception, exception.__traceback__)
return request.Response(
code=500, text='Internal Server Error')
return self.default_error_handler(request, exception)
def _get_idle_and_busy_connections(self):
# FIXME if there is buffered data in gather the connections should be
# considered busy, now it's idle
return \
[c for c in self._connections if c.pipeline_empty], \
[c for c in self._connections if not c.pipeline_empty]
async def drain(self):
# TODO idle connections will close connection with half-read requests
idle, busy = self._get_idle_and_busy_connections()
for c in idle:
c.transport.close()
# for c in busy_connections:
# need to implement something that makes protocol.on_data
# start rejecting incoming data
# this closes transport unfortunately
# sock = c.transport.get_extra_info('socket')
# sock.shutdown(socket.SHUT_RD)
if idle or busy:
print('Draining connections...')
else:
return
if idle:
print('{} idle connections closed immediately'.format(len(idle)))
if busy:
print('{} connections busy, read-end closed'.format(len(busy)))
for x in range(5, 0, -1):
await asyncio.sleep(1)
idle, busy = self._get_idle_and_busy_connections()
for c in idle:
c.transport.close()
if not busy:
break
else:
print(
"{} seconds remaining, {} connections still busy"
.format(x, len(busy)))
_, busy = self._get_idle_and_busy_connections()
if busy:
print('Forcefully killing {} connections'.format(len(busy)))
for c in busy:
c.pipeline_cancel()
def extend_request(self, handler, *, name=None, property=False):
if not name:
name = handler.__name__
self._request_extensions[name] = (handler, property)
def serve(self, *, sock, host, port, reloader_pid):
faulthandler.enable()
self.__finalize()
loop = self.loop
asyncio.set_event_loop(loop)
server_coro = loop.create_server(
lambda: self._protocol_factory(self), sock=sock)
server = loop.run_until_complete(server_coro)
loop.add_signal_handler(signal.SIGTERM, loop.stop)
loop.add_signal_handler(signal.SIGINT, loop.stop)
if reloader_pid:
from japronto.reloader import ChangeDetector
detector = ChangeDetector(loop)
detector.start()
print('Accepting connections on http://{}:{}'.format(host, port))
try:
loop.run_forever()
finally:
server.close()
loop.run_until_complete(server.wait_closed())
loop.run_until_complete(self.drain())
self._reaper.stop()
loop.close()
# break reference and cleanup matcher buffer
del self._matcher
def _run(self, *, host, port, worker_num=None, reloader_pid=None,
debug=None):
self._debug = debug or self._debug
if self._debug and not self._log_request:
self._log_request = self._debug
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
os.set_inheritable(sock.fileno(), True)
workers = set()
terminating = False
def stop(sig, frame):
nonlocal terminating
if reloader_pid and sig == signal.SIGHUP:
print('Reload request received')
elif not terminating:
terminating = True
print('Termination request received')
for worker in workers:
worker.terminate()
signal.signal(signal.SIGINT, stop)
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGHUP, stop)
for _ in range(worker_num or 1):
worker = multiprocessing.Process(
target=self.serve,
kwargs=dict(sock=sock, host=host, port=port,
reloader_pid=reloader_pid))
worker.daemon = True
worker.start()
workers.add(worker)
# prevent further operations on socket in parent
sock.close()
for worker in workers:
worker.join()
if worker.exitcode > 0:
print('Worker exited with code {}'.format(worker.exitcode))
elif worker.exitcode < 0:
try:
signame = signames[-worker.exitcode]
except KeyError:
print(
'Worker crashed with unknown code {}!'
.format(worker.exitcode))
else:
print('Worker crashed on signal {}!'.format(signame))
def run(self, host='0.0.0.0', port=8080, *, worker_num=None, reload=False,
debug=False):
if os.environ.get('_JAPR_IGNORE_RUN'):
return
reloader_pid = None
if reload:
if '_JAPR_RELOADER' not in os.environ:
from japronto.reloader import exec_reloader
exec_reloader(host=host, port=port, worker_num=worker_num)
else:
reloader_pid = int(os.environ['_JAPR_RELOADER'])
self._run(
host=host, port=port, worker_num=worker_num,
reloader_pid=reloader_pid, debug=debug)
================================================
FILE: src/japronto/capsule.c
================================================
#include <Python.h>
void* get_ptr_from_mod(const char* module_name, const char* attr_name,
const char* capsule_name)
{
void* ptr;
PyObject* module = NULL;
PyObject* capsule = NULL;
module = PyImport_ImportModule(module_name);
if(!module)
goto error;
capsule = PyObject_GetAttrString(module, attr_name);
if(!capsule)
goto error;
ptr = PyCapsule_GetPointer(capsule, capsule_name);
if(!ptr)
goto error;
goto finally;
error:
ptr = NULL;
finally:
Py_XDECREF(capsule);
Py_XDECREF(module);
return ptr;
}
PyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name,
const char* capsule_name)
{
PyObject* capsule = NULL;
capsule = PyCapsule_New(ptr, capsule_name, NULL);
if(!capsule)
goto error;
if(PyModule_AddObject(m, attr_name, capsule) == -1)
goto error;
Py_INCREF(capsule);
goto finally;
error:
Py_XDECREF(capsule);
capsule = NULL;
finally:
return capsule;
}
================================================
FILE: src/japronto/capsule.h
================================================
#pragma once
void* get_ptr_from_mod(const char* module_name, const char* attr_name,
const char* capsule_name);
PyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name,
const char* capsule_name);
#define import_capi(module_name) \
get_ptr_from_mod(module_name, "_capi", module_name "._capi")
#define export_capi(m, module_name, capi) \
put_ptr_in_mod(m, capi, "_capi", module_name "._capi")
================================================
FILE: src/japronto/common.h
================================================
#pragma once
typedef enum {
KEEP_ALIVE_UNSET,
KEEP_ALIVE_TRUE,
KEEP_ALIVE_FALSE
} KEEP_ALIVE;
================================================
FILE: src/japronto/cpu_features.c
================================================
#include <cpuid.h>
#include <assert.h>
#include "cpu_features.h"
int supports_x86_sse42(void)
{
#if defined(__clang__)
unsigned int eax = 0, ebx = 0, ecx = 0, edx = 0;
__get_cpuid(1, &eax, &ebx, &ecx, &edx);
return ecx & bit_SSE42;
#else
__builtin_cpu_init();
return __builtin_cpu_supports("sse4.2");
#endif
}
================================================
FILE: src/japronto/cpu_features.h
================================================
#pragma once
int supports_x86_sse42(void);
================================================
FILE: src/japronto/parser/.gitignore
================================================
libpicohttpparser.c
================================================
FILE: src/japronto/parser/__init__.py
================================================
header_errors = [
'malformed_headers', 'incomplete_headers', 'invalid_headers',
'excessive_data']
body_errors = ['malformed_body', 'incomplete_body']
================================================
FILE: src/japronto/parser/build_libpicohttpparser.py
================================================
import distutils.log
distutils.log.set_verbosity(distutils.log.DEBUG)
import os.path
import cffi
ffibuilder = cffi.FFI()
shared_path = os.path.abspath(os.path.join(os.path.dirname(__file__),
'../../picohttpparser'))
print(shared_path)
ffibuilder.set_source("libpicohttpparser", """
#include "picohttpparser.h"
""", libraries=['picohttpparser'], include_dirs=[shared_path],
library_dirs=[shared_path],
extra_link_args=['-Wl,-rpath=' + shared_path])
# extra_objects=[os.path.join(shared_path, 'picohttpparser.o')],
# or a list of libraries to link with
# (more arguments like setup.py's Extension class:
# include_dirs=[..], extra_objects=[..], and so on)
ffibuilder.cdef("""
struct phr_header {
const char *name;
size_t name_len;
const char *value;
size_t value_len;
};
struct phr_chunked_decoder {
size_t bytes_left_in_chunk; /* number of bytes left in current chunk */
char consume_trailer; /* if trailing headers should be consumed */
char _hex_count;
char _state;
};
int phr_parse_request(const char *buf, size_t len, const char **method,
size_t *method_len, const char **path,
size_t *path_len, int *minor_version,
struct phr_header *headers, size_t *num_headers,
size_t last_len);
ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf,
size_t *bufsz);
""")
if __name__ == "__main__":
ffibuilder.compile(verbose=True)
================================================
FILE: src/japronto/parser/cffiparser.py
================================================
from parser.libpicohttpparser import ffi, lib
class HttpRequestParser(object):
def __init__(self, on_headers, on_body, on_error):
self.on_headers = on_headers
self.on_body = on_body
self.on_error = on_error
self.c_method = ffi.new('char **')
self.method_len = ffi.new('size_t *')
self.c_path = ffi.new('char **')
self.path_len = ffi.new('size_t *')
self.minor_version = ffi.new('int *')
self.c_headers = ffi.new('struct phr_header[10]')
self.num_headers = ffi.new('size_t *')
self.chunked_offset = ffi.new('size_t*')
self._reset_state(True)
def _reset_state(self, disconnect=False):
self.state = 'headers'
self.transfer = None
self.content_length = None
self.chunked_decoder = None
self.chunked_offset[0] = 0
if disconnect:
self.connection = None
self.buffer = bytearray()
def _parse_headers(self):
if self.connection == 'close':
self.on_error('excessive_data')
self._reset_state(True)
return -1
self.num_headers[0] = 10
# FIXME: More than 10 headers
result = lib.phr_parse_request(
ffi.from_buffer(self.buffer), len(self.buffer),
self.c_method, self.method_len,
self.c_path, self.path_len,
self.minor_version, self.c_headers, self.num_headers, 0)
if result == -2:
return result
elif result == -1:
self.on_error('malformed_headers')
self._reset_state(True)
return result
else:
self._reset_state()
method = ffi.cast(
'char[{}]'.format(self.method_len[0]), self.c_method[0])
path = ffi.cast(
'char[{}]'.format(self.path_len[0]), self.c_path[0])
headers = ffi.cast(
"struct phr_header[{}]".format(self.num_headers[0]),
self.c_headers)
if ffi.buffer(method)[:] in (b'GET', b'DELETE', b'HEAD'):
self.no_semantics = True
if self.minor_version[0] == 0:
self.connection = 'close'
else:
self.connection = 'keep-alive'
for header in headers:
header_name = ffi.string(header.name, header.name_len).title()
# maybe len + strcasecmp C style is faster?
if header_name == b'Transfer-Encoding':
self.transfer = ffi.string(
header.value, header.value_len).decode('ascii')
# FIXME comma separated and invalid values
elif header_name == b'Connection':
self.connection = ffi.string(
header.value, header.value_len).decode('ascii')
# FIXME other options for Connection like updgrade
elif header_name == b'Content-Length':
content_length_error = False
if not header.value_len:
content_length_error = True
if not content_length_error:
content_length = ffi.buffer(header.value, header.value_len)
if not content_length_error and content_length[0] in b'+-':
content_length_error = True
if not content_length_error:
try:
self.content_length = int(content_length[:])
except ValueError:
content_length_error = True
if content_length_error:
self.on_error('invalid_headers')
self._reset_state(True)
return -1
self.on_headers(method, path, self.minor_version[0], headers)
self.buffer = self.buffer[result:]
return result
def _parse_body(self):
if self.content_length is None and self.transfer is None:
self.on_body(None)
return 0
elif self.content_length == 0:
self.on_body(ffi.from_buffer(b""))
return 0
elif self.content_length is not None:
if self.content_length > len(self.buffer):
return -2
body = memoryview(self.buffer)[:self.content_length]
self.on_body(ffi.from_buffer(body))
self.buffer = self.buffer[self.content_length:]
result = self.content_length
return result
elif self.transfer == 'chunked':
if not self.chunked_decoder:
self.chunked_decoder = ffi.new('struct phr_chunked_decoder*')
self.chunked_decoder.consume_trailer = b'\x01'
chunked_offset_start = self.chunked_offset[0]
self.chunked_offset[0] = len(self.buffer) - self.chunked_offset[0]
result = lib.phr_decode_chunked(
self.chunked_decoder,
ffi.from_buffer(self.buffer) + chunked_offset_start,
self.chunked_offset)
self.chunked_offset[0] = self.chunked_offset[0] \
+ chunked_offset_start
if result == -2:
self.buffer = self.buffer[:self.chunked_offset[0]]
return result
elif result == -1:
self.on_error('malformed_body')
self._reset_state(True)
return result
body = memoryview(self.buffer)[:self.chunked_offset[0]]
self.on_body(ffi.from_buffer(body))
self.buffer = self.buffer[
self.chunked_offset[0]:self.chunked_offset[0] + result]
self._reset_state()
return result
def feed(self, data):
self.buffer += data
while self.buffer:
if self.state == 'headers':
result = self._parse_headers()
if result <= 0:
return None
self.state = 'body'
if self.state == 'body':
result = self._parse_body()
if result < 0:
return None
self.state = 'headers'
def feed_disconnect(self):
if self.state == 'headers' and self.buffer:
self.on_error('incomplete_headers')
elif self.state == 'body':
self.on_error('incomplete_body')
self._reset_state(True)
================================================
FILE: src/japronto/parser/cparser.c
================================================
#include <strings.h>
#include <sys/param.h>
#include "cparser.h"
#include "cpu_features.h"
#ifndef PARSER_STANDALONE
#include "cprotocol.h"
#endif
static PyObject* malformed_headers;
static PyObject* malformed_body;
static PyObject* incomplete_headers;
static PyObject* invalid_headers;
static PyObject* incomplete_body;
static PyObject* excessive_data;
//static PyObject* empty_body;
const char zero_body[] = "";
/*static PyObject* GET;
static PyObject* POST;
static PyObject* DELETE;
static PyObject* HEAD;
static PyObject* Host;
static PyObject* User_Agent;
static PyObject* Accept;
static PyObject* Accept_Language;
static PyObject* Accept_Encoding;
static PyObject* Accept_Charset;
static PyObject* Connection;
static PyObject* Cookie;
static PyObject* Content_Length;
static PyObject* Transfer_Encoding;
static PyObject* val_close;
static PyObject* keep_alive;*/
static unsigned long const CONTENT_LENGTH_UNSET = ULONG_MAX;
static void _reset_state(Parser* self, bool disconnect) {
self->state = PARSER_HEADERS;
self->transfer = PARSER_TRANSFER_UNSET;
self->content_length = CONTENT_LENGTH_UNSET;
memset(&self->chunked_decoder, 0, sizeof(struct phr_chunked_decoder));
self->chunked_decoder.consume_trailer = 1;
self->chunked_offset = 0;
if(disconnect) {
self->connection = PARSER_CONNECTION_UNSET;
self->buffer_start = 0;
self->buffer_end = 0;
}
}
#ifdef PARSER_STANDALONE
static PyObject *
Parser_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
#else
void
Parser_new(Parser* self)
#endif
{
#ifdef PARSER_STANDALONE
Parser *self = NULL;
self = (Parser *)type->tp_alloc(type, 0);
if (!self)
goto finally;
self->on_headers = NULL;
self->on_body = NULL;
self->on_error = NULL;
#endif
#ifdef PARSER_STANDALONE
finally:
return (PyObject *)self;
#endif
}
#ifdef PARSER_STANDALONE
static int
Parser_init(Parser *self, PyObject *args, PyObject *kwds)
#else
int
Parser_init(Parser* self, void* protocol)
#endif
{
#ifdef PARSER_STANDALONE
#ifdef DEBUG_PRINT
printf("__init__\n");
#endif
// FIXME: __init__ can be called many times
// FIXME: check argument types
int result = PyArg_ParseTuple(
args, "OOO", &self->on_headers, &self->on_body, &self->on_error);
if(!result)
return -1;
Py_INCREF(self->on_headers);
Py_INCREF(self->on_body);
Py_INCREF(self->on_error);
#else
self->protocol = protocol;
#endif
_reset_state(self, true);
self->buffer_capacity = PARSER_INITIAL_BUFFER_SIZE;
self->buffer = self->inline_buffer;
return 0;
}
#ifdef PARSER_STANDALONE
static void
Parser_dealloc(Parser* self)
#else
void
Parser_dealloc(Parser* self)
#endif
{
#ifdef PARSER_STANDALONE
#ifdef DEBUG_PRINT
printf("__del__\n");
#endif
#endif
if(self->buffer != self->inline_buffer)
free(self->buffer);
#ifdef PARSER_STANDALONE
Py_XDECREF(self->on_error);
Py_XDECREF(self->on_body);
Py_XDECREF(self->on_headers);
Py_TYPE(self)->tp_free((PyObject*)self);
#endif
}
static int
(*_phr_parse_request)(
const char *, size_t, const char **, size_t *, const char **, size_t *,
int *, struct phr_header *, size_t *, size_t);
static int _parse_headers(Parser* self) {
#ifdef PARSER_STANDALONE
PyObject* method_view = NULL;
PyObject* path_view = NULL;
PyObject* minor_version_long = NULL;
PyObject* headers_view = NULL;
#endif
PyObject* error;
int result = -1;
if(self->connection == PARSER_CLOSE) {
error = excessive_data;
goto on_error;
}
char* method;
size_t method_len;
char* path;
size_t path_len;
int minor_version;
struct phr_header headers[50];
size_t num_headers = sizeof(headers) / sizeof(struct phr_header);
result = _phr_parse_request(
self->buffer + self->buffer_start, self->buffer_end - self->buffer_start,
(const char**)&method, &method_len,
(const char**)&path, &path_len,
&minor_version, headers, &num_headers, 0);
// FIXME: More than 10 headers
#ifdef DEBUG_PRINT
printf("result: %d\n", result);
#endif
if(result == -2)
goto finally;
if(result == -1) {
error = malformed_headers;
goto on_error;
}
if(minor_version == 0) {
self->connection = PARSER_CLOSE;
} else {
self->connection = PARSER_KEEP_ALIVE;
}
#define header_name_equal(val) \
header->name_len == strlen(val) && strncasecmp(header->name, val, header->name_len) == 0
#define header_value_equal(val) \
header->value_len == strlen(val) && strncasecmp(header->value, val, header->value_len) == 0
/*#define cmp_and_set_header_name(name, val) \
if(header_name_equal(val)) { \
py_header_name = name; \
Py_INCREF(name); \
}
#define cmp_and_set_header_value(name, val) \
if(header_value_equal(val)) { \
py_header_value = name; \
Py_INCREF(name); \
}*/
for(struct phr_header* header = headers;
header < headers + num_headers;
header++) {
// TODO: common names and values static
/*PyObject* py_header_name = NULL;
PyObject* py_header_value = NULL;*/
if(header_name_equal("Transfer-Encoding")) {
if(header_value_equal("chunked"))
self->transfer = PARSER_CHUNKED;
else if(header_value_equal("identity"))
self->transfer = PARSER_IDENTITY;
else
/*TODO: handle incorrept values for protocol version, also comma sep*/;
/*py_header_name = Transfer_Encoding;
Py_INCREF(Transfer_Encoding);*/
} else if(header_name_equal("Content-Length")) {
if(!header->value_len) {
error = invalid_headers;
goto on_error;
}
if(*header->value == '+' || *header->value == '-') {
error = invalid_headers;
goto on_error;
}
char * endptr = (char *)header->value + header->value_len;
self->content_length = strtol(header->value, &endptr, 10);
// FIXME: overflow?
if(endptr != (char*)header->value + header->value_len) {
error = invalid_headers;
goto on_error;
}
} else if(header_name_equal("Connection")) {
if(header_value_equal("close"))
self->connection = PARSER_CLOSE;
else if(header_value_equal("keep-alive"))
self->connection = PARSER_KEEP_ALIVE;
else
/* FIXME: on_error*/;
/*py_header_name = Content_Length;
Py_INCREF(Content_Length);*/
}
/*else cmp_and_set_header_name(Host, "Host")
else cmp_and_set_header_name(User_Agent, "User-Agent")
else cmp_and_set_header_name(Accept, "Accept")
else cmp_and_set_header_name(Accept_Language, "Accept-Language")
else cmp_and_set_header_name(Accept_Encoding, "Accept-Encoding")
else cmp_and_set_header_name(Accept_Charset, "Accept-Charset")
else cmp_and_set_header_name(Connection, "Connection")
else cmp_and_set_header_name(Cookie, "Cookie")
else {
bool prev_alpha = false;
for(char* c = (char*)header.name; c < header.name + header.name_len; c++) {
if(*c >= 'A' && *c <= 'Z') {
if(prev_alpha)
*c |= 0x20;
prev_alpha = true;
} else if (*c >= 'a' && *c <= 'z')
prev_alpha = true;
else
prev_alpha = false;
}
// FIXME this should accept only ascii
py_header_name = PyUnicode_FromStringAndSize(
header.name, header.name_len);
if(!py_header_name) {
result = -3;
goto finally_loop;
}
}
if(py_header_name == Connection) {
cmp_and_set_header_value(keep_alive, "keep-alive")
else cmp_and_set_header_value(val_close, "close")
else FIXME: invalid Connection value;
} else {
// FIXME: this can return NULL on codec error
py_header_value = PyUnicode_DecodeLatin1(
header.value, header.value_len, NULL);
if(!py_header_value) {
result = -3;
goto finally_loop;
}
}
if(PyDict_SetItem(py_headers, py_header_name, py_header_value) == -1)
result = -3;
#ifdef DEBUG_PRINT
PyObject_Print(py_header_name, stdout, 0); printf(": ");
PyObject_Print(py_header_value, stdout, 0); printf("\n");
#endif
finally_loop:
Py_XDECREF(py_header_value);
Py_XDECREF(py_header_name);
if(result == -3)
goto finally;*/
}
#ifdef DEBUG_PRINT
if(self->content_length != CONTENT_LENGTH_UNSET)
printf("self->content_length: %ld\n", self->content_length);
if(self->transfer == PARSER_IDENTITY)
printf("self->transfer: identity\n");
else if(self->transfer == PARSER_CHUNKED)
printf("self->transfer: chunked\n");
#endif
#ifdef PARSER_STANDALONE
method_view = PyMemoryView_FromMemory(method, method_len, PyBUF_READ);
path_view = PyMemoryView_FromMemory(path, path_len, PyBUF_READ);
minor_version_long = PyLong_FromLong(minor_version);
headers_view = PyMemoryView_FromMemory((char*)headers, sizeof(struct phr_header) * num_headers, PyBUF_READ);
// FIXME the functions above can fail
PyObject* on_headers_result = PyObject_CallFunctionObjArgs(
self->on_headers, method_view, path_view, minor_version_long, headers_view, NULL);
if(!on_headers_result)
goto error;
Py_DECREF(on_headers_result);
#else
if(!Protocol_on_headers(
self->protocol, method, method_len,
path, path_len, minor_version, headers, num_headers))
goto error;
#endif
self->buffer_start += (size_t)result;
goto finally;
#ifdef PARSER_STANDALONE
PyObject* on_error_result;
on_error:
on_error_result = PyObject_CallFunctionObjArgs(
self->on_error, error, NULL);
if(!on_error_result)
goto error;
Py_DECREF(on_error_result);
#else
on_error:
if(!Protocol_on_error(self->protocol, error))
goto error;
#endif
_reset_state(self, true);
result = -1;
goto finally;
error:
result = -3;
finally:
#ifdef PARSER_STANDALONE
Py_XDECREF(headers_view);
Py_XDECREF(minor_version_long);
Py_XDECREF(path_view);
Py_XDECREF(method_view);
#endif
return result;
}
static int _parse_body(Parser* self) {
#ifdef PARSER_STANDALONE
PyObject* body_view = NULL;
#endif
char* body = NULL;
size_t body_len = 0;
int result = -2;
if(self->content_length == CONTENT_LENGTH_UNSET
&& self->transfer == PARSER_TRANSFER_UNSET) {
result = 0;
goto on_body;
}
if(self->content_length == 0) {
body = (char*)zero_body;
result = 0;
goto on_body;
}
if(self->content_length != CONTENT_LENGTH_UNSET) {
if(self->content_length > self->buffer_end - self->buffer_start) {
result = -2;
goto finally;
}
body = self->buffer + self->buffer_start;
body_len = self->content_length;
self->buffer_start += self->content_length;
// TODO result = self->content_length (long)
result = 1;
goto on_body;
}
if(self->transfer == PARSER_CHUNKED) {
size_t chunked_offset_start = self->chunked_offset;
self->chunked_offset = self->buffer_end - self->buffer_start - self->chunked_offset;
result = phr_decode_chunked(
&self->chunked_decoder,
self->buffer + self->buffer_start + chunked_offset_start,
&self->chunked_offset);
self->chunked_offset = self->chunked_offset + chunked_offset_start;
if(result == -2) {
self->buffer_end = self->buffer_start + self->chunked_offset;
goto finally;
}
if(result == -1)
goto on_error;
body = self->buffer + self->buffer_start;
body_len = self->chunked_offset;
self->buffer_start += self->chunked_offset;
self->buffer_end = self->buffer_start + (size_t)result;
goto on_body;
}
goto finally;
on_body:
if(body) {
#if 0
if(PyObject_SetAttrString(self->request, "body", body) == -1) {
result = -3;
goto finally;
}
#else
/*((Request*)(self->request))->body = body;
Py_INCREF(body);*/
#endif
#ifdef DEBUG_PRINT
printf("body: "); PyObject_Print(body, stdout, 0); printf("\n");
#endif
}
#ifdef PARSER_STANDALONE
if(body) {
body_view = PyMemoryView_FromMemory(body, body_len, PyBUF_READ);
if(!body_view)
goto error;
} else {
body_view = Py_None;
Py_INCREF(body_view);
}
PyObject* on_body_result = PyObject_CallFunctionObjArgs(
self->on_body, body_view, NULL);
if(!on_body_result)
goto error;
Py_DECREF(on_body_result);
#else
if(!Protocol_on_body(self->protocol, body, body_len, self->buffer_end - self->buffer_start))
goto error;
#endif
_reset_state(self, false);
goto finally;
#ifdef PARSER_STANDALONE
PyObject* on_error_result;
on_error:
on_error_result = PyObject_CallFunctionObjArgs(
self->on_error, malformed_body, NULL);
if(!on_error_result)
goto error;
Py_DECREF(on_error_result);
#else
on_error:
if(!Protocol_on_error(self->protocol, malformed_body))
goto error;
#endif
_reset_state(self, true);
result = -1;
goto finally;
error:
result = -3;
finally:
#ifdef PARSER_STANDALONE
Py_XDECREF(body_view);
#endif
return result;
}
#ifdef PARSER_STANDALONE
static PyObject *
Parser_feed(Parser* self, PyObject *args)
#else
Parser*
Parser_feed(Parser* self, PyObject* py_data)
#endif
{
char* data;
int iresult = 0;
#ifdef PARSER_STANDALONE
PyObject* result = Py_None;
// FIXME: can be called without __init__
#ifdef DEBUG_PRINT
printf("feed\n");
#endif
int data_len;
if(!PyArg_ParseTuple(args, "y#", &data, &data_len))
goto error;
#else
Parser* result = self;
Py_ssize_t data_len;
if(PyBytes_AsStringAndSize(py_data, &data, &data_len) == -1)
goto error;
#endif
if(self->buffer_start == self->buffer_end) {
self->buffer_start = 0;
self->buffer_end = 0;
} else if((size_t)data_len > self->buffer_capacity - self->buffer_end) {
memmove(self->buffer, self->buffer + self->buffer_start, self->buffer_end - self->buffer_start);
self->buffer_end -= self->buffer_start;
self->buffer_start = 0;
}
if((size_t)data_len > self->buffer_capacity - (self->buffer_end - self->buffer_start)) {
self->buffer_capacity = MAX(
self->buffer_capacity * 2,
self->buffer_end - self->buffer_start + data_len);
if(self->buffer == self->inline_buffer) {
self->buffer = malloc(self->buffer_capacity);
memcpy(self->buffer + self->buffer_start,
self->inline_buffer + self->buffer_start,
self->buffer_end - self->buffer_start);
} else
self->buffer = realloc(self->buffer, self->buffer_capacity);
if(!self->buffer)
goto error;
}
memcpy(self->buffer + self->buffer_end, data, (size_t)data_len);
self->buffer_end += (size_t)data_len;
while(self->buffer_start != self->buffer_end) {
if(self->state == PARSER_HEADERS) {
iresult = _parse_headers(self);
if(iresult == -3)
goto error;
if(iresult <= 0)
break;
self->state = PARSER_BODY;
}
if(self->state == PARSER_BODY) {
iresult = _parse_body(self);
if(iresult == -3)
goto error;
if(iresult < 0)
break;
self->state = PARSER_HEADERS;
}
}
#ifndef PARSER_STANDALONE
if(iresult == -2)
Protocol_on_incomplete(self->protocol);
#endif
goto finally;
error:
result = NULL;
finally:
#ifdef PARSER_STANDALONE
if(result)
Py_INCREF(result);
#endif
return result;
}
#ifdef PARSER_STANDALONE
static PyObject *
Parser_feed_disconnect(Parser* self)
#else
Parser*
Parser_feed_disconnect(Parser* self)
#endif
{
// FIXME: can be called without __init__
#ifdef DEBUG_PRINT
printf("feed_disconnect\n");
#endif
PyObject* error;
if(self->state == PARSER_HEADERS
&& self->buffer_start != self->buffer_end) {
error = incomplete_headers;
goto on_error;
}
if(self->state == PARSER_BODY) {
error = incomplete_body;
goto on_error;
}
goto finally;
#ifdef PARSER_STANDALONE
PyObject* on_error_result;
on_error:
on_error_result = PyObject_CallFunctionObjArgs(
self->on_error, error, NULL);
if(!on_error_result)
return NULL; /*FIXME maybe leak */
Py_DECREF(on_error_result);
#else
on_error:
if(!Protocol_on_error(self->protocol, error)) {
return NULL; /*FIXME maybe leak */
}
#endif
finally:
_reset_state(self, true);
#ifdef PARSER_STANDALONE
Py_RETURN_NONE;
#else
return self;
#endif
}
#ifdef PARSER_STANDALONE
static PyObject *
Parser_dump_buffer(Parser* self) {
// printf("buffer: "); PyObject_Print(self->buffer, stdout, 0); printf("\n");
Py_RETURN_NONE;
}
static PyMethodDef Parser_methods[] = {
{"feed", (PyCFunction)Parser_feed, METH_VARARGS, "feed"},
{"feed_disconnect", (PyCFunction)Parser_feed_disconnect,
METH_NOARGS,
"feed_disconnect"
},
{
"_dump_buffer", (PyCFunction)Parser_dump_buffer,
METH_NOARGS,
"_dump_buffer"
},
{NULL} /* Sentinel */
};
static PyTypeObject ParserType = {
PyVarObject_HEAD_INIT(NULL, 0)
"cparser.HttpRequestParser", /* tp_name */
sizeof(Parser), /* tp_basicsize */
0, /* tp_itemsize */
(destructor)Parser_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
0, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"HttpRequestParser", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Parser_methods, /* tp_methods */
0, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc)Parser_init, /* tp_init */
0, /* tp_alloc */
Parser_new, /* tp_new */
};
static PyModuleDef cparser = {
PyModuleDef_HEAD_INIT,
"cparser",
"cparser",
-1,
NULL, NULL, NULL, NULL, NULL
};
#endif
#ifdef PARSER_STANDALONE
PyMODINIT_FUNC
PyInit_cparser(void)
#else
int
cparser_init(void)
#endif
{
if(supports_x86_sse42()) {
_phr_parse_request = phr_parse_request_sse42;
} else {
printf("Warning: Host CPU doesn't support SSE 4.2, selecting slower implementation\n");
_phr_parse_request = phr_parse_request;
}
malformed_headers = NULL;
invalid_headers = NULL;
malformed_body = NULL;
incomplete_headers = NULL;
incomplete_body = NULL;
excessive_data = NULL;
/*empty_body = NULL;
GET = NULL;
POST = NULL;
DELETE = NULL;
HEAD = NULL;
Host = NULL;
User_Agent = NULL;
Accept = NULL;
Accept_Language = NULL;
Accept_Encoding = NULL;
Accept_Charset = NULL;
Connection = NULL;
Cookie = NULL;
Content_Length = NULL;
Transfer_Encoding = NULL;
val_close = NULL;
keep_alive = NULL;*/
#ifdef PARSER_STANDALONE
PyObject* m = NULL;
#else
int m = 0;
#endif
#ifdef PARSER_STANDALONE
if (PyType_Ready(&ParserType) < 0)
goto error;
m = PyModule_Create(&cparser);
if (!m)
goto error;
#endif
#define alloc_static(name) \
name = PyUnicode_FromString(#name); \
if(!name) \
goto error;
#define alloc_static2(name, val) \
name = PyUnicode_FromString(val); \
if(!name) \
goto error;
alloc_static(malformed_headers)
alloc_static(malformed_body)
alloc_static(incomplete_headers)
alloc_static(invalid_headers)
alloc_static(incomplete_body)
alloc_static(excessive_data)
/*empty_body = PyBytes_FromString("");
if(!empty_body)
goto error;
alloc_static(GET)
alloc_static(POST)
alloc_static(DELETE)
alloc_static(HEAD)
alloc_static(Host)
alloc_static2(User_Agent, "User-Agent")
alloc_static(Accept)
alloc_static2(Accept_Language, "Accept-Language")
alloc_static2(Accept_Encoding, "Accept-Encoding")
alloc_static2(Accept_Charset, "Accept-Charset")
alloc_static(Connection)
alloc_static(Cookie)
alloc_static2(Content_Length, "Content-Length")
alloc_static2(Transfer_Encoding, "Transfer-Encoding")
alloc_static2(val_close, "close")
alloc_static2(keep_alive, "keep-alive")*/
#undef alloc_static
#undef alloc_static2
#ifdef PARSER_STANDALONE
Py_INCREF(&ParserType);
PyModule_AddObject(
m, "HttpRequestParser", (PyObject *)&ParserType);
#endif
goto finally;
error:
/*Py_XDECREF(keep_alive);
Py_XDECREF(val_close);
Py_XDECREF(Transfer_Encoding);
Py_XDECREF(Content_Length);
Py_XDECREF(Cookie);
Py_XDECREF(Connection);
Py_XDECREF(Accept_Charset);
Py_XDECREF(Accept_Encoding);
Py_XDECREF(Accept_Language);
Py_XDECREF(Accept);
Py_XDECREF(User_Agent);
Py_XDECREF(Host);
Py_XDECREF(HEAD);
Py_XDECREF(DELETE);
Py_XDECREF(POST);
Py_XDECREF(GET);
Py_XDECREF(empty_body);*/
Py_XDECREF(incomplete_body);
Py_XDECREF(invalid_headers);
Py_XDECREF(incomplete_headers);
Py_XDECREF(malformed_body);
Py_XDECREF(malformed_headers);
#ifndef PARSER_STANDALONE
m = -1;
#endif
finally:
return m;
}
================================================
FILE: src/japronto/parser/cparser.h
================================================
#pragma once
#include <stdbool.h>
#include <Python.h>
#include "picohttpparser.h"
enum Parser_state {
PARSER_HEADERS,
PARSER_BODY
};
enum Parser_transfer {
PARSER_TRANSFER_UNSET,
PARSER_IDENTITY,
PARSER_CHUNKED
};
enum Parser_connection {
PARSER_CONNECTION_UNSET,
PARSER_CLOSE,
PARSER_KEEP_ALIVE
};
#define PARSER_INITIAL_BUFFER_SIZE 4096
typedef struct {
#ifdef PARSER_STANDALONE
PyObject_HEAD
#endif
enum Parser_state state;
enum Parser_transfer transfer;
enum Parser_connection connection;
unsigned long content_length;
struct phr_chunked_decoder chunked_decoder;
size_t chunked_offset;
char* buffer;
size_t buffer_start;
size_t buffer_end;
size_t buffer_capacity;
char inline_buffer[PARSER_INITIAL_BUFFER_SIZE];
#ifdef PARSER_STANDALONE
PyObject* on_headers;
PyObject* on_body;
PyObject* on_error;
#else
void* protocol;
#endif
} Parser;
#ifndef PARSER_STANDALONE
void
Parser_new(Parser* self);
int
Parser_init(Parser* self, void* protocol);
void
Parser_dealloc(Parser* self);
Parser*
Parser_feed(Parser* self, PyObject* py_data);
Parser*
Parser_feed_disconnect(Parser* self);
int
cparser_init(void);
#endif
================================================
FILE: src/japronto/parser/cparser_ext.py
================================================
from distutils.core import Extension
def get_extension():
return Extension(
'japronto.parser.cparser',
sources=['cparser.c', '../cpu_features.c'],
include_dirs=['../../picohttpparser', '..'],
extra_objects=[
'src/picohttpparser/picohttpparser.o',
'src/picohttpparser/ssepicohttpparser.o'],
define_macros=[('PARSER_STANDALONE', 1)])
================================================
FILE: src/japronto/parser/test_parser.py
================================================
from functools import partial
from itertools import zip_longest
import pytest
from cases import parametrize_cases
from parts import one_part, make_parts, geometric_series, fancy_series
from protocol.tracing import CTracingProtocol, CffiTracingProtocol
from parser import cffiparser, header_errors, body_errors
try:
from parser import cparser
except ImportError:
cparser = None
if cparser:
def make_c(protocol_factory=CTracingProtocol):
protocol = protocol_factory()
parser = cparser.HttpRequestParser(
protocol.on_headers, protocol.on_body, protocol.on_error)
return parser, protocol
def make_cffi(protocol_factory=CffiTracingProtocol):
protocol = protocol_factory()
parser = cffiparser.HttpRequestParser(
protocol.on_headers, protocol.on_body, protocol.on_error)
return parser, protocol
@pytest.mark.parametrize('data,get_size,dir,parts', [
(b'abcde', 2, 1, [b'ab', b'cd', b'e']),
(b'abcde', 2, -1, [b'a', b'bc', b'de']),
(b'aaBBBBccccCCCCd', geometric_series(), 1,
[b'aa', b'BBBB', b'ccccCCCC', b'd']),
(b'dCCCCccccBBBBaa', geometric_series(), -1,
[b'd', b'CCCCcccc', b'BBBB', b'aa'])
])
def test_make_parts(data, get_size, dir, parts):
assert make_parts(data, get_size, dir) == parts
def parametrize_make_parser():
ids = []
factories = []
if 'make_c' in globals():
factories.append(make_c)
ids.append('c')
factories.append(make_cffi)
ids.append('cffi')
return pytest.mark.parametrize('make_parser', factories, ids=ids)
def parametrize_do_parts():
funcs = [
one_part,
partial(make_parts, get_size=15),
partial(make_parts, get_size=geometric_series()),
partial(make_parts, get_size=geometric_series(), dir=-1),
partial(make_parts, get_size=fancy_series())
]
ids = ['one', 'const', 'geom', 'invgeom', 'fancy']
return pytest.mark.parametrize('do_parts', funcs, ids=ids)
_begin = object()
_end = object()
@parametrize_do_parts()
@parametrize_cases(
'base',
'10msg', '10msg!', '10get', '10get!', 'keep:10msg+10get',
'keep:10get+10msg',
'10malformed_headers1', '10malformed_headers2', '10incomplete_headers!',
'keep:10msg+10malformed_headers2', 'keep:10msg+10incomplete_headers!',
'keep:10get+10malformed_headers1', 'keep:10get+10malformed_headers2',
'10msg!+10get!', '10get!+10msg!',
'10msg!+keep:10get+keep:10msg+10get',
'10msg+e excessive_data:10get', '10get+e excessive_data:10msg')
@parametrize_make_parser()
def test_http10(make_parser, do_parts, cases):
parser, protocol = make_parser()
def flush():
nonlocal data
if not data:
return
parts = do_parts(data)
for part in parts:
parser.feed(part)
if protocol.error:
break
data = b''
data = b''
for case in cases:
data += case.data
if case.disconnect:
flush()
parser.feed_disconnect()
flush()
header_count = 0
error_count = 0
body_count = 0
for case, request in zip_longest(cases, protocol.requests):
if case.error:
assert protocol.error == case.error
if case.error in header_errors:
error_count += 1
break
header_count += 1
assert request.method == case.method
assert request.path == case.path
assert request.version == case.version
assert request.headers == case.headers
if case.error in body_errors:
error_count += 1
break
body_count += 1
assert request.body == case.body
assert protocol.on_headers_call_count == header_count
assert protocol.on_error_call_count == error_count
assert protocol.on_body_call_count == body_count
@parametrize_make_parser()
def test_empty(make_parser):
parser, protocol = make_parser()
parser.feed_disconnect()
parser.feed(b'')
parser.feed(b'')
parser.feed_disconnect()
parser.feed_disconnect()
parser.feed(b'')
assert not protocol.on_headers_call_count
assert not protocol.on_error_call_count
assert not protocol.on_body_call_count
@parametrize_do_parts()
@parametrize_cases(
'base',
'11get', '11getmsg', '11msg', '11msgzero', 'close:11get', 'close:11msg',
'11get!', '11getmsg!', '11msg!', 'close:11msgzero!',
'11msg+close:11msg', '11msg+11msg',
'close:11msg!+11msg', 'close:11msg!+close:11msg',
'11msg!+close:11msg', '11msg!+11msg',
'11get+close:11msg', '11msg+11get', '11getmsg+11get',
'11get+close:11msg!', '11msg!+11get', '11getmsg!+11get!',
'11msg+11msg+close:11msg',
'11msg+11msg+11msg',
'11msg+11msgzero+11msg',
'11msgzero+11msg+11msgzero',
'11msg+11get+11msgzero',
'11msgzero+11msgzero',
'11get+11getmsg+11get',
'close:11msg+e excessive_data:11msg',
'close:11msg+e excessive_data:close:11msg',
'close:11msg+e excessive_data:close:11msg+11msg',
'11msg+close:11msgzero+e excessive_data:11get',
'11clincomplete_headers!', '11clincomplete_body!',
'11clinvalid1', '11clinvalid2', '11clinvalid3',
'11clinvalid4', '11clinvalid5',
'11msg+11clincomplete_headers!', 'close:11msg!+11clincomplete_body!',
'11msgzero+11clincomplete_headers!', '11msgzero+11clincomplete_body!',
'close:11msg!+11msg+11clincomplete_body!',
'11get+11clincomplete_body!',
'11getmsg+11clincomplete_headers!'
)
@parametrize_make_parser()
def test_http11(make_parser, do_parts, cases):
parser, protocol = make_parser()
def flush():
nonlocal data
if not data:
return
parts = do_parts(data)
for part in parts:
parser.feed(part)
if protocol.error:
break
data = b''
data = b''
for case in cases:
data += case.data
if case.disconnect:
flush()
parser.feed_disconnect()
flush()
header_count = 0
error_count = 0
body_count = 0
for case, request in zip_longest(cases, protocol.requests):
if case.error:
assert protocol.error == case.error
if case.error in header_errors:
error_count += 1
break
header_count += 1
assert request.method == case.method
assert request.path == case.path
assert request.version == case.version
assert request.headers == case.headers
if case.error in body_errors:
error_count += 1
break
body_count += 1
assert request.body == case.body
assert protocol.on_headers_call_count == header_count
assert protocol.on_error_call_count == error_count
assert protocol.on_body_call_count == body_count
@parametrize_do_parts()
@parametrize_cases(
'base',
'11chunked1', '11chunked2', '11chunked3', '11chunkedzero',
'11chunked1+11chunked1',
'11chunked1+11chunked2',
'11chunked2+11chunked1',
'11chunked2+11chunked3',
'11chunked1+11chunked2+11chunked3',
'11chunked3+11chunked2+11chunked1',
'11chunked3+11chunked3+11chunked3',
'11chunkedincomplete_body!', '11chunkedmalformed_body',
'11chunked1+11chunkedincomplete_body!',
'11chunked1+11chunkedmalformed_body',
'11chunked2+11chunkedincomplete_body!',
'11chunked2+11chunkedmalformed_body',
'11chunked2+11chunked2+11chunkedincomplete_body!',
'11chunked3+11chunked1+11chunkedmalformed_body'
)
@parametrize_make_parser()
def test_http11_chunked(make_parser, do_parts, cases):
parser, protocol = make_parser()
def flush():
nonlocal data
if not data:
return
parts = do_parts(data)
for part in parts:
parser.feed(part)
if protocol.error:
break
data = b''
data = b''
for case in cases:
data += case.data
if case.disconnect:
flush()
parser.feed_disconnect()
flush()
header_count = 0
error_count = 0
body_count = 0
for case, request in zip_longest(cases, protocol.requests):
if case.error:
assert protocol.error == case.error
if case.error in header_errors:
error_count += 1
break
header_count += 1
assert request.method == case.method
assert request.path == case.path
assert request.version == case.version
assert request.headers == case.headers
if case.error in body_errors:
error_count += 1
break
body_count += 1
assert request.body == case.body
assert protocol.on_headers_call_count == header_count
assert protocol.on_error_call_count == error_count
assert protocol.on_body_call_count == body_count
@parametrize_do_parts()
@parametrize_cases(
'base',
'11chunked1+11msgzero',
'11msg+11chunked2',
'11chunked2+close:11msg',
'11msgzero+11chunked3',
'close:11msg+e excessive_data:11chunked1+11chunked3',
'11chunked3+11msg+close:11msg',
'11chunked3+11chunked3+close:11msg'
)
@parametrize_make_parser()
def test_http11_mixed(make_parser, do_parts, cases):
parser, protocol = make_parser()
def flush():
nonlocal data
if not data:
return
parts = do_parts(data)
for part in parts:
parser.feed(part)
if protocol.error:
break
data = b''
data = b''
for case in cases:
data += case.data
if case.disconnect:
flush()
parser.feed_disconnect()
flush()
header_count = 0
error_count = 0
body_count = 0
for case, request in zip_longest(cases, protocol.requests):
if case.error:
assert protocol.error == case.error
if case.error in header_errors:
error_count += 1
break
header_count += 1
assert request.method == case.method
assert request.path == case.path
assert request.version == case.version
assert request.headers == case.headers
if case.error in body_errors:
error_count += 1
break
body_count += 1
assert request.body == case.body
assert protocol.on_headers_call_count == header_count
assert protocol.on_error_call_count == error_count
assert protocol.on_body_call_count == body_count
================================================
FILE: src/japronto/pipeline/__init__.py
================================================
class Pipeline:
def __init__(self, ready):
self._queue = []
self._ready = ready
@property
def empty(self):
return not self._queue
def queue(self, task):
print("queued")
self._queue.append(task)
task.add_done_callback(self._task_done)
def _task_done(self, task):
print('Done', task.result())
pop_idx = 0
for task in self._queue:
if not task.done():
break
self.write(task)
pop_idx += 1
if pop_idx:
self._queue[:pop_idx] = []
def write(self, task):
self._ready(task)
print('Written', task.result())
if __name__ == '__main__':
import asyncio
async def coro(sleep):
await asyncio.sleep(sleep)
return sleep
from uvloop import new_event_loop
loop = new_event_loop()
asyncio.set_event_loop(loop)
pipeline = Pipeline()
def queue(x):
t = loop.create_task(coro(x))
pipeline.queue(t)
loop.call_later(2, lambda: queue(2))
loop.call_later(12, lambda: queue(2))
queue(1)
queue(10)
queue(5)
queue(1)
loop.run_forever()
================================================
FILE: src/japronto/pipeline/cpipeline.c
================================================
#include <Python.h>
#include "structmember.h"
#include "cpipeline.h"
static PyTypeObject PipelineType;
#ifdef PIPELINE_OPAQUE
static PyObject*
Pipeline_new(PyTypeObject* type, PyObject* args, PyObject* kw)
#else
PyObject*
Pipeline_new(Pipeline* self)
#endif
{
#ifdef PIPELINE_OPAQUE
Pipeline* self = NULL;
self = (Pipeline*)type->tp_alloc(type, 0);
if(!self)
goto finally;
#else
((PyObject*)self)->ob_refcnt = 1;
((PyObject*)self)->ob_type = &PipelineType;
#endif
self->ready = NULL;
self->task_done = NULL;
#ifdef PIPELINE_OPAQUE
finally:
#endif
return (PyObject*)self;
}
#ifdef PIPELINE_OPAQUE
static void
#else
void
#endif
Pipeline_dealloc(Pipeline* self)
{
#ifdef PIPELINE_OPAQUE
Py_XDECREF(self->ready);
#endif
Py_XDECREF(self->task_done);
#ifdef PIPELINE_OPAQUE
Py_TYPE(self)->tp_free((PyObject*)self);
#endif
}
#ifdef PIPELINE_OPAQUE
static int
Pipeline_init(Pipeline* self, PyObject *args, PyObject* kw)
#else
int
Pipeline_init(Pipeline* self, void* (*ready)(PipelineEntry, PyObject*), PyObject* protocol)
#endif
{
int result = 0;
#ifdef PIPELINE_OPAQUE
if(!PyArg_ParseTuple(args, "O", &self->ready))
goto error;
Py_INCREF(self->ready);
#else
self->ready = ready;
self->protocol = protocol;
#endif
if(!(self->task_done = PyObject_GetAttrString((PyObject*)self, "_task_done")))
goto error;
self->queue_start = 0;
self->queue_end = 0;
goto finally;
error:
result = -1;
finally:
return result;
}
static PyObject*
Pipeline__task_done(Pipeline* self, PyObject* task)
{
PyObject* result = Py_True;
PipelineEntry *queue_entry;
for(queue_entry = self->queue + self->queue_start;
queue_entry < self->queue + self->queue_end; queue_entry++) {
PyObject* done = NULL;
PyObject* done_result = NULL;
result = Py_True;
if(PipelineEntry_is_task(*queue_entry)) {
task = PipelineEntry_get_task(*queue_entry);
if(!(done = PyObject_GetAttrString(task, "done")))
goto loop_error;
if(!(done_result = PyObject_CallFunctionObjArgs(done, NULL)))
goto loop_error;
if(done_result == Py_False) {
result = Py_False;
goto loop_finally;
}
}
#ifdef PIPELINE_OPAQUE
PyObject* tmp;
if(!(tmp = PyObject_CallFunctionObjArgs(self->ready, *queue_entry, NULL)))
goto loop_error;
Py_DECREF(tmp);
#else
if(!self->ready(*queue_entry, self->protocol))
goto loop_error;
#endif
PipelineEntry_DECREF(*queue_entry);
goto loop_finally;
loop_error:
result = NULL;
loop_finally:
Py_XDECREF(done_result);
Py_XDECREF(done);
if(!result)
goto error;
if(result == Py_False)
break;
}
self->queue_start = queue_entry - self->queue;
#ifndef PIPELINE_OPAQUE
if(PIPELINE_EMPTY(self))
// we became empty so release protocol
Py_DECREF(self->protocol);
#endif
goto finally;
error:
result = NULL;
finally:
Py_XINCREF(result);
return result;
}
#ifdef PIPELINE_OPAQUE
static PyObject*
#else
PyObject*
#endif
Pipeline_queue(Pipeline* self, PipelineEntry entry)
{
PyObject* result = Py_None;
PyObject* add_done_callback = NULL;
if(PIPELINE_EMPTY(self)) {
self->queue_start = self->queue_end = 0;
#ifndef PIPELINE_OPAQUE
// we will become non empty so hold a reference to protocol
Py_INCREF(self->protocol);
#endif
}
assert(self->queue_end < sizeof(self->queue) / sizeof(self->queue[0]));
PipelineEntry* queue_entry = self->queue + self->queue_end;
*queue_entry = entry;
PipelineEntry_INCREF(*queue_entry);
self->queue_end++;
if(PipelineEntry_is_task(entry)) {
PyObject* task = PipelineEntry_get_task(entry);
if(!(add_done_callback = PyObject_GetAttrString(task, "add_done_callback")))
goto error;
PyObject* tmp;
if(!(tmp = PyObject_CallFunctionObjArgs(add_done_callback, self->task_done, NULL)))
goto error;
Py_DECREF(tmp);
}
goto finally;
error:
result = NULL;
finally:
Py_XDECREF(add_done_callback);
#ifdef PIPELINE_OPAQUE
Py_XINCREF(result);
#endif
return result;
}
#ifndef PIPELINE_OPAQUE
void*
Pipeline_cancel(Pipeline* self)
{
void* result = self;
PipelineEntry *queue_entry;
for(queue_entry = self->queue + self->queue_start;
queue_entry < self->queue + self->queue_end; queue_entry++) {
if(!PipelineEntry_is_task(*queue_entry))
continue;
PyObject* task = PipelineEntry_get_task(*queue_entry);
PyObject* cancel = NULL;
if(!(cancel = PyObject_GetAttrString(task, "cancel")))
goto loop_error;
PyObject* tmp;
if(!(tmp = PyObject_CallFunctionObjArgs(cancel, NULL)))
goto loop_error;
Py_DECREF(tmp);
goto loop_finally;
loop_error:
result = NULL;
loop_finally:
Py_XDECREF(cancel);
if(!result)
break;
}
return result;
}
#endif
#ifdef PIPELINE_OPAQUE
static PyObject*
Pipeline_get_empty(Pipeline* self, void* closure) {
PyObject* result = PIPELINE_EMPTY(self) ? Py_True : Py_False;
Py_INCREF(result);
return result;
}
#endif
static PyMethodDef Pipeline_methods[] = {
#ifdef PIPELINE_OPAQUE
{"queue", (PyCFunction)Pipeline_queue, METH_O, ""},
#endif
{"_task_done", (PyCFunction)Pipeline__task_done, METH_O, ""},
{NULL}
};
#ifdef PIPELINE_OPAQUE
static PyGetSetDef Pipeline_getset[] = {
{"empty", (getter)Pipeline_get_empty, NULL, "", NULL},
{NULL}
};
#endif
static PyTypeObject PipelineType = {
PyVarObject_HEAD_INIT(NULL, 0)
"cpipeline.Pipeline", /* tp_name */
sizeof(Pipeline), /* tp_basicsize */
0, /* tp_itemsize */
#ifdef PIPELINE_OPAQUE
(destructor)Pipeline_dealloc, /* tp_dealloc */
#else
0, /* tp_dealloc */
#endif
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
0, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"Pipeline", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
Pipeline_methods, /* tp_methods */
0, /* tp_members */
#ifdef PIPELINE_OPAQUE
Pipeline_getset, /* tp_getset */
#else
0, /* tp_getset */
#endif
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
#ifdef PIPELINE_OPAQUE
(initproc)Pipeline_init, /* tp_init */
#else
0, /* tp_init */
#endif
0, /* tp_alloc */
#ifdef PIPELINE_OPAQUE
Pipeline_new, /* tp_new */
#else
0, /* tp_new */
#endif
};
#ifdef PIPELINE_OPAQUE
static PyModuleDef cpipeline = {
PyModuleDef_HEAD_INIT,
"cpipeline",
"cpipeline",
-1,
NULL, NULL, NULL, NULL, NULL
};
#endif
#ifdef PIPELINE_OPAQUE
PyMODINIT_FUNC
PyInit_cpipeline(void)
#else
void*
cpipeline_init(void)
#endif
{
#ifdef PIPELINE_OPAQUE
PyObject* m = NULL;
#else
void* m = &PipelineType;
#endif
if(PyType_Ready(&PipelineType) < 0)
goto error;
#ifdef PIPELINE_OPAQUE
if(!(m = PyModule_Create(&cpipeline)))
goto error;
Py_INCREF(&PipelineType);
PyModule_AddObject(m, "Pipeline", (PyObject*)&PipelineType);
#endif
goto finally;
error:
m = NULL;
finally:
return m;
}
================================================
FILE: src/japronto/pipeline/cpipeline.h
================================================
#pragma once
#include <stdbool.h>
#include <Python.h>
#ifdef PIPELINE_PAIR
typedef struct {
bool is_task;
PyObject* request;
PyObject* task;
} PipelineEntry;
static inline bool
PipelineEntry_is_task(PipelineEntry entry)
{
return entry.is_task;
}
static inline void
PipelineEntry_DECREF(PipelineEntry entry)
{
Py_DECREF(entry.request);
// if not real task this was response,
// that was inside request that was already freed above
if(entry.is_task)
Py_XDECREF(entry.task);
}
static inline void
PipelineEntry_INCREF(PipelineEntry entry)
{
Py_INCREF(entry.request);
Py_XINCREF(entry.task);
}
static inline PyObject*
PipelineEntry_get_task(PipelineEntry entry)
{
return entry.task;
}
#else
typedef PyObject* PipelineEntry;
static inline bool
PipelineEntry_is_task(PipelineEntry entry)
{
return true;
}
static inline void
PipelineEntry_DECREF(PipelineEntry entry)
{
Py_DECREF(entry);
}
static inline void
PipelineEntry_INCREF(PipelineEntry entry)
{
Py_INCREF(entry);
}
static inline PyObject*
PipelineEntry_get_task(PipelineEntry entry)
{
return entry;
}
#endif
typedef struct {
PyObject_HEAD
#ifdef PIPELINE_OPAQUE
PyObject* ready;
#else
void* (*ready)(PipelineEntry, PyObject*);
PyObject* protocol;
#endif
PyObject* task_done;
PipelineEntry queue[10];
size_t queue_start;
size_t queue_end;
} Pipeline;
#define PIPELINE_EMPTY(p) ((p)->queue_start == (p)->queue_end)
#ifndef PIPELINE_OPAQUE
PyObject*
Pipeline_new(Pipeline* self);
void
Pipeline_dealloc(Pipeline* self);
int
Pipeline_init(Pipeline* self, void* (*ready)(PipelineEntry, PyObject*), PyObject* protocol);
PyObject*
Pipeline_queue(Pipeline* self, PipelineEntry entry);
void*
Pipeline_cancel(Pipeline* self);
void*
cpipeline_init(void);
#endif
================================================
FILE: src/japronto/pipeline/cpipeline_ext.py
================================================
from distutils.core import Extension
def get_extension():
return Extension(
'japronto.pipeline.cpipeline',
sources=['cpipeline.c'],
include_dirs=[],
libraries=[], library_dirs=[],
extra_link_args=[],
define_macros=[('PIPELINE_OPAQUE', 1)])
================================================
FILE: src/japronto/pipeline/test_pipeline.py
================================================
import asyncio
import gc
import sys
from collections import namedtuple
from functools import partial
import pytest
import uvloop
from japronto.pipeline import Pipeline
from japronto.pipeline.cpipeline import Pipeline as CPipeline
Example = namedtuple('Example', 'value,delay')
class FakeLoop:
def call_soon(self, callback, val):
callback(val)
def get_debug(self):
return False
def create_future(self):
return asyncio.Future(loop=self)
class FakeFuture:
cnt = 0
def __new__(cls):
print('new')
cls.cnt += 1
return object.__new__(cls)
def __del__(self):
type(self).cnt -= 1
print('del')
def __init__(self):
self.callbacks = []
def add_done_callback(self, cb):
self.callbacks.append(cb)
def done(self):
return hasattr(self, '_result')
def result(self):
return self._result
def set_result(self, result):
self._result = result
for cb in self.callbacks:
cb(self)
self.callbacks = []
def parametrize_make_pipeline():
def make_pipeline(cls):
results = []
def append(task):
results.append(task.result())
return cls(append), results
return pytest.mark.parametrize(
'make_pipeline',
[partial(make_pipeline, CPipeline), partial(make_pipeline, Pipeline)],
ids=['c', 'py'])
def parametrize_case(examples):
cases = [parse_case(i) for i in examples]
return pytest.mark.parametrize('case', cases, ids=examples)
def parse_example(e, accum):
value, delay = map(int, e.split('@')) if '@' in e else (int(e), 0)
return Example(value, delay + accum)
def parse_case(case):
results = []
delay = 0
for c in case.split('+'):
e = parse_example(c, delay)
results.append(e)
delay = e.delay
return results
def create_futures(resolves, case):
futures = [None] * len(case)
case = case[:]
for c in sorted(case):
idx = case.index(c)
futures[idx] = resolves[idx]()
case[idx] = None
return tuple(futures)
@parametrize_case([
'1',
'1+5', '5+1',
'1+5+10', '10+5+1', '5+1+10', '10+1+5',
'1+10+5+1', '1+1+10+5', '10+5+1+1', '1+1+5+10'
])
@parametrize_make_pipeline()
def test_fake_future(make_pipeline, case):
pipeline, results = make_pipeline()
def queue(x):
fut = FakeFuture()
pipeline.queue(fut)
def resolve():
fut.set_result(x)
return fut
return resolve
resolves = tuple(queue(v) for v in case)
futures = create_futures(resolves, case)
assert pipeline.empty
del resolves
# this loop is not pythonic on purpose
# carefully don't create extra references
for i in range(len(futures)):
print(sys.getrefcount(futures[i]))
del i
assert results == case
gc.collect()
del futures
gc.set_debug(gc.DEBUG_LEAK)
gc.collect()
print(gc.garbage)
gc.set_debug(0)
assert FakeFuture.cnt == 0
def parametrize_loop():
return pytest.mark.parametrize(
'loop', [uvloop.new_event_loop(), asyncio.new_event_loop()],
ids=['uv', 'aio'])
@parametrize_case([
'1', '1@1',
'1+2', '2+1', '2+1@1', '1@1+2',
'1+2+3', '3+2+1', '2+1+3', '3+1+2',
'1+3+2+1', '1+3+2+1+1@1', '1+1+3+2', '3+2+1+1', '1+1+2+3'
])
@parametrize_make_pipeline()
@parametrize_loop()
def test_real_task(loop, make_pipeline, case):
DIVISOR = 1000
pipeline, results = make_pipeline()
async def coro(example):
await asyncio.sleep(example.value / DIVISOR, loop=loop)
return example
def queue(x):
task = loop.create_task(coro(x))
pipeline.queue(task)
for v in case:
if v.delay:
loop.call_later(v.delay / DIVISOR, partial(queue, v))
else:
queue(v)
duration = max((e.value + e.delay) / DIVISOR for e in case)
loop.run_until_complete(asyncio.sleep(duration, loop=loop))
# timing issue, wait a little bit more so we collect all the results
if len(results) < len(case):
loop.run_until_complete(asyncio.sleep(10 / DIVISOR, loop=loop))
assert pipeline.empty
assert results == case
================================================
FILE: src/japronto/protocol/__init__.py
================================================
================================================
FILE: src/japronto/protocol/cprotocol.c
================================================
#include <Python.h>
#include "cprotocol.h"
#include "cmatcher.h"
#include "crequest.h"
#include "cresponse.h"
#include "capsule.h"
#include "match_dict.h"
#ifdef PARSER_STANDALONE
static PyObject* Parser;
#endif
static PyObject* PyRequest;
static PyObject* RouteNotFoundException;
static Request_CAPI* request_capi;
static Matcher_CAPI* matcher_capi;
static Response_CAPI* response_capi;
static PyObject *
Protocol_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
Protocol* self = NULL;
self = (Protocol*)type->tp_alloc(type, 0);
if(!self)
goto finally;
#ifdef PARSER_STANDALONE
self->feed = NULL;
self->feed_disconnect = NULL;
#else
Parser_new(&self->parser);
#endif
Pipeline_new(&self->pipeline);
Request_new(request_capi->RequestType, &self->static_request);
self->app = NULL;
self->matcher = NULL;
self->error_handler = NULL;
self->transport = NULL;
self->write = NULL;
self->create_task = NULL;
self->request_logger = NULL;
self->gather.prev_buffer = NULL;
finally:
return (PyObject*)self;
}
static void
Protocol_dealloc(Protocol* self)
{
Py_XDECREF(self->gather.prev_buffer);
Py_XDECREF(self->request_logger);
Py_XDECREF(self->create_task);
Py_XDECREF(self->write);
Py_XDECREF(self->writelines);
Py_XDECREF(self->transport);
Py_XDECREF(self->error_handler);
Py_XDECREF(self->matcher);
Py_XDECREF(self->app);
Request_dealloc(&self->static_request);
Pipeline_dealloc(&self->pipeline);
#ifdef PARSER_STANDALONE
Py_XDECREF(self->feed_disconnect);
Py_XDECREF(self->feed);
#else
Parser_dealloc(&self->parser);
#endif
Py_TYPE(self)->tp_free((PyObject*)self);
}
static void* Protocol_pipeline_ready(PipelineEntry entry, PyObject* protocol);
static int
Protocol_init(Protocol* self, PyObject *args, PyObject *kw)
{
int result = 0;
PyObject* loop = NULL;
PyObject* log_request = NULL;
#ifdef PARSER_STANDALONE
PyObject* parser = NULL;
PyObject* on_headers = PyObject_GetAttrString((PyObject*)self, "on_headers");
if(!on_headers) // FIXME leak
goto error;
PyObject* on_body = PyObject_GetAttrString((PyObject*)self, "on_body");
if(!on_body) // FIXME leak
goto error;
PyObject* on_error = PyObject_GetAttrString((PyObject*)self, "on_error");
if(!on_error) // FIXME leak
goto error;
parser = PyObject_CallFunctionObjArgs(
Parser, on_headers, on_body, on_error, NULL);
if(!parser)
goto error;
self->feed = PyObject_GetAttrString(parser, "feed");
if(!self->feed)
goto error;
self->feed_disconnect = PyObject_GetAttrString(parser, "feed_disconnect");
if(!self->feed_disconnect)
goto error;
#else
if(Parser_init(&self->parser, self) == -1)
goto error;
#endif
if(Pipeline_init(&self->pipeline, Protocol_pipeline_ready, (PyObject*)self) == -1)
goto error;
if(!PyArg_ParseTuple(args, "O", &self->app))
goto error;
Py_INCREF(self->app);
self->matcher = PyObject_GetAttrString(self->app, "_matcher");
if(!self->matcher)
goto error;
self->error_handler = PyObject_GetAttrString(self->app, "error_handler");
if(!self->error_handler)
goto error;
loop = PyObject_GetAttrString(self->app, "_loop");
if(!loop)
goto error;
self->create_task = PyObject_GetAttrString(loop, "create_task");
if(!self->create_task)
goto error;
if(!(log_request = PyObject_GetAttrString(self->app, "_log_request")))
goto error;
if(log_request == Py_True) {
if(!(self->request_logger = PyObject_GetAttrString(self->app, "default_request_logger")))
goto error;
}
self->gather.responses_end = 0;
self->gather.len = 0;
goto finally;
error:
result = -1;
finally:
Py_XDECREF(log_request);
Py_XDECREF(loop);
#ifdef PARSER_STANDALONE
Py_XDECREF(parser);
#endif
return result;
}
static PyObject*
Protocol_connection_made(Protocol* self, PyObject* transport)
{
#ifdef PROTOCOL_TRACK_REFCNT
printf("made: %ld, %ld, %ld, ",
(size_t)Py_REFCNT(Py_None), (size_t)Py_REFCNT(Py_True), (size_t)Py_REFCNT(Py_False));
self->none_cnt = Py_REFCNT(Py_None);
self->true_cnt = Py_REFCNT(Py_True);
self->false_cnt = Py_REFCNT(Py_False);
#endif
PyObject* connections = NULL;
self->transport = transport;
Py_INCREF(self->transport);
if(!(self->write = PyObject_GetAttrString(transport, "write")))
goto error;
if(!(self->writelines = PyObject_GetAttrString(transport, "writelines")))
goto error;
if(!(connections = PyObject_GetAttrString(self->app, "_connections")))
goto error;
#ifdef REAPER_ENABLED
self->idle_time = 0;
self->read_ops = 0;
self->last_read_ops = 0;
#endif
if(PySet_Add(connections, (PyObject*)self) == -1)
goto error;
self->closed = false;
goto finally;
error:
return NULL;
finally:
Py_XDECREF(connections);
Py_RETURN_NONE;
}
static void*
Protocol_close(Protocol* self)
{
void* result = self;
PyObject* close = NULL;
close = PyObject_GetAttrString(self->transport, "close");
if(!close)
goto error;
PyObject* tmp = PyObject_CallFunctionObjArgs(close, NULL);
if(!tmp)
goto error;
Py_DECREF(tmp);
goto finally;
error:
result = NULL;
finally:
Py_XDECREF(close);
return result;
}
static PyObject*
Protocol_connection_lost(Protocol* self, PyObject* args)
{
self->closed = true;
PyObject* connections = NULL;
PyObject* result = Py_None;
#ifdef PARSER_STANDALONE
PyObject* result = PyObject_CallFunctionObjArgs(
self->feed_disconnect, NULL);
if(!result)
goto error;
Py_DECREF(result); // FIXME: result can leak
#else
if(!Parser_feed_disconnect(&self->parser))
goto error;
#endif
if(!(connections = PyObject_GetAttrString(self->app, "_connections")))
goto error;
if(PySet_Discard(connections, (PyObject*)self) == -1)
goto error;
if(!Pipeline_cancel(&self->pipeline))
goto error;
#ifdef PROTOCOL_TRACK_REFCNT
printf("lost: %ld, %ld, %ld\n",
(size_t)Py_REFCNT(Py_None), (size_t)Py_REFCNT(Py_True), (size_t)Py_REFCNT(Py_False));
assert(Py_REFCNT(Py_None) == self->none_cnt);
assert(Py_REFCNT(Py_True) == self->true_cnt);
assert(Py_REFCNT(Py_False) >= self->false_cnt);
#endif
goto finally;
error:
result = NULL;
finally:
Py_XDECREF(connections);
Py_XINCREF(result);
return result;
}
static PyObject*
Protocol_data_received(Protocol* self, PyObject* data)
{
#ifdef REAPER_ENABLED
self->read_ops++;
#endif
#ifdef PARSER_STANDALONE
PyObject* result = PyObject_CallFunctionObjArgs(
self->feed, data, NULL);
if(!result)
goto error;
Py_DECREF(result);
#else
if(!Parser_feed(&self->parser, data))
goto error;
#endif
goto finally;
error:
return NULL;
finally:
Py_RETURN_NONE;
}
static inline PyObject* Gather_flush(Gather* gather);
#ifndef PARSER_STANDALONE
Protocol*
Protocol_on_incomplete(Protocol* self)
{
Gather* gather = &self->gather;
PyObject* gather_buffer = NULL;
if(!gather->len)
goto finally;
if(!(gather_buffer = Gather_flush(gather)))
goto error;
PyObject* tmp;
if(!(tmp = PyObject_CallFunctionObjArgs(self->write, gather_buffer, NULL)))
goto error;
Py_DECREF(tmp);
goto finally;
error:
self = NULL;
finally:
Py_XDECREF(gather_buffer);
return self;
}
#endif
#ifdef PARSER_STANDALONE
static PyObject*
Protocol_on_headers(Protocol* self, PyObject *args)
{
Py_RETURN_NONE;
}
#else
Protocol*
Protocol_on_headers(Protocol* self, char* method, size_t method_len,
char* path, size_t path_len, int minor_version,
void* headers, size_t num_headers)
{
Protocol* result = self;
Request_dealloc(&self->static_request);
Request_new(request_capi->RequestType, &self->static_request);
request
gitextract_jn3kd_d7/
├── .gitignore
├── .gitmodules
├── .travis.yml
├── CHANGELOG.md
├── LICENSE.txt
├── MANIFEST.in
├── README.md
├── benchmarks/
│ ├── aiohttp/
│ │ ├── micro.py
│ │ └── requirements.txt
│ ├── gevent/
│ │ ├── micro.py
│ │ └── requirements.txt
│ ├── golang/
│ │ ├── README.md
│ │ └── micro.go
│ ├── golang-fasthttp/
│ │ ├── README.md
│ │ └── micro.go
│ ├── japronto/
│ │ └── micro.py
│ ├── meinheld/
│ │ ├── micro.py
│ │ └── requirements.txt
│ ├── nodejs/
│ │ └── micro.js
│ ├── results.ods
│ ├── sanic/
│ │ ├── micro.py
│ │ └── requirements.txt
│ └── tornado/
│ ├── micro.py
│ └── requirements.txt
├── build.py
├── cases/
│ ├── __init__.py
│ ├── base.toml
│ └── websites.toml
├── conftest.py
├── do_wrk.py
├── examples/
│ ├── 1_hello/
│ │ └── hello.py
│ ├── 2_async/
│ │ └── async.py
│ ├── 3_router/
│ │ └── router.py
│ ├── 4_request/
│ │ └── request.py
│ ├── 5_response/
│ │ └── response.py
│ ├── 6_exceptions/
│ │ └── exceptions.py
│ ├── 7_extend/
│ │ └── extend.py
│ ├── 8_template/
│ │ ├── index.html
│ │ └── template.py
│ └── todo_api/
│ ├── .gitignore
│ └── todo_api.py
├── integration_tests/
│ ├── __init__.py
│ ├── common.py
│ ├── drain.py
│ ├── dump.py
│ ├── experiments.py
│ ├── generators.py
│ ├── longrun.py
│ ├── noleak.py
│ ├── reaper.py
│ ├── strategies.py
│ ├── test_drain.py
│ ├── test_noleak.py
│ ├── test_perror.py
│ ├── test_reaper.py
│ └── test_request.py
├── misc/
│ ├── __init__.py
│ ├── bootstrap.sh
│ ├── buggers.py
│ ├── cleanup_script.py
│ ├── client.py
│ ├── collector.py
│ ├── cpu.py
│ ├── do_perf.py
│ ├── docker/
│ │ └── Dockerfile
│ ├── parts.py
│ ├── perf.md
│ ├── pipeline.lua
│ ├── report.py
│ ├── requirements-test.txt
│ ├── requirements.txt
│ ├── rpm-requirements.txt
│ ├── runpytest.py
│ ├── simple.py
│ ├── suppr.txt
│ └── travis/
│ ├── before_install.sh
│ ├── install.sh
│ └── script.sh
├── setup.py
├── src/
│ ├── japronto/
│ │ ├── __init__.py
│ │ ├── __main__.py
│ │ ├── app/
│ │ │ └── __init__.py
│ │ ├── capsule.c
│ │ ├── capsule.h
│ │ ├── common.h
│ │ ├── cpu_features.c
│ │ ├── cpu_features.h
│ │ ├── parser/
│ │ │ ├── .gitignore
│ │ │ ├── __init__.py
│ │ │ ├── build_libpicohttpparser.py
│ │ │ ├── cffiparser.py
│ │ │ ├── cparser.c
│ │ │ ├── cparser.gcda
│ │ │ ├── cparser.h
│ │ │ ├── cparser_ext.py
│ │ │ └── test_parser.py
│ │ ├── pipeline/
│ │ │ ├── __init__.py
│ │ │ ├── cpipeline.c
│ │ │ ├── cpipeline.h
│ │ │ ├── cpipeline_ext.py
│ │ │ └── test_pipeline.py
│ │ ├── protocol/
│ │ │ ├── __init__.py
│ │ │ ├── cprotocol.c
│ │ │ ├── cprotocol.h
│ │ │ ├── cprotocol_ext.py
│ │ │ ├── creaper.c
│ │ │ ├── creaper_ext.py
│ │ │ ├── generator.c
│ │ │ ├── generator.h
│ │ │ ├── generator_ext.py
│ │ │ ├── handler.py
│ │ │ ├── null.py
│ │ │ └── tracing.py
│ │ ├── reloader.py
│ │ ├── request/
│ │ │ ├── __init__.py
│ │ │ ├── crequest.c
│ │ │ ├── crequest.h
│ │ │ └── crequest_ext.py
│ │ ├── response/
│ │ │ ├── __init__.py
│ │ │ ├── cresponse.c
│ │ │ ├── cresponse.h
│ │ │ ├── cresponse_ext.py
│ │ │ ├── py.py
│ │ │ └── reasons.h
│ │ ├── router/
│ │ │ ├── __init__.py
│ │ │ ├── analyzer.py
│ │ │ ├── cmatcher.c
│ │ │ ├── cmatcher.h
│ │ │ ├── cmatcher_ext.py
│ │ │ ├── match_dict.c
│ │ │ ├── match_dict.h
│ │ │ ├── matcher.py
│ │ │ ├── route.py
│ │ │ ├── test_analyzer.py
│ │ │ ├── test_matcher.py
│ │ │ └── test_route.py
│ │ └── runner.py
│ └── picohttpparser/
│ ├── build
│ ├── picohttpparser.c
│ └── picohttpparser.h
└── tutorial/
├── 1_hello.md
├── 2_async.md
├── 3_router.md
├── 4_request.md
├── 5_response.md
├── 6_exceptions.md
├── 7_extend.md
└── 8_template.md
SYMBOL INDEX (540 symbols across 93 files)
FILE: benchmarks/aiohttp/micro.py
function hello (line 10) | async def hello(request):
FILE: benchmarks/gevent/micro.py
function hello (line 4) | def hello(environ, start_response):
FILE: benchmarks/golang-fasthttp/micro.go
function hello (line 5) | func hello(ctx *fasthttp.RequestCtx) {
function main (line 14) | func main() {
FILE: benchmarks/golang/micro.go
function hello (line 10) | func hello(w http.ResponseWriter, r *http.Request) {
function main (line 19) | func main() {
FILE: benchmarks/japronto/micro.py
function hello (line 4) | def hello(request):
FILE: benchmarks/meinheld/micro.py
function hello (line 4) | def hello(environ, start_response):
FILE: benchmarks/sanic/micro.py
function hello (line 8) | async def hello(request):
FILE: benchmarks/tornado/micro.py
class MainHandler (line 13) | class MainHandler(web.RequestHandler):
method get (line 14) | def get(self):
method set_etag_header (line 18) | def set_etag_header(self):
method check_etag_header (line 21) | def check_etag_header(self):
method clear (line 25) | def clear(self):
FILE: build.py
class BuildSystem (line 23) | class BuildSystem:
method __init__ (line 24) | def __init__(self, args, relative_source=False):
method get_extension_by_path (line 29) | def get_extension_by_path(self, path):
method discover_extensions (line 67) | def discover_extensions(self):
method dest_folder (line 76) | def dest_folder(self, mod_name):
method build_toml (line 79) | def build_toml(self, mod_name):
method get_so (line 82) | def get_so(self, ext):
method flags_changed (line 86) | def flags_changed(self, ext):
method should_rebuild (line 102) | def should_rebuild(self, ext):
function prune (line 121) | def prune(dest):
function profile_clean (line 128) | def profile_clean():
function get_includes (line 134) | def get_includes(ext):
function symlink_python_files (line 158) | def symlink_python_files(dest):
function get_parser (line 193) | def get_parser():
function get_platform (line 239) | def get_platform():
class custom_build_ext (line 251) | class custom_build_ext(build_ext):
method build_extensions (line 252) | def build_extensions(self):
function compile_c (line 273) | def compile_c(compiler, cfile, ofile, *, options=None):
function main (line 284) | def main():
FILE: cases/__init__.py
function parse_casesel (line 14) | def parse_casesel(suite, casesel):
function parametrize_cases (line 29) | def parametrize_cases(suite, *args):
function load_casefile (line 36) | def load_casefile(path):
function load_cases (line 54) | def load_cases():
function keep_alive (line 64) | def keep_alive(case):
function close (line 73) | def close(case):
function should_keep_alive (line 80) | def should_keep_alive(case):
function set_error (line 86) | def set_error(case, error):
function disconnect (line 90) | def disconnect(case):
function update_case (line 94) | def update_case(case, headers=False, error=False, disconnect=None):
FILE: conftest.py
function add_build (line 11) | def add_build(mark):
function execute_builds (line 20) | def execute_builds():
function add_coverage (line 33) | def add_coverage(mark):
function setup_coverage (line 38) | def setup_coverage():
function make_coverage (line 48) | def make_coverage():
function pytest_itemcollected (line 76) | def pytest_itemcollected(item):
function pytest_collection_modifyitems (line 84) | def pytest_collection_modifyitems(config, items):
function pytest_unconfigure (line 89) | def pytest_unconfigure():
FILE: do_wrk.py
function run_wrk (line 15) | def run_wrk(loop, endpoint=None):
function cpu_usage (line 41) | def cpu_usage(p):
function connections (line 45) | def connections(process):
function memory (line 51) | def memory(p):
FILE: examples/1_hello/hello.py
function hello (line 6) | def hello(request):
FILE: examples/2_async/async.py
function synchronous (line 6) | def synchronous(request):
function asynchronous (line 14) | async def asynchronous(request):
FILE: examples/3_router/router.py
function slash (line 10) | def slash(request):
function get_love (line 19) | def get_love(request):
function methods (line 28) | def methods(request):
function params (line 40) | def params(request):
FILE: examples/4_request/request.py
function basic (line 13) | def basic(request):
function body (line 39) | def body(request):
function misc (line 68) | def misc(request):
FILE: examples/5_response/response.py
function text (line 9) | def text(request):
function encoding (line 14) | def encoding(request):
function mime (line 19) | def mime(request):
function body (line 31) | def body(request):
function json (line 38) | def json(request):
function code (line 43) | def code(request):
function headers (line 48) | def headers(request):
function cookies (line 56) | def cookies(request):
FILE: examples/6_exceptions/exceptions.py
class KittyError (line 5) | class KittyError(Exception):
method __init__ (line 6) | def __init__(self):
class DoggieError (line 10) | class DoggieError(Exception):
method __init__ (line 11) | def __init__(self):
function cat (line 17) | def cat(request):
function dog (line 21) | def dog(request):
function unhandled (line 27) | def unhandled(request):
function handle_cat (line 40) | def handle_cat(request, exception):
function handle_dog (line 44) | def handle_dog(request, exception):
function handle_not_found (line 49) | def handle_not_found(request, exception):
FILE: examples/7_extend/extend.py
function extended_hello (line 6) | def extended_hello(request):
function with_callback (line 17) | def with_callback(request):
function reversed_agent (line 27) | def reversed_agent(request):
function host_startswith (line 34) | def host_startswith(request, prefix):
FILE: examples/8_template/template.py
function index (line 7) | def index(request):
function example (line 13) | def example(request):
function jinja (line 20) | def jinja(request):
FILE: examples/todo_api/todo_api.py
function add_todo (line 8) | def add_todo(request):
function list_todos (line 18) | def list_todos(request):
function show_todo (line 26) | def show_todo(request):
function delete_todo (line 38) | def delete_todo(request):
function maybe_create_schema (line 54) | def maybe_create_schema():
function cursor (line 66) | def cursor(request):
FILE: integration_tests/common.py
function start_server (line 10) | def start_server(script, *, stdout=None, path=None, sanitize=True, wait=...
FILE: integration_tests/drain.py
function slash (line 6) | def slash(request):
function sleep (line 10) | async def sleep(request):
FILE: integration_tests/dump.py
class ForcedException (line 8) | class ForcedException(Exception):
function dump (line 12) | def dump(request, exception=None):
function adump (line 39) | async def adump(request):
FILE: integration_tests/experiments.py
function my_fix (line 5) | def my_fix(request):
function size_k (line 11) | def size_k(request):
function test1 (line 29) | def test1(size_k):
FILE: integration_tests/generators.py
function generate_body (line 6) | def generate_body(body, size_k):
function makeval (line 17) | def makeval(v, default_st, default=None):
function print_request (line 30) | def print_request(request):
function generate_request (line 42) | def generate_request(*, method=None, path=None, query_string=None,
function generate_combinations (line 54) | def generate_combinations(reverse=False):
function send_requests (line 68) | def send_requests(conn, number, **kwargs):
FILE: integration_tests/longrun.py
function setup (line 16) | def setup():
function run (line 44) | def run():
function main (line 58) | def main():
FILE: integration_tests/noleak.py
function noleak (line 10) | def noleak(request):
function noleak (line 13) | def noleak(request):
function noleak (line 16) | def noleak(request):
function noleak (line 19) | def noleak(request):
function noleak (line 22) | def noleak(request):
function noleak (line 25) | def noleak(request):
function noleak (line 28) | def noleak(request):
function noleak (line 31) | def noleak(request):
FILE: integration_tests/test_drain.py
function server (line 13) | def server():
function server_terminate (line 19) | def server_terminate(server):
function connect (line 34) | def connect():
function test_no_connections (line 51) | def test_no_connections(server_terminate):
function test_unclosed_connections (line 58) | def test_unclosed_connections(num, connect, server_terminate):
function test_closed_connections (line 68) | def test_closed_connections(num, connect, server_terminate):
function test_unclosed_requests (line 79) | def test_unclosed_requests(num, connect, server_terminate):
function test_closed_requests (line 91) | def test_closed_requests(num, connect, server_terminate):
function test_pipelined (line 105) | def test_pipelined(num, connect, server_terminate):
function test_pipelined_timeout (line 123) | def test_pipelined_timeout(num, connect, server_terminate):
function test_refuse (line 140) | def test_refuse(connect, server):
FILE: integration_tests/test_noleak.py
function server (line 12) | def server(request):
function connection (line 25) | def connection(server):
function test_method (line 32) | def test_method(connection):
function test_path (line 44) | def test_path(connection):
function test_match_dict (line 56) | def test_match_dict(connection):
function test_query_string (line 68) | def test_query_string(connection):
function test_headers (line 80) | def test_headers(connection):
function test_body (line 94) | def test_body(connection):
function test_keep_alive (line 108) | def test_keep_alive(request):
function test_route (line 128) | def test_route(connection):
FILE: integration_tests/test_perror.py
function server (line 16) | def server():
function line_getter (line 31) | def line_getter(server):
function connect (line 50) | def connect(request):
function make_truncated_request_line (line 57) | def make_truncated_request_line(cut):
function test_truncated_request_line (line 67) | def test_truncated_request_line(line_getter, connect, request_line):
function test_truncated_request_line_disconnect (line 82) | def test_truncated_request_line_disconnect(line_getter, connect, request...
function make_truncated_header (line 94) | def make_truncated_header(cut):
function test_truncated_header (line 104) | def test_truncated_header(line_getter, connect, header_line):
function test_truncated_header_disconnect (line 120) | def test_truncated_header_disconnect(line_getter, connect, header_line):
function test_invalid_content_length (line 136) | def test_invalid_content_length(line_getter, connect, value):
FILE: integration_tests/test_reaper.py
function get_connections_and_wait (line 14) | def get_connections_and_wait(request):
function test_empty (line 30) | def test_empty(get_connections_and_wait):
function test_request (line 46) | def test_request(get_connections_and_wait):
FILE: integration_tests/test_request.py
function server (line 17) | def server():
function connect (line 28) | def connect(request):
function prefix (line 40) | def prefix(request):
function test_method (line 46) | def test_method(prefix, connect, method):
function test_route (line 63) | def test_route(prefix, connect, route_prefix):
function test_match_dict (line 77) | def test_match_dict(prefix, connect, param1, param2):
function test_query_string (line 91) | def test_query_string(prefix, connect, query_string):
function test_headers (line 108) | def test_headers(prefix, connect, headers):
function test_error (line 129) | def test_error(prefix, connect, error):
function test_body (line 150) | def test_body(prefix, connect, size_k, body):
function test_chunked (line 173) | def test_chunked(prefix, connect, size_k, body):
function test_all (line 207) | def test_all(prefix, connect, size_k, method, error, route_prefix,
function test_pipeline (line 271) | def test_pipeline(requests):
function format_sleep_qs (line 322) | def format_sleep_qs(val):
function test_async_pipeline (line 339) | def test_async_pipeline(requests):
FILE: misc/buggers.py
function silence (line 8) | def silence():
FILE: misc/cleanup_script.py
function main (line 4) | def main():
FILE: misc/client.py
function readline (line 6) | def readline(sock):
function readexact (line 14) | def readexact(sock, size):
class Response (line 24) | class Response:
method __init__ (line 25) | def __init__(self, sock):
method read_status_line (line 32) | def read_status_line(self):
method read_headers (line 39) | def read_headers(self):
method encoding (line 53) | def encoding(self):
method read_body (line 64) | def read_body(self):
method json (line 69) | def json(self):
class Connection (line 73) | class Connection:
method __init__ (line 74) | def __init__(self, addr):
method maybe_connect (line 78) | def maybe_connect(self):
method putline (line 91) | def putline(self, line=None):
method putclose (line 98) | def putclose(self, data):
method putrequest (line 105) | def putrequest(self, method, path, query_string=None):
method request (line 114) | def request(self, method, path, query_string=None, headers=None,
method putheader (line 128) | def putheader(self, name, value):
method endheaders (line 132) | def endheaders(self, body=None):
method getresponse (line 142) | def getresponse(self):
method close (line 145) | def close(self):
function chunked_encoder (line 149) | def chunked_encoder(data):
FILE: misc/collector.py
function get_connections (line 9) | def get_connections(process):
function get_memory (line 15) | def get_memory(p):
function sample_process (line 20) | def sample_process(pid):
function main (line 39) | def main():
FILE: misc/cpu.py
function save (line 14) | def save():
function change (line 48) | def change(governor, freq=None):
function available_freq (line 78) | def available_freq():
function min_freq (line 91) | def min_freq():
function max_freq (line 98) | def max_freq():
function dump (line 105) | def dump():
FILE: misc/do_perf.py
function get_http10long (line 12) | def get_http10long():
function get_websites (line 16) | def get_websites(size=2 ** 18):
FILE: misc/parts.py
function make_parts (line 6) | def make_parts(value, get_size, dir=1):
function one_part (line 26) | def one_part(value):
function geometric_series (line 30) | def geometric_series():
function fancy_series (line 37) | def fancy_series(minimum=2):
FILE: misc/report.py
function report (line 7) | def report(samples, pid):
function load (line 36) | def load(filepath):
function order (line 46) | def order(samples):
function normalize_time (line 50) | def normalize_time(samples):
function main (line 59) | def main():
FILE: misc/simple.py
function slash (line 15) | def slash(request):
function hello (line 19) | def hello(request):
function sleep (line 23) | async def sleep(request):
function loop (line 28) | async def loop(request):
function dump (line 38) | def dump(request):
FILE: src/japronto/__main__.py
function main (line 7) | def main():
FILE: src/japronto/app/__init__.py
class Application (line 22) | class Application:
method __init__ (line 23) | def __init__(self, *, reaper_settings=None, log_request=None,
method loop (line 36) | def loop(self):
method router (line 43) | def router(self):
method __finalize (line 49) | def __finalize(self):
method protocol_error_handler (line 56) | def protocol_error_handler(self, error):
method default_request_logger (line 68) | def default_request_logger(self, request):
method add_error_handler (line 71) | def add_error_handler(self, typ, handler):
method default_error_handler (line 74) | def default_error_handler(self, request, exception):
method error_handler (line 88) | def error_handler(self, request, exception):
method _get_idle_and_busy_connections (line 106) | def _get_idle_and_busy_connections(self):
method drain (line 113) | async def drain(self):
method extend_request (line 153) | def extend_request(self, handler, *, name=None, property=False):
method serve (line 159) | def serve(self, *, sock, host, port, reloader_pid):
method _run (line 193) | def _run(self, *, host, port, worker_num=None, reloader_pid=None,
method run (line 249) | def run(self, host='0.0.0.0', port=8080, *, worker_num=None, reload=Fa...
FILE: src/japronto/capsule.c
function PyObject (line 35) | PyObject* put_ptr_in_mod(PyObject* m, void* ptr, const char* attr_name,
FILE: src/japronto/common.h
type KEEP_ALIVE (line 4) | typedef enum {
FILE: src/japronto/cpu_features.c
function supports_x86_sse42 (line 6) | int supports_x86_sse42(void)
FILE: src/japronto/parser/cffiparser.py
class HttpRequestParser (line 4) | class HttpRequestParser(object):
method __init__ (line 5) | def __init__(self, on_headers, on_body, on_error):
method _reset_state (line 21) | def _reset_state(self, disconnect=False):
method _parse_headers (line 31) | def _parse_headers(self):
method _parse_body (line 115) | def _parse_body(self):
method feed (line 164) | def feed(self, data):
method feed_disconnect (line 184) | def feed_disconnect(self):
FILE: src/japronto/parser/cparser.c
function _reset_state (line 42) | static void _reset_state(Parser* self, bool disconnect) {
function Parser_new (line 60) | void
function Parser_init (line 86) | int
function Parser_dealloc (line 120) | void
type phr_header (line 144) | struct phr_header
function _parse_headers (line 146) | static int _parse_headers(Parser* self) {
function _parse_body (line 381) | static int _parse_body(Parser* self) {
function Parser (line 522) | Parser*
function Parser (line 617) | Parser*
function PyObject (line 667) | static PyObject *
function cparser_init (line 744) | int
FILE: src/japronto/parser/cparser.h
type Parser_state (line 9) | enum Parser_state {
type Parser_transfer (line 15) | enum Parser_transfer {
type Parser_connection (line 22) | enum Parser_connection {
type Parser (line 30) | typedef struct {
FILE: src/japronto/parser/cparser_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/parser/test_parser.py
function make_c (line 17) | def make_c(protocol_factory=CTracingProtocol):
function make_cffi (line 25) | def make_cffi(protocol_factory=CffiTracingProtocol):
function test_make_parts (line 41) | def test_make_parts(data, get_size, dir, parts):
function parametrize_make_parser (line 45) | def parametrize_make_parser():
function parametrize_do_parts (line 58) | def parametrize_do_parts():
function test_http10 (line 90) | def test_http10(make_parser, do_parts, cases):
function test_empty (line 149) | def test_empty(make_parser):
function test_http11 (line 197) | def test_http11(make_parser, do_parts, cases):
function test_http11_chunked (line 276) | def test_http11_chunked(make_parser, do_parts, cases):
function test_http11_mixed (line 346) | def test_http11_mixed(make_parser, do_parts, cases):
FILE: src/japronto/pipeline/__init__.py
class Pipeline (line 1) | class Pipeline:
method __init__ (line 2) | def __init__(self, ready):
method empty (line 7) | def empty(self):
method queue (line 10) | def queue(self, task):
method _task_done (line 17) | def _task_done(self, task):
method write (line 32) | def write(self, task):
function coro (line 40) | async def coro(sleep):
function queue (line 52) | def queue(x):
FILE: src/japronto/pipeline/cpipeline.c
function PyObject (line 12) | PyObject*
function Pipeline_dealloc (line 39) | void
function Pipeline_init (line 57) | int
function PyObject (line 89) | static PyObject*
function PyObject (line 252) | static PyObject*
FILE: src/japronto/pipeline/cpipeline.h
type PipelineEntry (line 9) | typedef struct {
function PipelineEntry_is_task (line 15) | static inline bool
function PipelineEntry_DECREF (line 21) | static inline void
function PipelineEntry_INCREF (line 31) | static inline void
function PyObject (line 38) | static inline PyObject*
type PyObject (line 44) | typedef PyObject* PipelineEntry;
function PipelineEntry_is_task (line 46) | static inline bool
function PipelineEntry_DECREF (line 52) | static inline void
function PipelineEntry_INCREF (line 58) | static inline void
function PyObject (line 64) | static inline PyObject*
type Pipeline (line 72) | typedef struct {
FILE: src/japronto/pipeline/cpipeline_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/pipeline/test_pipeline.py
class FakeLoop (line 17) | class FakeLoop:
method call_soon (line 18) | def call_soon(self, callback, val):
method get_debug (line 21) | def get_debug(self):
method create_future (line 24) | def create_future(self):
class FakeFuture (line 28) | class FakeFuture:
method __new__ (line 31) | def __new__(cls):
method __del__ (line 36) | def __del__(self):
method __init__ (line 40) | def __init__(self):
method add_done_callback (line 43) | def add_done_callback(self, cb):
method done (line 46) | def done(self):
method result (line 49) | def result(self):
method set_result (line 52) | def set_result(self, result):
function parametrize_make_pipeline (line 61) | def parametrize_make_pipeline():
function parametrize_case (line 76) | def parametrize_case(examples):
function parse_example (line 82) | def parse_example(e, accum):
function parse_case (line 88) | def parse_case(case):
function create_futures (line 99) | def create_futures(resolves, case):
function test_fake_future (line 117) | def test_fake_future(make_pipeline, case):
function parametrize_loop (line 158) | def parametrize_loop():
function test_real_task (line 172) | def test_real_task(loop, make_pipeline, case):
FILE: src/japronto/protocol/cprotocol.c
function PyObject (line 22) | static PyObject *
function Protocol_dealloc (line 54) | static void
function Protocol_init (line 82) | static int
function PyObject (line 166) | static PyObject*
function PyObject (line 237) | static PyObject*
function PyObject (line 284) | static PyObject*
function Protocol (line 314) | Protocol*
function PyObject (line 344) | static PyObject*
function Protocol (line 350) | Protocol*
function PyBytesObject (line 390) | static inline PyBytesObject*
function PyObject (line 405) | static inline
function Protocol (line 456) | static inline Protocol*
function Protocol (line 602) | static inline Protocol*
function Protocol (line 629) | Protocol*
function PyObject (line 732) | static PyObject*
function Protocol (line 738) | Protocol*
function PyObject (line 773) | static PyObject*
function PyObject (line 797) | static PyObject*
function PyObject (line 807) | static PyObject*
function PyMODINIT_FUNC (line 874) | PyMODINIT_FUNC
FILE: src/japronto/protocol/cprotocol.h
type Gather (line 13) | typedef struct {
type Protocol (line 22) | typedef struct {
type Protocol_CAPI (line 66) | typedef struct {
FILE: src/japronto/protocol/cprotocol_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/protocol/creaper.c
type Reaper (line 6) | typedef struct {
function PyObject (line 30) | static PyObject*
function Reaper_dealloc (line 50) | static void
function PyObject (line 75) | static PyObject*
function Reaper_init (line 105) | static int
function PyObject (line 167) | static PyObject*
function PyMODINIT_FUNC (line 275) | PyMODINIT_FUNC
FILE: src/japronto/protocol/creaper_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/protocol/generator.c
type Generator (line 6) | typedef struct _Generator {
function Generator_dealloc (line 44) | void
function Generator_init (line 58) | int
function PyObject (line 84) | static PyObject*
function PyObject (line 93) | static PyObject*
FILE: src/japronto/protocol/generator.h
type _Generator (line 4) | struct _Generator
type _Generator (line 12) | struct _Generator
type _Generator (line 15) | struct _Generator
FILE: src/japronto/protocol/generator_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/protocol/handler.py
function make_class (line 17) | def make_class(flavor):
function handle_requests (line 88) | async def handle_requests(queue, transport):
function handle_request (line 97) | async def handle_request(request, transport):
function handle_request_block (line 103) | def handle_request_block(request, transport, response):
function handle_dump (line 109) | def handle_dump(request, transport, response):
FILE: src/japronto/protocol/null.py
class NullProtocol (line 1) | class NullProtocol:
method on_headers (line 2) | def on_headers(self, *args):
method on_body (line 5) | def on_body(self, body):
method on_error (line 8) | def on_error(self, error):
FILE: src/japronto/protocol/tracing.py
class TracingProtocol (line 7) | class TracingProtocol:
method __init__ (line 8) | def __init__(self, on_headers_adapter: callable,
method on_headers (line 20) | def on_headers(self, *args):
method on_body (line 27) | def on_body(self, body):
method on_error (line 32) | def on_error(self, error: str):
function _request_from_cprotocol (line 38) | def _request_from_cprotocol(method: memoryview, path: memoryview, versio...
function _body_from_cprotocol (line 53) | def _body_from_cprotocol(body: memoryview):
function _request_from_cffiprotocol (line 57) | def _request_from_cffiprotocol(method: "char[]", path: "char[]", version...
function _body_from_cffiprotocol (line 68) | def _body_from_cffiprotocol(body: "char[]"):
function _extract_headers (line 72) | def _extract_headers(headers_cdata: "struct phr_header[]"):
FILE: src/japronto/reloader.py
function main (line 9) | def main():
function exec_reloader (line 39) | def exec_reloader(*, host, port, worker_num):
function change_detector (line 49) | def change_detector():
class ChangeDetector (line 77) | class ChangeDetector(threading.Thread):
method __init__ (line 78) | def __init__(self, loop):
method run (line 82) | def run(self):
FILE: src/japronto/request/__init__.py
class HttpRequest (line 9) | class HttpRequest(object):
method __init__ (line 12) | def __init__(self, method, path, version, headers):
method dump_headers (line 19) | def dump_headers(self):
method __repr__ (line 26) | def __repr__(self):
function memoize (line 31) | def memoize(func):
function text (line 48) | def text(request):
function json (line 56) | def json(request):
function query (line 64) | def query(request):
function remote_addr (line 71) | def remote_addr(request):
function parsed_content_type (line 76) | def parsed_content_type(request):
function mime_type (line 84) | def mime_type(request):
function encoding (line 88) | def encoding(request):
function parsed_form_and_files (line 93) | def parsed_form_and_files(request):
function form (line 103) | def form(request):
function files (line 107) | def files(request):
function hostname_and_port (line 112) | def hostname_and_port(request):
function port (line 123) | def port(request):
function hostname (line 127) | def hostname(request):
function parse_cookie (line 131) | def parse_cookie(cookie):
function cookies (line 154) | def cookies(request):
function parse_multipart_form (line 169) | def parse_multipart_form(body, boundary):
FILE: src/japronto/request/crequest.c
function PyObject (line 30) | PyObject*
function Request_dealloc (line 74) | void
function Request_init (line 104) | int
function PyObject (line 116) | static PyObject*
function PyObject (line 176) | static PyObject*
type RequestCopy (line 206) | typedef enum {
type phr_header (line 226) | struct phr_header
type phr_header (line 228) | struct phr_header
type phr_header (line 285) | struct phr_header
type phr_header (line 286) | struct phr_header
type phr_header (line 299) | struct phr_header
type phr_header (line 300) | struct phr_header
function Request_from_raw (line 352) | static void
function Request_set_match_dict_entries (line 374) | static void
function Request_set_body (line 384) | static void
function percent_decode (line 401) | static inline size_t percent_decode(char* data, ssize_t length, size_t* ...
function title_case (line 467) | static inline void title_case(char* data, size_t len)
function PyObject (line 485) | static inline PyObject*
function PyObject (line 541) | static PyObject*
function PyObject (line 554) | static PyObject*
function PyObject (line 568) | static PyObject*
function PyObject (line 586) | static PyObject*
function PyObject (line 595) | static PyObject*
function PyObject (line 605) | static PyObject*
function PyObject (line 617) | static PyObject*
function PyObject (line 631) | static PyObject*
function KEEP_ALIVE (line 639) | static KEEP_ALIVE
function PyObject (line 676) | static PyObject*
function PyObject (line 686) | static PyObject*
function PyObject (line 697) | static PyObject*
function PyObject (line 708) | static PyObject*
function PyObject (line 716) | static PyObject*
function PyObject (line 771) | static PyObject*
function PyObject (line 812) | static PyObject*
FILE: src/japronto/request/crequest.h
type Request (line 12) | typedef struct {
type Request_CAPI (line 61) | typedef struct {
FILE: src/japronto/request/crequest_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/response/cresponse.c
function PyObject (line 24) | PyObject*
function Response_dealloc (line 62) | void
function Response_init (line 90) | int
type CacheEntry (line 221) | typedef struct {
type Cache (line 229) | typedef struct {
function PyObject (line 243) | static inline PyObject*
function Response_cache (line 263) | static inline void
function PyObject (line 281) | PyObject*
function PyMODINIT_FUNC (line 561) | PyMODINIT_FUNC
FILE: src/japronto/response/cresponse.h
type Response (line 10) | typedef struct {
type Response_CAPI (line 30) | typedef struct {
FILE: src/japronto/response/cresponse_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/response/py.py
function factory (line 4) | def factory(status_code=200, text='', mime_type='text/plain',
function dispose (line 22) | def dispose(response):
class Response (line 26) | class Response:
method __init__ (line 29) | def __init__(self, status_code=200, text='', mime_type='text/plain',
method render (line 36) | def render(self):
FILE: src/japronto/response/reasons.h
type ReasonRange (line 59) | typedef struct {
FILE: src/japronto/router/__init__.py
class Router (line 5) | class Router:
method __init__ (line 6) | def __init__(self, matcher_factory=Matcher):
method add_route (line 10) | def add_route(self, pattern, handler, method=None, methods=None):
method get_matcher (line 26) | def get_matcher(self):
FILE: src/japronto/router/analyzer.py
function is_simple (line 9) | def is_simple(fun):
function is_pointless_coroutine (line 35) | def is_pointless_coroutine(fun):
function coroutine_to_func (line 43) | def coroutine_to_func(f):
FILE: src/japronto/router/cmatcher.c
type _Matcher (line 9) | struct _Matcher {
type SegmentType (line 17) | typedef enum {
type ExactSegment (line 23) | typedef struct {
type PlaceholderSegment (line 29) | typedef struct {
type Segment (line 35) | typedef struct {
function PyObject (line 50) | static PyObject *
function Matcher_dealloc (line 86) | static void
function Matcher_init (line 101) | static int
function MatcherEntry (line 138) | MatcherEntry*
function PyObject (line 267) | static PyObject*
function PyMODINIT_FUNC (line 362) | PyMODINIT_FUNC
FILE: src/japronto/router/cmatcher.h
type MatcherEntry (line 8) | typedef struct {
type Matcher (line 20) | typedef struct _Matcher Matcher;
type Matcher_CAPI (line 23) | typedef struct {
FILE: src/japronto/router/cmatcher_ext.py
function get_extension (line 4) | def get_extension():
FILE: src/japronto/router/match_dict.c
function PyObject (line 6) | PyObject*
FILE: src/japronto/router/match_dict.h
type MatchDictEntry (line 5) | typedef struct {
FILE: src/japronto/router/matcher.py
class Matcher (line 1) | class Matcher:
method __init__ (line 2) | def __init__(self, routes):
method match_request (line 5) | def match_request(self, request):
FILE: src/japronto/router/route.py
class RouteNotFoundException (line 8) | class RouteNotFoundException(Exception):
class Route (line 12) | class Route:
method __init__ (line 13) | def __init__(self, pattern, handler, methods):
method __repr__ (line 21) | def __repr__(self):
method describe (line 25) | def describe(self):
method __eq__ (line 29) | def __eq__(self, other):
function parse (line 33) | def parse(pattern):
class SegmentType (line 69) | class SegmentType(IntEnum):
function roundto8 (line 121) | def roundto8(v):
function padto8 (line 125) | def padto8(data):
function compile (line 137) | def compile(route):
function compile_all (line 167) | def compile_all(routes):
FILE: src/japronto/router/test_analyzer.py
function test_is_simple (line 35) | def test_is_simple(code, simple):
function test_is_pointless (line 53) | def test_is_pointless(code, pointless):
FILE: src/japronto/router/test_matcher.py
class FakeRequest (line 10) | class FakeRequest:
method __init__ (line 11) | def __init__(self, method, path):
method from_str (line 16) | def from_str(cls, value):
class TracingRoute (line 20) | class TracingRoute(Route):
method __new__ (line 23) | def __new__(cls, *args, **kw):
method __init__ (line 28) | def __init__(self, pattern, methods):
method __del__ (line 31) | def __del__(self):
function route_from_str (line 36) | def route_from_str(value):
function parametrize_make_matcher (line 44) | def parametrize_make_matcher():
function parametrize_request_route_and_dict (line 63) | def parametrize_request_route_and_dict(cases):
function test_matcher (line 80) | def test_matcher(make_matcher, req, route, match_dict):
function parametrize_request (line 90) | def parametrize_request(requests):
function test_matcher_not_found (line 105) | def test_matcher_not_found(make_matcher, req):
FILE: src/japronto/router/test_route.py
function test_parse (line 19) | def test_parse(pattern, result):
function test_parse_error (line 29) | def test_parse_error(pattern, error):
function decompile (line 40) | def decompile(buffer):
function handler (line 65) | def handler():
function coro (line 69) | async def coro():
function test_compile (line 81) | def test_compile(route):
FILE: src/japronto/runner.py
function get_parser (line 15) | def get_parser():
function verify (line 37) | def verify(args):
function run (line 75) | def run(attribute, args):
FILE: src/picohttpparser/picohttpparser.c
type phr_header (line 264) | struct phr_header
type phr_header (line 331) | struct phr_header
function PHR_PARSE_REQUEST (line 370) | int PHR_PARSE_REQUEST(const char *buf_start, size_t len, const char **me...
type phr_header (line 401) | struct phr_header
function phr_parse_response (line 429) | int phr_parse_response(const char *buf_start, size_t len, int *minor_ver...
function phr_parse_headers (line 455) | int phr_parse_headers(const char *buf_start, size_t len, struct phr_head...
function decode_hex (line 487) | static int decode_hex(int ch)
function phr_decode_chunked (line 500) | ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *bu...
function phr_decode_chunked_is_in_data (line 618) | int phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder)
FILE: src/picohttpparser/picohttpparser.h
type phr_header (line 44) | struct phr_header {
type phr_header (line 54) | struct phr_header
type phr_header (line 57) | struct phr_header
type phr_header (line 64) | struct phr_header
type phr_header (line 67) | struct phr_header
type phr_chunked_decoder (line 71) | struct phr_chunked_decoder {
type phr_chunked_decoder (line 87) | struct phr_chunked_decoder
type phr_chunked_decoder (line 90) | struct phr_chunked_decoder
Condensed preview — 148 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (370K chars).
[
{
"path": ".gitignore",
"chars": 77,
"preview": "__pycache__\n.hypothesis\n.test\ncoverage.info\n*.egg-info\n*.build.toml\n*.so\n*.o\n"
},
{
"path": ".gitmodules",
"chars": 93,
"preview": "[submodule \"misc/terryfy\"]\n\tpath = misc/terryfy\n\turl = https://github.com/MacPython/terryfy/\n"
},
{
"path": ".travis.yml",
"chars": 1087,
"preview": "language: python\n\nsudo: required\n\nservices:\n - docker\n\nenv:\n global:\n - secure: \"HO3zCuv0FtNFTQ7kkBpIqKYAZW8sYZPfc1"
},
{
"path": "CHANGELOG.md",
"chars": 146,
"preview": "0.1.1 - Feb 9 2017\n------------------\n\n- Native support for OSX\n- Support for older hardware without SSE4.2\n- Better cra"
},
{
"path": "LICENSE.txt",
"chars": 1068,
"preview": "Copyright (c) 2017 Paweł Piotr Przeradowski\n\nPermission is hereby granted, free of charge, to any person obtaining a cop"
},
{
"path": "MANIFEST.in",
"chars": 102,
"preview": "include README.md LICENSE.txt src/picohttpparser/picohttpparser.c src/picohttpparser/picohttpparser.h\n"
},
{
"path": "README.md",
"chars": 4691,
"preview": "# Japronto!\n\n[](https://webchat.freenode."
},
{
"path": "benchmarks/aiohttp/micro.py",
"chars": 308,
"preview": "from aiohttp import web\nimport asyncio\nimport uvloop\n\n\nloop = uvloop.new_event_loop()\nasyncio.set_event_loop(loop)\n\n\nasy"
},
{
"path": "benchmarks/aiohttp/requirements.txt",
"chars": 15,
"preview": "aiohttp==1.2.0\n"
},
{
"path": "benchmarks/gevent/micro.py",
"chars": 562,
"preview": "from gevent.pywsgi import WSGIServer\n\n\ndef hello(environ, start_response):\n if(environ['PATH_INFO'] == '/' and enviro"
},
{
"path": "benchmarks/gevent/requirements.txt",
"chars": 14,
"preview": "gevent==1.2.1\n"
},
{
"path": "benchmarks/golang/README.md",
"chars": 38,
"preview": "```\ngo build .\nGOMAXPROCS=1 ./bin\n```\n"
},
{
"path": "benchmarks/golang/micro.go",
"chars": 377,
"preview": "package main\n\nimport \"net/http\"\n\nvar (\n\thelloResp = []byte(\"Hello world!\")\n\tnotFoundResp = []byte(\"Not Found\")\n)\n\nfun"
},
{
"path": "benchmarks/golang-fasthttp/README.md",
"chars": 38,
"preview": "```\ngo build .\nGOMAXPROCS=1 ./bin\n```\n"
},
{
"path": "benchmarks/golang-fasthttp/micro.go",
"chars": 291,
"preview": "package main\n\nimport \"github.com/valyala/fasthttp\"\n\nfunc hello(ctx *fasthttp.RequestCtx) {\n\tif string(ctx.Path()) != \"/\""
},
{
"path": "benchmarks/japronto/micro.py",
"chars": 191,
"preview": "from japronto import Application\n\n\ndef hello(request):\n return request.Response(text='Hello world!')\n\n\napp = Applicat"
},
{
"path": "benchmarks/meinheld/micro.py",
"chars": 596,
"preview": "from meinheld import server\n\n\ndef hello(environ, start_response):\n if(environ['PATH_INFO'] == '/' and environ['REQUES"
},
{
"path": "benchmarks/meinheld/requirements.txt",
"chars": 16,
"preview": "meinheld==0.6.1\n"
},
{
"path": "benchmarks/nodejs/micro.js",
"chars": 387,
"preview": "const http = require('http');\n\n\nvar srv = http.createServer( (req, res) => {\n res.sendDate = false;\n if(req.url == '/'"
},
{
"path": "benchmarks/sanic/micro.py",
"chars": 191,
"preview": "from sanic import Sanic\nfrom sanic.response import text\n\napp = Sanic(__name__)\n\n\n@app.route(\"/\")\nasync def hello(request"
},
{
"path": "benchmarks/sanic/requirements.txt",
"chars": 13,
"preview": "sanic==0.2.0\n"
},
{
"path": "benchmarks/tornado/micro.py",
"chars": 854,
"preview": "from tornado import web\nfrom tornado.httputil import HTTPHeaders, responses\nfrom tornado.platform.asyncio import AsyncIO"
},
{
"path": "benchmarks/tornado/requirements.txt",
"chars": 15,
"preview": "tornado==4.4.2\n"
},
{
"path": "build.py",
"chars": 12611,
"preview": "import argparse\nimport distutils\nfrom distutils.command.build_ext import build_ext, CompileError\nfrom distutils.core imp"
},
{
"path": "cases/__init__.py",
"chars": 3220,
"preview": "from collections import namedtuple\nimport glob\nimport os.path\n\nimport pytoml\nimport pytest\n\n\ntestcase_fields = 'data,met"
},
{
"path": "cases/base.toml",
"chars": 5282,
"preview": "[10msg]\ndata = \"\"\"\\\nPOST \\\n/wp-content/uploads/2010/03/hello-kitty-darth-vader-pink.jpg \\\nHTTP/1.0\\r\\n\\\nHOST: www.kittyh"
},
{
"path": "cases/websites.toml",
"chars": 2446,
"preview": "[github]\ndata = \"\"\"\\\nGET / HTTP/1.1\\r\\n\\\nHost: github.com\\r\\n\\\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:49"
},
{
"path": "conftest.py",
"chars": 2343,
"preview": "import subprocess\nimport sys\nimport os\nimport shutil\n\n\nbuilds = []\ncoverages = set()\n\n\ndef add_build(mark):\n global b"
},
{
"path": "do_wrk.py",
"chars": 3491,
"preview": "import argparse\nimport sys\nimport asyncio as aio\nimport os\nfrom asyncio.subprocess import PIPE, STDOUT\nimport statistics"
},
{
"path": "examples/1_hello/hello.py",
"chars": 672,
"preview": "from japronto import Application\n\n\n# Views handle logic, take request as a parameter and\n# return the Response object ba"
},
{
"path": "examples/2_async/async.py",
"chars": 754,
"preview": "import asyncio\nfrom japronto import Application\n\n\n# This is a synchronous handler.\ndef synchronous(request):\n return "
},
{
"path": "examples/3_router/router.py",
"chars": 1228,
"preview": "from japronto import Application\n\n\napp = Application()\nr = app.router\n\n\n# Requests with the path set exactly to `/` and "
},
{
"path": "examples/4_request/request.py",
"chars": 3166,
"preview": "from json import JSONDecodeError\n\nfrom japronto import Application\n\n\n# Request line and headers.\n# This represents the p"
},
{
"path": "examples/5_response/response.py",
"chars": 2290,
"preview": "import random\nfrom http.cookies import SimpleCookie\n\nfrom japronto.app import Application\n\n\n# Providing just a text argu"
},
{
"path": "examples/6_exceptions/exceptions.py",
"chars": 1465,
"preview": "from japronto import Application, RouteNotFoundException\n\n\n# These are our custom exceptions we want to turn into 200 re"
},
{
"path": "examples/7_extend/extend.py",
"chars": 1317,
"preview": "from japronto import Application\n\n\n# This view accesses custom method host_startswith\n# and a custom property reversed_a"
},
{
"path": "examples/8_template/index.html",
"chars": 237,
"preview": "<!doctype html>\n\n<html lang=\"en\">\n<head>\n <meta charset=\"utf-8\">\n\n <title>japronto</title>\n <meta name=\"description\" "
},
{
"path": "examples/8_template/template.py",
"chars": 899,
"preview": "# examples/8_template/template.py\nfrom japronto import Application\nfrom jinja2 import Template\n\n\n# A view can read HTML "
},
{
"path": "examples/todo_api/.gitignore",
"chars": 12,
"preview": "todo.sqlite\n"
},
{
"path": "examples/todo_api/todo_api.py",
"chars": 2161,
"preview": "import os.path\nimport sqlite3\nfrom functools import partial\n\nfrom japronto import Application\n\n\ndef add_todo(request):\n "
},
{
"path": "integration_tests/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "integration_tests/common.py",
"chars": 1313,
"preview": "import os\nimport subprocess\nimport ctypes.util\nimport sys\nimport time\n\nimport psutil\n\n\ndef start_server(script, *, stdou"
},
{
"path": "integration_tests/drain.py",
"chars": 363,
"preview": "import asyncio\n\nfrom japronto.app import Application\n\n\ndef slash(request):\n return request.Response()\n\n\nasync def sle"
},
{
"path": "integration_tests/dump.py",
"chars": 1397,
"preview": "import base64\nimport asyncio\n\n\nfrom japronto.app import Application\n\n\nclass ForcedException(Exception):\n pass\n\n\ndef d"
},
{
"path": "integration_tests/experiments.py",
"chars": 462,
"preview": "import pytest\n\n\n@pytest.fixture(autouse=True)\ndef my_fix(request):\n print('auto')\n pytest.set_trace()\n\n\n@pytest.fi"
},
{
"path": "integration_tests/generators.py",
"chars": 2001,
"preview": "from integration_tests import strategies as st\n\nfrom hypothesis.strategies import SearchStrategy\n\n\ndef generate_body(bod"
},
{
"path": "integration_tests/longrun.py",
"chars": 1584,
"preview": "import subprocess\nimport sys\nimport signal\nimport atexit\nimport os\nimport time\n\nsys.path.insert(0, '.')\n\nimport integrat"
},
{
"path": "integration_tests/noleak.py",
"chars": 1029,
"preview": "import sys\n\n\nfrom japronto.app import Application\n\n\nprop = sys.argv[1]\n\nif prop == 'method':\n def noleak(request):\n "
},
{
"path": "integration_tests/reaper.py",
"chars": 243,
"preview": "import sys\n\nfrom japronto.app import Application\n\nreaper_settings = {\n 'check_interval': int(sys.argv[1]),\n 'idle_"
},
{
"path": "integration_tests/strategies.py",
"chars": 1276,
"preview": "import string\nimport re\n\nfrom hypothesis import strategies as st\nsampled_from = st.sampled_from\nfixed_dictionaries = st."
},
{
"path": "integration_tests/test_drain.py",
"chars": 3584,
"preview": "import pytest\nimport subprocess\nimport time\n\nfrom misc import client\nimport integration_tests.common\n\n\npytestmark = pyte"
},
{
"path": "integration_tests/test_noleak.py",
"chars": 3585,
"preview": "import pytest\n\nfrom misc import client\nimport integration_tests.common\n\n\npytestmark = pytest.mark.needs_build(\n '--ex"
},
{
"path": "integration_tests/test_perror.py",
"chars": 3744,
"preview": "import pytest\nfrom hypothesis import given, strategies as st, settings, Verbosity\nimport subprocess\nimport queue\nimport "
},
{
"path": "integration_tests/test_reaper.py",
"chars": 1352,
"preview": "from functools import partial\nimport time\n\nimport pytest\n\nfrom misc import client\nimport integration_tests.common\n\n\npyte"
},
{
"path": "integration_tests/test_request.py",
"chars": 11580,
"preview": "import pytest\nimport json\nimport base64\nfrom functools import partial\n\nfrom hypothesis import given, settings, Verbosity"
},
{
"path": "misc/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "misc/bootstrap.sh",
"chars": 770,
"preview": "#!/bin/bash\n\ncd ~\n\nsudo apt-get update\nsudo apt-get install -y python3 git libbz2-dev libz-dev libsqlite3-dev libssl-dev"
},
{
"path": "misc/buggers.py",
"chars": 397,
"preview": "import atexit\nimport psutil\n\nnoisy = ['atom', 'chrome', 'firefox', 'dropbox', 'opera', 'spotify',\n 'gnome-docume"
},
{
"path": "misc/cleanup_script.py",
"chars": 397,
"preview": "import sys\n\n\ndef main():\n fp = open(sys.argv[1])\n\n for line in fp:\n line = line.rstrip()\n if line.st"
},
{
"path": "misc/client.py",
"chars": 4046,
"preview": "import socket\nimport urllib.parse\nimport json\n\n\ndef readline(sock):\n line = b''\n while not line.endswith(b'\\r\\n'):"
},
{
"path": "misc/collector.py",
"chars": 1126,
"preview": "import psutil\nimport sys\nimport time\nimport os\nimport json\nfrom functools import partial\n\n\ndef get_connections(process):"
},
{
"path": "misc/cpu.py",
"chars": 3315,
"preview": "\"\"\"\nCPU file\n\"\"\"\n\n\n# module imports\nimport subprocess\n\n\n# cpu location\nCPU_PREFIX = '/sys/devices/system/cpu/'\n\n\ndef sav"
},
{
"path": "misc/do_perf.py",
"chars": 1901,
"preview": "import subprocess\nimport os\nimport sys\nimport argparse\n\nimport parsers\nimport cases\nimport buggers\nimport cpu\n\n\ndef get_"
},
{
"path": "misc/docker/Dockerfile",
"chars": 97,
"preview": "FROM python:3.6.0-slim\n\nRUN pip3 install japronto\nENV PYTHONUNBUFFERED=1\nENTRYPOINT [\"japronto\"]\n"
},
{
"path": "misc/parts.py",
"chars": 780,
"preview": "from functools import partial\nimport types\nimport math\n\n\ndef make_parts(value, get_size, dir=1):\n parts = []\n\n lef"
},
{
"path": "misc/perf.md",
"chars": 796,
"preview": "Capturing performance data with perf\n====================================\n\nFor best results the C source should be built"
},
{
"path": "misc/pipeline.lua",
"chars": 915,
"preview": "-- example script demonstrating HTTP pipelining\n\ninit = function(args)\n local r = {}\n r[1] = wrk.format(nil, \"/\")\n "
},
{
"path": "misc/report.py",
"chars": 1579,
"preview": "import matplotlib.pyplot as plt\nimport sys\nimport os\nimport json\n\n\ndef report(samples, pid):\n plt.figure(figsize=(25,"
},
{
"path": "misc/requirements-test.txt",
"chars": 107,
"preview": "-r requirements.txt\nhypothesis==3.80.0\npsutil==5.6.6\npytest==3.9.2\npytoml==0.1.20\nperf==1.5.1\ncffi==1.11.5\n"
},
{
"path": "misc/requirements.txt",
"chars": 15,
"preview": "uvloop>=0.11.3\n"
},
{
"path": "misc/rpm-requirements.txt",
"chars": 22,
"preview": "libasan\nlcov\nlibubsan\n"
},
{
"path": "misc/runpytest.py",
"chars": 32,
"preview": "from pytest import main\n\nmain()\n"
},
{
"path": "misc/simple.py",
"chars": 1764,
"preview": "import asyncio\nimport argparse\nimport os.path\nimport sys\nimport socket\n\nsys.path.insert(0, os.path.abspath(os.path.dirna"
},
{
"path": "misc/suppr.txt",
"chars": 166,
"preview": "# This is a known leak.\n# Python leaks a little, we use malloc directly\nleak:PyMem_RawMalloc\nleak:_PyObject_GC_Resize\nle"
},
{
"path": "misc/travis/before_install.sh",
"chars": 459,
"preview": "#!/bin/bash\n\nexport JAPR_MSG=`git show -s --format=%B | xargs`\nexport JAPR_WHEEL=`[[ $JAPR_MSG == *\"[travis-wheel]\"* ]] "
},
{
"path": "misc/travis/install.sh",
"chars": 398,
"preview": "#!/bin/bash\n\nset -ex\n\nif [[ $JAPR_OS == \"Darwin\" ]]; then\n /usr/bin/clang --version\n source misc/terryfy/travis_tools."
},
{
"path": "misc/travis/script.sh",
"chars": 572,
"preview": "#!/bin/bash\n\nset -ex\n\nif [[ $JAPR_WHEEL == \"1\" ]]; then\n if [[ $JAPR_OS == \"Linux\" ]]; then\n docker run --rm -u `id "
},
{
"path": "setup.py",
"chars": 1905,
"preview": "\"\"\"\nJapronto\n\"\"\"\nimport codecs\nimport os\nimport re\n\nfrom setuptools import setup, find_packages\n\nimport build\n\n\nwith cod"
},
{
"path": "src/japronto/__init__.py",
"chars": 112,
"preview": "from .app import Application # noqa\nfrom .router import RouteNotFoundException # noqa\n\n\n__version__ = '0.1.2'\n"
},
{
"path": "src/japronto/__main__.py",
"chars": 528,
"preview": "import sys\nimport os\n\nfrom .runner import get_parser, verify, run\n\n\ndef main():\n parser = get_parser()\n args = par"
},
{
"path": "src/japronto/app/__init__.py",
"chars": 8668,
"preview": "import signal\nimport asyncio\nimport traceback\nimport socket\nimport os\nimport sys\nimport multiprocessing\nimport faulthand"
},
{
"path": "src/japronto/capsule.c",
"chars": 1004,
"preview": "#include <Python.h>\n\n\nvoid* get_ptr_from_mod(const char* module_name, const char* attr_name,\n cons"
},
{
"path": "src/japronto/capsule.h",
"chars": 462,
"preview": "#pragma once\n\n\nvoid* get_ptr_from_mod(const char* module_name, const char* attr_name,\n const char*"
},
{
"path": "src/japronto/common.h",
"chars": 102,
"preview": "#pragma once\n\n\ntypedef enum {\n KEEP_ALIVE_UNSET,\n KEEP_ALIVE_TRUE,\n KEEP_ALIVE_FALSE\n} KEEP_ALIVE;\n"
},
{
"path": "src/japronto/cpu_features.c",
"chars": 322,
"preview": "#include <cpuid.h>\n#include <assert.h>\n\n#include \"cpu_features.h\"\n\nint supports_x86_sse42(void)\n{\n#if defined(__clang__)"
},
{
"path": "src/japronto/cpu_features.h",
"chars": 44,
"preview": "#pragma once\n\nint supports_x86_sse42(void);\n"
},
{
"path": "src/japronto/parser/.gitignore",
"chars": 20,
"preview": "libpicohttpparser.c\n"
},
{
"path": "src/japronto/parser/__init__.py",
"chars": 158,
"preview": "header_errors = [\n 'malformed_headers', 'incomplete_headers', 'invalid_headers',\n 'excessive_data']\nbody_errors = "
},
{
"path": "src/japronto/parser/build_libpicohttpparser.py",
"chars": 1635,
"preview": "import distutils.log\ndistutils.log.set_verbosity(distutils.log.DEBUG)\n\nimport os.path\n\nimport cffi\nffibuilder = cffi.FFI"
},
{
"path": "src/japronto/parser/cffiparser.py",
"chars": 6367,
"preview": "from parser.libpicohttpparser import ffi, lib\n\n\nclass HttpRequestParser(object):\n def __init__(self, on_headers, on_b"
},
{
"path": "src/japronto/parser/cparser.c",
"chars": 21897,
"preview": "#include <strings.h>\n#include <sys/param.h>\n\n#include \"cparser.h\"\n#include \"cpu_features.h\"\n\n#ifndef PARSER_STANDALONE\n#"
},
{
"path": "src/japronto/parser/cparser.h",
"chars": 1211,
"preview": "#pragma once\n\n#include <stdbool.h>\n#include <Python.h>\n\n#include \"picohttpparser.h\"\n\n\nenum Parser_state {\n PARSER_HEADE"
},
{
"path": "src/japronto/parser/cparser_ext.py",
"chars": 402,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.parser.cparser',\n "
},
{
"path": "src/japronto/parser/test_parser.py",
"chars": 10516,
"preview": "from functools import partial\nfrom itertools import zip_longest\n\nimport pytest\n\nfrom cases import parametrize_cases\nfrom"
},
{
"path": "src/japronto/pipeline/__init__.py",
"chars": 1196,
"preview": "class Pipeline:\n def __init__(self, ready):\n self._queue = []\n self._ready = ready\n\n @property\n d"
},
{
"path": "src/japronto/pipeline/cpipeline.c",
"chars": 8145,
"preview": "#include <Python.h>\n#include \"structmember.h\"\n\n#include \"cpipeline.h\"\n\nstatic PyTypeObject PipelineType;\n\n#ifdef PIPELIN"
},
{
"path": "src/japronto/pipeline/cpipeline.h",
"chars": 1789,
"preview": "#pragma once\n\n#include <stdbool.h>\n\n#include <Python.h>\n\n#ifdef PIPELINE_PAIR\n\ntypedef struct {\n bool is_task;\n PyObje"
},
{
"path": "src/japronto/pipeline/cpipeline_ext.py",
"chars": 294,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.pipeline.cpipeline',"
},
{
"path": "src/japronto/pipeline/test_pipeline.py",
"chars": 4280,
"preview": "import asyncio\nimport gc\nimport sys\nfrom collections import namedtuple\nfrom functools import partial\n\nimport pytest\nimpo"
},
{
"path": "src/japronto/protocol/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "src/japronto/protocol/cprotocol.c",
"chars": 22269,
"preview": "#include <Python.h>\n\n\n#include \"cprotocol.h\"\n#include \"cmatcher.h\"\n#include \"crequest.h\"\n#include \"cresponse.h\"\n#include"
},
{
"path": "src/japronto/protocol/cprotocol.h",
"chars": 1523,
"preview": "#pragma once\n\n#ifndef PARSER_STANDALONE\n#include \"cparser.h\"\n#endif\n\n#include \"cpipeline.h\"\n#include \"crequest.h\"\n#inclu"
},
{
"path": "src/japronto/protocol/cprotocol_ext.py",
"chars": 843,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n cparser = system.get_extension_by_path('japronto/parser/"
},
{
"path": "src/japronto/protocol/creaper.c",
"chars": 7133,
"preview": "#include <Python.h>\n\n#include \"cprotocol.h\"\n#include \"capsule.h\"\n\ntypedef struct {\n PyObject_HEAD\n\n PyObject* connecti"
},
{
"path": "src/japronto/protocol/creaper_ext.py",
"chars": 493,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n define_macros = [('PIPELINE_PAIR', 1)]\n if system.arg"
},
{
"path": "src/japronto/protocol/generator.c",
"chars": 4003,
"preview": "#include <Python.h>\n\n#include \"generator.h\"\n\n\ntypedef struct _Generator {\n PyObject_HEAD\n\n PyObject* object;\n} Generat"
},
{
"path": "src/japronto/protocol/generator.h",
"chars": 277,
"preview": "#pragma once\n\n#ifndef GENERATOR_OPAQUE\nstruct _Generator;\n\n#define GENERATOR struct _Generator\n\nPyObject*\nGenerator_new("
},
{
"path": "src/japronto/protocol/generator_ext.py",
"chars": 219,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'protocol.generator',\n "
},
{
"path": "src/japronto/protocol/handler.py",
"chars": 3400,
"preview": "import asyncio\nfrom asyncio.queues import Queue\n\n\nfrom japronto.response.cresponse import Response\nfrom japronto.protoco"
},
{
"path": "src/japronto/protocol/null.py",
"chars": 154,
"preview": "class NullProtocol:\n def on_headers(self, *args):\n pass\n\n def on_body(self, body):\n pass\n\n def on"
},
{
"path": "src/japronto/protocol/tracing.py",
"chars": 2664,
"preview": "from functools import partial\n\nfrom parser.libpicohttpparser import ffi\nfrom request import HttpRequest\n\n\nclass TracingP"
},
{
"path": "src/japronto/reloader.py",
"chars": 2246,
"preview": "import sys\nimport os.path\nimport os\nimport time\nimport threading\nimport signal\n\n\ndef main():\n import subprocess\n\n "
},
{
"path": "src/japronto/request/__init__.py",
"chars": 5437,
"preview": "import urllib.parse\nfrom json import loads as json_loads\nimport cgi\nimport encodings.idna\nimport collections\nfrom http.c"
},
{
"path": "src/japronto/request/crequest.c",
"chars": 22656,
"preview": "#include <stddef.h>\n#include <sys/param.h>\n#include <strings.h>\n#include <string.h>\n\n#include \"crequest.h\"\n\n#include \"cr"
},
{
"path": "src/japronto/request/crequest.h",
"chars": 1852,
"preview": "#pragma once\n\n#include <Python.h>\n#include <stdbool.h>\n\n#include \"cmatcher.h\"\n#include \"cresponse.h\"\n#include \"common.h\""
},
{
"path": "src/japronto/request/crequest_ext.py",
"chars": 387,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.request.crequest',\n "
},
{
"path": "src/japronto/response/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "src/japronto/response/cresponse.c",
"chars": 14622,
"preview": "#include <Python.h>\n#include <sys/param.h>\n\n#include \"cresponse.h\"\n#include \"capsule.h\"\n#include \"reasons.h\"\n\n#ifdef RES"
},
{
"path": "src/japronto/response/cresponse.h",
"chars": 756,
"preview": "#pragma once\n\n#include <Python.h>\n#include <stdbool.h>\n#include \"common.h\"\n\n\n#define RESPONSE_INITIAL_BUFFER_LEN 1024\n\nt"
},
{
"path": "src/japronto/response/cresponse_ext.py",
"chars": 376,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n define_macros = [('RESPONSE_OPAQUE', 1)]\n if system.a"
},
{
"path": "src/japronto/response/py.py",
"chars": 1258,
"preview": "_responses = None\n\n\ndef factory(status_code=200, text='', mime_type='text/plain',\n encoding='utf-8'):\n glo"
},
{
"path": "src/japronto/response/reasons.h",
"chars": 1547,
"preview": "#pragma once\n\nstatic const char* reasons_1xx[] = {\n \"Continue\", //100\n \"Switching Protocols\", //101\n};\n\nstatic const c"
},
{
"path": "src/japronto/router/__init__.py",
"chars": 694,
"preview": "from .route import Route, RouteNotFoundException\nfrom .cmatcher import Matcher\n\n\nclass Router:\n def __init__(self, ma"
},
{
"path": "src/japronto/router/analyzer.py",
"chars": 1704,
"preview": "import dis\nimport functools\nimport types\n\n\nFLAG_COROUTINE = 128\n\n\ndef is_simple(fun):\n \"\"\"A heuristic to find out if "
},
{
"path": "src/japronto/router/cmatcher.c",
"chars": 9440,
"preview": "#include <Python.h>\n#include <stdbool.h>\n\n#include \"cmatcher.h\"\n#include \"crequest.h\"\n#include \"capsule.h\"\n\n\nstruct _Mat"
},
{
"path": "src/japronto/router/cmatcher.h",
"chars": 497,
"preview": "#pragma once\n\n#include <Python.h>\n#include <stdbool.h>\n\n#include \"match_dict.h\"\n\ntypedef struct {\n PyObject* route;\n P"
},
{
"path": "src/japronto/router/cmatcher_ext.py",
"chars": 245,
"preview": "from distutils.core import Extension\n\n\ndef get_extension():\n return Extension(\n 'japronto.router.cmatcher',\n "
},
{
"path": "src/japronto/router/match_dict.c",
"chars": 927,
"preview": "#include <Python.h>\n\n#include \"match_dict.h\"\n\n\nPyObject*\nMatchDict_entries_to_dict(MatchDictEntry* entries, size_t lengt"
},
{
"path": "src/japronto/router/match_dict.h",
"chars": 221,
"preview": "#pragma once\n\n#include <Python.h>\n\ntypedef struct {\n char* key;\n size_t key_length;\n char* value;\n size_t value_leng"
},
{
"path": "src/japronto/router/matcher.py",
"chars": 1078,
"preview": "class Matcher:\n def __init__(self, routes):\n self._routes = routes\n\n def match_request(self, request):\n "
},
{
"path": "src/japronto/router/route.py",
"chars": 3983,
"preview": "import asyncio\nfrom enum import IntEnum\nfrom struct import Struct\n\nfrom . import analyzer\n\n\nclass RouteNotFoundException"
},
{
"path": "src/japronto/router/test_analyzer.py",
"chars": 1427,
"preview": "from collections import OrderedDict\n\nimport pytest\n\nfrom . import analyzer\n\n\nsimple_fixtures = OrderedDict([\n ('empty"
},
{
"path": "src/japronto/router/test_matcher.py",
"chars": 2728,
"preview": "from functools import partial\n\nimport pytest\n\nfrom . import Route\nfrom .matcher import Matcher\nfrom .cmatcher import Mat"
},
{
"path": "src/japronto/router/test_route.py",
"chars": 2695,
"preview": "import asyncio\nfrom collections import namedtuple\n\nimport pytest\n\nfrom .route import parse, MatcherEntry, Segment, Segme"
},
{
"path": "src/japronto/runner.py",
"chars": 2344,
"preview": "from argparse import ArgumentParser, SUPPRESS\nfrom importlib import import_module\nimport os\nimport sys\nimport runpy\n\nfro"
},
{
"path": "src/picohttpparser/build",
"chars": 221,
"preview": "#!/bin/bash\n\nset -ex\n\ngcc -c picohttpparser.c -O3 -fPIC -msse4.2 -o ssepicohttpparser.o\ngcc -c picohttpparser.c -O3 -fPI"
},
{
"path": "src/picohttpparser/picohttpparser.c",
"chars": 23364,
"preview": "/*\n * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase,\n * Shigeo Mitsunari\n"
},
{
"path": "src/picohttpparser/picohttpparser.h",
"chars": 3795,
"preview": "/*\n * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase,\n * Shigeo Mitsunari\n"
},
{
"path": "tutorial/1_hello.md",
"chars": 2046,
"preview": "# Getting Started\n\nMake sure you have both [pip](https://pip.pypa.io/en/stable/installing/) and at\nleast version 3.5 of "
},
{
"path": "tutorial/2_async.md",
"chars": 1447,
"preview": "# Asynchronous handlers\n\nWith Japronto you can freely combine synchronous and asynchronous handlers and\nfully take advan"
},
{
"path": "tutorial/3_router.md",
"chars": 1857,
"preview": "# Router\n\nThe router is a subsystem responsible for directing incoming requests to\nparticular handlers based on some con"
},
{
"path": "tutorial/4_request.md",
"chars": 3747,
"preview": "# Request Object\n\nRequest represents an incoming HTTP request with a rich set of properties. They can be divided into\nth"
},
{
"path": "tutorial/5_response.md",
"chars": 2889,
"preview": "# Response object\n\nHandlers return Response instances to fulfill requests. They can contain status code, headers and alm"
},
{
"path": "tutorial/6_exceptions.md",
"chars": 2023,
"preview": "# Handling exceptions\n\nThere may be cases where you may want to respond with a custom response instead of a 500 Internal"
},
{
"path": "tutorial/7_extend.md",
"chars": 1952,
"preview": "# Extending Request object\n\nYou can register custom properties and methods on Request object. This is typically done to "
},
{
"path": "tutorial/8_template.md",
"chars": 1391,
"preview": "# Responding with HTML\n\nServing HTML from japronto is as simple as adding a MIME type of `text/html` to the Response. Ji"
}
]
// ... and 2 more files (download for full content)
About this extraction
This page contains the full source code of the squeaky-pl/japronto GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 148 files (337.7 KB), approximately 90.5k tokens, and a symbol index with 540 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.