Showing preview only (216K chars total). Download the full file or copy to clipboard to get everything.
Repository: openresty/replace-filter-nginx-module
Branch: master
Commit: 0b432c649d2d
Files: 27
Total size: 206.9 KB
Directory structure:
gitextract_mewcfnqx/
├── .gitattributes
├── .gitignore
├── .travis.yml
├── README.markdown
├── config
├── src/
│ ├── ddebug.h
│ ├── ngx_http_replace_filter_module.c
│ ├── ngx_http_replace_filter_module.h
│ ├── ngx_http_replace_parse.c
│ ├── ngx_http_replace_parse.h
│ ├── ngx_http_replace_script.c
│ ├── ngx_http_replace_script.h
│ ├── ngx_http_replace_util.c
│ └── ngx_http_replace_util.h
├── t/
│ ├── 01-sanity.t
│ ├── 02-max-buffered.t
│ ├── 03-var.t
│ ├── 04-capturing.t
│ ├── 05-capturing-max-buffered.t
│ ├── 06-if.t
│ ├── 07-multi.t
│ ├── 08-gzip.t
│ ├── 09-unused.t
│ ├── 10-last-modified.t
│ └── 11-skip.t
├── util/
│ └── build.sh
└── valgrind.suppress
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
*.t linguist-language=Text
================================================
FILE: .gitignore
================================================
*.mobi
genmobi.sh
.libs
*.swp
*.slo
*.la
*.swo
*.lo
*~
*.o
print.txt
.rsync
*.tar.gz
dist
build[789]
build
build10
tags
update-readme
*.tmp
go
t/t.sh
releng
reset
*.t_
ctags
src/stream.h
nginx
keepalive
reindex
src/module.[ch]
all
t/servroot/
buildroot/
build1[0-9]
re
*.plist
Makefile
src/script.[ch]
src/parse.[ch]
src/util.[ch]
*.patch
================================================
FILE: .travis.yml
================================================
sudo: required
dist: focal
os: linux
language: c
compiler:
- gcc
addons:
apt:
packages:
- axel
- cpanminus
- libgd-dev
- libpcre3-dev
cache:
apt: true
env:
global:
- JOBS=3
- NGX_BUILD_JOBS=$JOBS
- LUAJIT_PREFIX=/opt/luajit21
- LUAJIT_LIB=$LUAJIT_PREFIX/lib
- LUAJIT_INC=$LUAJIT_PREFIX/include/luajit-2.1
- LUA_INCLUDE_DIR=$LUAJIT_INC
- LD_LIBRARY_PATH=$LUAJIT_LIB:$LD_LIBRARY_PATH
jobs:
- NGINX_VERSION=1.21.4
- NGINX_VERSION=1.25.1 NGX_EXTRA_OPT=--without-pcre2
before_install:
- sudo cpanm --notest Test::Nginx > build.log 2>&1 || (cat build.log && exit 1)
install:
- git clone https://github.com/openresty/openresty.git ../openresty
- git clone https://github.com/openresty/no-pool-nginx.git ../no-pool-nginx
- git clone https://github.com/openresty/openresty-devel-utils.git
- git clone https://github.com/openresty/lua-nginx-module.git ../lua-nginx-module
- git clone https://github.com/openresty/echo-nginx-module.git ../echo-nginx-module
- git clone https://github.com/simpl/ngx_devel_kit.git ../ndk-nginx-module
- git clone https://github.com/openresty/lua-resty-core.git ../lua-resty-core
- git clone https://github.com/openresty/lua-resty-lrucache.git ../lua-resty-lrucache
- git clone -b v2.1-agentzh https://github.com/openresty/luajit2.git luajit2
- git clone https://github.com/openresty/sregex.git
script:
- cd sregex && sudo make PREFIX=/usr install && cd ..
- cd luajit2/
- make -j$JOBS CCDEBUG=-g Q= PREFIX=$LUAJIT_PREFIX CC=$CC XCFLAGS='-DLUA_USE_APICHECK -DLUA_USE_ASSERT -msse4.2' > build.log 2>&1 || (cat build.log && exit 1)
- sudo make install PREFIX=$LUAJIT_PREFIX > build.log 2>&1 || (cat build.log && exit 1)
- cd ..
- export PATH=$PWD/work/nginx/sbin:$PWD/openresty-devel-utils:$PATH
- sh util/build.sh $NGINX_VERSION > build.log 2>&1 || (cat build.log && exit 1)
- nginx -V
- prove -r t
================================================
FILE: README.markdown
================================================
Name
====
ngx_replace_filter - Streaming regular expression replacement in response bodies.
*This module is not distributed with the Nginx source.* See [the installation instructions](#installation).
Table of Contents
=================
* [Name](#name)
* [Status](#status)
* [Synopsis](#synopsis)
* [Description](#description)
* [Directives](#directives)
* [replace_filter](#replace_filter)
* [replace_filter_types](#replace_filter_types)
* [replace_filter_max_buffered_size](#replace_filter_max_buffered_size)
* [replace_filter_last_modified](#replace_filter_last_modified)
* [replace_filter_skip](#replace_filter_skip)
* [Installation](#installation)
* [Trouble Shooting](#trouble-shooting)
* [TODO](#todo)
* [Community](#community)
* [English Mailing List](#english-mailing-list)
* [Chinese Mailing List](#chinese-mailing-list)
* [Bugs and Patches](#bugs-and-patches)
* [Author](#author)
* [Copyright and License](#copyright-and-license)
* [See Also](#see-also)
Status
======
This module is already quite usable though still at the early phase of development
and is considered experimental.
Synopsis
========
```nginx
location /t {
default_type text/html;
echo abc;
replace_filter 'ab|abc' X;
}
location / {
# proxy_pass/fastcgi_pass/...
# caseless global substitution:
replace_filter '\d+' 'blah blah' 'ig';
replace_filter_types text/plain text/css;
}
location /a {
# proxy_pass/fastcgi_pass/root/...
# remove line-leading spaces and line-trailing spaces,
# as well as blank lines:
replace_filter '^\s+|\s+$' '' g;
}
location /b {
# proxy_pass/fastcgi_pass/root/...
# only remove line-leading spaces and line-trailing spaces:
replace_filter '^[ \f\t]+|[ \f\t]+$' '' g;
}
location ~ '\.cpp$' {
# proxy_pass/fastcgi_pass/root/...
replace_filter_types text/plain;
# skip C/C++ string literals:
replace_filter "'(?:\\\\[^\n]|[^'\n])*'" $& g;
replace_filter '"(?:\\\\[^\n]|[^"\n])*"' $& g;
# remove all those ugly C/C++ comments:
replace_filter '/\*.*?\*/|//[^\n]*' '' g;
}
```
Description
===========
This Nginx output filter module tries to do regular expression substitutions in
a non-buffered manner wherever possible.
This module does *not* use traditional backtracking regular expression engines like PCRE, rather,
it uses the new [sregex](https://github.com/agentzh/sregex) library implemented by the author himself, which was designed with streaming processing in mind from the very beginning:
A good common subset of Perl 5 regular expressions is supported by `sregex`. For the complete
feature list, check out sregex's documentation:
https://github.com/agentzh/sregex#syntax-supported
Response body data is only buffered when absolutely necessary, like facing an incomplete capture that belongs to a possible match near the data chunk boundaries.
[Back to TOC](#table-of-contents)
Directives
==========
[Back to TOC](#table-of-contents)
replace_filter
--------------
**syntax:** *replace_filter <regex> <replace>*
**syntax:** *replace_filter <regex> <replace> <options>*
**default:** *no*
**context:** *http, server, location, location if*
**phase:** *output body filter*
Specifies the regex pattern and text to be replaced, with optional regex flags.
By default, the filter stops matching after the first match is found. This behavior can be changed by specifying the `g` regex option.
The following regex options are supported:
* `g`
for global search and substitution (default off)
* `i`
for case-insensitive matching (default off)
Multiple options can be combined in a single string argument, for example:
```nginx
replace_filter hello hiya ig;
```
Nginx variables can be interpolated into the text to be replaced, for example:
```nginx
replace_filter \w+ "[$foo,$bar]";
```
If you want to use the literal dollar sign character (`$`), use the `$$` sequence for that,
for instance:
```nginx
replace_filter \w "$$";
```
Use of submatch capturing variables like `$&`, `$1`, `$2`, and etc are also supported, for example,
```nginx
replace_filter [bc]|d [$&-$1-$2] g;
```
The semantics of the submatch capturing variables is exactly the same as in the Perl 5 language.
Multiple `replace_filter` directives in the same scope is also supported.
All the patterns will be applied at the same time as in a tokenizer.
We will *not* use the longest token match semantics, but rather, patterns will be prioritized according to their order in
the configure file.
Here is an example for removing all the C/C++ comments from a C/C++ source code file:
```nginx
replace_filter "'(?:\\\\[^\n]|[^'\n])*'" $& g;
replace_filter '"(?:\\\\[^\n]|[^"\n])*"' $& g;
replace_filter '/\*.*?\*/|//[^\n]*' '' g;
```
When the `Content-Encoding` response header is not empty (like `gzip`), the response
body will always remain intact. So usually you want to disable the gzip compression
in your backend servers' responses by adding the following line to your `nginx.conf`
if you are the ngx_proxy module:
```nginx
proxy_set_header Accept-Encoding '';
```
Your responses can still be gzip compressed on the Nginx server level though.
[Back to TOC](#table-of-contents)
replace_filter_types
--------------------
**syntax:** *replace_filter_types <mime-type> ...*
**default:** *replace_filter_types text/html*
**context:** *http, server, location, location if*
**phase:** *output body filter*
Specify one or more MIME types (in the `Content-Type` response header) to be processed.
By default, only `text/html` typed responses are processed.
[Back to TOC](#table-of-contents)
replace_filter_max_buffered_size
---------------------------------
**syntax:** *replace_filter_max_buffered_size <size>*
**default:** *replace_filter_max_buffered_size 8k*
**context:** *http, server, location, location if*
**phase:** *output body filter*
Limits the total size of the data buffered by the module at runtime. Default to `8k`.
When the limit is reached, `replace_filter` will immediately stop processing and
leave all the remaining response body data intact.
[Back to TOC](#table-of-contents)
replace_filter_last_modified
----------------------------
**syntax:** *replace_filter_last_modifiled keep | clear*
**default:** *replace_filter_last_modified clear*
**context:** *http, server, location, location if*
**phase:** *output body filter*
Controls how to deal with the existing Last-Modified response header.
By default, this module will clear the `Last-Modified` response header if there is any. You can specify
```nginx
replace_filter_last_modified keep;
```
to always keep the original `Last-Modified` response header.
[Back to TOC](#table-of-contents)
replace_filter_skip
-------------------
**syntax:** *replace_filter_skip $var*
**default:** *no*
**context:** *http, server, location, location if*
**phase:** *output header filter*
This directive controls whether to skip all the `replace_filter` rules on a per-request basis.
Both constant values or strings containing NGINX variables are supported.
When the value is evaluated to an empty value ("") or the value "0" in the request output header phase, no `replace_filter` rules will be skipped for the current request. Otherwise all the `replace_filter` rules will be skipped for the current request.
Below is a trivial example for this:
```nginx
set $skip '';
location /t {
content_by_lua '
ngx.var.skip = 1
ngx.say("abcabd")
';
replace_filter_skip $skip;
replace_filter abcabd X;
}
```
[Back to TOC](#table-of-contents)
Installation
============
You need to install the sregex library first:
https://github.com/agentzh/sregex
And then rebuild your Nginx like this:
```bash
./configure --add-module=/path/to/replace-filter-nginx-module
```
If sregex is not installed to the default prefix (i.e., `/usr/local`), then
you should specify the locations of your sregex installation via
the `SREGEX_INC` and `SREGEX_LIB` environments before running the
`./configure` script, as in
```bash
export SREGEX_INC=/opt/sregex/include
export SREGEX_LIB=/opt/sregex/lib
```
assuming that your sregex is installed to the prefix `/opt/sregex`.
Starting from NGINX 1.9.11, you can also compile this module as a dynamic module, by using the `--add-dynamic-module=PATH` option instead of `--add-module=PATH` on the
`./configure` command line above. And then you can explicitly load the module in your `nginx.conf` via the [load_module](http://nginx.org/en/docs/ngx_core_module.html#load_module)
directive, for example,
```nginx
load_module /path/to/modules/ngx_http_replace_filter_module.so;
```
[Back to TOC](#table-of-contents)
Trouble Shooting
================
* If you are seeing the error "error while loading shared libraries: libsregex.so.0: cannot open shared object file: No such file or directory"
while starting nginx, then it means that the installation path of your libsregex library
is not in your system's default library search path. You can solve this issue by passing the option `--with-ld-opt='-Wl,-rpath,/usr/local/lib'` to nginx's `./configure` command. Alternatively, you can just add the path of your libsregex.so.0 to the `LD_LIBRARY_PATH` environment value before starting your nginx server.
[Back to TOC](#table-of-contents)
TODO
====
* optimize the special case for verbatim substitutions, i.e., `replace_filter <regex> $&;`.
* implement the `replace_filter_skip $var` directive to control whether to enable the filter on the fly.
* reduce the amount of data that has to be buffered for when an partial match is already found.
* recycle the memory blocks used to buffer the pending capture data and "complex values" for replacement.
* allow use of inlined Lua code as the `replacement` argument of the `replace_filter` directive to generate the text to be replaced on-the-fly.
[Back to TOC](#table-of-contents)
Community
=========
[Back to TOC](#table-of-contents)
English Mailing List
--------------------
The [openresty-en](https://groups.google.com/group/openresty-en) mailing list is for English speakers.
[Back to TOC](#table-of-contents)
Chinese Mailing List
--------------------
The [openresty](https://groups.google.com/group/openresty) mailing list is for Chinese speakers.
[Back to TOC](#table-of-contents)
Bugs and Patches
================
Please submit bug reports, wishlists, or patches by
1. creating a ticket on the [GitHub Issue Tracker](http://github.com/agentzh/replace-filter-nginx-module/issues),
1. or posting to the [OpenResty community](http://wiki.nginx.org/HttpLuaModule#Community).
[Back to TOC](#table-of-contents)
Author
======
Yichun "agentzh" Zhang (章亦春) <agentzh@gmail.com>, OpenResty Inc.
[Back to TOC](#table-of-contents)
Copyright and License
=====================
This module is licensed under the BSD license.
Copyright (C) 2012-2017, by Yichun "agentzh" Zhang (章亦春), OpenResty Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
[Back to TOC](#table-of-contents)
See Also
========
* agentzh's sregex library: https://github.com/agentzh/sregex
* The standard ngx_sub_filter module: http://nginx.org/en/docs/http/ngx_http_sub_module.html
* Slides for my talk "sregex: matching Perl 5 regexes on data streams": http://agentzh.org/misc/slides/yapc-na-2013-sregex.pdf
[Back to TOC](#table-of-contents)
================================================
FILE: config
================================================
ngx_feature="agentzh's sregex library"
ngx_feature_libs="-lsregex"
ngx_feature_name=
ngx_feature_run=no
ngx_feature_incs="#include <sregex/sregex.h>"
ngx_feature_path=
ngx_feature_test="sre_regex_t *re;
sre_program_t *prog;
sre_pool_t *pool;
u_char s[] = {'a', 'b', 'c'};
sre_int_t rc, err_offset, ovector;
sre_int_t *pending_matched;
sre_uint_t ncaps;
sre_vm_pike_ctx_t *pctx;
pool = sre_create_pool(1024);
re = sre_regex_parse(pool, s, &ncaps, 0, &err_offset);
prog = sre_regex_compile(pool, re);
pctx = sre_vm_pike_create_ctx(pool, prog, &ovector, 0);
rc = sre_vm_pike_exec(pctx, s, 32, 1, &pending_matched);
sre_destroy_pool(pool)"
if [ -n "$SREGEX_INC" -o -n "$SREGEX_LIB" ]; then
# explicitly set sregex lib path
ngx_feature="agentzh's sregex library in $SREGEX_LIB and $SREGEX_INC (specified by the SREGEX_LIB and SREGEX_INC env)"
ngx_feature_path="$SREGEX_INC"
if [ $NGX_RPATH = YES ]; then
ngx_feature_libs="-R$SREGEX_LIB -L$SREGEX_LIB -lsregex"
else
ngx_feature_libs="-L$SREGEX_LIB -lsregex"
fi
. auto/feature
if [ $ngx_found = no ]; then
cat << END
$0: error: ngx_http_replace_filter_module requires agentzh's sregex library and SREGEX_LIB is defined as $SREGEX_LIB and SREGEX_INC (path for sregex/sregex.h) $SREGEX_INC, but we cannot find sregex there.
END
exit 1
fi
else
# auto-discovery
ngx_feature="agentzh's sregex library"
ngx_feature_libs="-lsregex"
. auto/feature
if [ $ngx_found = no ]; then
# default installation prefix in sregex
ngx_feature="agentzh's sregex library in /usr/local/"
ngx_feature_path="/usr/local/include"
if [ $NGX_RPATH = YES ]; then
ngx_feature_libs="-R/usr/local/lib -L/usr/local/lib -lsregex"
else
ngx_feature_libs="-L/usr/local/lib -lsregex"
fi
. auto/feature
fi
fi
if [ $ngx_found = no ]; then
cat << END
$0: error: ngx_http_replace_filter_module requires agentzh's sregex library.
END
exit 1
fi
REPLACE_FILTER_SRCS="$ngx_addon_dir/src/ngx_http_replace_filter_module.c \
$ngx_addon_dir/src/ngx_http_replace_script.c \
$ngx_addon_dir/src/ngx_http_replace_parse.c \
$ngx_addon_dir/src/ngx_http_replace_util.c"
REPLACE_FILTER_DEPS="$ngx_addon_dir/src/ngx_http_replace_filter_module.h \
$ngx_addon_dir/src/ngx_http_replace_script.h \
$ngx_addon_dir/src/ngx_http_replace_parse.h \
$ngx_addon_dir/src/ngx_http_replace_util.h"
ngx_addon_name=ngx_http_replace_filter_module
if test -n "$ngx_module_link"; then
ngx_module_type=HTTP_AUX_FILTER
ngx_module_name=$ngx_addon_name
ngx_module_srcs="$REPLACE_FILTER_SRCS"
ngx_module_deps="$REPLACE_FILTER_DEPS"
ngx_module_incs=$ngx_feature_path
ngx_module_libs=$ngx_feature_libs
. auto/module
else
HTTP_AUX_FILTER_MODULES="$HTTP_AUX_FILTER_MODULES $ngx_addon_name"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS $REPLACE_FILTER_SRCS"
NGX_ADDON_DEPS="$NGX_ADDON_DEPS $REPLACE_FILTER_DEPS"
CORE_INCS="$CORE_INCS $ngx_feature_path"
CORE_LIBS="$CORE_LIBS $ngx_feature_libs"
fi
================================================
FILE: src/ddebug.h
================================================
#ifndef DDEBUG_H
#define DDEBUG_H
#include <ngx_core.h>
#include <ngx_http.h>
#include <assert.h>
#if defined(DDEBUG) && (DDEBUG)
# if (NGX_HAVE_VARIADIC_MACROS)
# define dd(...) fprintf(stderr, "replace_filter *** %s: ", __func__); \
fprintf(stderr, __VA_ARGS__); \
fprintf(stderr, " at %s line %d.\n", __FILE__, __LINE__)
# else
#include <stdarg.h>
#include <stdio.h>
#include <stdarg.h>
static void dd(const char * fmt, ...) {
}
# endif
# if DDEBUG > 1
# define dd_enter() dd_enter_helper(r, __func__)
static void dd_enter_helper(ngx_http_request_t *r, const char *func) {
ngx_http_posted_request_t *pr;
fprintf(stderr, ">enter %s %.*s %.*s?%.*s c:%d m:%p r:%p ar:%p pr:%p",
func,
(int) r->method_name.len, r->method_name.data,
(int) r->uri.len, r->uri.data,
(int) r->args.len, r->args.data,
0/*(int) r->main->count*/, r->main,
r, r->connection->data, r->parent);
if (r->posted_requests) {
fprintf(stderr, " posted:");
for (pr = r->posted_requests; pr; pr = pr->next) {
fprintf(stderr, "%p,", pr);
}
}
fprintf(stderr, "\n");
}
# else
# define dd_enter()
# endif
#else
# if (NGX_HAVE_VARIADIC_MACROS)
# define dd(...)
# define dd_enter()
# else
#include <stdarg.h>
static void dd(const char * fmt, ...) {
}
static void dd_enter() {
}
# endif
#endif
#if defined(DDEBUG) && (DDEBUG)
#define dd_check_read_event_handler(r) \
dd("r->read_event_handler = %s", \
r->read_event_handler == ngx_http_block_reading ? \
"ngx_http_block_reading" : \
r->read_event_handler == ngx_http_test_reading ? \
"ngx_http_test_reading" : \
r->read_event_handler == ngx_http_request_empty_handler ? \
"ngx_http_request_empty_handler" : "UNKNOWN")
#define dd_check_write_event_handler(r) \
dd("r->write_event_handler = %s", \
r->write_event_handler == ngx_http_handler ? \
"ngx_http_handler" : \
r->write_event_handler == ngx_http_core_run_phases ? \
"ngx_http_core_run_phases" : \
r->write_event_handler == ngx_http_request_empty_handler ? \
"ngx_http_request_empty_handler" : "UNKNOWN")
#else
#define dd_check_read_event_handler(r)
#define dd_check_write_event_handler(r)
#endif
#endif /* DDEBUG_H */
================================================
FILE: src/ngx_http_replace_filter_module.c
================================================
/*
* Copyright (C) Yichun Zhang (agentzh)
* Copyright (C) Igor Sysoev
* Copyright (C) Nginx, Inc.
*/
#ifndef DDEBUG
#define DDEBUG 0
#endif
#include "ddebug.h"
#include "ngx_http_replace_filter_module.h"
#include "ngx_http_replace_parse.h"
#include "ngx_http_replace_script.h"
#include "ngx_http_replace_util.h"
enum {
SREGEX_COMPILER_POOL_SIZE = 4096
};
static ngx_int_t ngx_http_replace_output(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx);
static char *ngx_http_replace_filter(ngx_conf_t *cf, ngx_command_t *cmd,
void *conf);
static void *ngx_http_replace_create_loc_conf(ngx_conf_t *cf);
static char *ngx_http_replace_merge_loc_conf(ngx_conf_t *cf,
void *parent, void *child);
static ngx_int_t ngx_http_replace_filter_init(ngx_conf_t *cf);
static void ngx_http_replace_cleanup_pool(void *data);
static void *ngx_http_replace_create_main_conf(ngx_conf_t *cf);
#define ngx_http_replace_regex_is_disabled(ctx) \
((ctx)->disabled[(ctx)->regex_id / 8] & (1 << ((ctx)->regex_id % 8)))
#define ngx_http_replace_regex_set_disabled(ctx) \
(ctx)->disabled[(ctx)->regex_id / 8] |= (1 << ((ctx)->regex_id % 8))
static volatile ngx_cycle_t *ngx_http_replace_prev_cycle = NULL;
#define NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED 0
#define NGX_HTTP_REPLACE_KEEP_LAST_MODIFIED 1
static ngx_conf_enum_t ngx_http_replace_filter_last_modified[] = {
{ ngx_string("clear"), NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED },
{ ngx_string("keep"), NGX_HTTP_REPLACE_KEEP_LAST_MODIFIED },
{ ngx_null_string, 0 }
};
static ngx_command_t ngx_http_replace_filter_commands[] = {
{ ngx_string("replace_filter"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF
|NGX_CONF_TAKE23,
ngx_http_replace_filter,
NGX_HTTP_LOC_CONF_OFFSET,
0,
NULL },
{ ngx_string("replace_filter_types"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF
|NGX_CONF_1MORE,
ngx_http_types_slot,
NGX_HTTP_LOC_CONF_OFFSET,
offsetof(ngx_http_replace_loc_conf_t, types_keys),
&ngx_http_html_default_types[0] },
{ ngx_string("replace_filter_max_buffered_size"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF
|NGX_CONF_TAKE1,
ngx_conf_set_size_slot,
NGX_HTTP_LOC_CONF_OFFSET,
offsetof(ngx_http_replace_loc_conf_t, max_buffered_size),
NULL },
{ ngx_string("replace_filter_last_modified"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF
|NGX_CONF_1MORE,
ngx_conf_set_enum_slot,
NGX_HTTP_LOC_CONF_OFFSET,
offsetof(ngx_http_replace_loc_conf_t, last_modified),
&ngx_http_replace_filter_last_modified },
{ ngx_string("replace_filter_skip"),
NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF
|NGX_CONF_TAKE1,
ngx_http_set_complex_value_slot,
NGX_HTTP_LOC_CONF_OFFSET,
offsetof(ngx_http_replace_loc_conf_t, skip),
NULL },
ngx_null_command
};
static ngx_http_module_t ngx_http_replace_filter_module_ctx = {
NULL, /* preconfiguration */
ngx_http_replace_filter_init, /* postconfiguration */
ngx_http_replace_create_main_conf, /* create main configuration */
NULL, /* init main configuration */
NULL, /* create server configuration */
NULL, /* merge server configuration */
ngx_http_replace_create_loc_conf, /* create location configuration */
ngx_http_replace_merge_loc_conf /* merge location configuration */
};
ngx_module_t ngx_http_replace_filter_module = {
NGX_MODULE_V1,
&ngx_http_replace_filter_module_ctx, /* module context */
ngx_http_replace_filter_commands, /* module directives */
NGX_HTTP_MODULE, /* module type */
NULL, /* init master */
NULL, /* init module */
NULL, /* init process */
NULL, /* init thread */
NULL, /* exit thread */
NULL, /* exit process */
NULL, /* exit master */
NGX_MODULE_V1_PADDING
};
static ngx_http_output_header_filter_pt ngx_http_next_header_filter;
static ngx_http_output_body_filter_pt ngx_http_next_body_filter;
static ngx_int_t
ngx_http_replace_header_filter(ngx_http_request_t *r)
{
size_t size;
ngx_str_t skip;
ngx_pool_cleanup_t *cln;
ngx_http_replace_ctx_t *ctx;
ngx_http_replace_loc_conf_t *rlcf;
rlcf = ngx_http_get_module_loc_conf(r, ngx_http_replace_filter_module);
dd("replace header filter");
if (rlcf->regexes.nelts == 0
|| r->headers_out.content_length_n == 0
|| (r->headers_out.content_encoding
&& r->headers_out.content_encoding->value.len)
|| ngx_http_test_content_type(r, &rlcf->types) == NULL)
{
return ngx_http_next_header_filter(r);
}
dd("skip: %p", rlcf->skip);
if (rlcf->skip != NULL) {
if (ngx_http_complex_value(r, rlcf->skip, &skip) != NGX_OK) {
return NGX_ERROR;
}
if (skip.len && (skip.len != 1 || skip.data[0] != '0')) {
return ngx_http_next_header_filter(r);
}
}
ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_replace_ctx_t));
if (ctx == NULL) {
return NGX_ERROR;
}
ctx->last_special = &ctx->special;
ctx->last_pending = &ctx->pending;
ctx->last_pending2 = &ctx->pending2;
ctx->last_captured = &ctx->captured;
ctx->sub = ngx_pcalloc(r->pool,
rlcf->multi_replace.nelts * sizeof(ngx_str_t));
if (ctx->sub == NULL) {
return NGX_ERROR;
}
ctx->ovector = ngx_palloc(r->pool, rlcf->ovecsize);
if (ctx->ovector == NULL) {
return NGX_ERROR;
}
size = ngx_align(rlcf->regexes.nelts, 8) / 8;
ctx->disabled = ngx_pcalloc(r->pool, size);
if (ctx->disabled == NULL) {
return NGX_ERROR;
}
ctx->vm_pool = sre_create_pool(1024);
if (ctx->vm_pool == NULL) {
return NGX_ERROR;
}
dd("created vm pool %p", ctx->vm_pool);
cln = ngx_pool_cleanup_add(r->pool, 0);
if (cln == NULL) {
sre_destroy_pool(ctx->vm_pool);
return NGX_ERROR;
}
cln->data = ctx->vm_pool;
cln->handler = ngx_http_replace_cleanup_pool;
ctx->vm_ctx = sre_vm_pike_create_ctx(ctx->vm_pool, rlcf->program,
ctx->ovector, rlcf->ovecsize);
if (ctx->vm_ctx == NULL) {
return NGX_ERROR;
}
ngx_http_set_ctx(r, ctx, ngx_http_replace_filter_module);
ctx->last_out = &ctx->out;
r->filter_need_in_memory = 1;
if (r == r->main) {
ngx_http_clear_content_length(r);
if (rlcf->last_modified == NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED) {
ngx_http_clear_last_modified(r);
}
}
return ngx_http_next_header_filter(r);
}
static void
ngx_http_replace_cleanup_pool(void *data)
{
sre_pool_t *pool = data;
if (pool) {
dd("destroy sre pool %p", pool);
sre_destroy_pool(pool);
}
}
static ngx_int_t
ngx_http_replace_body_filter(ngx_http_request_t *r, ngx_chain_t *in)
{
ngx_int_t rc;
ngx_buf_t *b;
ngx_str_t *sub;
ngx_chain_t *cl, *cur = NULL, *rematch = NULL;
ngx_http_replace_ctx_t *ctx;
ngx_http_replace_loc_conf_t *rlcf;
rlcf = ngx_http_get_module_loc_conf(r, ngx_http_replace_filter_module);
ctx = ngx_http_get_module_ctx(r, ngx_http_replace_filter_module);
if (ctx == NULL) {
return ngx_http_next_body_filter(r, in);
}
if ((in == NULL
&& ctx->buf == NULL
&& ctx->in == NULL
&& ctx->busy == NULL))
{
return ngx_http_next_body_filter(r, in);
}
if ((ctx->once || ctx->vm_done) && (ctx->buf == NULL || ctx->in == NULL)) {
if (ctx->busy) {
if (ngx_http_replace_output(r, ctx) == NGX_ERROR) {
return NGX_ERROR;
}
}
return ngx_http_next_body_filter(r, in);
}
/* add the incoming chain to the chain ctx->in */
if (in) {
if (ngx_chain_add_copy(r->pool, &ctx->in, in) != NGX_OK) {
return NGX_ERROR;
}
}
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http sub filter \"%V\"", &r->uri);
while (ctx->in || ctx->buf) {
if (ctx->buf == NULL) {
cur = ctx->in;
ctx->buf = cur->buf;
ctx->in = cur->next;
ctx->pos = ctx->buf->pos;
ctx->special_buf = ngx_buf_special(ctx->buf);
ctx->last_buf = (ctx->buf->last_buf || ctx->buf->last_in_chain);
dd("=== new incoming buf: size=%d, special=%u, last=%u",
(int) ngx_buf_size(ctx->buf), ctx->special_buf,
ctx->last_buf);
}
b = NULL;
while (ctx->pos < ctx->buf->last
|| (ctx->special_buf && ctx->last_buf))
{
rc = rlcf->parse_buf(r, ctx, rematch);
ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"replace filter parse: %d, %p-%p",
rc, ctx->copy_start, ctx->copy_end);
if (rc == NGX_ERROR) {
return rc;
}
if (rc == NGX_DECLINED) {
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (!ctx->special_buf) {
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
} else {
ctx->copy_start = NULL;
ctx->copy_end = NULL;
}
sre_reset_pool(ctx->vm_pool);
ctx->vm_done = 1;
}
dd("copy_end - copy_start: %d, special: %u",
(int) (ctx->copy_end - ctx->copy_start), ctx->special_buf);
if (ctx->copy_start != ctx->copy_end && !ctx->special_buf) {
dd("copy: %.*s", (int) (ctx->copy_end - ctx->copy_start),
ctx->copy_start);
cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free);
if (cl == NULL) {
return NGX_ERROR;
}
b = cl->buf;
b->memory = 1;
b->pos = ctx->copy_start;
b->last = ctx->copy_end;
*ctx->last_out = cl;
ctx->last_out = &cl->next;
}
if (rc == NGX_AGAIN) {
if (ctx->special_buf && ctx->last_buf) {
break;
}
continue;
}
if (rc == NGX_DECLINED) {
break;
}
/* rc == NGX_OK || rc == NGX_BUSY */
cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free);
if (cl == NULL) {
return NGX_ERROR;
}
b = cl->buf;
dd("free data buf: %p", b);
sub = &ctx->sub[ctx->regex_id];
if (sub->data == NULL
|| rlcf->parse_buf == ngx_http_replace_capturing_parse)
{
ngx_http_replace_complex_value_t *cv;
if (ngx_http_replace_regex_is_disabled(ctx)) {
cv = &rlcf->verbatim;
} else {
cv = rlcf->multi_replace.elts;
cv = &cv[ctx->regex_id];
}
if (ngx_http_replace_complex_value(r, ctx->captured,
rlcf->ncaps,
ctx->ovector,
cv, sub)
!= NGX_OK)
{
return NGX_ERROR;
}
/* release ctx->captured */
if (ctx->captured) {
dd("release ctx captured: %p", ctx->captured);
*ctx->last_captured = ctx->free;
ctx->free = ctx->captured;
ctx->captured = NULL;
ctx->last_captured = &ctx->captured;
}
}
dd("emit replaced value: \"%.*s\"", (int) sub->len, sub->data);
if (sub->len) {
b->memory = 1;
b->pos = sub->data;
b->last = sub->data + sub->len;
} else {
b->sync = 1;
}
cl->buf = b;
cl->next = NULL;
*ctx->last_out = cl;
ctx->last_out = &cl->next;
if (!ctx->once && !ngx_http_replace_regex_is_disabled(ctx)) {
uint8_t *once;
once = rlcf->multi_once.elts;
if (rlcf->regexes.nelts == 1) {
ctx->once = once[0];
} else {
if (once[ctx->regex_id]) {
ngx_http_replace_regex_set_disabled(ctx);
if (!rlcf->seen_global
&& ++ctx->disabled_count == rlcf->regexes.nelts)
{
ctx->once = 1;
}
}
}
}
if (rc == NGX_BUSY) {
dd("goto rematch");
goto rematch;
}
if (ctx->special_buf) {
break;
}
continue;
}
if ((ctx->buf->flush || ctx->last_buf || ngx_buf_in_memory(ctx->buf))
&& cur)
{
if (b == NULL) {
cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free);
if (cl == NULL) {
return NGX_ERROR;
}
b = cl->buf;
b->sync = 1;
*ctx->last_out = cl;
ctx->last_out = &cl->next;
}
dd("setting shadow and last buf: %d", (int) ctx->buf->last_buf);
b->last_buf = ctx->buf->last_buf;
b->last_in_chain = ctx->buf->last_in_chain;
b->flush = ctx->buf->flush;
b->shadow = ctx->buf;
b->recycled = ctx->buf->recycled;
}
if (!ctx->special_buf) {
ctx->stream_pos += ctx->buf->last - ctx->buf->pos;
}
if (rematch) {
rematch->next = ctx->free;
ctx->free = rematch;
rematch = NULL;
}
rematch:
dd("ctx->rematch: %p", ctx->rematch);
if (ctx->rematch == NULL) {
ctx->buf = NULL;
cur = NULL;
} else {
if (cur) {
ctx->in = cur;
cur = NULL;
}
ctx->buf = ctx->rematch->buf;
dd("ctx->buf set to rematch buf %p, len=%d, next=%p",
ctx->buf, (int) ngx_buf_size(ctx->buf), ctx->rematch->next);
rematch = ctx->rematch;
ctx->rematch = rematch->next;
ctx->pos = ctx->buf->pos;
ctx->special_buf = ngx_buf_special(ctx->buf);
ctx->last_buf = (ctx->buf->last_buf || ctx->buf->last_in_chain);
ctx->stream_pos = ctx->buf->file_pos;
}
#if (DDEBUG)
/*
ngx_http_replace_dump_chain("ctx->pending", &ctx->pending,
ctx->last_pending);
ngx_http_replace_dump_chain("ctx->pending2", &ctx->pending2,
ctx->last_pending2);
*/
#endif
} /* while */
if (ctx->out == NULL && ctx->busy == NULL) {
return NGX_OK;
}
return ngx_http_replace_output(r, ctx);
}
static ngx_int_t
ngx_http_replace_output(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx)
{
ngx_int_t rc;
ngx_buf_t *b;
ngx_chain_t *cl;
#if (DDEBUG)
b = NULL;
for (cl = ctx->out; cl; cl = cl->next) {
ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"replace out: %p %p", cl->buf, cl->buf->pos);
if (cl->buf == b) {
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"the same buf was used in sub");
ngx_debug_point();
return NGX_ERROR;
}
b = cl->buf;
}
/* ngx_http_replace_dump_chain("ctx->out", &ctx->out, ctx->last_out); */
#endif
rc = ngx_http_next_body_filter(r, ctx->out);
/* we are essentially duplicating the logic of
* ngx_chain_update_chains below,
* with our own optimizations */
if (ctx->busy == NULL) {
ctx->busy = ctx->out;
} else {
for (cl = ctx->busy; cl->next; cl = cl->next) { /* void */ }
cl->next = ctx->out;
}
ctx->out = NULL;
ctx->last_out = &ctx->out;
while (ctx->busy) {
cl = ctx->busy;
b = cl->buf;
if (ngx_buf_size(b) != 0) {
break;
}
if (cl->buf->tag != (ngx_buf_tag_t) &ngx_http_replace_filter_module) {
ctx->busy = cl->next;
ngx_free_chain(r->pool, cl);
continue;
}
if (b->shadow) {
b->shadow->pos = b->shadow->last;
b->shadow->file_pos = b->shadow->file_last;
}
ctx->busy = cl->next;
if (ngx_buf_special(b)) {
/* collect special bufs to ctx->special, which may still be busy */
cl->next = NULL;
*ctx->last_special = cl;
ctx->last_special = &cl->next;
} else {
/* add ctx->special to ctx->free because they cannot be busy at
* this point */
*ctx->last_special = ctx->free;
ctx->free = ctx->special;
ctx->special = NULL;
ctx->last_special = &ctx->special;
#if 0
/* free the temporary buf's data block if it is big enough */
if (b->temporary
&& b->start != NULL
&& b->end - b->start > (ssize_t) r->pool->max)
{
ngx_pfree(r->pool, b->start);
}
#endif
/* add the data buf itself to the free buf chain */
cl->next = ctx->free;
ctx->free = cl;
}
}
if (ctx->in || ctx->buf) {
r->buffered |= NGX_HTTP_SUB_BUFFERED;
} else {
r->buffered &= ~NGX_HTTP_SUB_BUFFERED;
}
return rc;
}
static char *
ngx_http_replace_filter(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
ngx_http_replace_loc_conf_t *rlcf = conf;
ngx_http_replace_main_conf_t *rmcf;
int *flags;
u_char *p, **re;
ngx_str_t *value;
ngx_uint_t i;
uint8_t *once;
ngx_pool_cleanup_t *cln;
ngx_http_replace_complex_value_t *cv;
ngx_http_replace_compile_complex_value_t ccv;
value = cf->args->elts;
re = ngx_array_push(&rlcf->regexes);
if (re == NULL) {
return NGX_CONF_ERROR;
}
*re = value[1].data;
cv = ngx_array_push(&rlcf->multi_replace);
if (cv == NULL) {
return NGX_CONF_ERROR;
}
ngx_memzero(cv, sizeof(ngx_http_replace_complex_value_t));
ngx_memzero(&ccv, sizeof(ngx_http_replace_compile_complex_value_t));
ccv.cf = cf;
ccv.value = &value[2];
ccv.complex_value = cv;
if (ngx_http_replace_compile_complex_value(&ccv) != NGX_OK) {
return NGX_CONF_ERROR;
}
/* check variable usage in the "replace" argument */
if (cv->capture_variables) {
rlcf->parse_buf = ngx_http_replace_capturing_parse;
} else if (rlcf->parse_buf == NULL) {
rlcf->parse_buf = ngx_http_replace_non_capturing_parse;
}
#if 0
rlcf->parse_buf = ngx_http_replace_capturing_parse;
#endif
flags = ngx_array_push(&rlcf->multi_flags);
if (flags == NULL) {
return NGX_CONF_ERROR;
}
*flags = 0;
once = ngx_array_push(&rlcf->multi_once);
if (once == NULL) {
return NGX_CONF_ERROR;
}
*once = 1; /* default to once */
if (cf->args->nelts == 4) {
/* 3 user args */
p = value[3].data;
for (i = 0; i < value[3].len; i++) {
switch (p[i]) {
case 'i':
*flags |= SRE_REGEX_CASELESS;
break;
case 'g':
*once = 0;
break;
default:
return "specifies an unrecognized regex flag";
}
}
}
if (*once) {
rlcf->seen_once = 1;
} else {
rlcf->seen_global = 1;
}
if (rlcf->seen_once && rlcf->regexes.nelts > 1) {
rlcf->parse_buf = ngx_http_replace_capturing_parse;
if (rlcf->verbatim.value.data == NULL) {
ngx_str_t v = ngx_string("$&");
ngx_memzero(&ccv, sizeof(ngx_http_replace_compile_complex_value_t));
ccv.cf = cf;
ccv.value = &v;
ccv.complex_value = &rlcf->verbatim;
if (ngx_http_replace_compile_complex_value(&ccv) != NGX_OK) {
return NGX_CONF_ERROR;
}
}
}
rmcf =
ngx_http_conf_get_module_main_conf(cf, ngx_http_replace_filter_module);
if (rmcf->compiler_pool == NULL) {
rmcf->compiler_pool = sre_create_pool(SREGEX_COMPILER_POOL_SIZE);
if (rmcf->compiler_pool == NULL) {
return NGX_CONF_ERROR;
}
cln = ngx_pool_cleanup_add(cf->pool, 0);
if (cln == NULL) {
sre_destroy_pool(rmcf->compiler_pool);
rmcf->compiler_pool = NULL;
return NGX_CONF_ERROR;
}
cln->data = rmcf->compiler_pool;
cln->handler = ngx_http_replace_cleanup_pool;
}
return NGX_CONF_OK;
}
static void *
ngx_http_replace_create_loc_conf(ngx_conf_t *cf)
{
ngx_http_replace_loc_conf_t *conf;
conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_replace_loc_conf_t));
if (conf == NULL) {
return NULL;
}
/*
* set by ngx_pcalloc():
*
* conf->types = { NULL };
* conf->types_keys = NULL;
* conf->program = NULL;
* conf->ncaps = 0;
* conf->ovecsize = 0;
* conf->parse_buf = NULL;
* conf->verbatim = { {0, NULL}, NULL, NULL, 0 };
* conf->seen_once = 0;
* conf->seen_global = 0;
* conf->skip = NULL;
*/
conf->max_buffered_size = NGX_CONF_UNSET_SIZE;
conf->last_modified = NGX_CONF_UNSET_UINT;
ngx_array_init(&conf->multi_replace, cf->pool, 4,
sizeof(ngx_http_replace_complex_value_t));
ngx_array_init(&conf->multi_flags, cf->pool, 4, sizeof(int));
ngx_array_init(&conf->regexes, cf->pool, 4, sizeof(u_char *));
ngx_array_init(&conf->multi_once, cf->pool, 4, sizeof(uint8_t));
return conf;
}
static char *
ngx_http_replace_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
u_char **value;
sre_int_t err_offset, err_regex_id;
ngx_str_t prefix, suffix;
sre_pool_t *ppool; /* parser pool */
sre_regex_t *re;
sre_program_t *prog;
ngx_http_replace_main_conf_t *rmcf;
ngx_http_replace_loc_conf_t *prev = parent;
ngx_http_replace_loc_conf_t *conf = child;
ngx_conf_merge_size_value(conf->max_buffered_size,
prev->max_buffered_size,
8192);
ngx_conf_merge_uint_value(conf->last_modified,
prev->last_modified,
NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED);
if (ngx_http_merge_types(cf, &conf->types_keys, &conf->types,
&prev->types_keys, &prev->types,
ngx_http_html_default_types)
!= NGX_OK)
{
return NGX_CONF_ERROR;
}
if (conf->skip == NULL) {
conf->skip = prev->skip;
}
if (conf->regexes.nelts > 0 && conf->program == NULL) {
dd("parsing and compiling %d regexes", (int) conf->regexes.nelts);
ppool = sre_create_pool(1024);
if (ppool == NULL) {
return NGX_CONF_ERROR;
}
value = conf->regexes.elts;
re = sre_regex_parse_multi(ppool, value, conf->regexes.nelts,
&conf->ncaps, conf->multi_flags.elts,
&err_offset, &err_regex_id);
if (re == NULL) {
if (err_offset >= 0) {
prefix.data = value[err_regex_id];
prefix.len = err_offset;
suffix.data = value[err_regex_id] + err_offset;
suffix.len = ngx_strlen(value[err_regex_id]) - err_offset;
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to parse regex at offset %i: "
"syntax error; marked by <-- HERE in "
"\"%V <-- HERE %V\"",
(ngx_int_t) err_offset, &prefix, &suffix);
} else {
if (err_regex_id >= 0) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to parse regex \"%s\"",
value[err_regex_id]);
} else {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to parse regex \"%s\" "
"and its siblings",
value[0]);
}
}
sre_destroy_pool(ppool);
return NGX_CONF_ERROR;
}
rmcf = ngx_http_conf_get_module_main_conf(cf,
ngx_http_replace_filter_module);
prog = sre_regex_compile(rmcf->compiler_pool, re);
sre_destroy_pool(ppool);
if (prog == NULL) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to compile regex \"%s\" and its "
"siblings", value[0]);
return NGX_CONF_ERROR;
}
conf->program = prog;
conf->ovecsize = 2 * (conf->ncaps + 1) * sizeof(sre_int_t);
} else {
conf->regexes = prev->regexes;
conf->multi_once = prev->multi_once;
conf->multi_flags = prev->multi_flags;
conf->multi_replace = prev->multi_replace;
conf->parse_buf = prev->parse_buf;
conf->verbatim = prev->verbatim;
conf->program = prev->program;
conf->ncaps = prev->ncaps;
conf->ovecsize = prev->ovecsize;
conf->seen_once = prev->seen_once;
conf->seen_global = prev->seen_global;
}
return NGX_CONF_OK;
}
static ngx_int_t
ngx_http_replace_filter_init(ngx_conf_t *cf)
{
int multi_http_blocks;
ngx_http_replace_main_conf_t *rmcf;
rmcf =
ngx_http_conf_get_module_main_conf(cf, ngx_http_replace_filter_module);
if (ngx_http_replace_prev_cycle != ngx_cycle) {
ngx_http_replace_prev_cycle = ngx_cycle;
multi_http_blocks = 0;
} else {
multi_http_blocks = 1;
}
if (multi_http_blocks || rmcf->compiler_pool != NULL) {
ngx_http_next_header_filter = ngx_http_top_header_filter;
ngx_http_top_header_filter = ngx_http_replace_header_filter;
ngx_http_next_body_filter = ngx_http_top_body_filter;
ngx_http_top_body_filter = ngx_http_replace_body_filter;
return NGX_OK;
}
return NGX_OK;
}
static void *
ngx_http_replace_create_main_conf(ngx_conf_t *cf)
{
ngx_http_replace_main_conf_t *rmcf;
rmcf = ngx_pcalloc(cf->pool, sizeof(ngx_http_replace_main_conf_t));
if (rmcf == NULL) {
return NULL;
}
/* set by ngx_pcalloc:
* rmcf->compiler_pool = NULL;
*/
return rmcf;
}
================================================
FILE: src/ngx_http_replace_filter_module.h
================================================
#ifndef _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_
#define _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_
#include "ngx_http_replace_script.h"
#include <ngx_core.h>
#include <ngx_http.h>
#include <nginx.h>
#include <sregex/sregex.h>
extern ngx_module_t ngx_http_replace_filter_module;
typedef struct {
sre_int_t regex_id;
sre_int_t stream_pos;
sre_int_t *ovector;
sre_pool_t *vm_pool;
sre_vm_pike_ctx_t *vm_ctx;
ngx_chain_t *pending; /* pending data before the
pending matched capture */
ngx_chain_t **last_pending;
ngx_chain_t *pending2; /* pending data after the pending
matched capture */
ngx_chain_t **last_pending2;
ngx_buf_t *buf;
ngx_str_t *sub;
u_char *pos;
u_char *copy_start;
u_char *copy_end;
ngx_chain_t *in;
ngx_chain_t *out;
ngx_chain_t **last_out;
ngx_chain_t *busy;
ngx_chain_t *free;
ngx_chain_t *special;
ngx_chain_t **last_special;
ngx_chain_t *rematch;
ngx_chain_t *captured;
ngx_chain_t **last_captured;
uint8_t *disabled;
sre_uint_t disabled_count;
size_t total_buffered;
unsigned once:1;
unsigned vm_done:1;
unsigned special_buf:1;
unsigned last_buf:1;
} ngx_http_replace_ctx_t;
typedef ngx_int_t (*ngx_http_replace_parse_buf_pt)(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch);
typedef struct {
sre_pool_t *compiler_pool;
} ngx_http_replace_main_conf_t;
typedef struct {
sre_uint_t ncaps;
size_t ovecsize;
ngx_array_t multi_once; /* of uint8_t */
ngx_array_t regexes; /* of u_char* */
ngx_array_t multi_flags; /* of int */
ngx_array_t multi_replace;
/* of ngx_http_replace_complex_value_t */
sre_program_t *program;
ngx_hash_t types;
ngx_array_t *types_keys;
size_t max_buffered_size;
ngx_uint_t last_modified;
/* replace_filter_last_modified */
ngx_http_replace_parse_buf_pt parse_buf;
ngx_http_replace_complex_value_t verbatim;
ngx_http_complex_value_t *skip;
unsigned seen_once; /* :1 */
unsigned seen_global; /* :1 */
} ngx_http_replace_loc_conf_t;
#endif /* _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_ */
================================================
FILE: src/ngx_http_replace_parse.c
================================================
/*
* Copyright (C) Yichun Zhang (agentzh)
* Copyright (C) Igor Sysoev
* Copyright (C) Nginx, Inc.
*/
#ifndef DDEBUG
#define DDEBUG 0
#endif
#include "ddebug.h"
#include "ngx_http_replace_parse.h"
#include "ngx_http_replace_util.h"
static void ngx_http_replace_check_total_buffered(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, sre_int_t len, sre_int_t mlen);
ngx_int_t
ngx_http_replace_capturing_parse(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch)
{
sre_int_t ret, from, to;
ngx_int_t rc;
ngx_chain_t *new_rematch = NULL;
ngx_chain_t *cl;
ngx_chain_t **last_rematch, **last;
size_t len;
dd("replace capturing parse");
if (ctx->once || ctx->vm_done) {
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "once");
return NGX_AGAIN;
}
len = ctx->buf->last - ctx->pos;
dd("=== process data chunk %p len=%d, pos=%u, special=%u, "
"last=%u, \"%.*s\"", ctx->buf, (int) (ctx->buf->last - ctx->pos),
(int) (ctx->pos - ctx->buf->pos + ctx->stream_pos),
ctx->special_buf, ctx->last_buf,
(int) (ctx->buf->last - ctx->pos), ctx->pos);
ret = sre_vm_pike_exec(ctx->vm_ctx, ctx->pos, len, ctx->last_buf, NULL);
dd("vm pike exec: %d", (int) ret);
if (ret >= 0) {
ctx->regex_id = ret;
ctx->total_buffered = 0;
from = ctx->ovector[0];
to = ctx->ovector[1];
dd("pike vm ok: (%d, %d)", (int) from, (int) to);
if (from >= ctx->stream_pos) {
/* the match is completely on the current buf */
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
/* prepare ctx->captured */
cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free);
if (cl == NULL) {
return NGX_ERROR;
}
cl->buf->pos = ctx->buf->pos;
cl->buf->last = ctx->buf->last;
cl->buf->memory = 1;
cl->buf->file_pos = ctx->stream_pos;
cl->buf->file_last = ctx->stream_pos
+ (cl->buf->last - cl->buf->pos);
*ctx->last_captured = cl;
ctx->last_captured = &cl->next;
dd("ctx captured: %p", ctx->captured);
/* prepare copy-out data */
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos);
dd("copy len: %d", (int) (ctx->copy_end - ctx->copy_start));
ctx->pos = ctx->buf->pos + (to - ctx->stream_pos);
return NGX_OK;
}
/* from < ctx->stream_pos */
if (ctx->pending) {
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending,
&ctx->last_pending, from,
&cl, &last, 1)
!= NGX_OK)
{
return NGX_ERROR;
}
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (cl) {
if (to >= ctx->stream_pos) {
/* no pending data to be rematched */
if (to == ctx->stream_pos) {
*ctx->last_captured = cl;
ctx->last_captured = &cl->next;
} else {
*ctx->last_captured = cl;
ctx->last_captured = last;
cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free);
if (cl == NULL) {
return NGX_ERROR;
}
cl->buf->pos = ctx->buf->pos;
cl->buf->last = ctx->buf->last;
cl->buf->memory = 1;
cl->buf->file_pos = ctx->stream_pos;
cl->buf->file_last = ctx->stream_pos
+ (cl->buf->last - cl->buf->pos);
*ctx->last_captured = cl;
ctx->last_captured = &cl->next;
}
} else {
/* there's pending data to be rematched */
if (ngx_http_replace_split_chain(r, ctx, &cl,
&last,
to, &new_rematch,
&last_rematch, 1)
!= NGX_OK)
{
return NGX_ERROR;
}
if (cl) {
*ctx->last_captured = cl;
ctx->last_captured = last;
}
if (new_rematch) {
if (rematch) {
ctx->rematch = rematch;
}
/* prepend cl to ctx->rematch */
*last_rematch = ctx->rematch;
ctx->rematch = new_rematch;
}
}
}
}
#if (DDEBUG)
ngx_http_replace_dump_chain("ctx->rematch", &ctx->rematch, NULL);
#endif
ctx->copy_start = NULL;
ctx->copy_end = NULL;
ctx->pos = ctx->buf->pos + (to - ctx->stream_pos);
return new_rematch ? NGX_BUSY : NGX_OK;
}
switch (ret) {
case SRE_AGAIN:
from = ctx->ovector[0];
to = ctx->ovector[1];
dd("pike vm again: (%d, %d)", (int) from, (int) to);
if (from == -1) {
from = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos);
}
if (to == -1) {
to = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos);
}
dd("pike vm again (adjusted): stream pos:%d, (%d, %d)",
(int) ctx->stream_pos, (int) from, (int) to);
if (from > to) {
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"invalid capture range: %i > %i", (ngx_int_t) from,
(ngx_int_t) to);
return NGX_ERROR;
}
if (from == to) {
if (ctx->pending) {
ctx->total_buffered = 0;
dd("output pending");
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos);
ctx->pos = ctx->copy_end;
return NGX_AGAIN;
}
/*
* append the existing ctx->pending data right before
* the $& capture to ctx->out.
*/
if (from >= ctx->stream_pos) {
/* the match is completely on the current buf */
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos);
if (ctx->pending) {
ctx->total_buffered = 0;
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
dd("create ctx->pending as (%ld, %ld)", (long) from, (long) to);
rc = ngx_http_replace_new_pending_buf(r, ctx, from, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
#if 1
if (rc == NGX_BUSY) {
dd("stop processing because of buffer size limit reached");
ctx->once = 1;
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
#endif
*ctx->last_pending = cl;
ctx->last_pending = &cl->next;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
dd("from < ctx->stream_pos");
if (ctx->pending) {
/* split ctx->pending into ctx->out and ctx->pending */
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending,
&ctx->last_pending, from, &cl,
&last, 1)
!= NGX_OK)
{
return NGX_ERROR;
}
if (ctx->pending) {
dd("adjust pending: pos=%d, from=%d",
(int) ctx->pending->buf->file_pos, (int) from);
ctx->total_buffered -= (size_t)
(from - ctx->pending->buf->file_pos);
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (cl) {
dd("splitted ctx->pending into ctx->out and ctx->pending: %d",
(int) ctx->total_buffered);
ctx->pending = cl;
ctx->last_pending = last;
}
}
/* new pending data to buffer to ctx->pending */
rc = ngx_http_replace_new_pending_buf(r, ctx, ctx->pos
- ctx->buf->pos
+ ctx->stream_pos, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
#if 1
if (rc == NGX_BUSY) {
ctx->once = 1;
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
#endif
*ctx->last_pending = cl;
ctx->last_pending = &cl->next;
ctx->copy_start = NULL;
ctx->copy_end = NULL;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
case SRE_DECLINED:
ctx->total_buffered = 0;
return NGX_DECLINED;
default:
/* SRE_ERROR */
return NGX_ERROR;
}
/* cannot reach here */
}
ngx_int_t
ngx_http_replace_non_capturing_parse(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch)
{
sre_int_t ret, from, to, mfrom = -1, mto = -1;
ngx_int_t rc;
ngx_chain_t *new_rematch = NULL;
ngx_chain_t *cl;
ngx_chain_t **last_rematch, **last;
size_t len;
sre_int_t *pending_matched;
dd("replace non capturing parse");
if (ctx->once || ctx->vm_done) {
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "once");
return NGX_AGAIN;
}
len = ctx->buf->last - ctx->pos;
dd("=== process data chunk %p len=%d, pos=%u, special=%u, "
"last=%u, \"%.*s\"", ctx->buf, (int) (ctx->buf->last - ctx->pos),
(int) (ctx->pos - ctx->buf->pos + ctx->stream_pos),
ctx->special_buf, ctx->last_buf,
(int) (ctx->buf->last - ctx->pos), ctx->pos);
ret = sre_vm_pike_exec(ctx->vm_ctx, ctx->pos, len, ctx->last_buf,
&pending_matched);
dd("vm pike exec: %d", (int) ret);
if (ret >= 0) {
ctx->regex_id = ret;
ctx->total_buffered = 0;
from = ctx->ovector[0];
to = ctx->ovector[1];
dd("pike vm ok: (%d, %d)", (int) from, (int) to);
if (from >= ctx->stream_pos) {
/* the match is completely on the current buf */
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (ctx->pending2) {
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"assertion failed: ctx->pending2 is not NULL "
"when the match is completely on the current "
"buf");
return NGX_ERROR;
}
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos);
dd("copy len: %d", (int) (ctx->copy_end - ctx->copy_start));
ctx->pos = ctx->buf->pos + (to - ctx->stream_pos);
return NGX_OK;
}
/* from < ctx->stream_pos */
if (ctx->pending) {
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending,
&ctx->last_pending, from,
&cl, &last, 0)
!= NGX_OK)
{
return NGX_ERROR;
}
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (cl) {
*last = ctx->free;
ctx->free = cl;
}
}
if (ctx->pending2) {
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending2,
&ctx->last_pending2,
to, &new_rematch, &last_rematch, 1)
!= NGX_OK)
{
return NGX_ERROR;
}
if (ctx->pending2) {
*ctx->last_pending2 = ctx->free;
ctx->free = ctx->pending2;
ctx->pending2 = NULL;
ctx->last_pending2 = &ctx->pending2;
}
if (new_rematch) {
if (rematch) {
ctx->rematch = rematch;
}
/* prepend cl to ctx->rematch */
*last_rematch = ctx->rematch;
ctx->rematch = new_rematch;
}
}
#if (DDEBUG)
ngx_http_replace_dump_chain("ctx->rematch", &ctx->rematch, NULL);
#endif
ctx->copy_start = NULL;
ctx->copy_end = NULL;
ctx->pos = ctx->buf->pos + (to - ctx->stream_pos);
return new_rematch ? NGX_BUSY : NGX_OK;
}
switch (ret) {
case SRE_AGAIN:
from = ctx->ovector[0];
to = ctx->ovector[1];
dd("pike vm again: (%d, %d)", (int) from, (int) to);
if (from == -1) {
from = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos);
}
if (to == -1) {
to = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos);
}
dd("pike vm again (adjusted): stream pos:%d, (%d, %d)",
(int) ctx->stream_pos, (int) from, (int) to);
if (from > to) {
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"invalid capture range: %i > %i", (ngx_int_t) from,
(ngx_int_t) to);
return NGX_ERROR;
}
if (pending_matched) {
mfrom = pending_matched[0];
mto = pending_matched[1];
dd("pending matched: (%ld, %ld)", (long) mfrom, (long) mto);
}
if (from == to) {
if (ctx->pending) {
ctx->total_buffered = 0;
dd("output pending");
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos);
ctx->pos = ctx->copy_end;
ngx_http_replace_check_total_buffered(r, ctx, to - from,
mto - mfrom);
return NGX_AGAIN;
}
/*
* append the existing ctx->pending data right before
* the $& capture to ctx->out.
*/
if (from >= ctx->stream_pos) {
/* the match is completely on the current buf */
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos);
if (ctx->pending) {
ctx->total_buffered = 0;
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (ctx->pending2) {
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"assertion failed: ctx->pending2 is not NULL "
"when the match is completely on the current "
"buf");
return NGX_ERROR;
}
if (pending_matched) {
if (from < mfrom) {
/* create ctx->pending as (from, mfrom) */
rc = ngx_http_replace_new_pending_buf(r, ctx, from, mfrom,
&cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
if (rc == NGX_BUSY) {
dd("stop processing because of buffer size limit "
"reached");
ctx->once = 1;
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
*ctx->last_pending = cl;
ctx->last_pending = &cl->next;
}
if (mto < to) {
/* create ctx->pending2 as (mto, to) */
rc = ngx_http_replace_new_pending_buf(r, ctx, mto, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
#if 1
if (rc == NGX_BUSY) {
dd("stop processing because of buffer size limit "
"reached");
ctx->once = 1;
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
#endif
*ctx->last_pending2 = cl;
ctx->last_pending2 = &cl->next;
}
} else {
dd("create ctx->pending as (%ld, %ld)", (long) from, (long) to);
rc = ngx_http_replace_new_pending_buf(r, ctx, from, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
#if 1
if (rc == NGX_BUSY) {
dd("stop processing because of buffer size limit reached");
ctx->once = 1;
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
#endif
*ctx->last_pending = cl;
ctx->last_pending = &cl->next;
}
ctx->pos = ctx->buf->last;
ngx_http_replace_check_total_buffered(r, ctx, to - from,
mto - mfrom);
return NGX_AGAIN;
}
dd("from < ctx->stream_pos");
if (ctx->pending) {
/* split ctx->pending into ctx->out and ctx->pending */
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending,
&ctx->last_pending, from, &cl,
&last, 1)
!= NGX_OK)
{
return NGX_ERROR;
}
if (ctx->pending) {
dd("adjust pending: pos=%d, from=%d",
(int) ctx->pending->buf->file_pos, (int) from);
ctx->total_buffered -= (size_t)
(from - ctx->pending->buf->file_pos);
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
if (cl) {
dd("splitted ctx->pending into ctx->out and ctx->pending: %d",
(int) ctx->total_buffered);
ctx->pending = cl;
ctx->last_pending = last;
}
if (pending_matched && !ctx->pending2 && mto >= ctx->stream_pos) {
dd("splitting ctx->pending into ctx->pending and ctx->free");
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending,
&ctx->last_pending, mfrom, &cl,
&last, 0)
!= NGX_OK)
{
return NGX_ERROR;
}
if (cl) {
ctx->total_buffered -= (size_t) (ctx->stream_pos - mfrom);
dd("splitted ctx->pending into ctx->pending and ctx->free");
*last = ctx->free;
ctx->free = cl;
}
}
}
if (ctx->pending2) {
if (pending_matched) {
dd("splitting ctx->pending2 into ctx->free and ctx->pending2");
if (ngx_http_replace_split_chain(r, ctx, &ctx->pending2,
&ctx->last_pending2,
mto, &cl, &last, 1)
!= NGX_OK)
{
return NGX_ERROR;
}
if (ctx->pending2) {
dd("total buffered reduced by %d (was %d)",
(int) (mto - ctx->pending2->buf->file_pos),
(int) ctx->total_buffered);
ctx->total_buffered -= (size_t)
(mto - ctx->pending2->buf->file_pos);
*ctx->last_pending2 = ctx->free;
ctx->free = ctx->pending2;
ctx->pending2 = NULL;
ctx->last_pending2 = &ctx->pending2;
}
if (cl) {
ctx->pending2 = cl;
ctx->last_pending2 = last;
}
}
if (mto < to) {
dd("new pending data to buffer to ctx->pending2: (%ld, %ld)",
(long) mto, (long) to);
rc = ngx_http_replace_new_pending_buf(r, ctx, mto, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
#if 1
if (rc == NGX_BUSY) {
ctx->once = 1;
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
ctx->copy_start = NULL;
ctx->copy_end = NULL;
if (ctx->pending2) {
new_rematch = ctx->pending2;
last_rematch = ctx->last_pending2;
if (rematch) {
ctx->rematch = rematch;
}
/* prepend cl to ctx->rematch */
*last_rematch = ctx->rematch;
ctx->rematch = new_rematch;
ctx->pending2 = NULL;
ctx->last_pending2 = &ctx->pending2;
}
ctx->pos = ctx->buf->pos + (mto - ctx->stream_pos);
return new_rematch ? NGX_BUSY : NGX_OK;
}
#endif
*ctx->last_pending2 = cl;
ctx->last_pending2 = &cl->next;
}
ctx->copy_start = NULL;
ctx->copy_end = NULL;
ctx->pos = ctx->buf->last;
ngx_http_replace_check_total_buffered(r, ctx, to - from,
mto - mfrom);
return NGX_AGAIN;
}
/* ctx->pending2 == NULL */
if (pending_matched) {
if (mto < to) {
/* new pending data to buffer to ctx->pending2 */
rc = ngx_http_replace_new_pending_buf(r, ctx, mto, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
if (rc == NGX_BUSY) {
ctx->once = 1;
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
ctx->copy_start = NULL;
ctx->copy_end = NULL;
ctx->pos = ctx->buf->pos + mto - ctx->stream_pos;
return NGX_OK;
}
*ctx->last_pending2 = cl;
ctx->last_pending2 = &cl->next;
}
/* otherwise no new data to buffer */
} else {
/* new pending data to buffer to ctx->pending */
rc = ngx_http_replace_new_pending_buf(r, ctx, ctx->pos
- ctx->buf->pos
+ ctx->stream_pos, to, &cl);
if (rc == NGX_ERROR) {
return NGX_ERROR;
}
#if 1
if (rc == NGX_BUSY) {
ctx->once = 1;
if (ctx->pending) {
*ctx->last_out = ctx->pending;
ctx->last_out = ctx->last_pending;
ctx->pending = NULL;
ctx->last_pending = &ctx->pending;
}
ctx->copy_start = ctx->pos;
ctx->copy_end = ctx->buf->last;
ctx->pos = ctx->buf->last;
return NGX_AGAIN;
}
#endif
*ctx->last_pending = cl;
ctx->last_pending = &cl->next;
}
ctx->copy_start = NULL;
ctx->copy_end = NULL;
ctx->pos = ctx->buf->last;
ngx_http_replace_check_total_buffered(r, ctx, to - from,
mto - mfrom);
return NGX_AGAIN;
case SRE_DECLINED:
ctx->total_buffered = 0;
return NGX_DECLINED;
default:
/* SRE_ERROR */
return NGX_ERROR;
}
/* cannot reach here */
}
static void
ngx_http_replace_check_total_buffered(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, sre_int_t len, sre_int_t mlen)
{
dd("total buffered: %d", (int) ctx->total_buffered);
if ((ssize_t) ctx->total_buffered != len - mlen) {
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"replace filter: ctx->total_buffered out of "
"sync: it is %i but should be %uz",
ctx->total_buffered, (ngx_int_t) (len - mlen));
#if (DDEBUG)
assert(0);
#endif
}
}
================================================
FILE: src/ngx_http_replace_parse.h
================================================
#ifndef _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_
#define _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_
#include "ngx_http_replace_filter_module.h"
ngx_int_t ngx_http_replace_non_capturing_parse(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch);
ngx_int_t ngx_http_replace_capturing_parse(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch);
#endif /* _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_ */
================================================
FILE: src/ngx_http_replace_script.c
================================================
/*
* Copyright (C) Yichun Zhang (agentzh)
*/
#ifndef DDEBUG
#define DDEBUG 0
#endif
#include "ddebug.h"
#include "ngx_http_replace_script.h"
static void *ngx_http_replace_script_add_code(ngx_array_t *codes, size_t size);
static size_t ngx_http_replace_script_copy_len_code(
ngx_http_replace_script_engine_t *e);
static size_t
ngx_http_replace_script_copy_code(ngx_http_replace_script_engine_t *e);
static ngx_int_t ngx_http_replace_script_add_copy_code(
ngx_http_replace_script_compile_t *sc, ngx_str_t *value, ngx_uint_t last);
static ngx_int_t
ngx_http_replace_script_compile(ngx_http_replace_script_compile_t *sc);
static ngx_int_t ngx_http_replace_script_add_capture_code(
ngx_http_replace_script_compile_t *sc, ngx_uint_t n);
static size_t ngx_http_replace_script_copy_capture_len_code(
ngx_http_replace_script_engine_t *e);
static size_t ngx_http_replace_script_copy_capture_code(
ngx_http_replace_script_engine_t *e);
static ngx_int_t
ngx_http_replace_script_done(ngx_http_replace_script_compile_t *sc);
static ngx_int_t ngx_http_replace_script_init_arrays(
ngx_http_replace_script_compile_t *sc);
static ngx_int_t
ngx_http_replace_script_add_var_code(ngx_http_replace_script_compile_t *sc,
ngx_str_t *name);
static size_t
ngx_http_replace_script_copy_var_len_code(
ngx_http_replace_script_engine_t *e);
static size_t
ngx_http_replace_script_copy_var_code(ngx_http_replace_script_engine_t *e);
static void ngx_http_replace_count_variables(u_char *src, size_t len,
ngx_uint_t *ngxvars, ngx_uint_t *capvars);
ngx_int_t
ngx_http_replace_compile_complex_value(
ngx_http_replace_compile_complex_value_t *ccv)
{
ngx_str_t *v;
ngx_uint_t n, ngxvars, capvars;
ngx_array_t lengths, values, *pl, *pv;
ngx_http_replace_script_compile_t sc;
v = ccv->value;
ngx_http_replace_count_variables(v->data, v->len, &ngxvars, &capvars);
ccv->complex_value->value = *v;
ccv->complex_value->lengths = NULL;
ccv->complex_value->values = NULL;
if (capvars == 0 && ngxvars == 0) {
return NGX_OK;
}
n = capvars * (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_capture_code_t))
+ ngxvars * (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_var_code_t))
+ sizeof(uintptr_t);
if (ngx_array_init(&lengths, ccv->cf->pool, n, 1) != NGX_OK) {
return NGX_ERROR;
}
n = capvars * (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_capture_code_t))
+ ngxvars * (2 * sizeof(ngx_http_replace_script_var_code_t)
+ sizeof(ngx_http_replace_script_var_code_t))
+ sizeof(uintptr_t);
if (ngx_array_init(&values, ccv->cf->pool, n, 1) != NGX_OK) {
return NGX_ERROR;
}
pl = &lengths;
pv = &values;
ngx_memzero(&sc, sizeof(ngx_http_replace_script_compile_t));
sc.cf = ccv->cf;
sc.source = v;
sc.lengths = &pl;
sc.values = &pv;
if (ngx_http_replace_script_compile(&sc) != NGX_OK) {
ngx_array_destroy(&lengths);
ngx_array_destroy(&values);
return NGX_ERROR;
}
ccv->complex_value->lengths = lengths.elts;
ccv->complex_value->values = values.elts;
ccv->complex_value->capture_variables = sc.capture_variables;
return NGX_OK;
}
ngx_int_t
ngx_http_replace_complex_value(ngx_http_request_t *r,
ngx_chain_t *captured, sre_uint_t ncaps, sre_int_t *cap,
ngx_http_replace_complex_value_t *val, ngx_str_t *value)
{
size_t len;
ngx_http_replace_script_code_pt code;
ngx_http_replace_script_len_code_pt lcode;
ngx_http_replace_script_engine_t e;
if (val->lengths == NULL) {
*value = val->value;
return NGX_OK;
}
ngx_memzero(&e, sizeof(ngx_http_replace_script_engine_t));
e.request = r;
e.ncaptures = (ncaps + 1) * 2;
e.captures_data = captured;
e.captures = cap;
e.ip = val->lengths;
len = 0;
while (*(uintptr_t *) e.ip) {
lcode = *(ngx_http_replace_script_len_code_pt *) e.ip;
len += lcode(&e);
}
value->len = len;
value->data = ngx_pnalloc(r->pool, len);
if (value->data == NULL) {
return NGX_ERROR;
}
e.ip = val->values;
e.pos = value->data;
while (*(uintptr_t *) e.ip) {
code = *(ngx_http_replace_script_code_pt *) e.ip;
code((ngx_http_replace_script_engine_t *) &e);
}
return NGX_OK;
}
static ngx_int_t
ngx_http_replace_script_compile(ngx_http_replace_script_compile_t *sc)
{
u_char ch;
ngx_str_t name;
ngx_uint_t i, bracket;
unsigned num_var;
ngx_uint_t n = 0;
if (ngx_http_replace_script_init_arrays(sc) != NGX_OK) {
return NGX_ERROR;
}
for (i = 0; i < sc->source->len; /* void */ ) {
name.len = 0;
if (sc->source->data[i] == '$') {
if (++i == sc->source->len) {
goto invalid_variable;
}
if (sc->source->data[i] == '$') {
name.data = &sc->source->data[i];
i++;
name.len++;
sc->size += name.len;
if (ngx_http_replace_script_add_copy_code(sc, &name,
(i == sc->source->len))
!= NGX_OK)
{
return NGX_ERROR;
}
continue;
}
if ((sc->source->data[i] >= '1' && sc->source->data[i] <= '9')
|| sc->source->data[i] == '&')
{
num_var = 1;
n = 0;
} else {
num_var = 0;
}
if (sc->source->data[i] == '{') {
bracket = 1;
if (++i == sc->source->len) {
goto invalid_variable;
}
if ((sc->source->data[i] >= '1' && sc->source->data[i] <= '9')
|| sc->source->data[i] == '&')
{
num_var = 1;
n = 0;
}
name.data = &sc->source->data[i];
} else {
bracket = 0;
name.data = &sc->source->data[i];
}
for ( /* void */ ; i < sc->source->len; i++, name.len++) {
ch = sc->source->data[i];
if (ch == '}' && bracket) {
i++;
bracket = 0;
break;
}
if (num_var) {
if (ch >= '0' && ch <= '9') {
n = n * 10 + (ch - '0');
continue;
}
if (ch == '&') {
i++;
name.len++;
}
break;
}
/* not a number variable like $1, $2, etc */
if ((ch >= 'A' && ch <= 'Z')
|| (ch >= 'a' && ch <= 'z')
|| (ch >= '0' && ch <= '9')
|| ch == '_')
{
continue;
}
break;
}
if (bracket) {
ngx_log_error(NGX_LOG_ERR, sc->cf->log, 0,
"the closing bracket in \"%V\" "
"variable is missing", &name);
return NGX_ERROR;
}
if (name.len == 0) {
goto invalid_variable;
}
if (num_var) {
dd("found numbered capturing variable \"%.*s\"",
(int) name.len, name.data);
sc->capture_variables++;
if (ngx_http_replace_script_add_capture_code(sc, n) != NGX_OK) {
return NGX_ERROR;
}
} else {
sc->nginx_variables++;
if (ngx_http_replace_script_add_var_code(sc, &name) != NGX_OK) {
return NGX_ERROR;
}
}
continue;
}
name.data = &sc->source->data[i];
while (i < sc->source->len) {
if (sc->source->data[i] == '$') {
break;
}
i++;
name.len++;
}
sc->size += name.len;
if (ngx_http_replace_script_add_copy_code(sc, &name,
(i == sc->source->len))
!= NGX_OK)
{
return NGX_ERROR;
}
}
return ngx_http_replace_script_done(sc);
invalid_variable:
ngx_log_error(NGX_LOG_ERR, sc->cf->log, 0,
"replace script: invalid capturing variable name found "
"in \"%V\"", sc->source);
return NGX_ERROR;
}
static ngx_int_t
ngx_http_replace_script_add_copy_code(ngx_http_replace_script_compile_t *sc,
ngx_str_t *value, ngx_uint_t last)
{
size_t size, len;
ngx_http_replace_script_copy_code_t *code;
len = value->len;
code = ngx_http_replace_script_add_code(*sc->lengths,
sizeof(ngx_http_replace_script_copy_code_t));
if (code == NULL) {
return NGX_ERROR;
}
code->code = (ngx_http_replace_script_code_pt)
ngx_http_replace_script_copy_len_code;
code->len = len;
size = (sizeof(ngx_http_replace_script_copy_code_t) + len +
sizeof(uintptr_t) - 1) & ~(sizeof(uintptr_t) - 1);
code = ngx_http_replace_script_add_code(*sc->values, size);
if (code == NULL) {
return NGX_ERROR;
}
code->code = (ngx_http_replace_script_code_pt)
ngx_http_replace_script_copy_code;
code->len = len;
ngx_memcpy((u_char *) code + sizeof(ngx_http_replace_script_copy_code_t),
value->data, value->len);
return NGX_OK;
}
static size_t
ngx_http_replace_script_copy_len_code(ngx_http_replace_script_engine_t *e)
{
ngx_http_replace_script_copy_code_t *code;
code = (ngx_http_replace_script_copy_code_t *) e->ip;
e->ip += sizeof(ngx_http_replace_script_copy_code_t);
return code->len;
}
static size_t
ngx_http_replace_script_copy_code(ngx_http_replace_script_engine_t *e)
{
u_char *p;
ngx_http_replace_script_copy_code_t *code;
code = (ngx_http_replace_script_copy_code_t *) e->ip;
p = e->pos;
if (!e->skip) {
e->pos = ngx_copy(p, e->ip
+ sizeof(ngx_http_replace_script_copy_code_t),
code->len);
}
e->ip += sizeof(ngx_http_replace_script_copy_code_t)
+ ((code->len + sizeof(uintptr_t) - 1) & ~(sizeof(uintptr_t) - 1));
ngx_log_debug2(NGX_LOG_DEBUG_HTTP, e->request->connection->log, 0,
"replace script copy: \"%*s\"", e->pos - p, p);
return 0;
}
static ngx_int_t
ngx_http_replace_script_add_capture_code(ngx_http_replace_script_compile_t *sc,
ngx_uint_t n)
{
ngx_http_replace_script_capture_code_t *code;
code = ngx_http_replace_script_add_code(*sc->lengths,
sizeof(ngx_http_replace_script_capture_code_t));
if (code == NULL) {
return NGX_ERROR;
}
code->code = (ngx_http_replace_script_code_pt)
ngx_http_replace_script_copy_capture_len_code;
code->n = 2 * n;
code = ngx_http_replace_script_add_code(*sc->values,
sizeof(ngx_http_replace_script_capture_code_t));
if (code == NULL) {
return NGX_ERROR;
}
code->code = (ngx_http_replace_script_code_pt)
ngx_http_replace_script_copy_capture_code;
code->n = 2 * n;
return NGX_OK;
}
static size_t
ngx_http_replace_script_copy_capture_len_code(
ngx_http_replace_script_engine_t *e)
{
sre_int_t *cap;
ngx_uint_t n;
ngx_http_replace_script_capture_code_t *code;
code = (ngx_http_replace_script_capture_code_t *) e->ip;
e->ip += sizeof(ngx_http_replace_script_capture_code_t);
n = code->n;
dd("group index: %d, ncaptures: %d", (int) n, (int) e->ncaptures);
if (n + 1 < e->ncaptures) {
cap = e->captures;
return cap[n + 1] - cap[n];
}
return 0;
}
static size_t
ngx_http_replace_script_copy_capture_code(ngx_http_replace_script_engine_t *e)
{
sre_int_t *cap, from, to, len;
u_char *p;
#if (NGX_DEBUG)
u_char *pos;
#endif
ngx_uint_t n;
ngx_chain_t *cl;
ngx_http_replace_script_capture_code_t *code;
code = (ngx_http_replace_script_capture_code_t *) e->ip;
e->ip += sizeof(ngx_http_replace_script_capture_code_t);
n = code->n;
#if (NGX_DEBUG)
pos = e->pos;
#endif
if (n < e->ncaptures) {
cap = e->captures;
from = cap[n];
to = cap[n + 1];
dd("captures data: %p", e->captures_data);
for (cl = e->captures_data; cl; cl = cl->next) {
if (from >= cl->buf->file_last) {
continue;
}
/* from < cl->buf->file_last */
if (to <= cl->buf->file_pos) {
break;
}
p = cl->buf->pos + (from - cl->buf->file_pos);
len = ngx_min(cl->buf->file_last, to) - from;
e->pos = ngx_copy(e->pos, p, len);
from += len;
}
}
#if (NGX_DEBUG)
ngx_log_debug2(NGX_LOG_DEBUG_HTTP, e->request->connection->log, 0,
"replace script capture: \"%*s\"", e->pos - pos, pos);
#endif
return 0;
}
static ngx_int_t
ngx_http_replace_script_init_arrays(ngx_http_replace_script_compile_t *sc)
{
ngx_uint_t n;
if (*sc->lengths == NULL) {
n = sc->capture_variables
* (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_capture_code_t))
+ sc->nginx_variables
* (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_var_code_t))
+ sizeof(uintptr_t);
*sc->lengths = ngx_array_create(sc->cf->pool, n, 1);
if (*sc->lengths == NULL) {
return NGX_ERROR;
}
}
if (*sc->values == NULL) {
n = sc->capture_variables
* (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_capture_code_t))
+ sc->nginx_variables
* (2 * sizeof(ngx_http_replace_script_copy_code_t)
+ sizeof(ngx_http_replace_script_var_code_t))
+ sizeof(uintptr_t);
*sc->values = ngx_array_create(sc->cf->pool, n, 1);
if (*sc->values == NULL) {
return NGX_ERROR;
}
}
sc->nginx_variables = 0;
sc->capture_variables = 0;
return NGX_OK;
}
static ngx_int_t
ngx_http_replace_script_done(ngx_http_replace_script_compile_t *sc)
{
uintptr_t *code;
code = ngx_http_replace_script_add_code(*sc->lengths,
sizeof(uintptr_t));
if (code == NULL) {
return NGX_ERROR;
}
*code = (uintptr_t) NULL;
code = ngx_http_replace_script_add_code(*sc->values, sizeof(uintptr_t));
if (code == NULL) {
return NGX_ERROR;
}
*code = (uintptr_t) NULL;
return NGX_OK;
}
static void *
ngx_http_replace_script_add_code(ngx_array_t *codes, size_t size)
{
return ngx_array_push_n(codes, size);
}
static ngx_int_t
ngx_http_replace_script_add_var_code(ngx_http_replace_script_compile_t *sc,
ngx_str_t *name)
{
ngx_int_t index;
ngx_http_replace_script_var_code_t *code;
index = ngx_http_get_variable_index(sc->cf, name);
if (index == NGX_ERROR) {
return NGX_ERROR;
}
code = ngx_http_replace_script_add_code(*sc->lengths,
sizeof(ngx_http_replace_script_var_code_t));
if (code == NULL) {
return NGX_ERROR;
}
code->code = (ngx_http_replace_script_code_pt)
ngx_http_replace_script_copy_var_len_code;
code->index = (uintptr_t) index;
code = ngx_http_replace_script_add_code(*sc->values,
sizeof(ngx_http_replace_script_var_code_t));
if (code == NULL) {
return NGX_ERROR;
}
code->code = (ngx_http_replace_script_code_pt)
ngx_http_replace_script_copy_var_code;
code->index = (uintptr_t) index;
return NGX_OK;
}
static size_t
ngx_http_replace_script_copy_var_len_code(ngx_http_replace_script_engine_t *e)
{
ngx_http_variable_value_t *value;
ngx_http_replace_script_var_code_t *code;
code = (ngx_http_replace_script_var_code_t *) e->ip;
e->ip += sizeof(ngx_http_replace_script_var_code_t);
value = ngx_http_get_indexed_variable(e->request, code->index);
if (value && !value->not_found) {
return value->len;
}
return 0;
}
static size_t
ngx_http_replace_script_copy_var_code(ngx_http_replace_script_engine_t *e)
{
u_char *p;
ngx_http_variable_value_t *value;
ngx_http_replace_script_var_code_t *code;
code = (ngx_http_replace_script_var_code_t *) e->ip;
e->ip += sizeof(ngx_http_replace_script_var_code_t);
if (!e->skip) {
value = ngx_http_get_indexed_variable(e->request, code->index);
if (value && !value->not_found) {
p = e->pos;
e->pos = ngx_copy(p, value->data, value->len);
ngx_log_debug2(NGX_LOG_DEBUG_HTTP,
e->request->connection->log, 0,
"http replace script var: \"%*s\"", e->pos - p, p);
}
}
return 0;
}
static void
ngx_http_replace_count_variables(u_char *src, size_t len,
ngx_uint_t *ngxvars, ngx_uint_t *capvars)
{
ngx_uint_t i;
unsigned var = 0;
u_char c;
*ngxvars = 0;
*capvars = 0;
for (i = 0; i < len; i++) {
c = src[i];
if (c == '$') {
if (var) {
var = 0;
} else {
var = 1;
}
} else if (var) {
if ((c >= '1' && c <= '9') || c == '&') {
(*capvars)++;
} else {
(*ngxvars)++;
}
var = 0;
}
}
}
/* vi:set ft=c ts=4 sw=4 et fdm=marker: */
================================================
FILE: src/ngx_http_replace_script.h
================================================
/*
* Copyright (C) Yichun Zhang (agentzh)
*/
#ifndef _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_
#define _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_
#include <nginx.h>
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>
#include <sregex/sregex.h>
typedef struct {
ngx_conf_t *cf;
ngx_str_t *source;
ngx_array_t **lengths;
ngx_array_t **values;
ngx_uint_t capture_variables; /* captures $1, $2, etc */
ngx_uint_t nginx_variables; /* nginx variables */
ngx_uint_t size;
} ngx_http_replace_script_compile_t;
typedef struct {
ngx_str_t value;
void *lengths;
void *values;
ngx_uint_t capture_variables;
} ngx_http_replace_complex_value_t;
typedef struct {
ngx_conf_t *cf;
ngx_str_t *value;
ngx_http_replace_complex_value_t *complex_value;
} ngx_http_replace_compile_complex_value_t;
typedef struct {
u_char *ip;
u_char *pos;
ngx_str_t buf;
sre_int_t *captures;
ngx_uint_t ncaptures;
ngx_chain_t *captures_data;
unsigned skip:1;
ngx_http_request_t *request;
} ngx_http_replace_script_engine_t;
typedef size_t (*ngx_http_replace_script_code_pt)
(ngx_http_replace_script_engine_t *e);
typedef size_t (*ngx_http_replace_script_len_code_pt)
(ngx_http_replace_script_engine_t *e);
typedef struct {
ngx_http_replace_script_code_pt code;
uintptr_t len;
} ngx_http_replace_script_copy_code_t;
typedef struct {
ngx_http_replace_script_code_pt code;
uintptr_t n;
} ngx_http_replace_script_capture_code_t;
typedef struct {
ngx_http_replace_script_code_pt code;
uintptr_t index;
} ngx_http_replace_script_var_code_t;
ngx_int_t ngx_http_replace_compile_complex_value(
ngx_http_replace_compile_complex_value_t *ccv);
ngx_int_t ngx_http_replace_complex_value(ngx_http_request_t *r,
ngx_chain_t *captured, sre_uint_t ncaps, sre_int_t *cap,
ngx_http_replace_complex_value_t *val, ngx_str_t *value);
#endif /* _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_ */
/* vi:set ft=c ts=4 sw=4 et fdm=marker: */
================================================
FILE: src/ngx_http_replace_util.c
================================================
/*
* Copyright (C) Yichun Zhang (agentzh)
*/
#ifndef DDEBUG
#define DDEBUG 0
#endif
#include "ddebug.h"
#include "ngx_http_replace_util.h"
ngx_chain_t *
ngx_http_replace_get_free_buf(ngx_pool_t *p, ngx_chain_t **free)
{
ngx_chain_t *cl;
cl = ngx_chain_get_free_buf(p, free);
if (cl == NULL) {
return cl;
}
ngx_memzero(cl->buf, sizeof(ngx_buf_t));
cl->buf->tag = (ngx_buf_tag_t) &ngx_http_replace_filter_module;
return cl;
}
ngx_int_t
ngx_http_replace_split_chain(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx,
ngx_chain_t **pa, ngx_chain_t ***plast_a, sre_int_t split, ngx_chain_t **pb,
ngx_chain_t ***plast_b, unsigned b_sane)
{
sre_int_t file_last;
ngx_chain_t *cl, *newcl, **ll;
#if 0
b_sane = 0;
#endif
ll = pa;
for (cl = *pa; cl; ll = &cl->next, cl = cl->next) {
if (cl->buf->file_last > split) {
/* found an overlap */
if (cl->buf->file_pos < split) {
dd("adjust cl buf (b_sane=%d): \"%.*s\"", b_sane,
(int) ngx_buf_size(cl->buf), cl->buf->pos);
file_last = cl->buf->file_last;
cl->buf->last -= file_last - split;
cl->buf->file_last = split;
dd("adjusted cl buf (next=%p): %.*s",
cl->next,
(int) ngx_buf_size(cl->buf), cl->buf->pos);
/* build the b chain */
if (b_sane) {
newcl = ngx_http_replace_get_free_buf(r->pool,
&ctx->free);
if (newcl == NULL) {
return NGX_ERROR;
}
newcl->buf->memory = 1;
newcl->buf->pos = cl->buf->last;
newcl->buf->last = cl->buf->last + file_last - split;
newcl->buf->file_pos = split;
newcl->buf->file_last = file_last;
newcl->next = cl->next;
*pb = newcl;
if (plast_b) {
if (cl->next) {
*plast_b = *plast_a;
} else {
*plast_b = &newcl->next;
}
}
} else {
*pb = cl->next;
if (plast_b) {
*plast_b = *plast_a;
}
}
/* truncate the a chain */
*plast_a = &cl->next;
cl->next = NULL;
return NGX_OK;
}
/* build the b chain */
*pb = cl;
if (plast_b) {
*plast_b = *plast_a;
}
/* truncate the a chain */
*plast_a = ll;
*ll = NULL;
return NGX_OK;
}
}
/* missed */
*pb = NULL;
if (plast_b) {
*plast_b = pb;
}
return NGX_OK;
}
ngx_int_t
ngx_http_replace_new_pending_buf(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, sre_int_t from, sre_int_t to,
ngx_chain_t **out)
{
size_t len;
ngx_buf_t *b;
ngx_chain_t *cl;
ngx_http_replace_loc_conf_t *rlcf;
if (from < ctx->stream_pos) {
from = ctx->stream_pos;
}
len = (size_t) (to - from);
if (len == 0) {
return NGX_ERROR;
}
ctx->total_buffered += len;
rlcf = ngx_http_get_module_loc_conf(r, ngx_http_replace_filter_module);
if (ctx->total_buffered > rlcf->max_buffered_size) {
#if 1
ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
"replace filter: exceeding "
"replace_filter_max_buffered_size (%uz): %uz",
rlcf->max_buffered_size, ctx->total_buffered);
return NGX_BUSY;
#endif
}
cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free);
if (cl == NULL) {
return NGX_ERROR;
}
b = cl->buf;
b->temporary = 1;
/* abuse the file_pos and file_last fields here */
b->file_pos = from;
b->file_last = to;
b->start = ngx_palloc(r->pool, len);
if (b->start == NULL) {
return NGX_ERROR;
}
b->end = b->start + len;
b->pos = b->start;
b->last = ngx_copy(b->pos, ctx->buf->pos + from - ctx->stream_pos, len);
dd("buffered pending data: stream_pos=%ld (%ld, %ld): %.*s",
(long) ctx->stream_pos, (long) from, (long) to,
(int) len, ctx->buf->pos + from - ctx->stream_pos);
*out = cl;
return NGX_OK;
}
#if (DDEBUG)
void
ngx_http_replace_dump_chain(const char *prefix, ngx_chain_t **pcl,
ngx_chain_t **last)
{
ngx_chain_t *cl;
if (*pcl == NULL) {
dd("%s buf empty", prefix);
if (last && last != pcl) {
dd("BAD last %s", prefix);
assert(0);
}
}
for (cl = *pcl; cl; cl = cl->next) {
dd("%s buf: \"%.*s\"", prefix, (int) ngx_buf_size(cl->buf),
cl->buf->pos);
if (cl->next == NULL) {
if (last && last != &cl->next) {
dd("BAD last %s", prefix);
assert(0);
}
}
}
}
#endif /* DDEBUG */
================================================
FILE: src/ngx_http_replace_util.h
================================================
#ifndef _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_
#define _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_
#include "ngx_http_replace_filter_module.h"
ngx_chain_t *ngx_http_replace_get_free_buf(ngx_pool_t *p,
ngx_chain_t **free);
ngx_int_t ngx_http_replace_split_chain(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, ngx_chain_t **pa, ngx_chain_t ***plast_a,
sre_int_t split, ngx_chain_t **pb, ngx_chain_t ***plast_b, unsigned b_sane);
ngx_int_t ngx_http_replace_new_pending_buf(ngx_http_request_t *r,
ngx_http_replace_ctx_t *ctx, sre_int_t from, sre_int_t to,
ngx_chain_t **out);
#if (DDEBUG)
void ngx_http_replace_dump_chain(const char *prefix, ngx_chain_t **pcl,
ngx_chain_t **last);
#endif
#endif /* _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_ */
================================================
FILE: t/01-sanity.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
#no_shuffle();
plan tests => repeat_each() * (blocks() * 4 + 1);
our $StapOutputChains = <<'_EOC_';
global active
F(ngx_http_handler) {
active = 1
}
/*
F(ngx_http_write_filter) {
if (active && pid() == target()) {
printf("http writer filter: %s\n", ngx_chain_dump($in))
}
}
*/
F(ngx_http_chunked_body_filter) {
if (active && pid() == target()) {
printf("http chunked filter: %s\n", ngx_chain_dump($in))
}
}
F(ngx_http_replace_output) {
if (active && pid() == target()) {
printf("http replace output: %s\n", ngx_chain_dump($ctx->out))
}
}
probe syscall.writev {
if (active && pid() == target()) {
printf("writev(%s)", ngx_iovec_dump($vec, $vlen))
/*
for (i = 0; i < $vlen; i++) {
printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len)))
}
*/
}
}
probe syscall.writev.return {
if (active && pid() == target()) {
printf(" = %s\n", retstr)
}
}
_EOC_
#no_diff();
#no_long_string();
run_tests();
__DATA__
=== TEST 1: ambiguous pattern
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcabde;
replace_filter abcabd X;
}
--- request
GET /t
--- stap
F(ngx_http_replace_non_capturing_parse) {
println("non capturing parse")
}
F(ngx_http_replace_capturing_parse) {
println("capturing parse")
}
F(ngx_http_replace_complex_value) {
println("complex value")
}
--- stap_out_like chop
^(non capturing parse\n)+$
--- response_body
abcXe
--- no_error_log
[alert]
[error]
=== TEST 2: ambiguous pattern
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n ababac;
replace_filter abac X;
}
--- request
GET /t
--- response_body chop
abX
--- no_error_log
[alert]
[error]
=== TEST 3: alt
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abc;
replace_filter 'ab|abc' X;
}
--- request
GET /t
--- response_body
Xc
--- no_error_log
[alert]
[error]
=== TEST 4: caseless
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcaBde;
replace_filter abCabd X i;
}
--- request
GET /t
--- response_body
abcXe
--- no_error_log
[alert]
[error]
=== TEST 5: case sensitive (no match)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcaBde;
replace_filter abCabd X;
}
--- request
GET /t
--- response_body
abcabcaBde
--- no_error_log
[alert]
[error]
=== TEST 6: 1-byte chain bufs
--- config
default_type text/html;
replace_filter_max_buffered_size 3;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
}
--- response_body
abXd
--- no_error_log
[alert]
[error]
=== TEST 7: 2-byte chain bufs
--- config
default_type text/html;
replace_filter_max_buffered_size 2;
location = /t {
echo -n ab;
echo -n ab;
echo -n ac;
echo d;
replace_filter abac X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
abXd
--- no_error_log
[alert]
[error]
=== TEST 8: 3-byte chain bufs
--- config
default_type text/html;
replace_filter_max_buffered_size 3;
location = /t {
echo -n aba;
echo -n bac;
echo d;
replace_filter abac X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
abXd
--- no_error_log
[alert]
[error]
=== TEST 9: 3-byte chain bufs (more)
--- config
default_type text/html;
replace_filter_max_buffered_size 4;
location = /t {
echo -n aba;
echo -n bac;
echo d;
replace_filter abacd X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
abX
--- no_error_log
[alert]
[error]
=== TEST 10: once by default (1st char matched)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /t {
echo abcabcabde;
replace_filter a X;
}
--- request
GET /t
--- response_body
Xbcabcabde
--- no_error_log
[alert]
[error]
=== TEST 11: once by default (2nd char matched)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcabde;
replace_filter b X;
}
--- request
GET /t
--- response_body
aXcabcabde
--- no_error_log
[alert]
[error]
=== TEST 12: global substitution
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo bbc;
replace_filter b X g;
}
--- request
GET /t
--- response_body
XXc
--- no_error_log
[alert]
[error]
=== TEST 13: global substitution
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcabde;
replace_filter b X g;
}
--- request
GET /t
--- response_body
aXcaXcaXde
--- no_error_log
[alert]
[error]
=== TEST 14: global substitution (empty captures)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n abcabcabde;
replace_filter [0-9]* X g;
}
--- request
GET /t
--- response_body chop
XaXbXcXaXbXcXaXbXdXeX
--- no_error_log
[alert]
[error]
=== TEST 15: global substitution (empty captures, splitted)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n ab;
echo -n cab;
echo -n c;
echo -n abde;
replace_filter [0-9]* X g;
}
--- request
GET /t
--- response_body chop
XaXbXcXaXbXcXaXbXdXeX
--- no_error_log
[alert]
[error]
=== TEST 16: global substitution (\d+)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo "hello1234, 56 world";
replace_filter \d+ X g;
}
--- request
GET /t
--- response_body
helloX, X world
--- no_error_log
[alert]
[error]
=== TEST 17: replace_filter_types default to text/html
--- config
default_type text/plain;
location /t {
echo abc;
replace_filter b X;
}
--- request
GET /t
--- response_body
abc
--- no_error_log
[alert]
[error]
=== TEST 18: custom replace_filter_types
--- config
default_type text/plain;
location /t {
echo abc;
replace_filter b X;
replace_filter_types text/plain;
}
--- request
GET /t
--- response_body
aXc
--- no_error_log
[alert]
[error]
=== TEST 19: multiple replace_filter_types settings
--- config
default_type text/plain;
location /t {
echo abc;
replace_filter b X;
replace_filter_types text/css text/plain;
}
--- request
GET /t
--- response_body
aXc
--- no_error_log
[alert]
[error]
=== TEST 20: trim leading spaces
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /a.html {
replace_filter '^\s+' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /a.html
--- response_body
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 21: trim trailing spaces
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /a.html {
replace_filter '\s+$' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /a.html
--- response_body chop
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 22: trim both leading and trailing spaces
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /a.html {
replace_filter '^\s+|\s+$' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /a.html
--- response_body chop
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 23: pure flush buf in the stream (no data)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location = /t {
echo_flush;
replace_filter 'a' 'X' g;
}
--- request
GET /t
--- response_body chop
--- no_error_log
[alert]
[error]
=== TEST 24: pure flush buf in the stream (with data)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location = /t {
echo a;
echo_flush;
replace_filter 'a' 'X' g;
}
--- request
GET /t
--- stap3 eval: $::StapOutputChains
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:539") {
printf("chain: %s", ngx_chain_dump($ctx->busy))
//print_ubacktrace()
}
--- response_body
X
--- no_error_log
[alert]
[error]
=== TEST 25: trim both leading and trailing spaces (1 byte at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
location = /t {
echo -n 'a';
echo ' ';
echo "b";
replace_filter '^\s+|\s+$' '' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body chop
a
b
--- no_error_log
[alert]
[error]
=== TEST 26: trim both leading and trailing spaces (1 byte at a time), no \s for $
--- config
replace_filter_max_buffered_size 1;
default_type text/html;
location = /t {
echo -n 'a';
echo ' ';
echo "b";
replace_filter '^\s+| +$' '' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a
b
--- no_error_log
[alert]
[error]
=== TEST 27: trim both leading and trailing spaces (1 byte at a time)
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- request
GET /t
--- response_body chop
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 28: \b at the border
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n a;
echo b;
replace_filter '\bb|a' X g;
}
--- request
GET /t
--- response_body
Xb
--- no_error_log
[alert]
[error]
=== TEST 29: \B at the border
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /t {
echo -n a;
echo ',';
replace_filter '\B,|a' X g;
}
--- request
GET /t
--- response_body
X,
--- no_error_log
[alert]
[error]
=== TEST 30: \A at the border
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n a;
echo 'b';
replace_filter '\Ab|a' X g;
}
--- request
GET /t
--- response_body
Xb
--- no_error_log
[alert]
[error]
=== TEST 31: memory bufs with last_buf=1
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
return 200 "abc";
replace_filter \w+ X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body chop
X
--- no_error_log
[alert]
[error]
=== TEST 32: trim both leading and trailing spaces (2 bytes at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 4;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
local len = string.len(txt)
i = 1
while i <= len do
if i == len then
ngx.print(string.sub(txt, i, i))
i = i + 1
else
ngx.print(string.sub(txt, i, i + 1))
i = i + 2
end
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2 eval: $::StapOutputChains
--- request
GET /t
--- response_body chop
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 33: trim both leading and trailing spaces (3 bytes at a time)
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
local len = string.len(txt)
i = 1
while i <= len do
if i == len then
ngx.print(string.sub(txt, i, i))
i = i + 1
elseif i == len - 1 then
ngx.print(string.sub(txt, i, i + 1))
i = i + 2
else
ngx.print(string.sub(txt, i, i + 2))
i = i + 3
end
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2 eval: $::StapOutputChains
--- request
GET /t
--- response_body chop
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 34: github issue #2: error "general look-ahead not supported"
--- config
replace_filter_max_buffered_size 3;
location /t {
charset utf-8;
default_type text/html;
echo "ABCabcABCabc";
#replace_filter_types text/plain;
replace_filter "a.+a" "X" "ig";
}
--- request
GET /t
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") {
print_ubacktrace()
}
--- response_body
Xbc
--- no_error_log
[alert]
[error]
=== TEST 35: backtrack to the middle of a pending capture (pending: output|capture + rematch)
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n ab;
echo -n c;
echo d;
replace_filter 'abce|b' 'X' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- no_error_log
[alert]
[error]
=== TEST 36: backtrack to the middle of a pending capture (pending: output + capture|rematch
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n a;
echo -n bc;
echo d;
replace_filter 'abce|b' 'X' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- no_error_log
[alert]
[error]
=== TEST 37: backtrack to the middle of a pending capture (pending: output + capture + rematch
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n a;
echo -n b;
echo -n c;
echo d;
replace_filter 'abce|b' 'X' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1522") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- no_error_log
[alert]
[error]
=== TEST 38: backtrack to the middle of a pending capture (pending: output|capture|rematch
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n abc;
echo d;
replace_filter 'abce|b' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- no_error_log
[alert]
[error]
=== TEST 39: backtrack to the middle of a pending capture (pending: output|capture|rematch(2)
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n abcc;
echo d;
replace_filter 'abcce|b' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXccd
--- no_error_log
[alert]
[error]
=== TEST 40: backtrack to the middle of a pending capture (pending: output|capture(2)|rematch
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n abbc;
echo d;
replace_filter 'abbce|bb' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- no_error_log
[alert]
[error]
=== TEST 41: backtrack to the middle of a pending capture (pending: output(2)|capture|rematch
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n aabc;
echo d;
replace_filter 'aabce|b' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aaXcd
--- no_error_log
[alert]
[error]
=== TEST 42: backtrack to the beginning of a pending capture (pending: output + capture|rematch(2)
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n a;
echo -n bcc;
echo d;
replace_filter 'abcce|b' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXccd
--- no_error_log
[alert]
[error]
=== TEST 43: backtrack to the beginning of a pending capture (pending: output + capture(2)|rematch
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n a;
echo -n bbc;
echo d;
replace_filter 'abbce|bb' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- no_error_log
[alert]
[error]
=== TEST 44: backtrack to the middle of a pending capture (pending: output(2) + capture|rematch
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n aa;
echo -n bc;
echo d;
replace_filter 'aabce|b' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aaXcd
--- no_error_log
[alert]
[error]
=== TEST 45: assertions across AGAIN
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n a;
echo -n "\n";
echo b;
replace_filter 'a\n^b' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
X
--- no_error_log
[alert]
[error]
=== TEST 46: assertions when capture backtracking happens
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n a;
echo -n b;
echo -n c;
echo -n d;
echo f;
#echo abcdf;
replace_filter 'abcde|b|\bc' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcdf
--- no_error_log
[alert]
[error]
=== TEST 47: assertions when capture backtracking happens (2 pending matches)
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n a;
echo -n b;
echo -n ' ';
echo -n d;
echo f;
#echo ab df;
replace_filter 'ab de|b|b |\b ' 'X' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXXdf
--- no_error_log
[alert]
[error]
=== TEST 48: github issue #2: error "general look-ahead not supported", no "g"
--- config
replace_filter_max_buffered_size 3;
location /t {
charset utf-8;
default_type text/html;
echo "ABCabcABCabc";
#replace_filter_types text/plain;
replace_filter "a.+a" "X" "i";
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
Xbc
--- no_error_log
[alert]
[error]
=== TEST 49: nested rematch bufs
--- config
replace_filter_max_buffered_size 4;
location /t {
default_type text/html;
echo -n a;
echo -n b;
echo -n c;
echo -n d;
echo -n e;
echo g;
#echo abcdeg;
replace_filter 'abcdef|b|cdf|c' X g;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
aXXdeg
--- no_error_log
[alert]
[error]
=== TEST 50: nested rematch bufs (splitting pending buf)
--- config
replace_filter_max_buffered_size 6;
location /t {
default_type text/html;
echo -n a;
echo -n b;
echo -n cd;
echo -n e;
echo -n f;
echo -n g;
echo i;
#echo abcdefh;
replace_filter 'abcdefgh|b|cdeg|d' X g;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
aXcXefgi
--- no_error_log
[alert]
[error]
=== TEST 51: remove C/C++ comments (1 byte at a time)
--- config
replace_filter_max_buffered_size 42;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '/\*.*?\*/|//[^\n]*' '' g;
}
--- user_files
>>> a.html
i don't know // hello // world /* */
hello world /** abc * b/c /*
hello ** // world
*
*/
blah /* hi */ */ b
//
///hi
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- request
GET /t
--- response_body eval
" i don't know
hello world
blah */ b
"
--- no_error_log
[alert]
[error]
=== TEST 52: remove C/C++ comments (all at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /a.html {
replace_filter '/\*.*?\*/|//[^\n]*' '' g;
}
--- user_files
>>> a.html
i don't know // hello // world /* */
hello world /** abc * b/c /*
hello ** // world
*
*/
blah /* hi */ */ b
//
///hi
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- request
GET /a.html
--- response_body eval
" i don't know
hello world
blah */ b
"
--- no_error_log
[alert]
[error]
=== TEST 53: remove C/C++ comments (all at a time) - server-level config
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
replace_filter '/\*.*?\*/|//[^\n]*' '' g;
--- user_files
>>> a.html
i don't know // hello // world /* */
hello world /** abc * b/c /*
hello ** // world
*
*/
blah /* hi */ */ b
//
///hi
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- request
GET /a.html
--- response_body eval
" i don't know
hello world
blah */ b
"
--- no_error_log
[alert]
[error]
=== TEST 54: multiple replace_filter_types settings (server level)
--- config
replace_filter_max_buffered_size 0;
default_type text/plain;
replace_filter_types text/css text/plain;
location /t {
echo abc;
replace_filter b X;
}
--- request
GET /t
--- response_body
aXc
--- no_error_log
[alert]
[error]
=== TEST 55: multiple replace_filter_types settings (server level, but overridding in location)
--- config
replace_filter_max_buffered_size 0;
default_type text/plain;
replace_filter_types text/css text/plain;
location /t {
echo abc;
replace_filter_types text/javascript;
replace_filter b X;
}
--- request
GET /t
--- response_body
abc
--- no_error_log
[alert]
[error]
=== TEST 56: do not use replace_filter at all
--- config
replace_filter_max_buffered_size 0;
default_type text/plain;
replace_filter_types text/css text/plain;
location /t {
echo abc;
replace_filter_types text/css;
}
--- request
GET /t
--- response_body
abc
--- no_error_log
[alert]
[error]
=== TEST 57: bad regex
--- config
default_type text/html;
location /t {
echo abc;
replace_filter '(a+b' '';
}
--- request
GET /t
--- response_body
abc
--- no_error_log
[alert]
[error]
--- SKIP
=== TEST 58: github issue #3: data lost in particular situation
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location /t {
default_type text/html;
echo "ABCabcABC";
echo "ABCabcABC";
#echo "ABCabcABC\nABCabcABC";
replace_filter "(a.+?c){2}" "X" "ig";
}
--- request
GET /t
--- response_body
XXABC
--- no_error_log
[alert]
[error]
=== TEST 59: variation
--- config
replace_filter_max_buffered_size 5;
default_type text/html;
location /t {
default_type text/html;
#echo "ABCabcABC";
#echo "ABCabcABC";
echo "ACacAC ACacAC";
replace_filter "(a.+?c){2}" "X" "ig";
}
--- request
GET /t
--- response_body
XacAC
--- no_error_log
[alert]
[error]
=== TEST 60: nested pending matched
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location /t {
default_type text/html;
echo -n a;
echo -n b;
echo -n c;
echo -n def;
echo -n gh;
echo -n i;
echo k;
#echo abcdefig;
replace_filter "abcdefghij|bcdefg|cd" "X" "ig";
}
--- request
GET /t
--- response_body
aXhik
--- no_error_log
[alert]
[error]
=== TEST 61: test split chain with b_sane=1, next=NULL
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n aba;
echo -n ba;
echo -n bac;
echo d;
#echo abababacd;
replace_filter abacd X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") {
print_ubacktrace()
}
--- response_body
ababX
--- no_error_log
[alert]
[error]
=== TEST 62: test split chain with b_sane=1, next not NULL
--- config
replace_filter_max_buffered_size 6;
default_type text/html;
location = /t {
echo -n aba;
echo -n ba;
echo -n ba;
echo -n bac;
echo d;
#echo abababacd;
replace_filter ababacd X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") {
print_ubacktrace()
}
--- response_body
ababX
--- no_error_log
[alert]
[error]
=== TEST 63: trim leading spaces (1 byte at a time)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /a.html {
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '^\s+' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /t
--- response_body
hello, world
blah yeah
hello
baby!
abc
--- no_error_log
[alert]
[error]
=== TEST 64: split ctx->pending into ctx->pending and ctx->free
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
#echo "abc\nd";
echo -n a;
echo -n b;
echo -n c;
echo -n "\n";
echo d;
replace_filter "abcd|bc\ne|c$" X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1482") {
print_ubacktrace()
}
--- response_body
abX
d
--- no_error_log
[alert]
[error]
=== TEST 65: trim both leading and trailing spaces (1 byte at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 2;
location /t {
echo -n 'a';
echo_sleep 0.001;
echo ' ';
echo_sleep 0.001;
echo '';
echo_sleep 0.001;
echo ' ';
echo_sleep 0.001;
echo "b";
echo_sleep 0.001;
echo " ";
replace_filter '^\s+|\s+$' '' g;
}
location = /main {
echo_location_async /t1;
echo_location_async /t2;
echo_location_async /t3;
echo_location_async /t4;
echo_location_async /t5;
echo_location_async /t6;
}
--- stap3 eval: $::StapOutputChains
--- request
GET /main
--- response_body
a
b
a
b
a
b
a
b
a
b
a
b
--- no_error_log
[alert]
[error]
================================================
FILE: t/02-max-buffered.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
#no_shuffle();
plan tests => repeat_each() * (blocks() * 4);
our $StapOutputChains = <<'_EOC_';
global active
F(ngx_http_handler) {
active = 1
}
/*
F(ngx_http_write_filter) {
if (active && pid() == target()) {
printf("http writer filter: %s\n", ngx_chain_dump($in))
}
}
*/
F(ngx_http_chunked_body_filter) {
if (active && pid() == target()) {
printf("http chunked filter: %s\n", ngx_chain_dump($in))
}
}
F(ngx_http_replace_output) {
if (active && pid() == target()) {
printf("http replace output: %s\n", ngx_chain_dump($ctx->out))
}
}
probe syscall.writev {
if (active && pid() == target()) {
printf("writev(%s)", ngx_iovec_dump($vec, $vlen))
/*
for (i = 0; i < $vlen; i++) {
printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len)))
}
*/
}
}
probe syscall.writev.return {
if (active && pid() == target()) {
printf(" = %s\n", retstr)
}
}
_EOC_
#no_diff();
no_long_string();
run_tests();
__DATA__
=== TEST 1: 1-byte chain bufs (0)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
}
--- response_body
ababacd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (0): 1
--- no_error_log
[error]
=== TEST 2: 1-byte chain bufs (1)
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac X;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1439") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- response_body
ababacd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (1): 2
--- no_error_log
[error]
=== TEST 3: trim both leading and trailing spaces (1 byte at a time) (2)
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- request
GET /t
--- response_body
hello, world
blah yeah
hello
baby!
abc
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (2): 3
--- no_error_log
[error]
=== TEST 4: github issue #2: error "general look-ahead not supported"
--- config
replace_filter_max_buffered_size 0;
location /t {
charset utf-8;
default_type text/html;
echo "ABCabcABCabc";
#replace_filter_types text/plain;
replace_filter "a.+a" "X" "ig";
}
--- request
GET /t
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1481") {
print_ubacktrace()
}
--- response_body
ABCabcABCabc
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (0): 2
--- no_error_log
[error]
=== TEST 5: backtrack to the middle of a pending capture (pending: output|capture + rematch) (0)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location = /t {
echo -n ab;
echo -n c;
echo d;
replace_filter 'abce|b' 'X' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
abcd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (0): 1
--- no_error_log
[error]
=== TEST 6: backtrack to the middle of a pending capture (pending: output|capture + rematch) (1)
--- config
replace_filter_max_buffered_size 1;
default_type text/html;
location = /t {
echo -n ab;
echo -n c;
echo d;
replace_filter 'abce|b' 'X' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aXcd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (1): 2
--- no_error_log
[error]
================================================
FILE: t/03-var.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
#no_shuffle();
plan tests => repeat_each() * (blocks() * 4 + 3);
our $StapOutputChains = <<'_EOC_';
global active
F(ngx_http_handler) {
active = 1
}
/*
F(ngx_http_write_filter) {
if (active && pid() == target()) {
printf("http writer filter: %s\n", ngx_chain_dump($in))
}
}
*/
F(ngx_http_chunked_body_filter) {
if (active && pid() == target()) {
printf("http chunked filter: %s\n", ngx_chain_dump($in))
}
}
F(ngx_http_replace_output) {
if (active && pid() == target()) {
printf("http replace output: %s\n", ngx_chain_dump($ctx->out))
}
}
probe syscall.writev {
if (active && pid() == target()) {
printf("writev(%s)", ngx_iovec_dump($vec, $vlen))
/*
for (i = 0; i < $vlen; i++) {
printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len)))
}
*/
}
}
probe syscall.writev.return {
if (active && pid() == target()) {
printf(" = %s\n", retstr)
}
}
_EOC_
#no_diff();
#no_long_string();
run_tests();
__DATA__
=== TEST 1: nginx vars (global)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo abc;
replace_filter . $foo g;
}
--- request
GET /t
--- stap
F(ngx_http_replace_non_capturing_parse) {
println("non capturing parse")
}
F(ngx_http_replace_capturing_parse) {
println("capturing parse")
}
--- stap_out_like chop
^(non capturing parse\n)+$
--- response_body chop
XXXX
--- no_error_log
[alert]
[error]
=== TEST 2: nginx vars (non-global)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo abc;
replace_filter . $foo;
}
--- request
GET /t
--- stap
F(ngx_http_replace_non_capturing_parse) {
println("non capturing parse")
}
F(ngx_http_replace_capturing_parse) {
println("capturing parse")
}
--- stap_out_like chop
^(non capturing parse\n)+$
--- response_body
Xbc
--- no_error_log
[alert]
[error]
=== TEST 3: undefined nginx vars
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abc;
replace_filter . $foo;
}
--- request
GET /t
--- response_body
Xbc
--- no_error_log
[alert]
[error]
--- SKIP
=== TEST 4: use of capturing variables
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abc;
replace_filter . $1;
}
--- request
GET /t
--- response_body
Xbc
--- no_error_log
[alert]
[error]
--- SKIP
=== TEST 5: more contexts
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo abc;
replace_filter . "[$foo]";
}
--- request
GET /t
--- response_body
[X]bc
--- no_error_log
[alert]
[error]
=== TEST 6: more nginx vars
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
set $bar Y;
echo abc;
replace_filter . "[$foo,$bar]";
}
--- request
GET /t
--- response_body
[X,Y]bc
--- no_error_log
[alert]
[error]
=== TEST 7: various lengths of nginx var values
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo XYZ;
set $bar "";
echo abc;
replace_filter . "[$foo,$bar]";
}
--- request
GET /t
--- response_body
[XYZ,]bc
--- no_error_log
[alert]
[error]
=== TEST 8: escaping the dollar sign
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
set $bar Y;
echo abc;
replace_filter . "[$foo,$$bar]";
}
--- request
GET /t
--- response_body
[X,$bar]bc
--- no_error_log
[alert]
[error]
=== TEST 9: \ is not an escaping sequence
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
set $bar Y;
echo abc;
replace_filter . "[\$foo,\$bar]";
}
--- request
GET /t
--- response_body
[\X,\Y]bc
--- no_error_log
[alert]
[error]
=== TEST 10: cached subs values
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo abc;
replace_filter . "$foo" g;
}
--- request
GET /t
--- response_body chop
XXXX
--- stap
F(ngx_http_replace_complex_value) {
println("complex value")
}
--- stap_out
complex value
--- no_error_log
[alert]
[error]
================================================
FILE: t/04-capturing.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
#no_shuffle();
plan tests => repeat_each() * (blocks() * 4 + 1);
our $StapOutputChains = <<'_EOC_';
global active
F(ngx_http_handler) {
active = 1
}
/*
F(ngx_http_write_filter) {
if (active && pid() == target()) {
printf("http writer filter: %s\n", ngx_chain_dump($in))
}
}
*/
F(ngx_http_chunked_body_filter) {
if (active && pid() == target()) {
printf("http chunked filter: %s\n", ngx_chain_dump($in))
}
}
F(ngx_http_replace_output) {
if (active && pid() == target()) {
printf("http replace output: %s\n", ngx_chain_dump($ctx->out))
}
}
probe syscall.writev {
if (active && pid() == target()) {
printf("writev(%s)", ngx_iovec_dump($vec, $vlen))
/*
for (i = 0; i < $vlen; i++) {
printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len)))
}
*/
}
}
probe syscall.writev.return {
if (active && pid() == target()) {
printf(" = %s\n", retstr)
}
}
_EOC_
#no_diff();
#no_long_string();
run_tests();
__DATA__
=== TEST 1: ambiguous pattern
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcabde;
replace_filter abcabd "[$&]";
}
--- request
GET /t
--- response_body
abc[abcabd]e
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 2: ambiguous pattern
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n ababac;
replace_filter abac "[$&]";
}
--- request
GET /t
--- response_body chop
ab[abac]
--- no_error_log
[alert]
[error]
=== TEST 3: alt
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abc;
replace_filter 'ab|abc' [$&];
}
--- request
GET /t
--- response_body
[ab]c
--- no_error_log
[alert]
[error]
=== TEST 4: caseless
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcaBde;
replace_filter abCabd [$&] i;
}
--- request
GET /t
--- response_body
abc[abcaBd]e
--- no_error_log
[alert]
[error]
=== TEST 5: case sensitive (no match)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcaBde;
replace_filter abCabd [$&];
}
--- request
GET /t
--- response_body
abcabcaBde
--- no_error_log
[alert]
[error]
=== TEST 6: 1-byte chain bufs
--- config
default_type text/html;
replace_filter_max_buffered_size 3;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
}
--- response_body
ab[abac]d
--- no_error_log
[alert]
[error]
=== TEST 7: 2-byte chain bufs
--- config
default_type text/html;
replace_filter_max_buffered_size 2;
location = /t {
echo -n ab;
echo -n ab;
echo -n ac;
echo d;
replace_filter abac [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
ab[abac]d
--- no_error_log
[alert]
[error]
=== TEST 8: 3-byte chain bufs
--- config
default_type text/html;
replace_filter_max_buffered_size 3;
location = /t {
echo -n aba;
echo -n bac;
echo d;
replace_filter abac [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
ab[abac]d
--- no_error_log
[alert]
[error]
=== TEST 9: 3-byte chain bufs (more)
--- config
default_type text/html;
replace_filter_max_buffered_size 4;
location = /t {
echo -n aba;
echo -n bac;
echo d;
replace_filter abacd [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
ab[abacd]
--- no_error_log
[alert]
[error]
=== TEST 10: once by default (1st char matched)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /t {
echo abcabcabde;
replace_filter a [$&];
}
--- request
GET /t
--- response_body
[a]bcabcabde
--- no_error_log
[alert]
[error]
=== TEST 11: once by default (2nd char matched)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcabde;
replace_filter b [$&];
}
--- request
GET /t
--- response_body
a[b]cabcabde
--- no_error_log
[alert]
[error]
=== TEST 12: global substitution
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo bbc;
replace_filter b [$&] g;
}
--- request
GET /t
--- response_body
[b][b]c
--- no_error_log
[alert]
[error]
=== TEST 13: global substitution
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo abcabcabde;
replace_filter b [$&] g;
}
--- request
GET /t
--- response_body
a[b]ca[b]ca[b]de
--- no_error_log
[alert]
[error]
=== TEST 14: global substitution (empty captures)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n abcabcabde;
replace_filter [0-9]* [$&] g;
}
--- request
GET /t
--- response_body chop
[]a[]b[]c[]a[]b[]c[]a[]b[]d[]e[]
--- no_error_log
[alert]
[error]
=== TEST 15: global substitution (empty captures, splitted)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n ab;
echo -n cab;
echo -n c;
echo -n abde;
replace_filter [0-9]* [$&] g;
}
--- request
GET /t
--- response_body chop
[]a[]b[]c[]a[]b[]c[]a[]b[]d[]e[]
--- no_error_log
[alert]
[error]
=== TEST 16: global substitution (\d+)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo "hello1234, 56 world";
replace_filter \d+ [$&] g;
}
--- request
GET /t
--- response_body
hello[1234], [56] world
--- no_error_log
[alert]
[error]
=== TEST 17: replace_filter_types default to text/html
--- config
default_type text/plain;
location /t {
echo abc;
replace_filter b [$&];
}
--- request
GET /t
--- response_body
abc
--- no_error_log
[alert]
[error]
=== TEST 18: custom replace_filter_types
--- config
default_type text/plain;
location /t {
echo abc;
replace_filter b [$&];
replace_filter_types text/plain;
}
--- request
GET /t
--- response_body
a[b]c
--- no_error_log
[alert]
[error]
=== TEST 19: multiple replace_filter_types settings
--- config
default_type text/plain;
location /t {
echo abc;
replace_filter b [$&];
replace_filter_types text/css text/plain;
}
--- request
GET /t
--- response_body
a[b]c
--- no_error_log
[alert]
[error]
=== TEST 20: trim leading spaces
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /a.html {
replace_filter '^\s+' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /a.html
--- response_body
[ ]hello, world
blah yeah
hello
[ ]baby!
[
]abc
--- no_error_log
[alert]
[error]
=== TEST 21: trim trailing spaces
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /a.html {
replace_filter '\s+$' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /a.html
--- response_body chop
hello, world[ ]
blah yeah
hello[ ]
baby![
]
abc[
]
--- no_error_log
[alert]
[error]
=== TEST 22: trim both leading and trailing spaces
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /a.html {
replace_filter '^\s+|\s+$' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /a.html
--- response_body chop
[ ]hello, world[ ]
blah yeah
hello[ ]
[ ]baby!
[
]abc[
]
--- no_error_log
[alert]
[error]
=== TEST 23: pure flush buf in the stream (no data)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location = /t {
echo_flush;
replace_filter 'a' '[$&]' g;
}
--- request
GET /t
--- response_body chop
--- no_error_log
[alert]
[error]
=== TEST 24: pure flush buf in the stream (with data)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location = /t {
echo a;
echo_flush;
replace_filter 'a' '[$&]' g;
}
--- request
GET /t
--- stap3 eval: $::StapOutputChains
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:539") {
printf("chain: %s", ngx_chain_dump($ctx->busy))
//print_ubacktrace()
}
--- response_body
[a]
--- no_error_log
[alert]
[error]
=== TEST 25: trim both leading and trailing spaces (1 byte at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 2;
location = /t {
echo -n 'a';
echo ' ';
echo "b";
replace_filter '^\s+|\s+$' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body chop
a[ ]
b[
]
--- no_error_log
[alert]
[error]
=== TEST 26: trim both leading and trailing spaces (1 byte at a time), no \s for $
--- config
replace_filter_max_buffered_size 1;
default_type text/html;
location = /t {
echo -n 'a';
echo ' ';
echo "b";
replace_filter '^\s+| +$' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[ ]
b
--- no_error_log
[alert]
[error]
=== TEST 27: trim both leading and trailing spaces (1 byte at a time)
--- config
replace_filter_max_buffered_size 7;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- request
GET /t
--- response_body chop
[ ]hello, world[ ]
blah yeah
hello[ ]
[ ]baby!
[
]abc[
]
--- no_error_log
[alert]
[error]
=== TEST 28: \b at the border
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n a;
echo b;
replace_filter '\bb|a' [$&] g;
}
--- request
GET /t
--- response_body
[a]b
--- no_error_log
[alert]
[error]
=== TEST 29: \B at the border
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location /t {
echo -n a;
echo ',';
replace_filter '\B,|a' [$&] g;
}
--- request
GET /t
--- response_body
[a],
--- no_error_log
[alert]
[error]
=== TEST 30: \A at the border
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n a;
echo 'b';
replace_filter '\Ab|a' [$&] g;
}
--- request
GET /t
--- response_body
[a]b
--- no_error_log
[alert]
[error]
=== TEST 31: memory bufs with last_buf=1
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
return 200 "abc";
replace_filter \w+ [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body chop
[abc]
--- no_error_log
[alert]
[error]
=== TEST 32: trim both leading and trailing spaces (2 bytes at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 7;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
local len = string.len(txt)
i = 1
while i <= len do
if i == len then
ngx.print(string.sub(txt, i, i))
i = i + 1
else
ngx.print(string.sub(txt, i, i + 1))
i = i + 2
end
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2 eval: $::StapOutputChains
--- request
GET /t
--- response_body chop
[ ]hello, world[ ]
blah yeah
hello[ ]
[ ]baby!
[
]abc[
]
--- no_error_log
[alert]
[error]
=== TEST 33: trim both leading and trailing spaces (3 bytes at a time)
--- config
replace_filter_max_buffered_size 5;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
local len = string.len(txt)
i = 1
while i <= len do
if i == len then
ngx.print(string.sub(txt, i, i))
i = i + 1
elseif i == len - 1 then
ngx.print(string.sub(txt, i, i + 1))
i = i + 2
else
ngx.print(string.sub(txt, i, i + 2))
i = i + 3
end
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2 eval: $::StapOutputChains
--- request
GET /t
--- response_body chop
[ ]hello, world[ ]
blah yeah
hello[ ]
[ ]baby!
[
]abc[
]
--- no_error_log
[alert]
[error]
=== TEST 34: github issue #2: error "general look-ahead not supported"
--- config
replace_filter_max_buffered_size 13;
location /t {
charset utf-8;
default_type text/html;
echo "ABCabcABCabc";
#replace_filter_types text/plain;
replace_filter "a.+a" "[$&]" "ig";
}
--- request
GET /t
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") {
print_ubacktrace()
}
--- response_body
[ABCabcABCa]bc
--- no_error_log
[alert]
[error]
=== TEST 35: backtrack to the middle of a pending capture (pending: output|capture + rematch)
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n ab;
echo -n c;
echo d;
replace_filter 'abce|b' '[$&]' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]cd
--- no_error_log
[alert]
[error]
=== TEST 36: backtrack to the middle of a pending capture (pending: output + capture|rematch
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n a;
echo -n bc;
echo d;
replace_filter 'abce|b' '[$&]' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]cd
--- no_error_log
[alert]
[error]
=== TEST 37: backtrack to the middle of a pending capture (pending: output + capture + rematch
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n a;
echo -n b;
echo -n c;
echo d;
replace_filter 'abce|b' '[$&]' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1522") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]cd
--- no_error_log
[alert]
[error]
=== TEST 38: backtrack to the middle of a pending capture (pending: output|capture|rematch
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
echo -n abc;
echo d;
replace_filter 'abce|b' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]cd
--- no_error_log
[alert]
[error]
=== TEST 39: backtrack to the middle of a pending capture (pending: output|capture|rematch(2)
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n abcc;
echo d;
replace_filter 'abcce|b' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]ccd
--- no_error_log
[alert]
[error]
=== TEST 40: backtrack to the middle of a pending capture (pending: output|capture(2)|rematch
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n abbc;
echo d;
replace_filter 'abbce|bb' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[bb]cd
--- no_error_log
[alert]
[error]
=== TEST 41: backtrack to the middle of a pending capture (pending: output(2)|capture|rematch
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n aabc;
echo d;
replace_filter 'aabce|b' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aa[b]cd
--- no_error_log
[alert]
[error]
=== TEST 42: backtrack to the beginning of a pending capture (pending: output + capture|rematch(2)
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n a;
echo -n bcc;
echo d;
replace_filter 'abcce|b' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]ccd
--- no_error_log
[alert]
[error]
=== TEST 43: backtrack to the beginning of a pending capture (pending: output + capture(2)|rematch
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n a;
echo -n bbc;
echo d;
replace_filter 'abbce|bb' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[bb]cd
--- no_error_log
[alert]
[error]
=== TEST 44: backtrack to the middle of a pending capture (pending: output(2) + capture|rematch
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n aa;
echo -n bc;
echo d;
replace_filter 'aabce|b' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
aa[b]cd
--- no_error_log
[alert]
[error]
=== TEST 45: assertions across AGAIN
--- config
replace_filter_max_buffered_size 2;
default_type text/html;
location = /t {
echo -n a;
echo -n "\n";
echo b;
replace_filter 'a\n^b' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
[a
b]
--- no_error_log
[alert]
[error]
=== TEST 46: assertions when capture backtracking happens
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n a;
echo -n b;
echo -n c;
echo -n d;
echo f;
#echo abcdf;
replace_filter 'abcde|b|\bc' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b]cdf
--- no_error_log
[alert]
[error]
=== TEST 47: assertions when capture backtracking happens (2 pending matches)
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n a;
echo -n b;
echo -n ' ';
echo -n d;
echo f;
#echo ab df;
replace_filter 'ab de|b|b |\b ' '[$&]' g;
}
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
a[b][ ]df
--- no_error_log
[alert]
[error]
=== TEST 48: github issue #2: error "general look-ahead not supported", no "g"
--- config
replace_filter_max_buffered_size 13;
location /t {
charset utf-8;
default_type text/html;
echo "ABCabcABCabc";
#replace_filter_types text/plain;
replace_filter "a.+a" "[$&]" "i";
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
[ABCabcABCa]bc
--- no_error_log
[alert]
[error]
=== TEST 49: nested rematch bufs
--- config
replace_filter_max_buffered_size 5;
location /t {
default_type text/html;
echo -n a;
echo -n b;
echo -n c;
echo -n d;
echo -n e;
echo g;
#echo abcdeg;
replace_filter 'abcdef|b|cdf|c' [$&] g;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
a[b][c]deg
--- no_error_log
[alert]
[error]
=== TEST 50: nested rematch bufs (splitting pending buf)
--- config
replace_filter_max_buffered_size 7;
location /t {
default_type text/html;
echo -n a;
echo -n b;
echo -n cd;
echo -n e;
echo -n f;
echo -n g;
echo i;
#echo abcdefh;
replace_filter 'abcdefgh|b|cdeg|d' [$&] g;
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- response_body
a[b]c[d]efgi
--- no_error_log
[alert]
[error]
=== TEST 51: remove C/C++ comments (1 byte at a time)
--- config
replace_filter_max_buffered_size 50;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '/\*.*?\*/|//[^\n]*' '[$&]' g;
}
--- user_files
>>> a.html
i don't know // hello // world /* */
hello world /** abc * b/c /*
hello ** // world
*
*/
blah /* hi */ */ b
//
///hi
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- request
GET /t
--- response_body eval
" i don't know [// hello // world /* */]
hello world [/** abc * b/c /*
hello ** // world
*
*/]
blah [/* hi */] */ b
[//]
[///hi]
"
--- no_error_log
[alert]
[error]
=== TEST 52: remove C/C++ comments (all at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /a.html {
replace_filter '/\*.*?\*/|//[^\n]*' '[$&]' g;
}
--- user_files
>>> a.html
i don't know // hello // world /* */
hello world /** abc * b/c /*
hello ** // world
*
*/
blah /* hi */ */ b
//
///hi
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- request
GET /a.html
--- response_body eval
" i don't know [// hello // world /* */]
hello world [/** abc * b/c /*
hello ** // world
*
*/]
blah [/* hi */] */ b
[//]
[///hi]
"
--- no_error_log
[alert]
[error]
=== TEST 53: remove C/C++ comments (all at a time) - server-level config
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
replace_filter '/\*.*?\*/|//[^\n]*' '[$&]' g;
--- user_files
>>> a.html
i don't know // hello // world /* */
hello world /** abc * b/c /*
hello ** // world
*
*/
blah /* hi */ */ b
//
///hi
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- request
GET /a.html
--- response_body eval
" i don't know [// hello // world /* */]
hello world [/** abc * b/c /*
hello ** // world
*
*/]
blah [/* hi */] */ b
[//]
[///hi]
"
--- no_error_log
[alert]
[error]
=== TEST 54: multiple replace_filter_types settings (server level)
--- config
replace_filter_max_buffered_size 0;
default_type text/plain;
replace_filter_types text/css text/plain;
location /t {
echo abc;
replace_filter b [$&];
}
--- request
GET /t
--- response_body
a[b]c
--- no_error_log
[alert]
[error]
=== TEST 55: multiple replace_filter_types settings (server level, but overridding in location)
--- config
replace_filter_max_buffered_size 0;
default_type text/plain;
replace_filter_types text/css text/plain;
location /t {
echo abc;
replace_filter_types text/javascript;
replace_filter b [$&];
}
--- request
GET /t
--- response_body
abc
--- no_error_log
[alert]
[error]
=== TEST 56: github issue #3: data lost in particular situation
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location /t {
default_type text/html;
echo "ABCabcABC";
echo "ABCabcABC";
#echo "ABCabcABC\nABCabcABC";
replace_filter "(a.+?c){2}" "[$&]" "ig";
}
--- request
GET /t
--- response_body
[ABCabc][ABC
ABCabc]ABC
--- no_error_log
[alert]
[error]
=== TEST 57: variation
--- config
replace_filter_max_buffered_size 5;
default_type text/html;
location /t {
default_type text/html;
#echo "ABCabcABC";
#echo "ABCabcABC";
echo "ACacAC ACacAC";
replace_filter "(a.+?c){2}" "[$&]" "ig";
}
--- request
GET /t
--- response_body
[ACacAC AC]acAC
--- no_error_log
[alert]
[error]
=== TEST 58: nested pending matched
--- config
replace_filter_max_buffered_size 9;
default_type text/html;
location /t {
default_type text/html;
echo -n a;
echo -n b;
echo -n c;
echo -n def;
echo -n gh;
echo -n i;
echo k;
#echo abcdefig;
replace_filter "abcdefghij|bcdefg|cd" "[$&]" "ig";
}
--- request
GET /t
--- response_body
a[bcdefg]hik
--- no_error_log
[alert]
[error]
=== TEST 59: test split chain with b_sane=1, next=NULL
--- config
replace_filter_max_buffered_size 4;
default_type text/html;
location = /t {
echo -n aba;
echo -n ba;
echo -n bac;
echo d;
#echo abababacd;
replace_filter abacd [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") {
print_ubacktrace()
}
--- response_body
abab[abacd]
--- no_error_log
[alert]
[error]
=== TEST 60: test split chain with b_sane=1, next not NULL
--- config
replace_filter_max_buffered_size 6;
default_type text/html;
location = /t {
echo -n aba;
echo -n ba;
echo -n ba;
echo -n bac;
echo d;
#echo abababacd;
replace_filter ababacd [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") {
print_ubacktrace()
}
--- response_body
abab[ababacd]
--- no_error_log
[alert]
[error]
=== TEST 61: trim leading spaces (1 byte at a time)
--- config
replace_filter_max_buffered_size 6;
default_type text/html;
location /a.html {
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '^\s+' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- request
GET /t
--- response_body
[ ]hello, world
blah yeah
hello
[ ]baby!
[
]abc
--- no_error_log
[alert]
[error]
=== TEST 62: split ctx->pending into ctx->pending and ctx->free
--- config
replace_filter_max_buffered_size 3;
default_type text/html;
location = /t {
#echo "abc\nd";
echo -n a;
echo -n b;
echo -n c;
echo -n "\n";
echo d;
replace_filter "abcd|bc\ne|c$" [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1482") {
print_ubacktrace()
}
--- response_body
ab[c]
d
--- no_error_log
[alert]
[error]
=== TEST 63: trim both leading and trailing spaces (1 byte at a time)
--- config
default_type text/html;
replace_filter_max_buffered_size 5;
location /t {
echo -n 'a';
echo_sleep 0.001;
echo ' ';
echo_sleep 0.001;
echo '';
echo_sleep 0.001;
echo ' ';
echo_sleep 0.001;
echo "b";
echo_sleep 0.001;
echo " ";
replace_filter '^\s+|\s+$' '[$&]' g;
}
location = /main {
echo_location_async /t1;
echo_location_async /t2;
echo_location_async /t3;
echo_location_async /t4;
echo_location_async /t5;
echo_location_async /t6;
}
--- stap3 eval: $::StapOutputChains
--- request
GET /main
--- response_body chop
a[
]
b
[
]a[
]
b
[
]a[
]
b
[
]a[
]
b
[
]a[
]
b
[
]a[
]
b
[
]
--- no_error_log
[alert]
[error]
=== TEST 64: global substitution + nginx vars
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo bbcd;
replace_filter ([bc])|(d) [$1-$2-$foo] g;
}
--- request
GET /t
--- response_body
[b--X][b--X][c--X][-d-X]
--- no_error_log
[alert]
[error]
=== TEST 65: global substitution + nginx vars (splitted bufs)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo -n b;
echo -n b;
echo -n c;
echo d;
replace_filter ([bc])|(d) [$1-$2-$foo] g;
}
--- request
GET /t
--- response_body
[b--X][b--X][c--X][-d-X]
--- no_error_log
[alert]
[error]
=== TEST 66: global substitution + nginx vars (out of captures)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
set $foo X;
echo -n b;
echo -n b;
echo -n c;
echo d;
replace_filter [bc]|d [$1-$2-$foo] g;
}
--- request
GET /t
--- stap
F(ngx_http_replace_complex_value) {
println("complex value")
}
--- stap_out
complex value
complex value
complex value
complex value
--- response_body
[--X][--X][--X][--X]
--- no_error_log
[alert]
[error]
================================================
FILE: t/05-capturing-max-buffered.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
#no_shuffle();
plan tests => repeat_each() * (blocks() * 4);
our $StapOutputChains = <<'_EOC_';
global active
F(ngx_http_handler) {
active = 1
}
/*
F(ngx_http_write_filter) {
if (active && pid() == target()) {
printf("http writer filter: %s\n", ngx_chain_dump($in))
}
}
*/
F(ngx_http_chunked_body_filter) {
if (active && pid() == target()) {
printf("http chunked filter: %s\n", ngx_chain_dump($in))
}
}
F(ngx_http_replace_output) {
if (active && pid() == target()) {
printf("http replace output: %s\n", ngx_chain_dump($ctx->out))
}
}
probe syscall.writev {
if (active && pid() == target()) {
printf("writev(%s)", ngx_iovec_dump($vec, $vlen))
/*
for (i = 0; i < $vlen; i++) {
printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len)))
}
*/
}
}
probe syscall.writev.return {
if (active && pid() == target()) {
printf(" = %s\n", retstr)
}
}
_EOC_
#no_diff();
no_long_string();
run_tests();
__DATA__
=== TEST 1: 1-byte chain bufs (0)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
}
--- response_body
ababacd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (0): 1
--- no_error_log
[error]
=== TEST 2: 1-byte chain bufs (1)
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1439") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- response_body
ababacd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (1): 2
--- no_error_log
[error]
=== TEST 3: 1-byte chain bufs (2)
--- config
default_type text/html;
replace_filter_max_buffered_size 2;
location = /t {
echo -n a;
echo -n b;
echo -n a;
echo -n b;
echo -n a;
echo -n c;
echo d;
replace_filter abac [$&];
}
--- request
GET /t
--- stap2 eval: $::StapOutputChains
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1439") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- response_body
ababacd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (2): 3
--- no_error_log
[error]
=== TEST 4: trim both leading and trailing spaces (1 byte at a time) (6)
--- config
replace_filter_max_buffered_size 6;
default_type text/html;
location /a.html {
internal;
}
location = /t {
content_by_lua '
local res = ngx.location.capture("/a.html")
local txt = res.body
for i = 1, string.len(txt) do
ngx.print(string.sub(txt, i, i))
ngx.flush(true)
end
';
replace_filter '^\s+|\s+$' '[$&]' g;
}
--- user_files
>>> a.html
hello, world
blah yeah
hello
baby!
abc
--- stap2
F(ngx_palloc) {
if ($size < 0) {
print_ubacktrace()
exit()
}
}
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") {
//printf("chain: %s", ngx_chain_dump($ctx->busy))
print_ubacktrace()
exit()
}
--- request
GET /t
--- response_body
[ ]hello, world[ ]
blah yeah
hello[ ]
[ ]baby!
abc
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (6): 7
--- no_error_log
[error]
=== TEST 5: github issue #2: error "general look-ahead not supported"
--- config
replace_filter_max_buffered_size 0;
location /t {
charset utf-8;
default_type text/html;
echo "ABCabcABCabc";
#replace_filter_types text/plain;
replace_filter "a.+a" "[$&]" "ig";
}
--- request
GET /t
--- stap3
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1481") {
print_ubacktrace()
}
--- response_body
ABCabcABCabc
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (0): 12
--- no_error_log
[error]
=== TEST 6: backtrack to the middle of a pending capture (pending: output|capture + rematch) (0)
--- config
replace_filter_max_buffered_size 0;
default_type text/html;
location = /t {
echo -n ab;
echo -n c;
echo d;
replace_filter 'abce|b' '[$&]' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
abcd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (0): 2
--- no_error_log
[error]
=== TEST 7: backtrack to the middle of a pending capture (pending: output|capture + rematch) (1)
--- config
replace_filter_max_buffered_size 1;
default_type text/html;
location = /t {
echo -n ab;
echo -n c;
echo d;
replace_filter 'abce|b' '[$&]' g;
}
--- stap2
probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") {
print_ubacktrace()
}
--- stap3 eval: $::StapOutputChains
--- request
GET /t
--- response_body
abcd
--- error_log
replace filter: exceeding replace_filter_max_buffered_size (1): 2
--- no_error_log
[error]
================================================
FILE: t/06-if.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
#no_shuffle();
plan tests => repeat_each() * (blocks() * 4);
#no_diff();
#no_long_string();
run_tests();
__DATA__
=== TEST 1: local if hit
--- config
location /t {
default_type text/plain;
echo abcabcabde;
if ($arg_disable = "") {
replace_filter_types text/plain;
replace_filter_max_buffered_size 0;
replace_filter abcabd X;
}
}
--- request
GET /t
--- response_body
abcXe
--- no_error_log
[alert]
[error]
=== TEST 2: local if miss
--- config
replace_filter_max_buffered_size 0;
location /t {
default_type text/plain;
echo abcabcabde;
if ($arg_disable = "") {
replace_filter_types text/plain;
replace_filter_max_buffered_size 0;
replace_filter abcabd X;
}
}
--- request
GET /t?disable=1
--- response_body
abcabcabde
--- no_error_log
[alert]
[error]
================================================
FILE: t/07-multi.t
================================================
# vim:set ft= ts=4 sw=4 et fdm=marker:
use lib 'lib';
use Test::Nginx::Socket;
#worker_connections(1014);
#master_on();
#workers(2);
#log_level('warn');
repeat_each(2);
no_shuffle();
plan tests => repeat_each() * (blocks() * 4 + 5);
our $StapOutputChains = <<'_EOC_';
global active
F(ngx_http_handler) {
active = 1
}
/*
F(ngx_http_write_filter) {
if (active && pid() == target()) {
printf("http writer filter: %s\n", ngx_chain_dump($in))
}
}
*/
F(ngx_http_chunked_body_filter) {
if (active && pid() == target()) {
printf("http chunked filter: %s\n", ngx_chain_dump($in))
}
}
F(ngx_http_replace_output) {
if (active && pid() == target()) {
printf("http replace output: %s\n", ngx_chain_dump($ctx->out))
}
}
probe syscall.writev {
if (active && pid() == target()) {
printf("writev(%s)", ngx_iovec_dump($vec, $vlen))
/*
for (i = 0; i < $vlen; i++) {
printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len)))
}
*/
}
}
probe syscall.writev.return {
if (active && pid() == target()) {
printf(" = %s\n", retstr)
}
}
_EOC_
#no_diff();
#no_long_string();
run_tests();
__DATA__
=== TEST 1: once patterns
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'hello world world hello';
replace_filter world "<$&>";
replace_filter hello "[$&]";
}
--- request
GET /t
--- response_body
[hello] <world> world hello
--- stap
F(ngx_http_replace_non_capturing_parse) {
println("non capturing parse")
}
F(ngx_http_replace_capturing_parse) {
println("capturing parse")
}
--- stap_out_like chop
^(capturing parse\n)+$
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 2: once patterns
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello world Hello world';
replace_filter world "<$&>";
replace_filter hello "[$&]";
}
--- request
GET /t
--- response_body
Hello <world> Hello world
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 3: case-insensitive patterns
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello world WORLD HELLO';
replace_filter world "<$&>";
replace_filter hello "[$&]" i;
}
--- request
GET /t
--- response_body
[Hello] <world> WORLD HELLO
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 4: global subs
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'hello world world hello';
replace_filter world "<$&>" g;
replace_filter hello "[$&]" g;
}
--- request
GET /t
--- response_body
[hello] <world> <world> [hello]
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 5: global subs (case sensitive)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello World worlD hellO';
replace_filter world "<$&>" g;
replace_filter hello "[$&]" g;
}
--- request
GET /t
--- response_body
Hello World worlD hellO
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 6: global subs (case insensitive)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello World worlD hellO';
replace_filter world "<$&>" ig;
replace_filter hello "[$&]" g;
}
--- request
GET /t
--- response_body
Hello <World> <worlD> hellO
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 7: global subs (case insensitive) (2)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello World worlD hellO';
replace_filter world "<$&>" g;
replace_filter hello "[$&]" ig;
}
--- request
GET /t
--- response_body
[Hello] World worlD [hellO]
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 8: global subs (case insensitive) (3)
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello World worlD hellO';
replace_filter world "<$&>" gi;
replace_filter hello "[$&]" ig;
}
--- request
GET /t
--- response_body
[Hello] <World> <worlD> [hellO]
--- stap
F(ngx_http_replace_non_capturing_parse) {
println("non capturing parse")
}
F(ngx_http_replace_capturing_parse) {
println("capturing parse")
}
--- stap_out_like chop
^(capturing parse\n)+$
--- no_error_log
[alert]
[error]
=== TEST 9: global subs (case insensitive) - non-capturing
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo 'Hello World';
replace_filter world "<>" gi;
replace_filter hello "[]" ig;
}
--- request
GET /t
--- response_body
[] <>
--- stap
F(ngx_http_replace_non_capturing_parse) {
println("non capturing parse")
}
F(ngx_http_replace_capturing_parse) {
println("capturing parse")
}
--- stap_out_like chop
^(non capturing parse\n)+$
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 10: working as a tokenizer
--- config
default_type text/html;
replace_filter_max_buffered_size 0;
location /t {
echo -n a;
echo b;
replace_filter a "[$&]" g;
replace_filter ab "<$&>" g;
}
--- request
GET /t
--- response_body
[a]b
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 11: working as a tokenizer (2)
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
location /t {
echo -n a;
echo b;
replace_filter ab "<$&>" g;
replace_filter a "[$&]" g;
}
--- request
GET /t
--- response_body
<ab>
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 12: on server level
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
replace_filter ab "<$&>" g;
replace_filter a "[$&]" g;
location /t {
echo -n a;
echo b;
}
--- request
GET /t
--- response_body
<ab>
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 13: mixing once and global patterns
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
location /t {
echo hello world hiya hiya world hello;
replace_filter hello "<$&>";
replace_filter hiya "{$&}";
replace_filter world "[$&]" g;
}
--- request
GET /t
--- response_body
<hello> [world] {hiya} hiya [world] hello
--- stap2 eval: $::StapOutputChains
--- no_error_log
[alert]
[error]
=== TEST 14: all once
--- config
default_type text/html;
replace_filter_max_buffered_size 1;
location /t {
echo hello world hiya hiya world hello;
replace_filter hello "<$&>";
replace_filter hiya "{$&}";
replace_filter world "[$&]";
}
--- request
GET /t
--- response_body
<hello> [world] {hiya} hiya worl
gitextract_mewcfnqx/ ├── .gitattributes ├── .gitignore ├── .travis.yml ├── README.markdown ├── config ├── src/ │ ├── ddebug.h │ ├── ngx_http_replace_filter_module.c │ ├── ngx_http_replace_filter_module.h │ ├── ngx_http_replace_parse.c │ ├── ngx_http_replace_parse.h │ ├── ngx_http_replace_script.c │ ├── ngx_http_replace_script.h │ ├── ngx_http_replace_util.c │ └── ngx_http_replace_util.h ├── t/ │ ├── 01-sanity.t │ ├── 02-max-buffered.t │ ├── 03-var.t │ ├── 04-capturing.t │ ├── 05-capturing-max-buffered.t │ ├── 06-if.t │ ├── 07-multi.t │ ├── 08-gzip.t │ ├── 09-unused.t │ ├── 10-last-modified.t │ └── 11-skip.t ├── util/ │ └── build.sh └── valgrind.suppress
SYMBOL INDEX (42 symbols across 7 files)
FILE: src/ddebug.h
function dd (line 25) | static void dd(const char * fmt, ...) {
function dd_enter_helper (line 34) | static void dd_enter_helper(ngx_http_request_t *r, const char *func) {
function dd (line 74) | static void dd(const char * fmt, ...) {
function dd_enter (line 77) | static void dd_enter() {
FILE: src/ngx_http_replace_filter_module.c
function ngx_int_t (line 141) | static ngx_int_t
function ngx_http_replace_cleanup_pool (line 242) | static void
function ngx_int_t (line 254) | static ngx_int_t
function ngx_int_t (line 568) | static ngx_int_t
function ngx_int_t (line 975) | static ngx_int_t
FILE: src/ngx_http_replace_filter_module.h
type ngx_http_replace_ctx_t (line 15) | typedef struct {
type ngx_int_t (line 60) | typedef ngx_int_t (*ngx_http_replace_parse_buf_pt)(ngx_http_request_t *r,
type ngx_http_replace_main_conf_t (line 64) | typedef struct {
type ngx_http_replace_loc_conf_t (line 69) | typedef struct {
FILE: src/ngx_http_replace_parse.c
function ngx_int_t (line 23) | ngx_int_t
function ngx_int_t (line 377) | ngx_int_t
function ngx_http_replace_check_total_buffered (line 921) | static void
FILE: src/ngx_http_replace_script.c
function ngx_int_t (line 47) | ngx_int_t
function ngx_int_t (line 113) | ngx_int_t
function ngx_int_t (line 161) | static ngx_int_t
function ngx_int_t (line 333) | static ngx_int_t
function ngx_http_replace_script_copy_len_code (line 371) | static size_t
function ngx_http_replace_script_copy_code (line 384) | static size_t
function ngx_int_t (line 411) | static ngx_int_t
function ngx_http_replace_script_copy_capture_len_code (line 441) | static size_t
function ngx_http_replace_script_copy_capture_code (line 467) | static size_t
function ngx_int_t (line 526) | static ngx_int_t
function ngx_int_t (line 568) | static ngx_int_t
function ngx_int_t (line 599) | static ngx_int_t
function ngx_http_replace_script_copy_var_len_code (line 637) | static size_t
function ngx_http_replace_script_copy_var_code (line 657) | static size_t
function ngx_http_replace_count_variables (line 686) | static void
FILE: src/ngx_http_replace_script.h
type ngx_http_replace_script_compile_t (line 18) | typedef struct {
type ngx_http_replace_complex_value_t (line 31) | typedef struct {
type ngx_http_replace_compile_complex_value_t (line 39) | typedef struct {
type ngx_http_replace_script_engine_t (line 47) | typedef struct {
type ngx_http_replace_script_copy_code_t (line 70) | typedef struct {
type ngx_http_replace_script_capture_code_t (line 76) | typedef struct {
type ngx_http_replace_script_var_code_t (line 82) | typedef struct {
FILE: src/ngx_http_replace_util.c
function ngx_chain_t (line 16) | ngx_chain_t *
function ngx_int_t (line 34) | ngx_int_t
function ngx_int_t (line 129) | ngx_int_t
function ngx_http_replace_dump_chain (line 194) | void
Condensed preview — 27 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (224K chars).
[
{
"path": ".gitattributes",
"chars": 27,
"preview": "*.t linguist-language=Text\n"
},
{
"path": ".gitignore",
"chars": 339,
"preview": "*.mobi\ngenmobi.sh\n.libs\n*.swp\n*.slo\n*.la\n*.swo\n*.lo\n*~\n*.o\nprint.txt\n.rsync\n*.tar.gz\ndist\nbuild[789]\nbuild\nbuild10\ntags\n"
},
{
"path": ".travis.yml",
"chars": 1940,
"preview": "sudo: required\ndist: focal\n\nos: linux\n\nlanguage: c\n\ncompiler:\n - gcc\n\naddons:\n apt:\n packages:\n - axel\n -"
},
{
"path": "README.markdown",
"chars": 12771,
"preview": "Name\n====\n\nngx_replace_filter - Streaming regular expression replacement in response bodies.\n\n*This module is not distri"
},
{
"path": "config",
"chars": 3499,
"preview": "ngx_feature=\"agentzh's sregex library\"\nngx_feature_libs=\"-lsregex\"\nngx_feature_name=\nngx_feature_run=no\nngx_feature_incs"
},
{
"path": "src/ddebug.h",
"chars": 2458,
"preview": "#ifndef DDEBUG_H\n#define DDEBUG_H\n\n\n#include <ngx_core.h>\n#include <ngx_http.h>\n#include <assert.h>\n\n\n#if defined(DDEBUG"
},
{
"path": "src/ngx_http_replace_filter_module.c",
"chars": 28908,
"preview": "\n/*\n * Copyright (C) Yichun Zhang (agentzh)\n * Copyright (C) Igor Sysoev\n * Copyright (C) Nginx, Inc.\n */\n\n\n#ifndef DDEB"
},
{
"path": "src/ngx_http_replace_filter_module.h",
"chars": 3062,
"preview": "#ifndef _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_\n#define _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_\n\n\n#include \"ngx_"
},
{
"path": "src/ngx_http_replace_parse.c",
"chars": 28382,
"preview": "\n/*\n * Copyright (C) Yichun Zhang (agentzh)\n * Copyright (C) Igor Sysoev\n * Copyright (C) Nginx, Inc.\n */\n\n\n#ifndef DDEB"
},
{
"path": "src/ngx_http_replace_parse.h",
"chars": 435,
"preview": "#ifndef _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_\n#define _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_\n\n\n#include \"ngx_http_replace_fil"
},
{
"path": "src/ngx_http_replace_script.c",
"chars": 18901,
"preview": "\n/*\n * Copyright (C) Yichun Zhang (agentzh)\n */\n\n\n#ifndef DDEBUG\n#define DDEBUG 0\n#endif\n#include \"ddebug.h\"\n\n\n#include "
},
{
"path": "src/ngx_http_replace_script.h",
"chars": 2502,
"preview": "\n/*\n * Copyright (C) Yichun Zhang (agentzh)\n */\n\n\n#ifndef _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_\n#define _NGX_HTTP_REPLACE"
},
{
"path": "src/ngx_http_replace_util.c",
"chars": 5334,
"preview": "\n/*\n * Copyright (C) Yichun Zhang (agentzh)\n */\n\n\n#ifndef DDEBUG\n#define DDEBUG 0\n#endif\n#include \"ddebug.h\"\n\n\n#include "
},
{
"path": "src/ngx_http_replace_util.h",
"chars": 755,
"preview": "#ifndef _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_\n#define _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_\n\n\n#include \"ngx_http_replace_filte"
},
{
"path": "t/01-sanity.t",
"chars": 30922,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/02-max-buffered.t",
"chars": 5443,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/03-var.t",
"chars": 4772,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/04-capturing.t",
"chars": 32383,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/05-capturing-max-buffered.t",
"chars": 6149,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/06-if.t",
"chars": 1087,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/07-multi.t",
"chars": 10735,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/08-gzip.t",
"chars": 682,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/09-unused.t",
"chars": 1855,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/10-last-modified.t",
"chars": 1594,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "t/11-skip.t",
"chars": 2880,
"preview": "# vim:set ft= ts=4 sw=4 et fdm=marker:\n\nuse lib 'lib';\nuse Test::Nginx::Socket;\n\n#worker_connections(1014);\n#master_on()"
},
{
"path": "util/build.sh",
"chars": 1167,
"preview": "#!/bin/bash\n\n# this file is mostly meant to be used by the author himself.\n\nroot=`pwd`\nhome=~\nversion=$1\nforce=$2\n\nngx-b"
},
{
"path": "valgrind.suppress",
"chars": 2905,
"preview": "{\n <insert_a_suppression_name_here>\n Memcheck:Addr1\n fun:ngx_init_cycle\n fun:ngx_master_process_cycle\n fun:mai"
}
]
About this extraction
This page contains the full source code of the openresty/replace-filter-nginx-module GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 27 files (206.9 KB), approximately 56.6k tokens, and a symbol index with 42 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.