Repository: openresty/replace-filter-nginx-module Branch: master Commit: 0b432c649d2d Files: 27 Total size: 206.9 KB Directory structure: gitextract_mewcfnqx/ ├── .gitattributes ├── .gitignore ├── .travis.yml ├── README.markdown ├── config ├── src/ │ ├── ddebug.h │ ├── ngx_http_replace_filter_module.c │ ├── ngx_http_replace_filter_module.h │ ├── ngx_http_replace_parse.c │ ├── ngx_http_replace_parse.h │ ├── ngx_http_replace_script.c │ ├── ngx_http_replace_script.h │ ├── ngx_http_replace_util.c │ └── ngx_http_replace_util.h ├── t/ │ ├── 01-sanity.t │ ├── 02-max-buffered.t │ ├── 03-var.t │ ├── 04-capturing.t │ ├── 05-capturing-max-buffered.t │ ├── 06-if.t │ ├── 07-multi.t │ ├── 08-gzip.t │ ├── 09-unused.t │ ├── 10-last-modified.t │ └── 11-skip.t ├── util/ │ └── build.sh └── valgrind.suppress ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitattributes ================================================ *.t linguist-language=Text ================================================ FILE: .gitignore ================================================ *.mobi genmobi.sh .libs *.swp *.slo *.la *.swo *.lo *~ *.o print.txt .rsync *.tar.gz dist build[789] build build10 tags update-readme *.tmp go t/t.sh releng reset *.t_ ctags src/stream.h nginx keepalive reindex src/module.[ch] all t/servroot/ buildroot/ build1[0-9] re *.plist Makefile src/script.[ch] src/parse.[ch] src/util.[ch] *.patch ================================================ FILE: .travis.yml ================================================ sudo: required dist: focal os: linux language: c compiler: - gcc addons: apt: packages: - axel - cpanminus - libgd-dev - libpcre3-dev cache: apt: true env: global: - JOBS=3 - NGX_BUILD_JOBS=$JOBS - LUAJIT_PREFIX=/opt/luajit21 - LUAJIT_LIB=$LUAJIT_PREFIX/lib - LUAJIT_INC=$LUAJIT_PREFIX/include/luajit-2.1 - LUA_INCLUDE_DIR=$LUAJIT_INC - LD_LIBRARY_PATH=$LUAJIT_LIB:$LD_LIBRARY_PATH jobs: - NGINX_VERSION=1.21.4 - NGINX_VERSION=1.25.1 NGX_EXTRA_OPT=--without-pcre2 before_install: - sudo cpanm --notest Test::Nginx > build.log 2>&1 || (cat build.log && exit 1) install: - git clone https://github.com/openresty/openresty.git ../openresty - git clone https://github.com/openresty/no-pool-nginx.git ../no-pool-nginx - git clone https://github.com/openresty/openresty-devel-utils.git - git clone https://github.com/openresty/lua-nginx-module.git ../lua-nginx-module - git clone https://github.com/openresty/echo-nginx-module.git ../echo-nginx-module - git clone https://github.com/simpl/ngx_devel_kit.git ../ndk-nginx-module - git clone https://github.com/openresty/lua-resty-core.git ../lua-resty-core - git clone https://github.com/openresty/lua-resty-lrucache.git ../lua-resty-lrucache - git clone -b v2.1-agentzh https://github.com/openresty/luajit2.git luajit2 - git clone https://github.com/openresty/sregex.git script: - cd sregex && sudo make PREFIX=/usr install && cd .. - cd luajit2/ - make -j$JOBS CCDEBUG=-g Q= PREFIX=$LUAJIT_PREFIX CC=$CC XCFLAGS='-DLUA_USE_APICHECK -DLUA_USE_ASSERT -msse4.2' > build.log 2>&1 || (cat build.log && exit 1) - sudo make install PREFIX=$LUAJIT_PREFIX > build.log 2>&1 || (cat build.log && exit 1) - cd .. - export PATH=$PWD/work/nginx/sbin:$PWD/openresty-devel-utils:$PATH - sh util/build.sh $NGINX_VERSION > build.log 2>&1 || (cat build.log && exit 1) - nginx -V - prove -r t ================================================ FILE: README.markdown ================================================ Name ==== ngx_replace_filter - Streaming regular expression replacement in response bodies. *This module is not distributed with the Nginx source.* See [the installation instructions](#installation). Table of Contents ================= * [Name](#name) * [Status](#status) * [Synopsis](#synopsis) * [Description](#description) * [Directives](#directives) * [replace_filter](#replace_filter) * [replace_filter_types](#replace_filter_types) * [replace_filter_max_buffered_size](#replace_filter_max_buffered_size) * [replace_filter_last_modified](#replace_filter_last_modified) * [replace_filter_skip](#replace_filter_skip) * [Installation](#installation) * [Trouble Shooting](#trouble-shooting) * [TODO](#todo) * [Community](#community) * [English Mailing List](#english-mailing-list) * [Chinese Mailing List](#chinese-mailing-list) * [Bugs and Patches](#bugs-and-patches) * [Author](#author) * [Copyright and License](#copyright-and-license) * [See Also](#see-also) Status ====== This module is already quite usable though still at the early phase of development and is considered experimental. Synopsis ======== ```nginx location /t { default_type text/html; echo abc; replace_filter 'ab|abc' X; } location / { # proxy_pass/fastcgi_pass/... # caseless global substitution: replace_filter '\d+' 'blah blah' 'ig'; replace_filter_types text/plain text/css; } location /a { # proxy_pass/fastcgi_pass/root/... # remove line-leading spaces and line-trailing spaces, # as well as blank lines: replace_filter '^\s+|\s+$' '' g; } location /b { # proxy_pass/fastcgi_pass/root/... # only remove line-leading spaces and line-trailing spaces: replace_filter '^[ \f\t]+|[ \f\t]+$' '' g; } location ~ '\.cpp$' { # proxy_pass/fastcgi_pass/root/... replace_filter_types text/plain; # skip C/C++ string literals: replace_filter "'(?:\\\\[^\n]|[^'\n])*'" $& g; replace_filter '"(?:\\\\[^\n]|[^"\n])*"' $& g; # remove all those ugly C/C++ comments: replace_filter '/\*.*?\*/|//[^\n]*' '' g; } ``` Description =========== This Nginx output filter module tries to do regular expression substitutions in a non-buffered manner wherever possible. This module does *not* use traditional backtracking regular expression engines like PCRE, rather, it uses the new [sregex](https://github.com/agentzh/sregex) library implemented by the author himself, which was designed with streaming processing in mind from the very beginning: A good common subset of Perl 5 regular expressions is supported by `sregex`. For the complete feature list, check out sregex's documentation: https://github.com/agentzh/sregex#syntax-supported Response body data is only buffered when absolutely necessary, like facing an incomplete capture that belongs to a possible match near the data chunk boundaries. [Back to TOC](#table-of-contents) Directives ========== [Back to TOC](#table-of-contents) replace_filter -------------- **syntax:** *replace_filter <regex> <replace>* **syntax:** *replace_filter <regex> <replace> <options>* **default:** *no* **context:** *http, server, location, location if* **phase:** *output body filter* Specifies the regex pattern and text to be replaced, with optional regex flags. By default, the filter stops matching after the first match is found. This behavior can be changed by specifying the `g` regex option. The following regex options are supported: * `g` for global search and substitution (default off) * `i` for case-insensitive matching (default off) Multiple options can be combined in a single string argument, for example: ```nginx replace_filter hello hiya ig; ``` Nginx variables can be interpolated into the text to be replaced, for example: ```nginx replace_filter \w+ "[$foo,$bar]"; ``` If you want to use the literal dollar sign character (`$`), use the `$$` sequence for that, for instance: ```nginx replace_filter \w "$$"; ``` Use of submatch capturing variables like `$&`, `$1`, `$2`, and etc are also supported, for example, ```nginx replace_filter [bc]|d [$&-$1-$2] g; ``` The semantics of the submatch capturing variables is exactly the same as in the Perl 5 language. Multiple `replace_filter` directives in the same scope is also supported. All the patterns will be applied at the same time as in a tokenizer. We will *not* use the longest token match semantics, but rather, patterns will be prioritized according to their order in the configure file. Here is an example for removing all the C/C++ comments from a C/C++ source code file: ```nginx replace_filter "'(?:\\\\[^\n]|[^'\n])*'" $& g; replace_filter '"(?:\\\\[^\n]|[^"\n])*"' $& g; replace_filter '/\*.*?\*/|//[^\n]*' '' g; ``` When the `Content-Encoding` response header is not empty (like `gzip`), the response body will always remain intact. So usually you want to disable the gzip compression in your backend servers' responses by adding the following line to your `nginx.conf` if you are the ngx_proxy module: ```nginx proxy_set_header Accept-Encoding ''; ``` Your responses can still be gzip compressed on the Nginx server level though. [Back to TOC](#table-of-contents) replace_filter_types -------------------- **syntax:** *replace_filter_types <mime-type> ...* **default:** *replace_filter_types text/html* **context:** *http, server, location, location if* **phase:** *output body filter* Specify one or more MIME types (in the `Content-Type` response header) to be processed. By default, only `text/html` typed responses are processed. [Back to TOC](#table-of-contents) replace_filter_max_buffered_size --------------------------------- **syntax:** *replace_filter_max_buffered_size <size>* **default:** *replace_filter_max_buffered_size 8k* **context:** *http, server, location, location if* **phase:** *output body filter* Limits the total size of the data buffered by the module at runtime. Default to `8k`. When the limit is reached, `replace_filter` will immediately stop processing and leave all the remaining response body data intact. [Back to TOC](#table-of-contents) replace_filter_last_modified ---------------------------- **syntax:** *replace_filter_last_modifiled keep | clear* **default:** *replace_filter_last_modified clear* **context:** *http, server, location, location if* **phase:** *output body filter* Controls how to deal with the existing Last-Modified response header. By default, this module will clear the `Last-Modified` response header if there is any. You can specify ```nginx replace_filter_last_modified keep; ``` to always keep the original `Last-Modified` response header. [Back to TOC](#table-of-contents) replace_filter_skip ------------------- **syntax:** *replace_filter_skip $var* **default:** *no* **context:** *http, server, location, location if* **phase:** *output header filter* This directive controls whether to skip all the `replace_filter` rules on a per-request basis. Both constant values or strings containing NGINX variables are supported. When the value is evaluated to an empty value ("") or the value "0" in the request output header phase, no `replace_filter` rules will be skipped for the current request. Otherwise all the `replace_filter` rules will be skipped for the current request. Below is a trivial example for this: ```nginx set $skip ''; location /t { content_by_lua ' ngx.var.skip = 1 ngx.say("abcabd") '; replace_filter_skip $skip; replace_filter abcabd X; } ``` [Back to TOC](#table-of-contents) Installation ============ You need to install the sregex library first: https://github.com/agentzh/sregex And then rebuild your Nginx like this: ```bash ./configure --add-module=/path/to/replace-filter-nginx-module ``` If sregex is not installed to the default prefix (i.e., `/usr/local`), then you should specify the locations of your sregex installation via the `SREGEX_INC` and `SREGEX_LIB` environments before running the `./configure` script, as in ```bash export SREGEX_INC=/opt/sregex/include export SREGEX_LIB=/opt/sregex/lib ``` assuming that your sregex is installed to the prefix `/opt/sregex`. Starting from NGINX 1.9.11, you can also compile this module as a dynamic module, by using the `--add-dynamic-module=PATH` option instead of `--add-module=PATH` on the `./configure` command line above. And then you can explicitly load the module in your `nginx.conf` via the [load_module](http://nginx.org/en/docs/ngx_core_module.html#load_module) directive, for example, ```nginx load_module /path/to/modules/ngx_http_replace_filter_module.so; ``` [Back to TOC](#table-of-contents) Trouble Shooting ================ * If you are seeing the error "error while loading shared libraries: libsregex.so.0: cannot open shared object file: No such file or directory" while starting nginx, then it means that the installation path of your libsregex library is not in your system's default library search path. You can solve this issue by passing the option `--with-ld-opt='-Wl,-rpath,/usr/local/lib'` to nginx's `./configure` command. Alternatively, you can just add the path of your libsregex.so.0 to the `LD_LIBRARY_PATH` environment value before starting your nginx server. [Back to TOC](#table-of-contents) TODO ==== * optimize the special case for verbatim substitutions, i.e., `replace_filter $&;`. * implement the `replace_filter_skip $var` directive to control whether to enable the filter on the fly. * reduce the amount of data that has to be buffered for when an partial match is already found. * recycle the memory blocks used to buffer the pending capture data and "complex values" for replacement. * allow use of inlined Lua code as the `replacement` argument of the `replace_filter` directive to generate the text to be replaced on-the-fly. [Back to TOC](#table-of-contents) Community ========= [Back to TOC](#table-of-contents) English Mailing List -------------------- The [openresty-en](https://groups.google.com/group/openresty-en) mailing list is for English speakers. [Back to TOC](#table-of-contents) Chinese Mailing List -------------------- The [openresty](https://groups.google.com/group/openresty) mailing list is for Chinese speakers. [Back to TOC](#table-of-contents) Bugs and Patches ================ Please submit bug reports, wishlists, or patches by 1. creating a ticket on the [GitHub Issue Tracker](http://github.com/agentzh/replace-filter-nginx-module/issues), 1. or posting to the [OpenResty community](http://wiki.nginx.org/HttpLuaModule#Community). [Back to TOC](#table-of-contents) Author ====== Yichun "agentzh" Zhang (章亦春) , OpenResty Inc. [Back to TOC](#table-of-contents) Copyright and License ===================== This module is licensed under the BSD license. Copyright (C) 2012-2017, by Yichun "agentzh" Zhang (章亦春), OpenResty Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. [Back to TOC](#table-of-contents) See Also ======== * agentzh's sregex library: https://github.com/agentzh/sregex * The standard ngx_sub_filter module: http://nginx.org/en/docs/http/ngx_http_sub_module.html * Slides for my talk "sregex: matching Perl 5 regexes on data streams": http://agentzh.org/misc/slides/yapc-na-2013-sregex.pdf [Back to TOC](#table-of-contents) ================================================ FILE: config ================================================ ngx_feature="agentzh's sregex library" ngx_feature_libs="-lsregex" ngx_feature_name= ngx_feature_run=no ngx_feature_incs="#include " ngx_feature_path= ngx_feature_test="sre_regex_t *re; sre_program_t *prog; sre_pool_t *pool; u_char s[] = {'a', 'b', 'c'}; sre_int_t rc, err_offset, ovector; sre_int_t *pending_matched; sre_uint_t ncaps; sre_vm_pike_ctx_t *pctx; pool = sre_create_pool(1024); re = sre_regex_parse(pool, s, &ncaps, 0, &err_offset); prog = sre_regex_compile(pool, re); pctx = sre_vm_pike_create_ctx(pool, prog, &ovector, 0); rc = sre_vm_pike_exec(pctx, s, 32, 1, &pending_matched); sre_destroy_pool(pool)" if [ -n "$SREGEX_INC" -o -n "$SREGEX_LIB" ]; then # explicitly set sregex lib path ngx_feature="agentzh's sregex library in $SREGEX_LIB and $SREGEX_INC (specified by the SREGEX_LIB and SREGEX_INC env)" ngx_feature_path="$SREGEX_INC" if [ $NGX_RPATH = YES ]; then ngx_feature_libs="-R$SREGEX_LIB -L$SREGEX_LIB -lsregex" else ngx_feature_libs="-L$SREGEX_LIB -lsregex" fi . auto/feature if [ $ngx_found = no ]; then cat << END $0: error: ngx_http_replace_filter_module requires agentzh's sregex library and SREGEX_LIB is defined as $SREGEX_LIB and SREGEX_INC (path for sregex/sregex.h) $SREGEX_INC, but we cannot find sregex there. END exit 1 fi else # auto-discovery ngx_feature="agentzh's sregex library" ngx_feature_libs="-lsregex" . auto/feature if [ $ngx_found = no ]; then # default installation prefix in sregex ngx_feature="agentzh's sregex library in /usr/local/" ngx_feature_path="/usr/local/include" if [ $NGX_RPATH = YES ]; then ngx_feature_libs="-R/usr/local/lib -L/usr/local/lib -lsregex" else ngx_feature_libs="-L/usr/local/lib -lsregex" fi . auto/feature fi fi if [ $ngx_found = no ]; then cat << END $0: error: ngx_http_replace_filter_module requires agentzh's sregex library. END exit 1 fi REPLACE_FILTER_SRCS="$ngx_addon_dir/src/ngx_http_replace_filter_module.c \ $ngx_addon_dir/src/ngx_http_replace_script.c \ $ngx_addon_dir/src/ngx_http_replace_parse.c \ $ngx_addon_dir/src/ngx_http_replace_util.c" REPLACE_FILTER_DEPS="$ngx_addon_dir/src/ngx_http_replace_filter_module.h \ $ngx_addon_dir/src/ngx_http_replace_script.h \ $ngx_addon_dir/src/ngx_http_replace_parse.h \ $ngx_addon_dir/src/ngx_http_replace_util.h" ngx_addon_name=ngx_http_replace_filter_module if test -n "$ngx_module_link"; then ngx_module_type=HTTP_AUX_FILTER ngx_module_name=$ngx_addon_name ngx_module_srcs="$REPLACE_FILTER_SRCS" ngx_module_deps="$REPLACE_FILTER_DEPS" ngx_module_incs=$ngx_feature_path ngx_module_libs=$ngx_feature_libs . auto/module else HTTP_AUX_FILTER_MODULES="$HTTP_AUX_FILTER_MODULES $ngx_addon_name" NGX_ADDON_SRCS="$NGX_ADDON_SRCS $REPLACE_FILTER_SRCS" NGX_ADDON_DEPS="$NGX_ADDON_DEPS $REPLACE_FILTER_DEPS" CORE_INCS="$CORE_INCS $ngx_feature_path" CORE_LIBS="$CORE_LIBS $ngx_feature_libs" fi ================================================ FILE: src/ddebug.h ================================================ #ifndef DDEBUG_H #define DDEBUG_H #include #include #include #if defined(DDEBUG) && (DDEBUG) # if (NGX_HAVE_VARIADIC_MACROS) # define dd(...) fprintf(stderr, "replace_filter *** %s: ", __func__); \ fprintf(stderr, __VA_ARGS__); \ fprintf(stderr, " at %s line %d.\n", __FILE__, __LINE__) # else #include #include #include static void dd(const char * fmt, ...) { } # endif # if DDEBUG > 1 # define dd_enter() dd_enter_helper(r, __func__) static void dd_enter_helper(ngx_http_request_t *r, const char *func) { ngx_http_posted_request_t *pr; fprintf(stderr, ">enter %s %.*s %.*s?%.*s c:%d m:%p r:%p ar:%p pr:%p", func, (int) r->method_name.len, r->method_name.data, (int) r->uri.len, r->uri.data, (int) r->args.len, r->args.data, 0/*(int) r->main->count*/, r->main, r, r->connection->data, r->parent); if (r->posted_requests) { fprintf(stderr, " posted:"); for (pr = r->posted_requests; pr; pr = pr->next) { fprintf(stderr, "%p,", pr); } } fprintf(stderr, "\n"); } # else # define dd_enter() # endif #else # if (NGX_HAVE_VARIADIC_MACROS) # define dd(...) # define dd_enter() # else #include static void dd(const char * fmt, ...) { } static void dd_enter() { } # endif #endif #if defined(DDEBUG) && (DDEBUG) #define dd_check_read_event_handler(r) \ dd("r->read_event_handler = %s", \ r->read_event_handler == ngx_http_block_reading ? \ "ngx_http_block_reading" : \ r->read_event_handler == ngx_http_test_reading ? \ "ngx_http_test_reading" : \ r->read_event_handler == ngx_http_request_empty_handler ? \ "ngx_http_request_empty_handler" : "UNKNOWN") #define dd_check_write_event_handler(r) \ dd("r->write_event_handler = %s", \ r->write_event_handler == ngx_http_handler ? \ "ngx_http_handler" : \ r->write_event_handler == ngx_http_core_run_phases ? \ "ngx_http_core_run_phases" : \ r->write_event_handler == ngx_http_request_empty_handler ? \ "ngx_http_request_empty_handler" : "UNKNOWN") #else #define dd_check_read_event_handler(r) #define dd_check_write_event_handler(r) #endif #endif /* DDEBUG_H */ ================================================ FILE: src/ngx_http_replace_filter_module.c ================================================ /* * Copyright (C) Yichun Zhang (agentzh) * Copyright (C) Igor Sysoev * Copyright (C) Nginx, Inc. */ #ifndef DDEBUG #define DDEBUG 0 #endif #include "ddebug.h" #include "ngx_http_replace_filter_module.h" #include "ngx_http_replace_parse.h" #include "ngx_http_replace_script.h" #include "ngx_http_replace_util.h" enum { SREGEX_COMPILER_POOL_SIZE = 4096 }; static ngx_int_t ngx_http_replace_output(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx); static char *ngx_http_replace_filter(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static void *ngx_http_replace_create_loc_conf(ngx_conf_t *cf); static char *ngx_http_replace_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child); static ngx_int_t ngx_http_replace_filter_init(ngx_conf_t *cf); static void ngx_http_replace_cleanup_pool(void *data); static void *ngx_http_replace_create_main_conf(ngx_conf_t *cf); #define ngx_http_replace_regex_is_disabled(ctx) \ ((ctx)->disabled[(ctx)->regex_id / 8] & (1 << ((ctx)->regex_id % 8))) #define ngx_http_replace_regex_set_disabled(ctx) \ (ctx)->disabled[(ctx)->regex_id / 8] |= (1 << ((ctx)->regex_id % 8)) static volatile ngx_cycle_t *ngx_http_replace_prev_cycle = NULL; #define NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED 0 #define NGX_HTTP_REPLACE_KEEP_LAST_MODIFIED 1 static ngx_conf_enum_t ngx_http_replace_filter_last_modified[] = { { ngx_string("clear"), NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED }, { ngx_string("keep"), NGX_HTTP_REPLACE_KEEP_LAST_MODIFIED }, { ngx_null_string, 0 } }; static ngx_command_t ngx_http_replace_filter_commands[] = { { ngx_string("replace_filter"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_TAKE23, ngx_http_replace_filter, NGX_HTTP_LOC_CONF_OFFSET, 0, NULL }, { ngx_string("replace_filter_types"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_1MORE, ngx_http_types_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_replace_loc_conf_t, types_keys), &ngx_http_html_default_types[0] }, { ngx_string("replace_filter_max_buffered_size"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_TAKE1, ngx_conf_set_size_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_replace_loc_conf_t, max_buffered_size), NULL }, { ngx_string("replace_filter_last_modified"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_1MORE, ngx_conf_set_enum_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_replace_loc_conf_t, last_modified), &ngx_http_replace_filter_last_modified }, { ngx_string("replace_filter_skip"), NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF |NGX_CONF_TAKE1, ngx_http_set_complex_value_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_replace_loc_conf_t, skip), NULL }, ngx_null_command }; static ngx_http_module_t ngx_http_replace_filter_module_ctx = { NULL, /* preconfiguration */ ngx_http_replace_filter_init, /* postconfiguration */ ngx_http_replace_create_main_conf, /* create main configuration */ NULL, /* init main configuration */ NULL, /* create server configuration */ NULL, /* merge server configuration */ ngx_http_replace_create_loc_conf, /* create location configuration */ ngx_http_replace_merge_loc_conf /* merge location configuration */ }; ngx_module_t ngx_http_replace_filter_module = { NGX_MODULE_V1, &ngx_http_replace_filter_module_ctx, /* module context */ ngx_http_replace_filter_commands, /* module directives */ NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ NULL, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; static ngx_http_output_header_filter_pt ngx_http_next_header_filter; static ngx_http_output_body_filter_pt ngx_http_next_body_filter; static ngx_int_t ngx_http_replace_header_filter(ngx_http_request_t *r) { size_t size; ngx_str_t skip; ngx_pool_cleanup_t *cln; ngx_http_replace_ctx_t *ctx; ngx_http_replace_loc_conf_t *rlcf; rlcf = ngx_http_get_module_loc_conf(r, ngx_http_replace_filter_module); dd("replace header filter"); if (rlcf->regexes.nelts == 0 || r->headers_out.content_length_n == 0 || (r->headers_out.content_encoding && r->headers_out.content_encoding->value.len) || ngx_http_test_content_type(r, &rlcf->types) == NULL) { return ngx_http_next_header_filter(r); } dd("skip: %p", rlcf->skip); if (rlcf->skip != NULL) { if (ngx_http_complex_value(r, rlcf->skip, &skip) != NGX_OK) { return NGX_ERROR; } if (skip.len && (skip.len != 1 || skip.data[0] != '0')) { return ngx_http_next_header_filter(r); } } ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_replace_ctx_t)); if (ctx == NULL) { return NGX_ERROR; } ctx->last_special = &ctx->special; ctx->last_pending = &ctx->pending; ctx->last_pending2 = &ctx->pending2; ctx->last_captured = &ctx->captured; ctx->sub = ngx_pcalloc(r->pool, rlcf->multi_replace.nelts * sizeof(ngx_str_t)); if (ctx->sub == NULL) { return NGX_ERROR; } ctx->ovector = ngx_palloc(r->pool, rlcf->ovecsize); if (ctx->ovector == NULL) { return NGX_ERROR; } size = ngx_align(rlcf->regexes.nelts, 8) / 8; ctx->disabled = ngx_pcalloc(r->pool, size); if (ctx->disabled == NULL) { return NGX_ERROR; } ctx->vm_pool = sre_create_pool(1024); if (ctx->vm_pool == NULL) { return NGX_ERROR; } dd("created vm pool %p", ctx->vm_pool); cln = ngx_pool_cleanup_add(r->pool, 0); if (cln == NULL) { sre_destroy_pool(ctx->vm_pool); return NGX_ERROR; } cln->data = ctx->vm_pool; cln->handler = ngx_http_replace_cleanup_pool; ctx->vm_ctx = sre_vm_pike_create_ctx(ctx->vm_pool, rlcf->program, ctx->ovector, rlcf->ovecsize); if (ctx->vm_ctx == NULL) { return NGX_ERROR; } ngx_http_set_ctx(r, ctx, ngx_http_replace_filter_module); ctx->last_out = &ctx->out; r->filter_need_in_memory = 1; if (r == r->main) { ngx_http_clear_content_length(r); if (rlcf->last_modified == NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED) { ngx_http_clear_last_modified(r); } } return ngx_http_next_header_filter(r); } static void ngx_http_replace_cleanup_pool(void *data) { sre_pool_t *pool = data; if (pool) { dd("destroy sre pool %p", pool); sre_destroy_pool(pool); } } static ngx_int_t ngx_http_replace_body_filter(ngx_http_request_t *r, ngx_chain_t *in) { ngx_int_t rc; ngx_buf_t *b; ngx_str_t *sub; ngx_chain_t *cl, *cur = NULL, *rematch = NULL; ngx_http_replace_ctx_t *ctx; ngx_http_replace_loc_conf_t *rlcf; rlcf = ngx_http_get_module_loc_conf(r, ngx_http_replace_filter_module); ctx = ngx_http_get_module_ctx(r, ngx_http_replace_filter_module); if (ctx == NULL) { return ngx_http_next_body_filter(r, in); } if ((in == NULL && ctx->buf == NULL && ctx->in == NULL && ctx->busy == NULL)) { return ngx_http_next_body_filter(r, in); } if ((ctx->once || ctx->vm_done) && (ctx->buf == NULL || ctx->in == NULL)) { if (ctx->busy) { if (ngx_http_replace_output(r, ctx) == NGX_ERROR) { return NGX_ERROR; } } return ngx_http_next_body_filter(r, in); } /* add the incoming chain to the chain ctx->in */ if (in) { if (ngx_chain_add_copy(r->pool, &ctx->in, in) != NGX_OK) { return NGX_ERROR; } } ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http sub filter \"%V\"", &r->uri); while (ctx->in || ctx->buf) { if (ctx->buf == NULL) { cur = ctx->in; ctx->buf = cur->buf; ctx->in = cur->next; ctx->pos = ctx->buf->pos; ctx->special_buf = ngx_buf_special(ctx->buf); ctx->last_buf = (ctx->buf->last_buf || ctx->buf->last_in_chain); dd("=== new incoming buf: size=%d, special=%u, last=%u", (int) ngx_buf_size(ctx->buf), ctx->special_buf, ctx->last_buf); } b = NULL; while (ctx->pos < ctx->buf->last || (ctx->special_buf && ctx->last_buf)) { rc = rlcf->parse_buf(r, ctx, rematch); ngx_log_debug3(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "replace filter parse: %d, %p-%p", rc, ctx->copy_start, ctx->copy_end); if (rc == NGX_ERROR) { return rc; } if (rc == NGX_DECLINED) { if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (!ctx->special_buf) { ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; } else { ctx->copy_start = NULL; ctx->copy_end = NULL; } sre_reset_pool(ctx->vm_pool); ctx->vm_done = 1; } dd("copy_end - copy_start: %d, special: %u", (int) (ctx->copy_end - ctx->copy_start), ctx->special_buf); if (ctx->copy_start != ctx->copy_end && !ctx->special_buf) { dd("copy: %.*s", (int) (ctx->copy_end - ctx->copy_start), ctx->copy_start); cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; } b = cl->buf; b->memory = 1; b->pos = ctx->copy_start; b->last = ctx->copy_end; *ctx->last_out = cl; ctx->last_out = &cl->next; } if (rc == NGX_AGAIN) { if (ctx->special_buf && ctx->last_buf) { break; } continue; } if (rc == NGX_DECLINED) { break; } /* rc == NGX_OK || rc == NGX_BUSY */ cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; } b = cl->buf; dd("free data buf: %p", b); sub = &ctx->sub[ctx->regex_id]; if (sub->data == NULL || rlcf->parse_buf == ngx_http_replace_capturing_parse) { ngx_http_replace_complex_value_t *cv; if (ngx_http_replace_regex_is_disabled(ctx)) { cv = &rlcf->verbatim; } else { cv = rlcf->multi_replace.elts; cv = &cv[ctx->regex_id]; } if (ngx_http_replace_complex_value(r, ctx->captured, rlcf->ncaps, ctx->ovector, cv, sub) != NGX_OK) { return NGX_ERROR; } /* release ctx->captured */ if (ctx->captured) { dd("release ctx captured: %p", ctx->captured); *ctx->last_captured = ctx->free; ctx->free = ctx->captured; ctx->captured = NULL; ctx->last_captured = &ctx->captured; } } dd("emit replaced value: \"%.*s\"", (int) sub->len, sub->data); if (sub->len) { b->memory = 1; b->pos = sub->data; b->last = sub->data + sub->len; } else { b->sync = 1; } cl->buf = b; cl->next = NULL; *ctx->last_out = cl; ctx->last_out = &cl->next; if (!ctx->once && !ngx_http_replace_regex_is_disabled(ctx)) { uint8_t *once; once = rlcf->multi_once.elts; if (rlcf->regexes.nelts == 1) { ctx->once = once[0]; } else { if (once[ctx->regex_id]) { ngx_http_replace_regex_set_disabled(ctx); if (!rlcf->seen_global && ++ctx->disabled_count == rlcf->regexes.nelts) { ctx->once = 1; } } } } if (rc == NGX_BUSY) { dd("goto rematch"); goto rematch; } if (ctx->special_buf) { break; } continue; } if ((ctx->buf->flush || ctx->last_buf || ngx_buf_in_memory(ctx->buf)) && cur) { if (b == NULL) { cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; } b = cl->buf; b->sync = 1; *ctx->last_out = cl; ctx->last_out = &cl->next; } dd("setting shadow and last buf: %d", (int) ctx->buf->last_buf); b->last_buf = ctx->buf->last_buf; b->last_in_chain = ctx->buf->last_in_chain; b->flush = ctx->buf->flush; b->shadow = ctx->buf; b->recycled = ctx->buf->recycled; } if (!ctx->special_buf) { ctx->stream_pos += ctx->buf->last - ctx->buf->pos; } if (rematch) { rematch->next = ctx->free; ctx->free = rematch; rematch = NULL; } rematch: dd("ctx->rematch: %p", ctx->rematch); if (ctx->rematch == NULL) { ctx->buf = NULL; cur = NULL; } else { if (cur) { ctx->in = cur; cur = NULL; } ctx->buf = ctx->rematch->buf; dd("ctx->buf set to rematch buf %p, len=%d, next=%p", ctx->buf, (int) ngx_buf_size(ctx->buf), ctx->rematch->next); rematch = ctx->rematch; ctx->rematch = rematch->next; ctx->pos = ctx->buf->pos; ctx->special_buf = ngx_buf_special(ctx->buf); ctx->last_buf = (ctx->buf->last_buf || ctx->buf->last_in_chain); ctx->stream_pos = ctx->buf->file_pos; } #if (DDEBUG) /* ngx_http_replace_dump_chain("ctx->pending", &ctx->pending, ctx->last_pending); ngx_http_replace_dump_chain("ctx->pending2", &ctx->pending2, ctx->last_pending2); */ #endif } /* while */ if (ctx->out == NULL && ctx->busy == NULL) { return NGX_OK; } return ngx_http_replace_output(r, ctx); } static ngx_int_t ngx_http_replace_output(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx) { ngx_int_t rc; ngx_buf_t *b; ngx_chain_t *cl; #if (DDEBUG) b = NULL; for (cl = ctx->out; cl; cl = cl->next) { ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "replace out: %p %p", cl->buf, cl->buf->pos); if (cl->buf == b) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "the same buf was used in sub"); ngx_debug_point(); return NGX_ERROR; } b = cl->buf; } /* ngx_http_replace_dump_chain("ctx->out", &ctx->out, ctx->last_out); */ #endif rc = ngx_http_next_body_filter(r, ctx->out); /* we are essentially duplicating the logic of * ngx_chain_update_chains below, * with our own optimizations */ if (ctx->busy == NULL) { ctx->busy = ctx->out; } else { for (cl = ctx->busy; cl->next; cl = cl->next) { /* void */ } cl->next = ctx->out; } ctx->out = NULL; ctx->last_out = &ctx->out; while (ctx->busy) { cl = ctx->busy; b = cl->buf; if (ngx_buf_size(b) != 0) { break; } if (cl->buf->tag != (ngx_buf_tag_t) &ngx_http_replace_filter_module) { ctx->busy = cl->next; ngx_free_chain(r->pool, cl); continue; } if (b->shadow) { b->shadow->pos = b->shadow->last; b->shadow->file_pos = b->shadow->file_last; } ctx->busy = cl->next; if (ngx_buf_special(b)) { /* collect special bufs to ctx->special, which may still be busy */ cl->next = NULL; *ctx->last_special = cl; ctx->last_special = &cl->next; } else { /* add ctx->special to ctx->free because they cannot be busy at * this point */ *ctx->last_special = ctx->free; ctx->free = ctx->special; ctx->special = NULL; ctx->last_special = &ctx->special; #if 0 /* free the temporary buf's data block if it is big enough */ if (b->temporary && b->start != NULL && b->end - b->start > (ssize_t) r->pool->max) { ngx_pfree(r->pool, b->start); } #endif /* add the data buf itself to the free buf chain */ cl->next = ctx->free; ctx->free = cl; } } if (ctx->in || ctx->buf) { r->buffered |= NGX_HTTP_SUB_BUFFERED; } else { r->buffered &= ~NGX_HTTP_SUB_BUFFERED; } return rc; } static char * ngx_http_replace_filter(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { ngx_http_replace_loc_conf_t *rlcf = conf; ngx_http_replace_main_conf_t *rmcf; int *flags; u_char *p, **re; ngx_str_t *value; ngx_uint_t i; uint8_t *once; ngx_pool_cleanup_t *cln; ngx_http_replace_complex_value_t *cv; ngx_http_replace_compile_complex_value_t ccv; value = cf->args->elts; re = ngx_array_push(&rlcf->regexes); if (re == NULL) { return NGX_CONF_ERROR; } *re = value[1].data; cv = ngx_array_push(&rlcf->multi_replace); if (cv == NULL) { return NGX_CONF_ERROR; } ngx_memzero(cv, sizeof(ngx_http_replace_complex_value_t)); ngx_memzero(&ccv, sizeof(ngx_http_replace_compile_complex_value_t)); ccv.cf = cf; ccv.value = &value[2]; ccv.complex_value = cv; if (ngx_http_replace_compile_complex_value(&ccv) != NGX_OK) { return NGX_CONF_ERROR; } /* check variable usage in the "replace" argument */ if (cv->capture_variables) { rlcf->parse_buf = ngx_http_replace_capturing_parse; } else if (rlcf->parse_buf == NULL) { rlcf->parse_buf = ngx_http_replace_non_capturing_parse; } #if 0 rlcf->parse_buf = ngx_http_replace_capturing_parse; #endif flags = ngx_array_push(&rlcf->multi_flags); if (flags == NULL) { return NGX_CONF_ERROR; } *flags = 0; once = ngx_array_push(&rlcf->multi_once); if (once == NULL) { return NGX_CONF_ERROR; } *once = 1; /* default to once */ if (cf->args->nelts == 4) { /* 3 user args */ p = value[3].data; for (i = 0; i < value[3].len; i++) { switch (p[i]) { case 'i': *flags |= SRE_REGEX_CASELESS; break; case 'g': *once = 0; break; default: return "specifies an unrecognized regex flag"; } } } if (*once) { rlcf->seen_once = 1; } else { rlcf->seen_global = 1; } if (rlcf->seen_once && rlcf->regexes.nelts > 1) { rlcf->parse_buf = ngx_http_replace_capturing_parse; if (rlcf->verbatim.value.data == NULL) { ngx_str_t v = ngx_string("$&"); ngx_memzero(&ccv, sizeof(ngx_http_replace_compile_complex_value_t)); ccv.cf = cf; ccv.value = &v; ccv.complex_value = &rlcf->verbatim; if (ngx_http_replace_compile_complex_value(&ccv) != NGX_OK) { return NGX_CONF_ERROR; } } } rmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_replace_filter_module); if (rmcf->compiler_pool == NULL) { rmcf->compiler_pool = sre_create_pool(SREGEX_COMPILER_POOL_SIZE); if (rmcf->compiler_pool == NULL) { return NGX_CONF_ERROR; } cln = ngx_pool_cleanup_add(cf->pool, 0); if (cln == NULL) { sre_destroy_pool(rmcf->compiler_pool); rmcf->compiler_pool = NULL; return NGX_CONF_ERROR; } cln->data = rmcf->compiler_pool; cln->handler = ngx_http_replace_cleanup_pool; } return NGX_CONF_OK; } static void * ngx_http_replace_create_loc_conf(ngx_conf_t *cf) { ngx_http_replace_loc_conf_t *conf; conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_replace_loc_conf_t)); if (conf == NULL) { return NULL; } /* * set by ngx_pcalloc(): * * conf->types = { NULL }; * conf->types_keys = NULL; * conf->program = NULL; * conf->ncaps = 0; * conf->ovecsize = 0; * conf->parse_buf = NULL; * conf->verbatim = { {0, NULL}, NULL, NULL, 0 }; * conf->seen_once = 0; * conf->seen_global = 0; * conf->skip = NULL; */ conf->max_buffered_size = NGX_CONF_UNSET_SIZE; conf->last_modified = NGX_CONF_UNSET_UINT; ngx_array_init(&conf->multi_replace, cf->pool, 4, sizeof(ngx_http_replace_complex_value_t)); ngx_array_init(&conf->multi_flags, cf->pool, 4, sizeof(int)); ngx_array_init(&conf->regexes, cf->pool, 4, sizeof(u_char *)); ngx_array_init(&conf->multi_once, cf->pool, 4, sizeof(uint8_t)); return conf; } static char * ngx_http_replace_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child) { u_char **value; sre_int_t err_offset, err_regex_id; ngx_str_t prefix, suffix; sre_pool_t *ppool; /* parser pool */ sre_regex_t *re; sre_program_t *prog; ngx_http_replace_main_conf_t *rmcf; ngx_http_replace_loc_conf_t *prev = parent; ngx_http_replace_loc_conf_t *conf = child; ngx_conf_merge_size_value(conf->max_buffered_size, prev->max_buffered_size, 8192); ngx_conf_merge_uint_value(conf->last_modified, prev->last_modified, NGX_HTTP_REPLACE_CLEAR_LAST_MODIFIED); if (ngx_http_merge_types(cf, &conf->types_keys, &conf->types, &prev->types_keys, &prev->types, ngx_http_html_default_types) != NGX_OK) { return NGX_CONF_ERROR; } if (conf->skip == NULL) { conf->skip = prev->skip; } if (conf->regexes.nelts > 0 && conf->program == NULL) { dd("parsing and compiling %d regexes", (int) conf->regexes.nelts); ppool = sre_create_pool(1024); if (ppool == NULL) { return NGX_CONF_ERROR; } value = conf->regexes.elts; re = sre_regex_parse_multi(ppool, value, conf->regexes.nelts, &conf->ncaps, conf->multi_flags.elts, &err_offset, &err_regex_id); if (re == NULL) { if (err_offset >= 0) { prefix.data = value[err_regex_id]; prefix.len = err_offset; suffix.data = value[err_regex_id] + err_offset; suffix.len = ngx_strlen(value[err_regex_id]) - err_offset; ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "failed to parse regex at offset %i: " "syntax error; marked by <-- HERE in " "\"%V <-- HERE %V\"", (ngx_int_t) err_offset, &prefix, &suffix); } else { if (err_regex_id >= 0) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "failed to parse regex \"%s\"", value[err_regex_id]); } else { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "failed to parse regex \"%s\" " "and its siblings", value[0]); } } sre_destroy_pool(ppool); return NGX_CONF_ERROR; } rmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_replace_filter_module); prog = sre_regex_compile(rmcf->compiler_pool, re); sre_destroy_pool(ppool); if (prog == NULL) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "failed to compile regex \"%s\" and its " "siblings", value[0]); return NGX_CONF_ERROR; } conf->program = prog; conf->ovecsize = 2 * (conf->ncaps + 1) * sizeof(sre_int_t); } else { conf->regexes = prev->regexes; conf->multi_once = prev->multi_once; conf->multi_flags = prev->multi_flags; conf->multi_replace = prev->multi_replace; conf->parse_buf = prev->parse_buf; conf->verbatim = prev->verbatim; conf->program = prev->program; conf->ncaps = prev->ncaps; conf->ovecsize = prev->ovecsize; conf->seen_once = prev->seen_once; conf->seen_global = prev->seen_global; } return NGX_CONF_OK; } static ngx_int_t ngx_http_replace_filter_init(ngx_conf_t *cf) { int multi_http_blocks; ngx_http_replace_main_conf_t *rmcf; rmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_replace_filter_module); if (ngx_http_replace_prev_cycle != ngx_cycle) { ngx_http_replace_prev_cycle = ngx_cycle; multi_http_blocks = 0; } else { multi_http_blocks = 1; } if (multi_http_blocks || rmcf->compiler_pool != NULL) { ngx_http_next_header_filter = ngx_http_top_header_filter; ngx_http_top_header_filter = ngx_http_replace_header_filter; ngx_http_next_body_filter = ngx_http_top_body_filter; ngx_http_top_body_filter = ngx_http_replace_body_filter; return NGX_OK; } return NGX_OK; } static void * ngx_http_replace_create_main_conf(ngx_conf_t *cf) { ngx_http_replace_main_conf_t *rmcf; rmcf = ngx_pcalloc(cf->pool, sizeof(ngx_http_replace_main_conf_t)); if (rmcf == NULL) { return NULL; } /* set by ngx_pcalloc: * rmcf->compiler_pool = NULL; */ return rmcf; } ================================================ FILE: src/ngx_http_replace_filter_module.h ================================================ #ifndef _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_ #define _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_ #include "ngx_http_replace_script.h" #include #include #include #include extern ngx_module_t ngx_http_replace_filter_module; typedef struct { sre_int_t regex_id; sre_int_t stream_pos; sre_int_t *ovector; sre_pool_t *vm_pool; sre_vm_pike_ctx_t *vm_ctx; ngx_chain_t *pending; /* pending data before the pending matched capture */ ngx_chain_t **last_pending; ngx_chain_t *pending2; /* pending data after the pending matched capture */ ngx_chain_t **last_pending2; ngx_buf_t *buf; ngx_str_t *sub; u_char *pos; u_char *copy_start; u_char *copy_end; ngx_chain_t *in; ngx_chain_t *out; ngx_chain_t **last_out; ngx_chain_t *busy; ngx_chain_t *free; ngx_chain_t *special; ngx_chain_t **last_special; ngx_chain_t *rematch; ngx_chain_t *captured; ngx_chain_t **last_captured; uint8_t *disabled; sre_uint_t disabled_count; size_t total_buffered; unsigned once:1; unsigned vm_done:1; unsigned special_buf:1; unsigned last_buf:1; } ngx_http_replace_ctx_t; typedef ngx_int_t (*ngx_http_replace_parse_buf_pt)(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch); typedef struct { sre_pool_t *compiler_pool; } ngx_http_replace_main_conf_t; typedef struct { sre_uint_t ncaps; size_t ovecsize; ngx_array_t multi_once; /* of uint8_t */ ngx_array_t regexes; /* of u_char* */ ngx_array_t multi_flags; /* of int */ ngx_array_t multi_replace; /* of ngx_http_replace_complex_value_t */ sre_program_t *program; ngx_hash_t types; ngx_array_t *types_keys; size_t max_buffered_size; ngx_uint_t last_modified; /* replace_filter_last_modified */ ngx_http_replace_parse_buf_pt parse_buf; ngx_http_replace_complex_value_t verbatim; ngx_http_complex_value_t *skip; unsigned seen_once; /* :1 */ unsigned seen_global; /* :1 */ } ngx_http_replace_loc_conf_t; #endif /* _NGX_HTTP_REPLACE_FILTER_MODULE_H_INCLUDED_ */ ================================================ FILE: src/ngx_http_replace_parse.c ================================================ /* * Copyright (C) Yichun Zhang (agentzh) * Copyright (C) Igor Sysoev * Copyright (C) Nginx, Inc. */ #ifndef DDEBUG #define DDEBUG 0 #endif #include "ddebug.h" #include "ngx_http_replace_parse.h" #include "ngx_http_replace_util.h" static void ngx_http_replace_check_total_buffered(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, sre_int_t len, sre_int_t mlen); ngx_int_t ngx_http_replace_capturing_parse(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch) { sre_int_t ret, from, to; ngx_int_t rc; ngx_chain_t *new_rematch = NULL; ngx_chain_t *cl; ngx_chain_t **last_rematch, **last; size_t len; dd("replace capturing parse"); if (ctx->once || ctx->vm_done) { ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "once"); return NGX_AGAIN; } len = ctx->buf->last - ctx->pos; dd("=== process data chunk %p len=%d, pos=%u, special=%u, " "last=%u, \"%.*s\"", ctx->buf, (int) (ctx->buf->last - ctx->pos), (int) (ctx->pos - ctx->buf->pos + ctx->stream_pos), ctx->special_buf, ctx->last_buf, (int) (ctx->buf->last - ctx->pos), ctx->pos); ret = sre_vm_pike_exec(ctx->vm_ctx, ctx->pos, len, ctx->last_buf, NULL); dd("vm pike exec: %d", (int) ret); if (ret >= 0) { ctx->regex_id = ret; ctx->total_buffered = 0; from = ctx->ovector[0]; to = ctx->ovector[1]; dd("pike vm ok: (%d, %d)", (int) from, (int) to); if (from >= ctx->stream_pos) { /* the match is completely on the current buf */ if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } /* prepare ctx->captured */ cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; } cl->buf->pos = ctx->buf->pos; cl->buf->last = ctx->buf->last; cl->buf->memory = 1; cl->buf->file_pos = ctx->stream_pos; cl->buf->file_last = ctx->stream_pos + (cl->buf->last - cl->buf->pos); *ctx->last_captured = cl; ctx->last_captured = &cl->next; dd("ctx captured: %p", ctx->captured); /* prepare copy-out data */ ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos); dd("copy len: %d", (int) (ctx->copy_end - ctx->copy_start)); ctx->pos = ctx->buf->pos + (to - ctx->stream_pos); return NGX_OK; } /* from < ctx->stream_pos */ if (ctx->pending) { if (ngx_http_replace_split_chain(r, ctx, &ctx->pending, &ctx->last_pending, from, &cl, &last, 1) != NGX_OK) { return NGX_ERROR; } if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (cl) { if (to >= ctx->stream_pos) { /* no pending data to be rematched */ if (to == ctx->stream_pos) { *ctx->last_captured = cl; ctx->last_captured = &cl->next; } else { *ctx->last_captured = cl; ctx->last_captured = last; cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; } cl->buf->pos = ctx->buf->pos; cl->buf->last = ctx->buf->last; cl->buf->memory = 1; cl->buf->file_pos = ctx->stream_pos; cl->buf->file_last = ctx->stream_pos + (cl->buf->last - cl->buf->pos); *ctx->last_captured = cl; ctx->last_captured = &cl->next; } } else { /* there's pending data to be rematched */ if (ngx_http_replace_split_chain(r, ctx, &cl, &last, to, &new_rematch, &last_rematch, 1) != NGX_OK) { return NGX_ERROR; } if (cl) { *ctx->last_captured = cl; ctx->last_captured = last; } if (new_rematch) { if (rematch) { ctx->rematch = rematch; } /* prepend cl to ctx->rematch */ *last_rematch = ctx->rematch; ctx->rematch = new_rematch; } } } } #if (DDEBUG) ngx_http_replace_dump_chain("ctx->rematch", &ctx->rematch, NULL); #endif ctx->copy_start = NULL; ctx->copy_end = NULL; ctx->pos = ctx->buf->pos + (to - ctx->stream_pos); return new_rematch ? NGX_BUSY : NGX_OK; } switch (ret) { case SRE_AGAIN: from = ctx->ovector[0]; to = ctx->ovector[1]; dd("pike vm again: (%d, %d)", (int) from, (int) to); if (from == -1) { from = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos); } if (to == -1) { to = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos); } dd("pike vm again (adjusted): stream pos:%d, (%d, %d)", (int) ctx->stream_pos, (int) from, (int) to); if (from > to) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "invalid capture range: %i > %i", (ngx_int_t) from, (ngx_int_t) to); return NGX_ERROR; } if (from == to) { if (ctx->pending) { ctx->total_buffered = 0; dd("output pending"); *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos); ctx->pos = ctx->copy_end; return NGX_AGAIN; } /* * append the existing ctx->pending data right before * the $& capture to ctx->out. */ if (from >= ctx->stream_pos) { /* the match is completely on the current buf */ ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos); if (ctx->pending) { ctx->total_buffered = 0; *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } dd("create ctx->pending as (%ld, %ld)", (long) from, (long) to); rc = ngx_http_replace_new_pending_buf(r, ctx, from, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } #if 1 if (rc == NGX_BUSY) { dd("stop processing because of buffer size limit reached"); ctx->once = 1; ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; return NGX_AGAIN; } #endif *ctx->last_pending = cl; ctx->last_pending = &cl->next; ctx->pos = ctx->buf->last; return NGX_AGAIN; } dd("from < ctx->stream_pos"); if (ctx->pending) { /* split ctx->pending into ctx->out and ctx->pending */ if (ngx_http_replace_split_chain(r, ctx, &ctx->pending, &ctx->last_pending, from, &cl, &last, 1) != NGX_OK) { return NGX_ERROR; } if (ctx->pending) { dd("adjust pending: pos=%d, from=%d", (int) ctx->pending->buf->file_pos, (int) from); ctx->total_buffered -= (size_t) (from - ctx->pending->buf->file_pos); *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (cl) { dd("splitted ctx->pending into ctx->out and ctx->pending: %d", (int) ctx->total_buffered); ctx->pending = cl; ctx->last_pending = last; } } /* new pending data to buffer to ctx->pending */ rc = ngx_http_replace_new_pending_buf(r, ctx, ctx->pos - ctx->buf->pos + ctx->stream_pos, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } #if 1 if (rc == NGX_BUSY) { ctx->once = 1; if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; return NGX_AGAIN; } #endif *ctx->last_pending = cl; ctx->last_pending = &cl->next; ctx->copy_start = NULL; ctx->copy_end = NULL; ctx->pos = ctx->buf->last; return NGX_AGAIN; case SRE_DECLINED: ctx->total_buffered = 0; return NGX_DECLINED; default: /* SRE_ERROR */ return NGX_ERROR; } /* cannot reach here */ } ngx_int_t ngx_http_replace_non_capturing_parse(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch) { sre_int_t ret, from, to, mfrom = -1, mto = -1; ngx_int_t rc; ngx_chain_t *new_rematch = NULL; ngx_chain_t *cl; ngx_chain_t **last_rematch, **last; size_t len; sre_int_t *pending_matched; dd("replace non capturing parse"); if (ctx->once || ctx->vm_done) { ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "once"); return NGX_AGAIN; } len = ctx->buf->last - ctx->pos; dd("=== process data chunk %p len=%d, pos=%u, special=%u, " "last=%u, \"%.*s\"", ctx->buf, (int) (ctx->buf->last - ctx->pos), (int) (ctx->pos - ctx->buf->pos + ctx->stream_pos), ctx->special_buf, ctx->last_buf, (int) (ctx->buf->last - ctx->pos), ctx->pos); ret = sre_vm_pike_exec(ctx->vm_ctx, ctx->pos, len, ctx->last_buf, &pending_matched); dd("vm pike exec: %d", (int) ret); if (ret >= 0) { ctx->regex_id = ret; ctx->total_buffered = 0; from = ctx->ovector[0]; to = ctx->ovector[1]; dd("pike vm ok: (%d, %d)", (int) from, (int) to); if (from >= ctx->stream_pos) { /* the match is completely on the current buf */ if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (ctx->pending2) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "assertion failed: ctx->pending2 is not NULL " "when the match is completely on the current " "buf"); return NGX_ERROR; } ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos); dd("copy len: %d", (int) (ctx->copy_end - ctx->copy_start)); ctx->pos = ctx->buf->pos + (to - ctx->stream_pos); return NGX_OK; } /* from < ctx->stream_pos */ if (ctx->pending) { if (ngx_http_replace_split_chain(r, ctx, &ctx->pending, &ctx->last_pending, from, &cl, &last, 0) != NGX_OK) { return NGX_ERROR; } if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (cl) { *last = ctx->free; ctx->free = cl; } } if (ctx->pending2) { if (ngx_http_replace_split_chain(r, ctx, &ctx->pending2, &ctx->last_pending2, to, &new_rematch, &last_rematch, 1) != NGX_OK) { return NGX_ERROR; } if (ctx->pending2) { *ctx->last_pending2 = ctx->free; ctx->free = ctx->pending2; ctx->pending2 = NULL; ctx->last_pending2 = &ctx->pending2; } if (new_rematch) { if (rematch) { ctx->rematch = rematch; } /* prepend cl to ctx->rematch */ *last_rematch = ctx->rematch; ctx->rematch = new_rematch; } } #if (DDEBUG) ngx_http_replace_dump_chain("ctx->rematch", &ctx->rematch, NULL); #endif ctx->copy_start = NULL; ctx->copy_end = NULL; ctx->pos = ctx->buf->pos + (to - ctx->stream_pos); return new_rematch ? NGX_BUSY : NGX_OK; } switch (ret) { case SRE_AGAIN: from = ctx->ovector[0]; to = ctx->ovector[1]; dd("pike vm again: (%d, %d)", (int) from, (int) to); if (from == -1) { from = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos); } if (to == -1) { to = ctx->stream_pos + (ctx->buf->last - ctx->buf->pos); } dd("pike vm again (adjusted): stream pos:%d, (%d, %d)", (int) ctx->stream_pos, (int) from, (int) to); if (from > to) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "invalid capture range: %i > %i", (ngx_int_t) from, (ngx_int_t) to); return NGX_ERROR; } if (pending_matched) { mfrom = pending_matched[0]; mto = pending_matched[1]; dd("pending matched: (%ld, %ld)", (long) mfrom, (long) mto); } if (from == to) { if (ctx->pending) { ctx->total_buffered = 0; dd("output pending"); *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos); ctx->pos = ctx->copy_end; ngx_http_replace_check_total_buffered(r, ctx, to - from, mto - mfrom); return NGX_AGAIN; } /* * append the existing ctx->pending data right before * the $& capture to ctx->out. */ if (from >= ctx->stream_pos) { /* the match is completely on the current buf */ ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->pos + (from - ctx->stream_pos); if (ctx->pending) { ctx->total_buffered = 0; *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (ctx->pending2) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "assertion failed: ctx->pending2 is not NULL " "when the match is completely on the current " "buf"); return NGX_ERROR; } if (pending_matched) { if (from < mfrom) { /* create ctx->pending as (from, mfrom) */ rc = ngx_http_replace_new_pending_buf(r, ctx, from, mfrom, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } if (rc == NGX_BUSY) { dd("stop processing because of buffer size limit " "reached"); ctx->once = 1; ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; return NGX_AGAIN; } *ctx->last_pending = cl; ctx->last_pending = &cl->next; } if (mto < to) { /* create ctx->pending2 as (mto, to) */ rc = ngx_http_replace_new_pending_buf(r, ctx, mto, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } #if 1 if (rc == NGX_BUSY) { dd("stop processing because of buffer size limit " "reached"); ctx->once = 1; ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; return NGX_AGAIN; } #endif *ctx->last_pending2 = cl; ctx->last_pending2 = &cl->next; } } else { dd("create ctx->pending as (%ld, %ld)", (long) from, (long) to); rc = ngx_http_replace_new_pending_buf(r, ctx, from, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } #if 1 if (rc == NGX_BUSY) { dd("stop processing because of buffer size limit reached"); ctx->once = 1; ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; return NGX_AGAIN; } #endif *ctx->last_pending = cl; ctx->last_pending = &cl->next; } ctx->pos = ctx->buf->last; ngx_http_replace_check_total_buffered(r, ctx, to - from, mto - mfrom); return NGX_AGAIN; } dd("from < ctx->stream_pos"); if (ctx->pending) { /* split ctx->pending into ctx->out and ctx->pending */ if (ngx_http_replace_split_chain(r, ctx, &ctx->pending, &ctx->last_pending, from, &cl, &last, 1) != NGX_OK) { return NGX_ERROR; } if (ctx->pending) { dd("adjust pending: pos=%d, from=%d", (int) ctx->pending->buf->file_pos, (int) from); ctx->total_buffered -= (size_t) (from - ctx->pending->buf->file_pos); *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } if (cl) { dd("splitted ctx->pending into ctx->out and ctx->pending: %d", (int) ctx->total_buffered); ctx->pending = cl; ctx->last_pending = last; } if (pending_matched && !ctx->pending2 && mto >= ctx->stream_pos) { dd("splitting ctx->pending into ctx->pending and ctx->free"); if (ngx_http_replace_split_chain(r, ctx, &ctx->pending, &ctx->last_pending, mfrom, &cl, &last, 0) != NGX_OK) { return NGX_ERROR; } if (cl) { ctx->total_buffered -= (size_t) (ctx->stream_pos - mfrom); dd("splitted ctx->pending into ctx->pending and ctx->free"); *last = ctx->free; ctx->free = cl; } } } if (ctx->pending2) { if (pending_matched) { dd("splitting ctx->pending2 into ctx->free and ctx->pending2"); if (ngx_http_replace_split_chain(r, ctx, &ctx->pending2, &ctx->last_pending2, mto, &cl, &last, 1) != NGX_OK) { return NGX_ERROR; } if (ctx->pending2) { dd("total buffered reduced by %d (was %d)", (int) (mto - ctx->pending2->buf->file_pos), (int) ctx->total_buffered); ctx->total_buffered -= (size_t) (mto - ctx->pending2->buf->file_pos); *ctx->last_pending2 = ctx->free; ctx->free = ctx->pending2; ctx->pending2 = NULL; ctx->last_pending2 = &ctx->pending2; } if (cl) { ctx->pending2 = cl; ctx->last_pending2 = last; } } if (mto < to) { dd("new pending data to buffer to ctx->pending2: (%ld, %ld)", (long) mto, (long) to); rc = ngx_http_replace_new_pending_buf(r, ctx, mto, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } #if 1 if (rc == NGX_BUSY) { ctx->once = 1; if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } ctx->copy_start = NULL; ctx->copy_end = NULL; if (ctx->pending2) { new_rematch = ctx->pending2; last_rematch = ctx->last_pending2; if (rematch) { ctx->rematch = rematch; } /* prepend cl to ctx->rematch */ *last_rematch = ctx->rematch; ctx->rematch = new_rematch; ctx->pending2 = NULL; ctx->last_pending2 = &ctx->pending2; } ctx->pos = ctx->buf->pos + (mto - ctx->stream_pos); return new_rematch ? NGX_BUSY : NGX_OK; } #endif *ctx->last_pending2 = cl; ctx->last_pending2 = &cl->next; } ctx->copy_start = NULL; ctx->copy_end = NULL; ctx->pos = ctx->buf->last; ngx_http_replace_check_total_buffered(r, ctx, to - from, mto - mfrom); return NGX_AGAIN; } /* ctx->pending2 == NULL */ if (pending_matched) { if (mto < to) { /* new pending data to buffer to ctx->pending2 */ rc = ngx_http_replace_new_pending_buf(r, ctx, mto, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } if (rc == NGX_BUSY) { ctx->once = 1; if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } ctx->copy_start = NULL; ctx->copy_end = NULL; ctx->pos = ctx->buf->pos + mto - ctx->stream_pos; return NGX_OK; } *ctx->last_pending2 = cl; ctx->last_pending2 = &cl->next; } /* otherwise no new data to buffer */ } else { /* new pending data to buffer to ctx->pending */ rc = ngx_http_replace_new_pending_buf(r, ctx, ctx->pos - ctx->buf->pos + ctx->stream_pos, to, &cl); if (rc == NGX_ERROR) { return NGX_ERROR; } #if 1 if (rc == NGX_BUSY) { ctx->once = 1; if (ctx->pending) { *ctx->last_out = ctx->pending; ctx->last_out = ctx->last_pending; ctx->pending = NULL; ctx->last_pending = &ctx->pending; } ctx->copy_start = ctx->pos; ctx->copy_end = ctx->buf->last; ctx->pos = ctx->buf->last; return NGX_AGAIN; } #endif *ctx->last_pending = cl; ctx->last_pending = &cl->next; } ctx->copy_start = NULL; ctx->copy_end = NULL; ctx->pos = ctx->buf->last; ngx_http_replace_check_total_buffered(r, ctx, to - from, mto - mfrom); return NGX_AGAIN; case SRE_DECLINED: ctx->total_buffered = 0; return NGX_DECLINED; default: /* SRE_ERROR */ return NGX_ERROR; } /* cannot reach here */ } static void ngx_http_replace_check_total_buffered(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, sre_int_t len, sre_int_t mlen) { dd("total buffered: %d", (int) ctx->total_buffered); if ((ssize_t) ctx->total_buffered != len - mlen) { ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "replace filter: ctx->total_buffered out of " "sync: it is %i but should be %uz", ctx->total_buffered, (ngx_int_t) (len - mlen)); #if (DDEBUG) assert(0); #endif } } ================================================ FILE: src/ngx_http_replace_parse.h ================================================ #ifndef _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_ #define _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_ #include "ngx_http_replace_filter_module.h" ngx_int_t ngx_http_replace_non_capturing_parse(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch); ngx_int_t ngx_http_replace_capturing_parse(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t *rematch); #endif /* _NGX_HTTP_REPLACE_PARSE_H_INCLUDED_ */ ================================================ FILE: src/ngx_http_replace_script.c ================================================ /* * Copyright (C) Yichun Zhang (agentzh) */ #ifndef DDEBUG #define DDEBUG 0 #endif #include "ddebug.h" #include "ngx_http_replace_script.h" static void *ngx_http_replace_script_add_code(ngx_array_t *codes, size_t size); static size_t ngx_http_replace_script_copy_len_code( ngx_http_replace_script_engine_t *e); static size_t ngx_http_replace_script_copy_code(ngx_http_replace_script_engine_t *e); static ngx_int_t ngx_http_replace_script_add_copy_code( ngx_http_replace_script_compile_t *sc, ngx_str_t *value, ngx_uint_t last); static ngx_int_t ngx_http_replace_script_compile(ngx_http_replace_script_compile_t *sc); static ngx_int_t ngx_http_replace_script_add_capture_code( ngx_http_replace_script_compile_t *sc, ngx_uint_t n); static size_t ngx_http_replace_script_copy_capture_len_code( ngx_http_replace_script_engine_t *e); static size_t ngx_http_replace_script_copy_capture_code( ngx_http_replace_script_engine_t *e); static ngx_int_t ngx_http_replace_script_done(ngx_http_replace_script_compile_t *sc); static ngx_int_t ngx_http_replace_script_init_arrays( ngx_http_replace_script_compile_t *sc); static ngx_int_t ngx_http_replace_script_add_var_code(ngx_http_replace_script_compile_t *sc, ngx_str_t *name); static size_t ngx_http_replace_script_copy_var_len_code( ngx_http_replace_script_engine_t *e); static size_t ngx_http_replace_script_copy_var_code(ngx_http_replace_script_engine_t *e); static void ngx_http_replace_count_variables(u_char *src, size_t len, ngx_uint_t *ngxvars, ngx_uint_t *capvars); ngx_int_t ngx_http_replace_compile_complex_value( ngx_http_replace_compile_complex_value_t *ccv) { ngx_str_t *v; ngx_uint_t n, ngxvars, capvars; ngx_array_t lengths, values, *pl, *pv; ngx_http_replace_script_compile_t sc; v = ccv->value; ngx_http_replace_count_variables(v->data, v->len, &ngxvars, &capvars); ccv->complex_value->value = *v; ccv->complex_value->lengths = NULL; ccv->complex_value->values = NULL; if (capvars == 0 && ngxvars == 0) { return NGX_OK; } n = capvars * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_capture_code_t)) + ngxvars * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_var_code_t)) + sizeof(uintptr_t); if (ngx_array_init(&lengths, ccv->cf->pool, n, 1) != NGX_OK) { return NGX_ERROR; } n = capvars * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_capture_code_t)) + ngxvars * (2 * sizeof(ngx_http_replace_script_var_code_t) + sizeof(ngx_http_replace_script_var_code_t)) + sizeof(uintptr_t); if (ngx_array_init(&values, ccv->cf->pool, n, 1) != NGX_OK) { return NGX_ERROR; } pl = &lengths; pv = &values; ngx_memzero(&sc, sizeof(ngx_http_replace_script_compile_t)); sc.cf = ccv->cf; sc.source = v; sc.lengths = &pl; sc.values = &pv; if (ngx_http_replace_script_compile(&sc) != NGX_OK) { ngx_array_destroy(&lengths); ngx_array_destroy(&values); return NGX_ERROR; } ccv->complex_value->lengths = lengths.elts; ccv->complex_value->values = values.elts; ccv->complex_value->capture_variables = sc.capture_variables; return NGX_OK; } ngx_int_t ngx_http_replace_complex_value(ngx_http_request_t *r, ngx_chain_t *captured, sre_uint_t ncaps, sre_int_t *cap, ngx_http_replace_complex_value_t *val, ngx_str_t *value) { size_t len; ngx_http_replace_script_code_pt code; ngx_http_replace_script_len_code_pt lcode; ngx_http_replace_script_engine_t e; if (val->lengths == NULL) { *value = val->value; return NGX_OK; } ngx_memzero(&e, sizeof(ngx_http_replace_script_engine_t)); e.request = r; e.ncaptures = (ncaps + 1) * 2; e.captures_data = captured; e.captures = cap; e.ip = val->lengths; len = 0; while (*(uintptr_t *) e.ip) { lcode = *(ngx_http_replace_script_len_code_pt *) e.ip; len += lcode(&e); } value->len = len; value->data = ngx_pnalloc(r->pool, len); if (value->data == NULL) { return NGX_ERROR; } e.ip = val->values; e.pos = value->data; while (*(uintptr_t *) e.ip) { code = *(ngx_http_replace_script_code_pt *) e.ip; code((ngx_http_replace_script_engine_t *) &e); } return NGX_OK; } static ngx_int_t ngx_http_replace_script_compile(ngx_http_replace_script_compile_t *sc) { u_char ch; ngx_str_t name; ngx_uint_t i, bracket; unsigned num_var; ngx_uint_t n = 0; if (ngx_http_replace_script_init_arrays(sc) != NGX_OK) { return NGX_ERROR; } for (i = 0; i < sc->source->len; /* void */ ) { name.len = 0; if (sc->source->data[i] == '$') { if (++i == sc->source->len) { goto invalid_variable; } if (sc->source->data[i] == '$') { name.data = &sc->source->data[i]; i++; name.len++; sc->size += name.len; if (ngx_http_replace_script_add_copy_code(sc, &name, (i == sc->source->len)) != NGX_OK) { return NGX_ERROR; } continue; } if ((sc->source->data[i] >= '1' && sc->source->data[i] <= '9') || sc->source->data[i] == '&') { num_var = 1; n = 0; } else { num_var = 0; } if (sc->source->data[i] == '{') { bracket = 1; if (++i == sc->source->len) { goto invalid_variable; } if ((sc->source->data[i] >= '1' && sc->source->data[i] <= '9') || sc->source->data[i] == '&') { num_var = 1; n = 0; } name.data = &sc->source->data[i]; } else { bracket = 0; name.data = &sc->source->data[i]; } for ( /* void */ ; i < sc->source->len; i++, name.len++) { ch = sc->source->data[i]; if (ch == '}' && bracket) { i++; bracket = 0; break; } if (num_var) { if (ch >= '0' && ch <= '9') { n = n * 10 + (ch - '0'); continue; } if (ch == '&') { i++; name.len++; } break; } /* not a number variable like $1, $2, etc */ if ((ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9') || ch == '_') { continue; } break; } if (bracket) { ngx_log_error(NGX_LOG_ERR, sc->cf->log, 0, "the closing bracket in \"%V\" " "variable is missing", &name); return NGX_ERROR; } if (name.len == 0) { goto invalid_variable; } if (num_var) { dd("found numbered capturing variable \"%.*s\"", (int) name.len, name.data); sc->capture_variables++; if (ngx_http_replace_script_add_capture_code(sc, n) != NGX_OK) { return NGX_ERROR; } } else { sc->nginx_variables++; if (ngx_http_replace_script_add_var_code(sc, &name) != NGX_OK) { return NGX_ERROR; } } continue; } name.data = &sc->source->data[i]; while (i < sc->source->len) { if (sc->source->data[i] == '$') { break; } i++; name.len++; } sc->size += name.len; if (ngx_http_replace_script_add_copy_code(sc, &name, (i == sc->source->len)) != NGX_OK) { return NGX_ERROR; } } return ngx_http_replace_script_done(sc); invalid_variable: ngx_log_error(NGX_LOG_ERR, sc->cf->log, 0, "replace script: invalid capturing variable name found " "in \"%V\"", sc->source); return NGX_ERROR; } static ngx_int_t ngx_http_replace_script_add_copy_code(ngx_http_replace_script_compile_t *sc, ngx_str_t *value, ngx_uint_t last) { size_t size, len; ngx_http_replace_script_copy_code_t *code; len = value->len; code = ngx_http_replace_script_add_code(*sc->lengths, sizeof(ngx_http_replace_script_copy_code_t)); if (code == NULL) { return NGX_ERROR; } code->code = (ngx_http_replace_script_code_pt) ngx_http_replace_script_copy_len_code; code->len = len; size = (sizeof(ngx_http_replace_script_copy_code_t) + len + sizeof(uintptr_t) - 1) & ~(sizeof(uintptr_t) - 1); code = ngx_http_replace_script_add_code(*sc->values, size); if (code == NULL) { return NGX_ERROR; } code->code = (ngx_http_replace_script_code_pt) ngx_http_replace_script_copy_code; code->len = len; ngx_memcpy((u_char *) code + sizeof(ngx_http_replace_script_copy_code_t), value->data, value->len); return NGX_OK; } static size_t ngx_http_replace_script_copy_len_code(ngx_http_replace_script_engine_t *e) { ngx_http_replace_script_copy_code_t *code; code = (ngx_http_replace_script_copy_code_t *) e->ip; e->ip += sizeof(ngx_http_replace_script_copy_code_t); return code->len; } static size_t ngx_http_replace_script_copy_code(ngx_http_replace_script_engine_t *e) { u_char *p; ngx_http_replace_script_copy_code_t *code; code = (ngx_http_replace_script_copy_code_t *) e->ip; p = e->pos; if (!e->skip) { e->pos = ngx_copy(p, e->ip + sizeof(ngx_http_replace_script_copy_code_t), code->len); } e->ip += sizeof(ngx_http_replace_script_copy_code_t) + ((code->len + sizeof(uintptr_t) - 1) & ~(sizeof(uintptr_t) - 1)); ngx_log_debug2(NGX_LOG_DEBUG_HTTP, e->request->connection->log, 0, "replace script copy: \"%*s\"", e->pos - p, p); return 0; } static ngx_int_t ngx_http_replace_script_add_capture_code(ngx_http_replace_script_compile_t *sc, ngx_uint_t n) { ngx_http_replace_script_capture_code_t *code; code = ngx_http_replace_script_add_code(*sc->lengths, sizeof(ngx_http_replace_script_capture_code_t)); if (code == NULL) { return NGX_ERROR; } code->code = (ngx_http_replace_script_code_pt) ngx_http_replace_script_copy_capture_len_code; code->n = 2 * n; code = ngx_http_replace_script_add_code(*sc->values, sizeof(ngx_http_replace_script_capture_code_t)); if (code == NULL) { return NGX_ERROR; } code->code = (ngx_http_replace_script_code_pt) ngx_http_replace_script_copy_capture_code; code->n = 2 * n; return NGX_OK; } static size_t ngx_http_replace_script_copy_capture_len_code( ngx_http_replace_script_engine_t *e) { sre_int_t *cap; ngx_uint_t n; ngx_http_replace_script_capture_code_t *code; code = (ngx_http_replace_script_capture_code_t *) e->ip; e->ip += sizeof(ngx_http_replace_script_capture_code_t); n = code->n; dd("group index: %d, ncaptures: %d", (int) n, (int) e->ncaptures); if (n + 1 < e->ncaptures) { cap = e->captures; return cap[n + 1] - cap[n]; } return 0; } static size_t ngx_http_replace_script_copy_capture_code(ngx_http_replace_script_engine_t *e) { sre_int_t *cap, from, to, len; u_char *p; #if (NGX_DEBUG) u_char *pos; #endif ngx_uint_t n; ngx_chain_t *cl; ngx_http_replace_script_capture_code_t *code; code = (ngx_http_replace_script_capture_code_t *) e->ip; e->ip += sizeof(ngx_http_replace_script_capture_code_t); n = code->n; #if (NGX_DEBUG) pos = e->pos; #endif if (n < e->ncaptures) { cap = e->captures; from = cap[n]; to = cap[n + 1]; dd("captures data: %p", e->captures_data); for (cl = e->captures_data; cl; cl = cl->next) { if (from >= cl->buf->file_last) { continue; } /* from < cl->buf->file_last */ if (to <= cl->buf->file_pos) { break; } p = cl->buf->pos + (from - cl->buf->file_pos); len = ngx_min(cl->buf->file_last, to) - from; e->pos = ngx_copy(e->pos, p, len); from += len; } } #if (NGX_DEBUG) ngx_log_debug2(NGX_LOG_DEBUG_HTTP, e->request->connection->log, 0, "replace script capture: \"%*s\"", e->pos - pos, pos); #endif return 0; } static ngx_int_t ngx_http_replace_script_init_arrays(ngx_http_replace_script_compile_t *sc) { ngx_uint_t n; if (*sc->lengths == NULL) { n = sc->capture_variables * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_capture_code_t)) + sc->nginx_variables * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_var_code_t)) + sizeof(uintptr_t); *sc->lengths = ngx_array_create(sc->cf->pool, n, 1); if (*sc->lengths == NULL) { return NGX_ERROR; } } if (*sc->values == NULL) { n = sc->capture_variables * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_capture_code_t)) + sc->nginx_variables * (2 * sizeof(ngx_http_replace_script_copy_code_t) + sizeof(ngx_http_replace_script_var_code_t)) + sizeof(uintptr_t); *sc->values = ngx_array_create(sc->cf->pool, n, 1); if (*sc->values == NULL) { return NGX_ERROR; } } sc->nginx_variables = 0; sc->capture_variables = 0; return NGX_OK; } static ngx_int_t ngx_http_replace_script_done(ngx_http_replace_script_compile_t *sc) { uintptr_t *code; code = ngx_http_replace_script_add_code(*sc->lengths, sizeof(uintptr_t)); if (code == NULL) { return NGX_ERROR; } *code = (uintptr_t) NULL; code = ngx_http_replace_script_add_code(*sc->values, sizeof(uintptr_t)); if (code == NULL) { return NGX_ERROR; } *code = (uintptr_t) NULL; return NGX_OK; } static void * ngx_http_replace_script_add_code(ngx_array_t *codes, size_t size) { return ngx_array_push_n(codes, size); } static ngx_int_t ngx_http_replace_script_add_var_code(ngx_http_replace_script_compile_t *sc, ngx_str_t *name) { ngx_int_t index; ngx_http_replace_script_var_code_t *code; index = ngx_http_get_variable_index(sc->cf, name); if (index == NGX_ERROR) { return NGX_ERROR; } code = ngx_http_replace_script_add_code(*sc->lengths, sizeof(ngx_http_replace_script_var_code_t)); if (code == NULL) { return NGX_ERROR; } code->code = (ngx_http_replace_script_code_pt) ngx_http_replace_script_copy_var_len_code; code->index = (uintptr_t) index; code = ngx_http_replace_script_add_code(*sc->values, sizeof(ngx_http_replace_script_var_code_t)); if (code == NULL) { return NGX_ERROR; } code->code = (ngx_http_replace_script_code_pt) ngx_http_replace_script_copy_var_code; code->index = (uintptr_t) index; return NGX_OK; } static size_t ngx_http_replace_script_copy_var_len_code(ngx_http_replace_script_engine_t *e) { ngx_http_variable_value_t *value; ngx_http_replace_script_var_code_t *code; code = (ngx_http_replace_script_var_code_t *) e->ip; e->ip += sizeof(ngx_http_replace_script_var_code_t); value = ngx_http_get_indexed_variable(e->request, code->index); if (value && !value->not_found) { return value->len; } return 0; } static size_t ngx_http_replace_script_copy_var_code(ngx_http_replace_script_engine_t *e) { u_char *p; ngx_http_variable_value_t *value; ngx_http_replace_script_var_code_t *code; code = (ngx_http_replace_script_var_code_t *) e->ip; e->ip += sizeof(ngx_http_replace_script_var_code_t); if (!e->skip) { value = ngx_http_get_indexed_variable(e->request, code->index); if (value && !value->not_found) { p = e->pos; e->pos = ngx_copy(p, value->data, value->len); ngx_log_debug2(NGX_LOG_DEBUG_HTTP, e->request->connection->log, 0, "http replace script var: \"%*s\"", e->pos - p, p); } } return 0; } static void ngx_http_replace_count_variables(u_char *src, size_t len, ngx_uint_t *ngxvars, ngx_uint_t *capvars) { ngx_uint_t i; unsigned var = 0; u_char c; *ngxvars = 0; *capvars = 0; for (i = 0; i < len; i++) { c = src[i]; if (c == '$') { if (var) { var = 0; } else { var = 1; } } else if (var) { if ((c >= '1' && c <= '9') || c == '&') { (*capvars)++; } else { (*ngxvars)++; } var = 0; } } } /* vi:set ft=c ts=4 sw=4 et fdm=marker: */ ================================================ FILE: src/ngx_http_replace_script.h ================================================ /* * Copyright (C) Yichun Zhang (agentzh) */ #ifndef _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_ #define _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_ #include #include #include #include #include typedef struct { ngx_conf_t *cf; ngx_str_t *source; ngx_array_t **lengths; ngx_array_t **values; ngx_uint_t capture_variables; /* captures $1, $2, etc */ ngx_uint_t nginx_variables; /* nginx variables */ ngx_uint_t size; } ngx_http_replace_script_compile_t; typedef struct { ngx_str_t value; void *lengths; void *values; ngx_uint_t capture_variables; } ngx_http_replace_complex_value_t; typedef struct { ngx_conf_t *cf; ngx_str_t *value; ngx_http_replace_complex_value_t *complex_value; } ngx_http_replace_compile_complex_value_t; typedef struct { u_char *ip; u_char *pos; ngx_str_t buf; sre_int_t *captures; ngx_uint_t ncaptures; ngx_chain_t *captures_data; unsigned skip:1; ngx_http_request_t *request; } ngx_http_replace_script_engine_t; typedef size_t (*ngx_http_replace_script_code_pt) (ngx_http_replace_script_engine_t *e); typedef size_t (*ngx_http_replace_script_len_code_pt) (ngx_http_replace_script_engine_t *e); typedef struct { ngx_http_replace_script_code_pt code; uintptr_t len; } ngx_http_replace_script_copy_code_t; typedef struct { ngx_http_replace_script_code_pt code; uintptr_t n; } ngx_http_replace_script_capture_code_t; typedef struct { ngx_http_replace_script_code_pt code; uintptr_t index; } ngx_http_replace_script_var_code_t; ngx_int_t ngx_http_replace_compile_complex_value( ngx_http_replace_compile_complex_value_t *ccv); ngx_int_t ngx_http_replace_complex_value(ngx_http_request_t *r, ngx_chain_t *captured, sre_uint_t ncaps, sre_int_t *cap, ngx_http_replace_complex_value_t *val, ngx_str_t *value); #endif /* _NGX_HTTP_REPLACE_SCRIPT_H_INCLUDED_ */ /* vi:set ft=c ts=4 sw=4 et fdm=marker: */ ================================================ FILE: src/ngx_http_replace_util.c ================================================ /* * Copyright (C) Yichun Zhang (agentzh) */ #ifndef DDEBUG #define DDEBUG 0 #endif #include "ddebug.h" #include "ngx_http_replace_util.h" ngx_chain_t * ngx_http_replace_get_free_buf(ngx_pool_t *p, ngx_chain_t **free) { ngx_chain_t *cl; cl = ngx_chain_get_free_buf(p, free); if (cl == NULL) { return cl; } ngx_memzero(cl->buf, sizeof(ngx_buf_t)); cl->buf->tag = (ngx_buf_tag_t) &ngx_http_replace_filter_module; return cl; } ngx_int_t ngx_http_replace_split_chain(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t **pa, ngx_chain_t ***plast_a, sre_int_t split, ngx_chain_t **pb, ngx_chain_t ***plast_b, unsigned b_sane) { sre_int_t file_last; ngx_chain_t *cl, *newcl, **ll; #if 0 b_sane = 0; #endif ll = pa; for (cl = *pa; cl; ll = &cl->next, cl = cl->next) { if (cl->buf->file_last > split) { /* found an overlap */ if (cl->buf->file_pos < split) { dd("adjust cl buf (b_sane=%d): \"%.*s\"", b_sane, (int) ngx_buf_size(cl->buf), cl->buf->pos); file_last = cl->buf->file_last; cl->buf->last -= file_last - split; cl->buf->file_last = split; dd("adjusted cl buf (next=%p): %.*s", cl->next, (int) ngx_buf_size(cl->buf), cl->buf->pos); /* build the b chain */ if (b_sane) { newcl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (newcl == NULL) { return NGX_ERROR; } newcl->buf->memory = 1; newcl->buf->pos = cl->buf->last; newcl->buf->last = cl->buf->last + file_last - split; newcl->buf->file_pos = split; newcl->buf->file_last = file_last; newcl->next = cl->next; *pb = newcl; if (plast_b) { if (cl->next) { *plast_b = *plast_a; } else { *plast_b = &newcl->next; } } } else { *pb = cl->next; if (plast_b) { *plast_b = *plast_a; } } /* truncate the a chain */ *plast_a = &cl->next; cl->next = NULL; return NGX_OK; } /* build the b chain */ *pb = cl; if (plast_b) { *plast_b = *plast_a; } /* truncate the a chain */ *plast_a = ll; *ll = NULL; return NGX_OK; } } /* missed */ *pb = NULL; if (plast_b) { *plast_b = pb; } return NGX_OK; } ngx_int_t ngx_http_replace_new_pending_buf(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, sre_int_t from, sre_int_t to, ngx_chain_t **out) { size_t len; ngx_buf_t *b; ngx_chain_t *cl; ngx_http_replace_loc_conf_t *rlcf; if (from < ctx->stream_pos) { from = ctx->stream_pos; } len = (size_t) (to - from); if (len == 0) { return NGX_ERROR; } ctx->total_buffered += len; rlcf = ngx_http_get_module_loc_conf(r, ngx_http_replace_filter_module); if (ctx->total_buffered > rlcf->max_buffered_size) { #if 1 ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, "replace filter: exceeding " "replace_filter_max_buffered_size (%uz): %uz", rlcf->max_buffered_size, ctx->total_buffered); return NGX_BUSY; #endif } cl = ngx_http_replace_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; } b = cl->buf; b->temporary = 1; /* abuse the file_pos and file_last fields here */ b->file_pos = from; b->file_last = to; b->start = ngx_palloc(r->pool, len); if (b->start == NULL) { return NGX_ERROR; } b->end = b->start + len; b->pos = b->start; b->last = ngx_copy(b->pos, ctx->buf->pos + from - ctx->stream_pos, len); dd("buffered pending data: stream_pos=%ld (%ld, %ld): %.*s", (long) ctx->stream_pos, (long) from, (long) to, (int) len, ctx->buf->pos + from - ctx->stream_pos); *out = cl; return NGX_OK; } #if (DDEBUG) void ngx_http_replace_dump_chain(const char *prefix, ngx_chain_t **pcl, ngx_chain_t **last) { ngx_chain_t *cl; if (*pcl == NULL) { dd("%s buf empty", prefix); if (last && last != pcl) { dd("BAD last %s", prefix); assert(0); } } for (cl = *pcl; cl; cl = cl->next) { dd("%s buf: \"%.*s\"", prefix, (int) ngx_buf_size(cl->buf), cl->buf->pos); if (cl->next == NULL) { if (last && last != &cl->next) { dd("BAD last %s", prefix); assert(0); } } } } #endif /* DDEBUG */ ================================================ FILE: src/ngx_http_replace_util.h ================================================ #ifndef _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_ #define _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_ #include "ngx_http_replace_filter_module.h" ngx_chain_t *ngx_http_replace_get_free_buf(ngx_pool_t *p, ngx_chain_t **free); ngx_int_t ngx_http_replace_split_chain(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, ngx_chain_t **pa, ngx_chain_t ***plast_a, sre_int_t split, ngx_chain_t **pb, ngx_chain_t ***plast_b, unsigned b_sane); ngx_int_t ngx_http_replace_new_pending_buf(ngx_http_request_t *r, ngx_http_replace_ctx_t *ctx, sre_int_t from, sre_int_t to, ngx_chain_t **out); #if (DDEBUG) void ngx_http_replace_dump_chain(const char *prefix, ngx_chain_t **pcl, ngx_chain_t **last); #endif #endif /* _NGX_HTTP_REPLACE_UTIL_H_INCLUDED_ */ ================================================ FILE: t/01-sanity.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4 + 1); our $StapOutputChains = <<'_EOC_'; global active F(ngx_http_handler) { active = 1 } /* F(ngx_http_write_filter) { if (active && pid() == target()) { printf("http writer filter: %s\n", ngx_chain_dump($in)) } } */ F(ngx_http_chunked_body_filter) { if (active && pid() == target()) { printf("http chunked filter: %s\n", ngx_chain_dump($in)) } } F(ngx_http_replace_output) { if (active && pid() == target()) { printf("http replace output: %s\n", ngx_chain_dump($ctx->out)) } } probe syscall.writev { if (active && pid() == target()) { printf("writev(%s)", ngx_iovec_dump($vec, $vlen)) /* for (i = 0; i < $vlen; i++) { printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len))) } */ } } probe syscall.writev.return { if (active && pid() == target()) { printf(" = %s\n", retstr) } } _EOC_ #no_diff(); #no_long_string(); run_tests(); __DATA__ === TEST 1: ambiguous pattern --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter abcabd X; } --- request GET /t --- stap F(ngx_http_replace_non_capturing_parse) { println("non capturing parse") } F(ngx_http_replace_capturing_parse) { println("capturing parse") } F(ngx_http_replace_complex_value) { println("complex value") } --- stap_out_like chop ^(non capturing parse\n)+$ --- response_body abcXe --- no_error_log [alert] [error] === TEST 2: ambiguous pattern --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n ababac; replace_filter abac X; } --- request GET /t --- response_body chop abX --- no_error_log [alert] [error] === TEST 3: alt --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abc; replace_filter 'ab|abc' X; } --- request GET /t --- response_body Xc --- no_error_log [alert] [error] === TEST 4: caseless --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcaBde; replace_filter abCabd X i; } --- request GET /t --- response_body abcXe --- no_error_log [alert] [error] === TEST 5: case sensitive (no match) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcaBde; replace_filter abCabd X; } --- request GET /t --- response_body abcabcaBde --- no_error_log [alert] [error] === TEST 6: 1-byte chain bufs --- config default_type text/html; replace_filter_max_buffered_size 3; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() } --- response_body abXd --- no_error_log [alert] [error] === TEST 7: 2-byte chain bufs --- config default_type text/html; replace_filter_max_buffered_size 2; location = /t { echo -n ab; echo -n ab; echo -n ac; echo d; replace_filter abac X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body abXd --- no_error_log [alert] [error] === TEST 8: 3-byte chain bufs --- config default_type text/html; replace_filter_max_buffered_size 3; location = /t { echo -n aba; echo -n bac; echo d; replace_filter abac X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body abXd --- no_error_log [alert] [error] === TEST 9: 3-byte chain bufs (more) --- config default_type text/html; replace_filter_max_buffered_size 4; location = /t { echo -n aba; echo -n bac; echo d; replace_filter abacd X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body abX --- no_error_log [alert] [error] === TEST 10: once by default (1st char matched) --- config replace_filter_max_buffered_size 0; default_type text/html; location /t { echo abcabcabde; replace_filter a X; } --- request GET /t --- response_body Xbcabcabde --- no_error_log [alert] [error] === TEST 11: once by default (2nd char matched) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter b X; } --- request GET /t --- response_body aXcabcabde --- no_error_log [alert] [error] === TEST 12: global substitution --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo bbc; replace_filter b X g; } --- request GET /t --- response_body XXc --- no_error_log [alert] [error] === TEST 13: global substitution --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter b X g; } --- request GET /t --- response_body aXcaXcaXde --- no_error_log [alert] [error] === TEST 14: global substitution (empty captures) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n abcabcabde; replace_filter [0-9]* X g; } --- request GET /t --- response_body chop XaXbXcXaXbXcXaXbXdXeX --- no_error_log [alert] [error] === TEST 15: global substitution (empty captures, splitted) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n ab; echo -n cab; echo -n c; echo -n abde; replace_filter [0-9]* X g; } --- request GET /t --- response_body chop XaXbXcXaXbXcXaXbXdXeX --- no_error_log [alert] [error] === TEST 16: global substitution (\d+) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo "hello1234, 56 world"; replace_filter \d+ X g; } --- request GET /t --- response_body helloX, X world --- no_error_log [alert] [error] === TEST 17: replace_filter_types default to text/html --- config default_type text/plain; location /t { echo abc; replace_filter b X; } --- request GET /t --- response_body abc --- no_error_log [alert] [error] === TEST 18: custom replace_filter_types --- config default_type text/plain; location /t { echo abc; replace_filter b X; replace_filter_types text/plain; } --- request GET /t --- response_body aXc --- no_error_log [alert] [error] === TEST 19: multiple replace_filter_types settings --- config default_type text/plain; location /t { echo abc; replace_filter b X; replace_filter_types text/css text/plain; } --- request GET /t --- response_body aXc --- no_error_log [alert] [error] === TEST 20: trim leading spaces --- config replace_filter_max_buffered_size 0; default_type text/html; location /a.html { replace_filter '^\s+' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /a.html --- response_body hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 21: trim trailing spaces --- config default_type text/html; replace_filter_max_buffered_size 0; location /a.html { replace_filter '\s+$' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /a.html --- response_body chop hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 22: trim both leading and trailing spaces --- config replace_filter_max_buffered_size 0; default_type text/html; location /a.html { replace_filter '^\s+|\s+$' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /a.html --- response_body chop hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 23: pure flush buf in the stream (no data) --- config replace_filter_max_buffered_size 0; default_type text/html; location = /t { echo_flush; replace_filter 'a' 'X' g; } --- request GET /t --- response_body chop --- no_error_log [alert] [error] === TEST 24: pure flush buf in the stream (with data) --- config replace_filter_max_buffered_size 0; default_type text/html; location = /t { echo a; echo_flush; replace_filter 'a' 'X' g; } --- request GET /t --- stap3 eval: $::StapOutputChains --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:539") { printf("chain: %s", ngx_chain_dump($ctx->busy)) //print_ubacktrace() } --- response_body X --- no_error_log [alert] [error] === TEST 25: trim both leading and trailing spaces (1 byte at a time) --- config default_type text/html; replace_filter_max_buffered_size 1; location = /t { echo -n 'a'; echo ' '; echo "b"; replace_filter '^\s+|\s+$' '' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body chop a b --- no_error_log [alert] [error] === TEST 26: trim both leading and trailing spaces (1 byte at a time), no \s for $ --- config replace_filter_max_buffered_size 1; default_type text/html; location = /t { echo -n 'a'; echo ' '; echo "b"; replace_filter '^\s+| +$' '' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a b --- no_error_log [alert] [error] === TEST 27: trim both leading and trailing spaces (1 byte at a time) --- config replace_filter_max_buffered_size 4; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '^\s+|\s+$' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- request GET /t --- response_body chop hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 28: \b at the border --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n a; echo b; replace_filter '\bb|a' X g; } --- request GET /t --- response_body Xb --- no_error_log [alert] [error] === TEST 29: \B at the border --- config replace_filter_max_buffered_size 0; default_type text/html; location /t { echo -n a; echo ','; replace_filter '\B,|a' X g; } --- request GET /t --- response_body X, --- no_error_log [alert] [error] === TEST 30: \A at the border --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n a; echo 'b'; replace_filter '\Ab|a' X g; } --- request GET /t --- response_body Xb --- no_error_log [alert] [error] === TEST 31: memory bufs with last_buf=1 --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { return 200 "abc"; replace_filter \w+ X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body chop X --- no_error_log [alert] [error] === TEST 32: trim both leading and trailing spaces (2 bytes at a time) --- config default_type text/html; replace_filter_max_buffered_size 4; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body local len = string.len(txt) i = 1 while i <= len do if i == len then ngx.print(string.sub(txt, i, i)) i = i + 1 else ngx.print(string.sub(txt, i, i + 1)) i = i + 2 end ngx.flush(true) end '; replace_filter '^\s+|\s+$' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 eval: $::StapOutputChains --- request GET /t --- response_body chop hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 33: trim both leading and trailing spaces (3 bytes at a time) --- config replace_filter_max_buffered_size 2; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body local len = string.len(txt) i = 1 while i <= len do if i == len then ngx.print(string.sub(txt, i, i)) i = i + 1 elseif i == len - 1 then ngx.print(string.sub(txt, i, i + 1)) i = i + 2 else ngx.print(string.sub(txt, i, i + 2)) i = i + 3 end ngx.flush(true) end '; replace_filter '^\s+|\s+$' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 eval: $::StapOutputChains --- request GET /t --- response_body chop hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 34: github issue #2: error "general look-ahead not supported" --- config replace_filter_max_buffered_size 3; location /t { charset utf-8; default_type text/html; echo "ABCabcABCabc"; #replace_filter_types text/plain; replace_filter "a.+a" "X" "ig"; } --- request GET /t --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") { print_ubacktrace() } --- response_body Xbc --- no_error_log [alert] [error] === TEST 35: backtrack to the middle of a pending capture (pending: output|capture + rematch) --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n ab; echo -n c; echo d; replace_filter 'abce|b' 'X' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- no_error_log [alert] [error] === TEST 36: backtrack to the middle of a pending capture (pending: output + capture|rematch --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n a; echo -n bc; echo d; replace_filter 'abce|b' 'X' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- no_error_log [alert] [error] === TEST 37: backtrack to the middle of a pending capture (pending: output + capture + rematch --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n a; echo -n b; echo -n c; echo d; replace_filter 'abce|b' 'X' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1522") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- no_error_log [alert] [error] === TEST 38: backtrack to the middle of a pending capture (pending: output|capture|rematch --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n abc; echo d; replace_filter 'abce|b' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- no_error_log [alert] [error] === TEST 39: backtrack to the middle of a pending capture (pending: output|capture|rematch(2) --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n abcc; echo d; replace_filter 'abcce|b' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXccd --- no_error_log [alert] [error] === TEST 40: backtrack to the middle of a pending capture (pending: output|capture(2)|rematch --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n abbc; echo d; replace_filter 'abbce|bb' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- no_error_log [alert] [error] === TEST 41: backtrack to the middle of a pending capture (pending: output(2)|capture|rematch --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n aabc; echo d; replace_filter 'aabce|b' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aaXcd --- no_error_log [alert] [error] === TEST 42: backtrack to the beginning of a pending capture (pending: output + capture|rematch(2) --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n a; echo -n bcc; echo d; replace_filter 'abcce|b' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXccd --- no_error_log [alert] [error] === TEST 43: backtrack to the beginning of a pending capture (pending: output + capture(2)|rematch --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n a; echo -n bbc; echo d; replace_filter 'abbce|bb' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- no_error_log [alert] [error] === TEST 44: backtrack to the middle of a pending capture (pending: output(2) + capture|rematch --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n aa; echo -n bc; echo d; replace_filter 'aabce|b' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aaXcd --- no_error_log [alert] [error] === TEST 45: assertions across AGAIN --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n a; echo -n "\n"; echo b; replace_filter 'a\n^b' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body X --- no_error_log [alert] [error] === TEST 46: assertions when capture backtracking happens --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n a; echo -n b; echo -n c; echo -n d; echo f; #echo abcdf; replace_filter 'abcde|b|\bc' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcdf --- no_error_log [alert] [error] === TEST 47: assertions when capture backtracking happens (2 pending matches) --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n a; echo -n b; echo -n ' '; echo -n d; echo f; #echo ab df; replace_filter 'ab de|b|b |\b ' 'X' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXXdf --- no_error_log [alert] [error] === TEST 48: github issue #2: error "general look-ahead not supported", no "g" --- config replace_filter_max_buffered_size 3; location /t { charset utf-8; default_type text/html; echo "ABCabcABCabc"; #replace_filter_types text/plain; replace_filter "a.+a" "X" "i"; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body Xbc --- no_error_log [alert] [error] === TEST 49: nested rematch bufs --- config replace_filter_max_buffered_size 4; location /t { default_type text/html; echo -n a; echo -n b; echo -n c; echo -n d; echo -n e; echo g; #echo abcdeg; replace_filter 'abcdef|b|cdf|c' X g; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body aXXdeg --- no_error_log [alert] [error] === TEST 50: nested rematch bufs (splitting pending buf) --- config replace_filter_max_buffered_size 6; location /t { default_type text/html; echo -n a; echo -n b; echo -n cd; echo -n e; echo -n f; echo -n g; echo i; #echo abcdefh; replace_filter 'abcdefgh|b|cdeg|d' X g; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body aXcXefgi --- no_error_log [alert] [error] === TEST 51: remove C/C++ comments (1 byte at a time) --- config replace_filter_max_buffered_size 42; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '/\*.*?\*/|//[^\n]*' '' g; } --- user_files >>> a.html i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /t --- response_body eval " i don't know hello world blah */ b " --- no_error_log [alert] [error] === TEST 52: remove C/C++ comments (all at a time) --- config default_type text/html; replace_filter_max_buffered_size 0; location /a.html { replace_filter '/\*.*?\*/|//[^\n]*' '' g; } --- user_files >>> a.html i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /a.html --- response_body eval " i don't know hello world blah */ b " --- no_error_log [alert] [error] === TEST 53: remove C/C++ comments (all at a time) - server-level config --- config replace_filter_max_buffered_size 0; default_type text/html; replace_filter '/\*.*?\*/|//[^\n]*' '' g; --- user_files >>> a.html i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /a.html --- response_body eval " i don't know hello world blah */ b " --- no_error_log [alert] [error] === TEST 54: multiple replace_filter_types settings (server level) --- config replace_filter_max_buffered_size 0; default_type text/plain; replace_filter_types text/css text/plain; location /t { echo abc; replace_filter b X; } --- request GET /t --- response_body aXc --- no_error_log [alert] [error] === TEST 55: multiple replace_filter_types settings (server level, but overridding in location) --- config replace_filter_max_buffered_size 0; default_type text/plain; replace_filter_types text/css text/plain; location /t { echo abc; replace_filter_types text/javascript; replace_filter b X; } --- request GET /t --- response_body abc --- no_error_log [alert] [error] === TEST 56: do not use replace_filter at all --- config replace_filter_max_buffered_size 0; default_type text/plain; replace_filter_types text/css text/plain; location /t { echo abc; replace_filter_types text/css; } --- request GET /t --- response_body abc --- no_error_log [alert] [error] === TEST 57: bad regex --- config default_type text/html; location /t { echo abc; replace_filter '(a+b' ''; } --- request GET /t --- response_body abc --- no_error_log [alert] [error] --- SKIP === TEST 58: github issue #3: data lost in particular situation --- config replace_filter_max_buffered_size 4; default_type text/html; location /t { default_type text/html; echo "ABCabcABC"; echo "ABCabcABC"; #echo "ABCabcABC\nABCabcABC"; replace_filter "(a.+?c){2}" "X" "ig"; } --- request GET /t --- response_body XXABC --- no_error_log [alert] [error] === TEST 59: variation --- config replace_filter_max_buffered_size 5; default_type text/html; location /t { default_type text/html; #echo "ABCabcABC"; #echo "ABCabcABC"; echo "ACacAC ACacAC"; replace_filter "(a.+?c){2}" "X" "ig"; } --- request GET /t --- response_body XacAC --- no_error_log [alert] [error] === TEST 60: nested pending matched --- config replace_filter_max_buffered_size 4; default_type text/html; location /t { default_type text/html; echo -n a; echo -n b; echo -n c; echo -n def; echo -n gh; echo -n i; echo k; #echo abcdefig; replace_filter "abcdefghij|bcdefg|cd" "X" "ig"; } --- request GET /t --- response_body aXhik --- no_error_log [alert] [error] === TEST 61: test split chain with b_sane=1, next=NULL --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n aba; echo -n ba; echo -n bac; echo d; #echo abababacd; replace_filter abacd X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") { print_ubacktrace() } --- response_body ababX --- no_error_log [alert] [error] === TEST 62: test split chain with b_sane=1, next not NULL --- config replace_filter_max_buffered_size 6; default_type text/html; location = /t { echo -n aba; echo -n ba; echo -n ba; echo -n bac; echo d; #echo abababacd; replace_filter ababacd X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") { print_ubacktrace() } --- response_body ababX --- no_error_log [alert] [error] === TEST 63: trim leading spaces (1 byte at a time) --- config replace_filter_max_buffered_size 0; default_type text/html; location /a.html { } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '^\s+' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /t --- response_body hello, world blah yeah hello baby! abc --- no_error_log [alert] [error] === TEST 64: split ctx->pending into ctx->pending and ctx->free --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { #echo "abc\nd"; echo -n a; echo -n b; echo -n c; echo -n "\n"; echo d; replace_filter "abcd|bc\ne|c$" X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1482") { print_ubacktrace() } --- response_body abX d --- no_error_log [alert] [error] === TEST 65: trim both leading and trailing spaces (1 byte at a time) --- config default_type text/html; replace_filter_max_buffered_size 2; location /t { echo -n 'a'; echo_sleep 0.001; echo ' '; echo_sleep 0.001; echo ''; echo_sleep 0.001; echo ' '; echo_sleep 0.001; echo "b"; echo_sleep 0.001; echo " "; replace_filter '^\s+|\s+$' '' g; } location = /main { echo_location_async /t1; echo_location_async /t2; echo_location_async /t3; echo_location_async /t4; echo_location_async /t5; echo_location_async /t6; } --- stap3 eval: $::StapOutputChains --- request GET /main --- response_body a b a b a b a b a b a b --- no_error_log [alert] [error] ================================================ FILE: t/02-max-buffered.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4); our $StapOutputChains = <<'_EOC_'; global active F(ngx_http_handler) { active = 1 } /* F(ngx_http_write_filter) { if (active && pid() == target()) { printf("http writer filter: %s\n", ngx_chain_dump($in)) } } */ F(ngx_http_chunked_body_filter) { if (active && pid() == target()) { printf("http chunked filter: %s\n", ngx_chain_dump($in)) } } F(ngx_http_replace_output) { if (active && pid() == target()) { printf("http replace output: %s\n", ngx_chain_dump($ctx->out)) } } probe syscall.writev { if (active && pid() == target()) { printf("writev(%s)", ngx_iovec_dump($vec, $vlen)) /* for (i = 0; i < $vlen; i++) { printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len))) } */ } } probe syscall.writev.return { if (active && pid() == target()) { printf(" = %s\n", retstr) } } _EOC_ #no_diff(); no_long_string(); run_tests(); __DATA__ === TEST 1: 1-byte chain bufs (0) --- config default_type text/html; replace_filter_max_buffered_size 0; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() } --- response_body ababacd --- error_log replace filter: exceeding replace_filter_max_buffered_size (0): 1 --- no_error_log [error] === TEST 2: 1-byte chain bufs (1) --- config default_type text/html; replace_filter_max_buffered_size 1; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac X; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1439") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- response_body ababacd --- error_log replace filter: exceeding replace_filter_max_buffered_size (1): 2 --- no_error_log [error] === TEST 3: trim both leading and trailing spaces (1 byte at a time) (2) --- config replace_filter_max_buffered_size 2; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '^\s+|\s+$' '' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- request GET /t --- response_body hello, world blah yeah hello baby! abc --- error_log replace filter: exceeding replace_filter_max_buffered_size (2): 3 --- no_error_log [error] === TEST 4: github issue #2: error "general look-ahead not supported" --- config replace_filter_max_buffered_size 0; location /t { charset utf-8; default_type text/html; echo "ABCabcABCabc"; #replace_filter_types text/plain; replace_filter "a.+a" "X" "ig"; } --- request GET /t --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1481") { print_ubacktrace() } --- response_body ABCabcABCabc --- error_log replace filter: exceeding replace_filter_max_buffered_size (0): 2 --- no_error_log [error] === TEST 5: backtrack to the middle of a pending capture (pending: output|capture + rematch) (0) --- config replace_filter_max_buffered_size 0; default_type text/html; location = /t { echo -n ab; echo -n c; echo d; replace_filter 'abce|b' 'X' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body abcd --- error_log replace filter: exceeding replace_filter_max_buffered_size (0): 1 --- no_error_log [error] === TEST 6: backtrack to the middle of a pending capture (pending: output|capture + rematch) (1) --- config replace_filter_max_buffered_size 1; default_type text/html; location = /t { echo -n ab; echo -n c; echo d; replace_filter 'abce|b' 'X' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aXcd --- error_log replace filter: exceeding replace_filter_max_buffered_size (1): 2 --- no_error_log [error] ================================================ FILE: t/03-var.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4 + 3); our $StapOutputChains = <<'_EOC_'; global active F(ngx_http_handler) { active = 1 } /* F(ngx_http_write_filter) { if (active && pid() == target()) { printf("http writer filter: %s\n", ngx_chain_dump($in)) } } */ F(ngx_http_chunked_body_filter) { if (active && pid() == target()) { printf("http chunked filter: %s\n", ngx_chain_dump($in)) } } F(ngx_http_replace_output) { if (active && pid() == target()) { printf("http replace output: %s\n", ngx_chain_dump($ctx->out)) } } probe syscall.writev { if (active && pid() == target()) { printf("writev(%s)", ngx_iovec_dump($vec, $vlen)) /* for (i = 0; i < $vlen; i++) { printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len))) } */ } } probe syscall.writev.return { if (active && pid() == target()) { printf(" = %s\n", retstr) } } _EOC_ #no_diff(); #no_long_string(); run_tests(); __DATA__ === TEST 1: nginx vars (global) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo abc; replace_filter . $foo g; } --- request GET /t --- stap F(ngx_http_replace_non_capturing_parse) { println("non capturing parse") } F(ngx_http_replace_capturing_parse) { println("capturing parse") } --- stap_out_like chop ^(non capturing parse\n)+$ --- response_body chop XXXX --- no_error_log [alert] [error] === TEST 2: nginx vars (non-global) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo abc; replace_filter . $foo; } --- request GET /t --- stap F(ngx_http_replace_non_capturing_parse) { println("non capturing parse") } F(ngx_http_replace_capturing_parse) { println("capturing parse") } --- stap_out_like chop ^(non capturing parse\n)+$ --- response_body Xbc --- no_error_log [alert] [error] === TEST 3: undefined nginx vars --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abc; replace_filter . $foo; } --- request GET /t --- response_body Xbc --- no_error_log [alert] [error] --- SKIP === TEST 4: use of capturing variables --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abc; replace_filter . $1; } --- request GET /t --- response_body Xbc --- no_error_log [alert] [error] --- SKIP === TEST 5: more contexts --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo abc; replace_filter . "[$foo]"; } --- request GET /t --- response_body [X]bc --- no_error_log [alert] [error] === TEST 6: more nginx vars --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; set $bar Y; echo abc; replace_filter . "[$foo,$bar]"; } --- request GET /t --- response_body [X,Y]bc --- no_error_log [alert] [error] === TEST 7: various lengths of nginx var values --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo XYZ; set $bar ""; echo abc; replace_filter . "[$foo,$bar]"; } --- request GET /t --- response_body [XYZ,]bc --- no_error_log [alert] [error] === TEST 8: escaping the dollar sign --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; set $bar Y; echo abc; replace_filter . "[$foo,$$bar]"; } --- request GET /t --- response_body [X,$bar]bc --- no_error_log [alert] [error] === TEST 9: \ is not an escaping sequence --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; set $bar Y; echo abc; replace_filter . "[\$foo,\$bar]"; } --- request GET /t --- response_body [\X,\Y]bc --- no_error_log [alert] [error] === TEST 10: cached subs values --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo abc; replace_filter . "$foo" g; } --- request GET /t --- response_body chop XXXX --- stap F(ngx_http_replace_complex_value) { println("complex value") } --- stap_out complex value --- no_error_log [alert] [error] ================================================ FILE: t/04-capturing.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4 + 1); our $StapOutputChains = <<'_EOC_'; global active F(ngx_http_handler) { active = 1 } /* F(ngx_http_write_filter) { if (active && pid() == target()) { printf("http writer filter: %s\n", ngx_chain_dump($in)) } } */ F(ngx_http_chunked_body_filter) { if (active && pid() == target()) { printf("http chunked filter: %s\n", ngx_chain_dump($in)) } } F(ngx_http_replace_output) { if (active && pid() == target()) { printf("http replace output: %s\n", ngx_chain_dump($ctx->out)) } } probe syscall.writev { if (active && pid() == target()) { printf("writev(%s)", ngx_iovec_dump($vec, $vlen)) /* for (i = 0; i < $vlen; i++) { printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len))) } */ } } probe syscall.writev.return { if (active && pid() == target()) { printf(" = %s\n", retstr) } } _EOC_ #no_diff(); #no_long_string(); run_tests(); __DATA__ === TEST 1: ambiguous pattern --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter abcabd "[$&]"; } --- request GET /t --- response_body abc[abcabd]e --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 2: ambiguous pattern --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n ababac; replace_filter abac "[$&]"; } --- request GET /t --- response_body chop ab[abac] --- no_error_log [alert] [error] === TEST 3: alt --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abc; replace_filter 'ab|abc' [$&]; } --- request GET /t --- response_body [ab]c --- no_error_log [alert] [error] === TEST 4: caseless --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcaBde; replace_filter abCabd [$&] i; } --- request GET /t --- response_body abc[abcaBd]e --- no_error_log [alert] [error] === TEST 5: case sensitive (no match) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcaBde; replace_filter abCabd [$&]; } --- request GET /t --- response_body abcabcaBde --- no_error_log [alert] [error] === TEST 6: 1-byte chain bufs --- config default_type text/html; replace_filter_max_buffered_size 3; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() } --- response_body ab[abac]d --- no_error_log [alert] [error] === TEST 7: 2-byte chain bufs --- config default_type text/html; replace_filter_max_buffered_size 2; location = /t { echo -n ab; echo -n ab; echo -n ac; echo d; replace_filter abac [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body ab[abac]d --- no_error_log [alert] [error] === TEST 8: 3-byte chain bufs --- config default_type text/html; replace_filter_max_buffered_size 3; location = /t { echo -n aba; echo -n bac; echo d; replace_filter abac [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body ab[abac]d --- no_error_log [alert] [error] === TEST 9: 3-byte chain bufs (more) --- config default_type text/html; replace_filter_max_buffered_size 4; location = /t { echo -n aba; echo -n bac; echo d; replace_filter abacd [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body ab[abacd] --- no_error_log [alert] [error] === TEST 10: once by default (1st char matched) --- config replace_filter_max_buffered_size 0; default_type text/html; location /t { echo abcabcabde; replace_filter a [$&]; } --- request GET /t --- response_body [a]bcabcabde --- no_error_log [alert] [error] === TEST 11: once by default (2nd char matched) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter b [$&]; } --- request GET /t --- response_body a[b]cabcabde --- no_error_log [alert] [error] === TEST 12: global substitution --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo bbc; replace_filter b [$&] g; } --- request GET /t --- response_body [b][b]c --- no_error_log [alert] [error] === TEST 13: global substitution --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter b [$&] g; } --- request GET /t --- response_body a[b]ca[b]ca[b]de --- no_error_log [alert] [error] === TEST 14: global substitution (empty captures) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n abcabcabde; replace_filter [0-9]* [$&] g; } --- request GET /t --- response_body chop []a[]b[]c[]a[]b[]c[]a[]b[]d[]e[] --- no_error_log [alert] [error] === TEST 15: global substitution (empty captures, splitted) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n ab; echo -n cab; echo -n c; echo -n abde; replace_filter [0-9]* [$&] g; } --- request GET /t --- response_body chop []a[]b[]c[]a[]b[]c[]a[]b[]d[]e[] --- no_error_log [alert] [error] === TEST 16: global substitution (\d+) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo "hello1234, 56 world"; replace_filter \d+ [$&] g; } --- request GET /t --- response_body hello[1234], [56] world --- no_error_log [alert] [error] === TEST 17: replace_filter_types default to text/html --- config default_type text/plain; location /t { echo abc; replace_filter b [$&]; } --- request GET /t --- response_body abc --- no_error_log [alert] [error] === TEST 18: custom replace_filter_types --- config default_type text/plain; location /t { echo abc; replace_filter b [$&]; replace_filter_types text/plain; } --- request GET /t --- response_body a[b]c --- no_error_log [alert] [error] === TEST 19: multiple replace_filter_types settings --- config default_type text/plain; location /t { echo abc; replace_filter b [$&]; replace_filter_types text/css text/plain; } --- request GET /t --- response_body a[b]c --- no_error_log [alert] [error] === TEST 20: trim leading spaces --- config replace_filter_max_buffered_size 0; default_type text/html; location /a.html { replace_filter '^\s+' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /a.html --- response_body [ ]hello, world blah yeah hello [ ]baby! [ ]abc --- no_error_log [alert] [error] === TEST 21: trim trailing spaces --- config default_type text/html; replace_filter_max_buffered_size 0; location /a.html { replace_filter '\s+$' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /a.html --- response_body chop hello, world[ ] blah yeah hello[ ] baby![ ] abc[ ] --- no_error_log [alert] [error] === TEST 22: trim both leading and trailing spaces --- config replace_filter_max_buffered_size 0; default_type text/html; location /a.html { replace_filter '^\s+|\s+$' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /a.html --- response_body chop [ ]hello, world[ ] blah yeah hello[ ] [ ]baby! [ ]abc[ ] --- no_error_log [alert] [error] === TEST 23: pure flush buf in the stream (no data) --- config replace_filter_max_buffered_size 0; default_type text/html; location = /t { echo_flush; replace_filter 'a' '[$&]' g; } --- request GET /t --- response_body chop --- no_error_log [alert] [error] === TEST 24: pure flush buf in the stream (with data) --- config replace_filter_max_buffered_size 0; default_type text/html; location = /t { echo a; echo_flush; replace_filter 'a' '[$&]' g; } --- request GET /t --- stap3 eval: $::StapOutputChains --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:539") { printf("chain: %s", ngx_chain_dump($ctx->busy)) //print_ubacktrace() } --- response_body [a] --- no_error_log [alert] [error] === TEST 25: trim both leading and trailing spaces (1 byte at a time) --- config default_type text/html; replace_filter_max_buffered_size 2; location = /t { echo -n 'a'; echo ' '; echo "b"; replace_filter '^\s+|\s+$' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body chop a[ ] b[ ] --- no_error_log [alert] [error] === TEST 26: trim both leading and trailing spaces (1 byte at a time), no \s for $ --- config replace_filter_max_buffered_size 1; default_type text/html; location = /t { echo -n 'a'; echo ' '; echo "b"; replace_filter '^\s+| +$' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[ ] b --- no_error_log [alert] [error] === TEST 27: trim both leading and trailing spaces (1 byte at a time) --- config replace_filter_max_buffered_size 7; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '^\s+|\s+$' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- request GET /t --- response_body chop [ ]hello, world[ ] blah yeah hello[ ] [ ]baby! [ ]abc[ ] --- no_error_log [alert] [error] === TEST 28: \b at the border --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n a; echo b; replace_filter '\bb|a' [$&] g; } --- request GET /t --- response_body [a]b --- no_error_log [alert] [error] === TEST 29: \B at the border --- config replace_filter_max_buffered_size 0; default_type text/html; location /t { echo -n a; echo ','; replace_filter '\B,|a' [$&] g; } --- request GET /t --- response_body [a], --- no_error_log [alert] [error] === TEST 30: \A at the border --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n a; echo 'b'; replace_filter '\Ab|a' [$&] g; } --- request GET /t --- response_body [a]b --- no_error_log [alert] [error] === TEST 31: memory bufs with last_buf=1 --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { return 200 "abc"; replace_filter \w+ [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body chop [abc] --- no_error_log [alert] [error] === TEST 32: trim both leading and trailing spaces (2 bytes at a time) --- config default_type text/html; replace_filter_max_buffered_size 7; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body local len = string.len(txt) i = 1 while i <= len do if i == len then ngx.print(string.sub(txt, i, i)) i = i + 1 else ngx.print(string.sub(txt, i, i + 1)) i = i + 2 end ngx.flush(true) end '; replace_filter '^\s+|\s+$' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 eval: $::StapOutputChains --- request GET /t --- response_body chop [ ]hello, world[ ] blah yeah hello[ ] [ ]baby! [ ]abc[ ] --- no_error_log [alert] [error] === TEST 33: trim both leading and trailing spaces (3 bytes at a time) --- config replace_filter_max_buffered_size 5; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body local len = string.len(txt) i = 1 while i <= len do if i == len then ngx.print(string.sub(txt, i, i)) i = i + 1 elseif i == len - 1 then ngx.print(string.sub(txt, i, i + 1)) i = i + 2 else ngx.print(string.sub(txt, i, i + 2)) i = i + 3 end ngx.flush(true) end '; replace_filter '^\s+|\s+$' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 eval: $::StapOutputChains --- request GET /t --- response_body chop [ ]hello, world[ ] blah yeah hello[ ] [ ]baby! [ ]abc[ ] --- no_error_log [alert] [error] === TEST 34: github issue #2: error "general look-ahead not supported" --- config replace_filter_max_buffered_size 13; location /t { charset utf-8; default_type text/html; echo "ABCabcABCabc"; #replace_filter_types text/plain; replace_filter "a.+a" "[$&]" "ig"; } --- request GET /t --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") { print_ubacktrace() } --- response_body [ABCabcABCa]bc --- no_error_log [alert] [error] === TEST 35: backtrack to the middle of a pending capture (pending: output|capture + rematch) --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n ab; echo -n c; echo d; replace_filter 'abce|b' '[$&]' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]cd --- no_error_log [alert] [error] === TEST 36: backtrack to the middle of a pending capture (pending: output + capture|rematch --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n a; echo -n bc; echo d; replace_filter 'abce|b' '[$&]' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]cd --- no_error_log [alert] [error] === TEST 37: backtrack to the middle of a pending capture (pending: output + capture + rematch --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n a; echo -n b; echo -n c; echo d; replace_filter 'abce|b' '[$&]' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1522") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]cd --- no_error_log [alert] [error] === TEST 38: backtrack to the middle of a pending capture (pending: output|capture|rematch --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { echo -n abc; echo d; replace_filter 'abce|b' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]cd --- no_error_log [alert] [error] === TEST 39: backtrack to the middle of a pending capture (pending: output|capture|rematch(2) --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n abcc; echo d; replace_filter 'abcce|b' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]ccd --- no_error_log [alert] [error] === TEST 40: backtrack to the middle of a pending capture (pending: output|capture(2)|rematch --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n abbc; echo d; replace_filter 'abbce|bb' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[bb]cd --- no_error_log [alert] [error] === TEST 41: backtrack to the middle of a pending capture (pending: output(2)|capture|rematch --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n aabc; echo d; replace_filter 'aabce|b' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aa[b]cd --- no_error_log [alert] [error] === TEST 42: backtrack to the beginning of a pending capture (pending: output + capture|rematch(2) --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n a; echo -n bcc; echo d; replace_filter 'abcce|b' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]ccd --- no_error_log [alert] [error] === TEST 43: backtrack to the beginning of a pending capture (pending: output + capture(2)|rematch --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n a; echo -n bbc; echo d; replace_filter 'abbce|bb' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[bb]cd --- no_error_log [alert] [error] === TEST 44: backtrack to the middle of a pending capture (pending: output(2) + capture|rematch --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n aa; echo -n bc; echo d; replace_filter 'aabce|b' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body aa[b]cd --- no_error_log [alert] [error] === TEST 45: assertions across AGAIN --- config replace_filter_max_buffered_size 2; default_type text/html; location = /t { echo -n a; echo -n "\n"; echo b; replace_filter 'a\n^b' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body [a b] --- no_error_log [alert] [error] === TEST 46: assertions when capture backtracking happens --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n a; echo -n b; echo -n c; echo -n d; echo f; #echo abcdf; replace_filter 'abcde|b|\bc' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b]cdf --- no_error_log [alert] [error] === TEST 47: assertions when capture backtracking happens (2 pending matches) --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n a; echo -n b; echo -n ' '; echo -n d; echo f; #echo ab df; replace_filter 'ab de|b|b |\b ' '[$&]' g; } --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body a[b][ ]df --- no_error_log [alert] [error] === TEST 48: github issue #2: error "general look-ahead not supported", no "g" --- config replace_filter_max_buffered_size 13; location /t { charset utf-8; default_type text/html; echo "ABCabcABCabc"; #replace_filter_types text/plain; replace_filter "a.+a" "[$&]" "i"; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body [ABCabcABCa]bc --- no_error_log [alert] [error] === TEST 49: nested rematch bufs --- config replace_filter_max_buffered_size 5; location /t { default_type text/html; echo -n a; echo -n b; echo -n c; echo -n d; echo -n e; echo g; #echo abcdeg; replace_filter 'abcdef|b|cdf|c' [$&] g; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body a[b][c]deg --- no_error_log [alert] [error] === TEST 50: nested rematch bufs (splitting pending buf) --- config replace_filter_max_buffered_size 7; location /t { default_type text/html; echo -n a; echo -n b; echo -n cd; echo -n e; echo -n f; echo -n g; echo i; #echo abcdefh; replace_filter 'abcdefgh|b|cdeg|d' [$&] g; } --- request GET /t --- stap2 eval: $::StapOutputChains --- response_body a[b]c[d]efgi --- no_error_log [alert] [error] === TEST 51: remove C/C++ comments (1 byte at a time) --- config replace_filter_max_buffered_size 50; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '/\*.*?\*/|//[^\n]*' '[$&]' g; } --- user_files >>> a.html i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /t --- response_body eval " i don't know [// hello // world /* */] hello world [/** abc * b/c /* hello ** // world * */] blah [/* hi */] */ b [//] [///hi] " --- no_error_log [alert] [error] === TEST 52: remove C/C++ comments (all at a time) --- config default_type text/html; replace_filter_max_buffered_size 0; location /a.html { replace_filter '/\*.*?\*/|//[^\n]*' '[$&]' g; } --- user_files >>> a.html i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /a.html --- response_body eval " i don't know [// hello // world /* */] hello world [/** abc * b/c /* hello ** // world * */] blah [/* hi */] */ b [//] [///hi] " --- no_error_log [alert] [error] === TEST 53: remove C/C++ comments (all at a time) - server-level config --- config replace_filter_max_buffered_size 0; default_type text/html; replace_filter '/\*.*?\*/|//[^\n]*' '[$&]' g; --- user_files >>> a.html i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /a.html --- response_body eval " i don't know [// hello // world /* */] hello world [/** abc * b/c /* hello ** // world * */] blah [/* hi */] */ b [//] [///hi] " --- no_error_log [alert] [error] === TEST 54: multiple replace_filter_types settings (server level) --- config replace_filter_max_buffered_size 0; default_type text/plain; replace_filter_types text/css text/plain; location /t { echo abc; replace_filter b [$&]; } --- request GET /t --- response_body a[b]c --- no_error_log [alert] [error] === TEST 55: multiple replace_filter_types settings (server level, but overridding in location) --- config replace_filter_max_buffered_size 0; default_type text/plain; replace_filter_types text/css text/plain; location /t { echo abc; replace_filter_types text/javascript; replace_filter b [$&]; } --- request GET /t --- response_body abc --- no_error_log [alert] [error] === TEST 56: github issue #3: data lost in particular situation --- config replace_filter_max_buffered_size 4; default_type text/html; location /t { default_type text/html; echo "ABCabcABC"; echo "ABCabcABC"; #echo "ABCabcABC\nABCabcABC"; replace_filter "(a.+?c){2}" "[$&]" "ig"; } --- request GET /t --- response_body [ABCabc][ABC ABCabc]ABC --- no_error_log [alert] [error] === TEST 57: variation --- config replace_filter_max_buffered_size 5; default_type text/html; location /t { default_type text/html; #echo "ABCabcABC"; #echo "ABCabcABC"; echo "ACacAC ACacAC"; replace_filter "(a.+?c){2}" "[$&]" "ig"; } --- request GET /t --- response_body [ACacAC AC]acAC --- no_error_log [alert] [error] === TEST 58: nested pending matched --- config replace_filter_max_buffered_size 9; default_type text/html; location /t { default_type text/html; echo -n a; echo -n b; echo -n c; echo -n def; echo -n gh; echo -n i; echo k; #echo abcdefig; replace_filter "abcdefghij|bcdefg|cd" "[$&]" "ig"; } --- request GET /t --- response_body a[bcdefg]hik --- no_error_log [alert] [error] === TEST 59: test split chain with b_sane=1, next=NULL --- config replace_filter_max_buffered_size 4; default_type text/html; location = /t { echo -n aba; echo -n ba; echo -n bac; echo d; #echo abababacd; replace_filter abacd [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") { print_ubacktrace() } --- response_body abab[abacd] --- no_error_log [alert] [error] === TEST 60: test split chain with b_sane=1, next not NULL --- config replace_filter_max_buffered_size 6; default_type text/html; location = /t { echo -n aba; echo -n ba; echo -n ba; echo -n bac; echo d; #echo abababacd; replace_filter ababacd [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1217") { print_ubacktrace() } --- response_body abab[ababacd] --- no_error_log [alert] [error] === TEST 61: trim leading spaces (1 byte at a time) --- config replace_filter_max_buffered_size 6; default_type text/html; location /a.html { } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '^\s+' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- request GET /t --- response_body [ ]hello, world blah yeah hello [ ]baby! [ ]abc --- no_error_log [alert] [error] === TEST 62: split ctx->pending into ctx->pending and ctx->free --- config replace_filter_max_buffered_size 3; default_type text/html; location = /t { #echo "abc\nd"; echo -n a; echo -n b; echo -n c; echo -n "\n"; echo d; replace_filter "abcd|bc\ne|c$" [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1482") { print_ubacktrace() } --- response_body ab[c] d --- no_error_log [alert] [error] === TEST 63: trim both leading and trailing spaces (1 byte at a time) --- config default_type text/html; replace_filter_max_buffered_size 5; location /t { echo -n 'a'; echo_sleep 0.001; echo ' '; echo_sleep 0.001; echo ''; echo_sleep 0.001; echo ' '; echo_sleep 0.001; echo "b"; echo_sleep 0.001; echo " "; replace_filter '^\s+|\s+$' '[$&]' g; } location = /main { echo_location_async /t1; echo_location_async /t2; echo_location_async /t3; echo_location_async /t4; echo_location_async /t5; echo_location_async /t6; } --- stap3 eval: $::StapOutputChains --- request GET /main --- response_body chop a[ ] b [ ]a[ ] b [ ]a[ ] b [ ]a[ ] b [ ]a[ ] b [ ]a[ ] b [ ] --- no_error_log [alert] [error] === TEST 64: global substitution + nginx vars --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo bbcd; replace_filter ([bc])|(d) [$1-$2-$foo] g; } --- request GET /t --- response_body [b--X][b--X][c--X][-d-X] --- no_error_log [alert] [error] === TEST 65: global substitution + nginx vars (splitted bufs) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo -n b; echo -n b; echo -n c; echo d; replace_filter ([bc])|(d) [$1-$2-$foo] g; } --- request GET /t --- response_body [b--X][b--X][c--X][-d-X] --- no_error_log [alert] [error] === TEST 66: global substitution + nginx vars (out of captures) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { set $foo X; echo -n b; echo -n b; echo -n c; echo d; replace_filter [bc]|d [$1-$2-$foo] g; } --- request GET /t --- stap F(ngx_http_replace_complex_value) { println("complex value") } --- stap_out complex value complex value complex value complex value --- response_body [--X][--X][--X][--X] --- no_error_log [alert] [error] ================================================ FILE: t/05-capturing-max-buffered.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4); our $StapOutputChains = <<'_EOC_'; global active F(ngx_http_handler) { active = 1 } /* F(ngx_http_write_filter) { if (active && pid() == target()) { printf("http writer filter: %s\n", ngx_chain_dump($in)) } } */ F(ngx_http_chunked_body_filter) { if (active && pid() == target()) { printf("http chunked filter: %s\n", ngx_chain_dump($in)) } } F(ngx_http_replace_output) { if (active && pid() == target()) { printf("http replace output: %s\n", ngx_chain_dump($ctx->out)) } } probe syscall.writev { if (active && pid() == target()) { printf("writev(%s)", ngx_iovec_dump($vec, $vlen)) /* for (i = 0; i < $vlen; i++) { printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len))) } */ } } probe syscall.writev.return { if (active && pid() == target()) { printf(" = %s\n", retstr) } } _EOC_ #no_diff(); no_long_string(); run_tests(); __DATA__ === TEST 1: 1-byte chain bufs (0) --- config default_type text/html; replace_filter_max_buffered_size 0; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1413") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() } --- response_body ababacd --- error_log replace filter: exceeding replace_filter_max_buffered_size (0): 1 --- no_error_log [error] === TEST 2: 1-byte chain bufs (1) --- config default_type text/html; replace_filter_max_buffered_size 1; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1439") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- response_body ababacd --- error_log replace filter: exceeding replace_filter_max_buffered_size (1): 2 --- no_error_log [error] === TEST 3: 1-byte chain bufs (2) --- config default_type text/html; replace_filter_max_buffered_size 2; location = /t { echo -n a; echo -n b; echo -n a; echo -n b; echo -n a; echo -n c; echo d; replace_filter abac [$&]; } --- request GET /t --- stap2 eval: $::StapOutputChains --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1439") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- response_body ababacd --- error_log replace filter: exceeding replace_filter_max_buffered_size (2): 3 --- no_error_log [error] === TEST 4: trim both leading and trailing spaces (1 byte at a time) (6) --- config replace_filter_max_buffered_size 6; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter '^\s+|\s+$' '[$&]' g; } --- user_files >>> a.html hello, world blah yeah hello baby! abc --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1438") { //printf("chain: %s", ngx_chain_dump($ctx->busy)) print_ubacktrace() exit() } --- request GET /t --- response_body [ ]hello, world[ ] blah yeah hello[ ] [ ]baby! abc --- error_log replace filter: exceeding replace_filter_max_buffered_size (6): 7 --- no_error_log [error] === TEST 5: github issue #2: error "general look-ahead not supported" --- config replace_filter_max_buffered_size 0; location /t { charset utf-8; default_type text/html; echo "ABCabcABCabc"; #replace_filter_types text/plain; replace_filter "a.+a" "[$&]" "ig"; } --- request GET /t --- stap3 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1481") { print_ubacktrace() } --- response_body ABCabcABCabc --- error_log replace filter: exceeding replace_filter_max_buffered_size (0): 12 --- no_error_log [error] === TEST 6: backtrack to the middle of a pending capture (pending: output|capture + rematch) (0) --- config replace_filter_max_buffered_size 0; default_type text/html; location = /t { echo -n ab; echo -n c; echo d; replace_filter 'abce|b' '[$&]' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1492") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body abcd --- error_log replace filter: exceeding replace_filter_max_buffered_size (0): 2 --- no_error_log [error] === TEST 7: backtrack to the middle of a pending capture (pending: output|capture + rematch) (1) --- config replace_filter_max_buffered_size 1; default_type text/html; location = /t { echo -n ab; echo -n c; echo d; replace_filter 'abce|b' '[$&]' g; } --- stap2 probe process("nginx").statement("*@ngx_http_replace_filter_module.c:1501") { print_ubacktrace() } --- stap3 eval: $::StapOutputChains --- request GET /t --- response_body abcd --- error_log replace filter: exceeding replace_filter_max_buffered_size (1): 2 --- no_error_log [error] ================================================ FILE: t/06-if.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4); #no_diff(); #no_long_string(); run_tests(); __DATA__ === TEST 1: local if hit --- config location /t { default_type text/plain; echo abcabcabde; if ($arg_disable = "") { replace_filter_types text/plain; replace_filter_max_buffered_size 0; replace_filter abcabd X; } } --- request GET /t --- response_body abcXe --- no_error_log [alert] [error] === TEST 2: local if miss --- config replace_filter_max_buffered_size 0; location /t { default_type text/plain; echo abcabcabde; if ($arg_disable = "") { replace_filter_types text/plain; replace_filter_max_buffered_size 0; replace_filter abcabd X; } } --- request GET /t?disable=1 --- response_body abcabcabde --- no_error_log [alert] [error] ================================================ FILE: t/07-multi.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); no_shuffle(); plan tests => repeat_each() * (blocks() * 4 + 5); our $StapOutputChains = <<'_EOC_'; global active F(ngx_http_handler) { active = 1 } /* F(ngx_http_write_filter) { if (active && pid() == target()) { printf("http writer filter: %s\n", ngx_chain_dump($in)) } } */ F(ngx_http_chunked_body_filter) { if (active && pid() == target()) { printf("http chunked filter: %s\n", ngx_chain_dump($in)) } } F(ngx_http_replace_output) { if (active && pid() == target()) { printf("http replace output: %s\n", ngx_chain_dump($ctx->out)) } } probe syscall.writev { if (active && pid() == target()) { printf("writev(%s)", ngx_iovec_dump($vec, $vlen)) /* for (i = 0; i < $vlen; i++) { printf(" %p [%s]", $vec[i]->iov_base, text_str(user_string_n($vec[i]->iov_base, $vec[i]->iov_len))) } */ } } probe syscall.writev.return { if (active && pid() == target()) { printf(" = %s\n", retstr) } } _EOC_ #no_diff(); #no_long_string(); run_tests(); __DATA__ === TEST 1: once patterns --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'hello world world hello'; replace_filter world "<$&>"; replace_filter hello "[$&]"; } --- request GET /t --- response_body [hello] world hello --- stap F(ngx_http_replace_non_capturing_parse) { println("non capturing parse") } F(ngx_http_replace_capturing_parse) { println("capturing parse") } --- stap_out_like chop ^(capturing parse\n)+$ --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 2: once patterns --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello world Hello world'; replace_filter world "<$&>"; replace_filter hello "[$&]"; } --- request GET /t --- response_body Hello Hello world --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 3: case-insensitive patterns --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello world WORLD HELLO'; replace_filter world "<$&>"; replace_filter hello "[$&]" i; } --- request GET /t --- response_body [Hello] WORLD HELLO --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 4: global subs --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'hello world world hello'; replace_filter world "<$&>" g; replace_filter hello "[$&]" g; } --- request GET /t --- response_body [hello] [hello] --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 5: global subs (case sensitive) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello World worlD hellO'; replace_filter world "<$&>" g; replace_filter hello "[$&]" g; } --- request GET /t --- response_body Hello World worlD hellO --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 6: global subs (case insensitive) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello World worlD hellO'; replace_filter world "<$&>" ig; replace_filter hello "[$&]" g; } --- request GET /t --- response_body Hello hellO --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 7: global subs (case insensitive) (2) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello World worlD hellO'; replace_filter world "<$&>" g; replace_filter hello "[$&]" ig; } --- request GET /t --- response_body [Hello] World worlD [hellO] --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 8: global subs (case insensitive) (3) --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello World worlD hellO'; replace_filter world "<$&>" gi; replace_filter hello "[$&]" ig; } --- request GET /t --- response_body [Hello] [hellO] --- stap F(ngx_http_replace_non_capturing_parse) { println("non capturing parse") } F(ngx_http_replace_capturing_parse) { println("capturing parse") } --- stap_out_like chop ^(capturing parse\n)+$ --- no_error_log [alert] [error] === TEST 9: global subs (case insensitive) - non-capturing --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo 'Hello World'; replace_filter world "<>" gi; replace_filter hello "[]" ig; } --- request GET /t --- response_body [] <> --- stap F(ngx_http_replace_non_capturing_parse) { println("non capturing parse") } F(ngx_http_replace_capturing_parse) { println("capturing parse") } --- stap_out_like chop ^(non capturing parse\n)+$ --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 10: working as a tokenizer --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo -n a; echo b; replace_filter a "[$&]" g; replace_filter ab "<$&>" g; } --- request GET /t --- response_body [a]b --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 11: working as a tokenizer (2) --- config default_type text/html; replace_filter_max_buffered_size 1; location /t { echo -n a; echo b; replace_filter ab "<$&>" g; replace_filter a "[$&]" g; } --- request GET /t --- response_body --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 12: on server level --- config default_type text/html; replace_filter_max_buffered_size 1; replace_filter ab "<$&>" g; replace_filter a "[$&]" g; location /t { echo -n a; echo b; } --- request GET /t --- response_body --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 13: mixing once and global patterns --- config default_type text/html; replace_filter_max_buffered_size 1; location /t { echo hello world hiya hiya world hello; replace_filter hello "<$&>"; replace_filter hiya "{$&}"; replace_filter world "[$&]" g; } --- request GET /t --- response_body [world] {hiya} hiya [world] hello --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 14: all once --- config default_type text/html; replace_filter_max_buffered_size 1; location /t { echo hello world hiya hiya world hello; replace_filter hello "<$&>"; replace_filter hiya "{$&}"; replace_filter world "[$&]"; } --- request GET /t --- response_body [world] {hiya} hiya world hello --- stap F(ngx_http_replace_complex_value) { println("complex value") } --- stap_out complex value complex value complex value --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 15: all once (server level) --- config default_type text/html; replace_filter_max_buffered_size 1; replace_filter hello "<$&>"; replace_filter hiya "{$&}"; replace_filter world "[$&]"; location /t { echo hello world hiya hiya world hello; } --- request GET /t --- response_body [world] {hiya} hiya world hello --- stap F(ngx_http_replace_complex_value) { println("complex value") } --- stap_out complex value complex value complex value --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] === TEST 16: remove C/C++ comments (1 byte at a time) --- config replace_filter_max_buffered_size 50; default_type text/html; location /a.html { internal; } location = /t { content_by_lua ' local res = ngx.location.capture("/a.html") local txt = res.body for i = 1, string.len(txt) do ngx.print(string.sub(txt, i, i)) ngx.flush(true) end '; replace_filter "'(?:\\\\[^\n]|[^'\n])*'" $& g; replace_filter '"(?:\\\\[^\n]|[^"\n])*"' $& g; replace_filter '/\*.*?\*/|//[^\n]*' '' g; } --- user_files >>> a.html b = '"'; /* blah */ c = '"' a = "h\"/* */"; i don't know // hello // world /* */ hello world /** abc * b/c /* hello ** // world * */ blah /* hi */ */ b // ///hi --- stap2 F(ngx_palloc) { if ($size < 0) { print_ubacktrace() exit() } } --- request GET /t --- response_body eval qq{b = '"'; c = '"' a = "h\\"/* */"; i don't know hello world blah */ b } --- no_error_log [alert] [error] === TEST 17: more patterns --- config default_type text/html; replace_filter_max_buffered_size 1; location /t { #echo hello world hiya hiya world hello; replace_filter a A; replace_filter b B; replace_filter c C; replace_filter d D; replace_filter e E; replace_filter f F; replace_filter g G; replace_filter h H; replace_filter i I; replace_filter j J; replace_filter k K; replace_filter l L; replace_filter m M; replace_filter n N; replace_filter o O; replace_filter p P; replace_filter q Q; replace_filter r R; replace_filter s S; replace_filter t T; replace_filter u U; replace_filter v V; replace_filter w W; replace_filter x X; replace_filter y Y; replace_filter z Z; } --- request GET /t --- user_files >>> t It'll be officially possible when the timer_by_lua directive is implemented in ngx_lua :) For now, people have been using some tricks to do something like that, i.e., using detached long-running requests (by calling ngx.eof early). See the related documentation for details: --- response_body IT'Ll BE OFfICiAllY PoSsible WHeN the tiMeR_by_lUa DirectiVe is implemented in nGX_lua :) For now, people have been using some tricKs to do something like that, i.e., using detached long-running reQuests (by calling ngx.eof early). See the related documentation for details: --- stap2 eval: $::StapOutputChains --- no_error_log [alert] [error] ================================================ FILE: t/08-gzip.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); no_shuffle(); plan tests => repeat_each() * (blocks() * 3); run_tests(); __DATA__ === TEST 1: once patterns --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { content_by_lua ' ngx.header.content_encoding = "gzip" ngx.say("hello world world hello"); '; replace_filter hello "[$&]"; replace_filter world "<$&>"; } --- request GET /t --- response_body hello world world hello --- no_error_log [error] ================================================ FILE: t/09-unused.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 5); run_tests(); __DATA__ === TEST 1: used --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter abcabd X; } --- request GET /t --- stap F(ngx_http_replace_header_filter) { println("replace header filter") } F(ngx_http_replace_body_filter) { println("replace body filter") } --- stap_out replace header filter replace body filter replace body filter --- response_body abcXe --- no_error_log [alert] [error] === TEST 2: unused --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; #replace_filter abcabd X; } --- request GET /t --- stap F(ngx_http_replace_header_filter) { println("replace header filter") } F(ngx_http_replace_body_filter) { println("replace body filter") } --- stap_out --- response_body abcabcabde --- no_error_log [alert] [error] === TEST 3: used (multi http {} blocks) This test case won't run with nginx 1.9.3+ since duplicate http {} blocks have been prohibited since then. --- SKIP --- config default_type text/html; replace_filter_max_buffered_size 0; location /t { echo abcabcabde; replace_filter abcabd X; } --- post_main_config http { } --- request GET /t --- stap F(ngx_http_replace_header_filter) { println("replace header filter") } F(ngx_http_replace_body_filter) { println("replace body filter") } --- stap_out replace header filter replace body filter replace body filter --- response_body abcXe --- no_error_log [alert] [error] ================================================ FILE: t/10-last-modified.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 5); run_tests(); __DATA__ === TEST 1: replace_filter_last_modified clear --- config default_type text/html; replace_filter_last_modified clear; location /t { content_by_lua ' ngx.header["Last-Modified"] = "Wed, 20 Nov 2013 05:30:35 GMT" ngx.say("ok") '; replace_filter abcabd X; } --- request GET /t --- response_body ok --- response_headers !Last-Modified --- no_error_log [alert] [error] === TEST 2: replace_filter_last_modified keep --- config default_type text/html; replace_filter_last_modified keep; location /t { content_by_lua ' ngx.header["Last-Modified"] = "Wed, 20 Nov 2013 05:30:35 GMT" ngx.say("ok") '; replace_filter abcabd X; } --- request GET /t --- response_body ok --- response_headers Last-Modified: Wed, 20 Nov 2013 05:30:35 GMT --- no_error_log [alert] [error] === TEST 3: replace_filter_last_modified default to clear --- config default_type text/html; #replace_filter_last_modified clear; location /t { content_by_lua ' ngx.header["Last-Modified"] = "Wed, 20 Nov 2013 05:30:35 GMT" ngx.say("ok") '; replace_filter abcabd X; } --- request GET /t --- response_body ok --- response_headers !Last-Modified --- no_error_log [alert] [error] ================================================ FILE: t/11-skip.t ================================================ # vim:set ft= ts=4 sw=4 et fdm=marker: use lib 'lib'; use Test::Nginx::Socket; #worker_connections(1014); #master_on(); #workers(2); #log_level('warn'); repeat_each(2); #no_shuffle(); plan tests => repeat_each() * (blocks() * 4); run_tests(); __DATA__ === TEST 1: skip true (constant) --- config default_type text/html; replace_filter_skip 1; location /t { content_by_lua ' ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body abcabd --- no_error_log [alert] [error] === TEST 2: skip false (constant 0) --- config default_type text/html; replace_filter_skip 0; location /t { content_by_lua ' ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body X --- no_error_log [alert] [error] === TEST 3: skip false (constant "") --- config default_type text/html; replace_filter_skip ""; location /t { content_by_lua ' ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body X --- no_error_log [alert] [error] === TEST 4: skip true (constant, random strings) --- config default_type text/html; replace_filter_skip ab; location /t { content_by_lua ' ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body abcabd --- no_error_log [alert] [error] === TEST 5: skip variable (1) --- config default_type text/html; set $skip ''; replace_filter_skip $skip; location /t { content_by_lua ' ngx.var.skip = 1 ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body abcabd --- no_error_log [alert] [error] === TEST 6: skip variable (0) --- config default_type text/html; set $skip ''; replace_filter_skip $skip; location /t { content_by_lua ' ngx.var.skip = 0 ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body X --- no_error_log [alert] [error] === TEST 7: skip variable ("") --- config default_type text/html; set $skip ''; replace_filter_skip $skip; location /t { content_by_lua ' ngx.var.skip = "" ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body X --- no_error_log [alert] [error] === TEST 8: skip variable (nil) --- config default_type text/html; set $skip ''; replace_filter_skip $skip; location /t { content_by_lua ' ngx.var.skip = nil ngx.say("abcabd") '; replace_filter abcabd X; } --- request GET /t --- response_body X --- no_error_log [alert] [error] ================================================ FILE: util/build.sh ================================================ #!/bin/bash # this file is mostly meant to be used by the author himself. root=`pwd` home=~ version=$1 force=$2 ngx-build $force $version \ $NGX_EXTRA_OPT \ --with-cc-opt="-I$PCRE_INC -I$OPENSSL_INC" \ --with-ld-opt="-L$PCRE_LIB -L$OPENSSL_LIB -Wl,-rpath,$SREGEX_LIB:$PCRE_LIB:$LIBDRIZZLE_LIB:$OPENSSL_LIB" \ --with-http_ssl_module \ --without-mail_pop3_module \ --without-mail_imap_module \ --without-mail_smtp_module \ --without-http_upstream_ip_hash_module \ --without-http_empty_gif_module \ --without-http_memcached_module \ --without-http_referer_module \ --without-http_autoindex_module \ --without-http_auth_basic_module \ --without-http_userid_module \ --add-module=$root $opts \ --add-module=$root/../ndk-nginx-module \ --add-module=$root/../lua-nginx-module \ --add-module=$root/../echo-nginx-module \ --with-debug #--with-cc-opt="-g3 -O0" #--without-http_ssi_module # we cannot disable ssi because echo_location_async depends on it (i dunno why?!) ================================================ FILE: valgrind.suppress ================================================ { Memcheck:Addr1 fun:ngx_init_cycle fun:ngx_master_process_cycle fun:main } { Memcheck:Addr4 fun:ngx_init_cycle fun:ngx_master_process_cycle fun:main } { Memcheck:Cond fun:ngx_vslprintf fun:ngx_snprintf fun:ngx_sock_ntop fun:ngx_event_accept fun:ngx_epoll_process_events fun:ngx_process_events_and_timers } { Memcheck:Cond fun:ngx_vslprintf fun:ngx_snprintf fun:ngx_sock_ntop fun:ngx_event_accept fun:ngx_epoll_process_events fun:ngx_process_events_and_timers } { Memcheck:Addr1 fun:ngx_vslprintf fun:ngx_snprintf fun:ngx_sock_ntop fun:ngx_event_accept } { Memcheck:Leak fun:malloc fun:ngx_alloc obj:* } { exp-sgcheck:SorG fun:ngx_http_lua_ndk_set_var_get } { exp-sgcheck:SorG fun:ngx_http_variables_init_vars fun:ngx_http_block } { exp-sgcheck:SorG fun:ngx_conf_parse } { exp-sgcheck:SorG fun:ngx_vslprintf fun:ngx_log_error_core } { Memcheck:Leak fun:malloc fun:ngx_alloc fun:ngx_calloc fun:ngx_event_process_init } { Memcheck:Param epoll_ctl(event) fun:epoll_ctl } { Memcheck:Leak fun:malloc fun:ngx_alloc fun:ngx_event_process_init } { Memcheck:Cond fun:ngx_conf_flush_files fun:ngx_single_process_cycle } { Memcheck:Cond fun:memcpy fun:ngx_vslprintf fun:ngx_log_error_core fun:ngx_http_charset_header_filter } { Memcheck:Param socketcall.setsockopt(optval) fun:setsockopt fun:drizzle_state_connect } { Memcheck:Leak fun:malloc fun:ngx_alloc fun:ngx_pool_cleanup_add } { Memcheck:Cond fun:ngx_conf_flush_files fun:ngx_single_process_cycle fun:main } { Memcheck:Cond fun:index fun:expand_dynamic_string_token fun:_dl_map_object fun:map_doit fun:_dl_catch_error fun:do_preload fun:dl_main } { Memcheck:Leak match-leak-kinds: definite fun:malloc fun:ngx_alloc fun:ngx_set_environment fun:ngx_single_process_cycle } { Memcheck:Leak match-leak-kinds: definite fun:malloc fun:ngx_alloc fun:ngx_set_environment fun:ngx_worker_process_init fun:ngx_worker_process_cycle }