Repository: xiaokai-wang/nginx-stream-upsync-module Branch: master Commit: d403a5b644c6 Files: 18 Total size: 302.8 KB Directory structure: gitextract_yli_f_d8/ ├── README.md ├── config ├── src/ │ ├── ngx_stream_http_parser.c │ ├── ngx_stream_http_parser.h │ ├── ngx_stream_json.c │ ├── ngx_stream_json.h │ ├── ngx_stream_upsync_module.c │ └── ngx_stream_upsync_module.h └── test/ ├── README ├── conf-server.sh ├── t/ │ ├── lib/ │ │ ├── Test/ │ │ │ ├── Nginx/ │ │ │ │ ├── IMAP.pm │ │ │ │ ├── POP3.pm │ │ │ │ └── SMTP.pm │ │ │ └── Nginx.pm │ │ └── Time/ │ │ ├── Parse.pm │ │ └── Zone.pm │ └── upsync.t └── test.sh ================================================ FILE CONTENTS ================================================ ================================================ FILE: README.md ================================================ Name ==== nginx-stream-upsync-module - Nginx C module, sync upstreams from consul or others, dynamically modify backend-servers attribute(weight, max_fails,...), needn't reload nginx. It may not always be convenient to modify configuration files and restart NGINX. For example, if you are experiencing large amounts of traffic and high load, restarting NGINX and reloading the configuration at that point further increases load on the system and can temporarily degrade performance. The module can be more smoothly expansion and constriction, and will not influence the performance. Another module, [nginx-upsync-module](https://github.com/weibocom/nginx-upsync-module) supports nginx http module(HTTP protocol), please be noticed. If you want to use [nginx-upsync-module](https://github.com/weibocom/nginx-upsync-module) and [nginx-stream-upsync-module](https://github.com/xiaokai-wang/nginx-stream-upsync-module) both, please refer to [nginx-upsync](https://github.com/CallMeFoxie/nginx-upsync). Table of Contents ================= * [Name](#name) * [Status](#status) * [Synopsis](#synopsis) * [Description](#description) * [Directives](#directives) * [upsync](#upsync) * [upsync_interval](#upsync_interval) * [upsync_timeout](#upsync_timeout) * [upsync_type](#upsync_type) * [strong_dependency](#strong_dependency) * [upsync_dump_path](#upsync_dump_path) * [upsync_lb](#upsync_lb) * [upstream_show](#upstream_show) * [Consul_interface](#consul_interface) * [Etcd_interface](#etcd_interface) * [TODO](#todo) * [Compatibility](#compatibility) * [Installation](#installation) * [Code style](#code-style) * [Author](#author) * [Copyright and License](#copyright-and-license) * [See Also](#see-also) * [Source Dependency](#source-dependency) Status ====== This module is still under active development and is considered production ready. Synopsis ======== nginx-consul: ```nginx-consul stream { upstream test { upsync 127.0.0.1:8500/v1/kv/upstreams/test/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off; upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf; include /usr/local/nginx/conf/servers/servers_test.conf; } upstream bar { server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3; } server { listen 12345; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass test; } server { listen 2345; upstream_show } server { listen 127.0.0.1:9091; proxy_responses 1; proxy_timeout 20s; proxy_pass bar; } } ``` nginx-etcd: ```nginx-etcd stream { upstream test { upsync 127.0.0.1:2379/v2/keys/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=etcd strong_dependency=off; upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf; include /usr/local/nginx/conf/servers/servers_test.conf; } upstream bar { server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3; } server { listen 12345; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass test; } server { listen 2345; upstream_show } server { listen 127.0.0.1:9091; proxy_responses 1; proxy_timeout 20s; proxy_pass bar; } } ``` upsync_lb: ```upsync_lb stream { upstream test { least_conn; //hash $uri consistent; upsync 127.0.0.1:8500/v1/kv/upstreams/test/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off; upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf; upsync_lb least_conn; //hash_ketama; include /usr/local/nginx/conf/servers/servers_test.conf; } upstream bar { server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3; } server { listen 12345; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass test; } server { listen 2345; upstream_show } server { listen 127.0.0.1:9091; proxy_responses 1; proxy_timeout 20s; proxy_pass bar; } } ``` NOTE: upstream: include command is neccesary, first time the dumped file should include all the servers. [Back to TOC](#table-of-contents) Description ====== This module provides a method to discover backend servers. Supporting dynamicly adding or deleting backend server through consul/etcd and dynamicly adjusting backend servers weight, module will timely pull new backend server list from consul/etcd to upsync nginx ip router. Nginx needn't reload. Having some advantages than others: * timely module send key to consul/etcd with index, consul/etcd will compare it with its index, if index doesn't change connection will hang five minutes, in the period any operation to the key-value, will feed back rightaway. * performance Pulling from consul/etcd equal a request to nginx, updating ip router nginx needn't reload, so affecting nginx performance is little. * stability Even if one pulling failed, it will pull next upsync_interval, so guaranteing backend server stably provides service. And support dumping the latest config to location, so even if consul/etcd hung up, and nginx can be reload anytime. [Back to TOC](#table-of-contents) Directives ====== upsync ----------- ``` syntax: upsync $consul/etcd.api.com:$port/v1/kv/upstreams/$upstream_name/ [upsync_type=consul/etcd] [upsync_interval=second/minutes] [upsync_timeout=second/minutes] [strong_dependency=off/on] ``` default: none, if parameters omitted, default parameters are upsync_interval=5s upsync_timeout=6m strong_dependency=off context: upstream description: Pull upstream servers from consul/etcd... . The parameters' meanings are: * upsync_interval pulling servers from consul/etcd interval time. * upsync_timeout pulling servers from consul/etcd request timeout. * upsync_type pulling servers from conf server type. * strong_dependency when nginx start up if strong_dependency is on that means servers will be depended on consul/etcd and will pull servers from consul/etcd. upsync_dump_path ----------- `syntax: upsync_dump_path $path` default: /tmp/servers_$host.conf context: upstream description: dump the upstream backends to the $path. upsync_lb ----------- `syntax: upsync_lb $load_balance` default: round_robin/ip_hash/hash modula context: upstream description: mainly for least_conn and hash consistent, when using one of them, you must point out using upsync_lb. upsync_show ----------- `syntax: upsync_show` default: none context: server description: show all upstreams. ```request curl http://localhost:2345/upstream_show show all upstreams ``` [Back to TOC](#table-of-contents) Consul_interface ====== Data can be taken from key/value store or service catalog. In the first case parameter upsync_type of directive must be *consul*. For example ```nginx-consul upsync 127.0.0.1:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off; ``` In the second case it must be *consul_services*. ```nginx-consul upsync 127.0.0.1:8500/v1/catalog/service/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul_services strong_dependency=off; ``` you can add or delete backend server through consul_ui or http_interface. Below are examples for key/value store. http_interface example: * add ``` curl -X PUT http://$consul_ip:$port/v1/kv/upstreams/$upstream_name/$backend_ip:$backend_port ``` default: weight=1 max_fails=2 fail_timeout=10 down=0 backup=0; ``` curl -X PUT -d "{\"weight\":1, \"max_fails\":2, \"fail_timeout\":10}" http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port or curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port ``` value support json format. * delete ``` curl -X DELETE http://$consul_ip:$port/v1/kv/upstreams/$upstream_name/$backend_ip:$backend_port ``` * adjust-weight ``` curl -X PUT -d "{\"weight\":2, \"max_fails\":2, \"fail_timeout\":10}" http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port or curl -X PUT -d '{"weight":2, "max_fails":2, "fail_timeout":10}' http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port ``` * mark server-down ``` curl -X PUT -d "{\"weight\":2, \"max_fails\":2, \"fail_timeout\":10, \"down\":1}" http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port or curl -X PUT -d '{"weight":2, "max_fails":2, "fail_timeout":10, "down":1}' http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port ``` * check ``` curl http://$consul_ip:$port/v1/kv/upstreams/$upstream_name?recurse ``` [Back to TOC](#table-of-contents) Etcd_interface ====== you can add or delete backend server through http_interface. mainly like etcd, http_interface example: * add ``` curl -X PUT http://$etcd_ip:$port/v2/keys/upstreams/$upstream_name/$backend_ip:$backend_port ``` default: weight=1 max_fails=2 fail_timeout=10 down=0 backup=0; ``` curl -X PUT -d value="{\"weight\":1, \"max_fails\":2, \"fail_timeout\":10}" http://$etcd_ip:$port/v2/keys/$dir1/$upstream_name/$backend_ip:$backend_port ``` value support json format. * delete ``` curl -X DELETE http://$etcd_ip:$port/v2/keys/upstreams/$upstream_name/$backend_ip:$backend_port ``` * adjust-weight ``` curl -X PUT -d "{\"weight\":2, \"max_fails\":2, \"fail_timeout\":10}" http://$etcd_ip:$port/v2/keys/$dir1/$upstream_name/$backend_ip:$backend_port ``` * mark server-down ``` curl -X PUT -d value="{\"weight\":2, \"max_fails\":2, \"fail_timeout\":10, \"down\":1}" http://$etcd_ip:$port/v2/keys/$dir1/$upstream_name/$backend_ip:$backend_port ``` * check ``` curl http://$etcd_ip:$port/v2/keys/upstreams/$upstream_name ``` [Back to TOC](#table-of-contents) TODO ==== * support zookeeper and so on [Back to TOC](#table-of-contents) Compatibility ============= The module was developed base on nginx-1.9.10. Master branch compatible with nginx-1.11.0+. Nginx-1.10.3- branch compatible with nginx-1.9.10 ~ nginx-1.11.0+. [Back to TOC](#table-of-contents) Installation ============ This module can be used independently, can be download[Github](https://github.com/xiaokai-wang/nginx-stream-upsync-module.git). Grab the nginx source code from [nginx.org](http://nginx.org/), for example, the version 1.8.0 (see nginx compatibility), and then build the source with this module: ```bash wget 'http://nginx.org/download/nginx-1.8.0.tar.gz' tar -xzvf nginx-1.8.0.tar.gz cd nginx-1.8.0/ ``` ```bash ./configure --add-module=/path/to/nginx-stream_upsync-module make make install ``` if you support nginx-upstream-check-module ```bash ./configure --add-module=/path/to/nginx-upstream-check-module --add-module=/path/to/nginx-stream_upsync-module make make install ``` [Back to TOC](#table-of-contents) Code style ====== Code style is mainly based on [style](http://tengine.taobao.org/book/appendix_a.html) [Back to TOC](#table-of-contents) Author ====== Xiaokai Wang (王晓开) , Weibo Inc. [Back to TOC](#table-of-contents) Copyright and License ===================== This README template copy from agentzh. This module is licensed under the BSD license. Copyright (C) 2014 by Xiaokai Wang All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. [Back to TOC](#table-of-contents) see also ======== * the nginx_upstream_check_module: https://github.com/alibaba/tengine/blob/master/src/http/ngx_http_upstream_check_module.c * the nginx_upstream_check_module patch: https://github.com/yaoweibin/nginx_upstream_check_module * or based on https://github.com/xiaokai-wang/nginx_upstream_check_module [back to toc](#table-of-contents) source dependency ======== * Cjson: https://github.com/kbranigan/cJSON * http-parser: https://github.com/nodejs/http-parser [back to toc](#table-of-contents) ================================================ FILE: config ================================================ ngx_addon_name=ngx_stream_upsync_module ngx_feature_libs="-lm" ngx_module_incs=$ngx_addon_dir/src _STREAM_UPSYNC_SRCS="\ $ngx_addon_dir/src/ngx_stream_upsync_module.c \ $ngx_addon_dir/src/ngx_stream_json.c \ $ngx_addon_dir/src/ngx_stream_http_parser.c \ " have=NGX_STREAM_UPSYNC . auto/have if test -n "$ngx_module_link"; then ngx_module_type=STREAM ngx_module_name=$ngx_addon_name ngx_module_srcs="$_STREAM_UPSYNC_SRCS" ngx_module_libs=$ngx_feature_libs . auto/module else NGX_ADDON_SRCS="$NGX_ADDON_SRCS $_STREAM_UPSYNC_SRCS" CORE_LIBS="$CORE_LIBS $ngx_feature_libs" CORE_INCS="$CORE_INCS $ngx_module_incs" STREAM_MODULES="$STREAM_MODULES $ngx_addon_name" fi ================================================ FILE: src/ngx_stream_http_parser.c ================================================ /* Based on src/http/ngx_http_parse.c from NGINX copyright Igor Sysoev * * Additional changes are licensed under the same terms as NGINX and * copyright Joyent, Inc. and other Node contributors. All rights reserved. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to * deal in the Software without restriction, including without limitation the * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or * sell copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS * IN THE SOFTWARE. */ #include #ifndef NGX_HTTP_UPSYNC #include "ngx_stream_http_parser.h" #include #include #include #include #include #include #ifndef ULLONG_MAX # define ULLONG_MAX ((uint64_t) -1) /* 2^64-1 */ #endif #ifndef MIN # define MIN(a,b) ((a) < (b) ? (a) : (b)) #endif #ifndef ARRAY_SIZE # define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) #endif #ifndef BIT_AT # define BIT_AT(a, i) \ (!!((unsigned int) (a)[(unsigned int) (i) >> 3] & \ (1 << ((unsigned int) (i) & 7)))) #endif #ifndef ELEM_AT # define ELEM_AT(a, i, v) ((unsigned int) (i) < ARRAY_SIZE(a) ? (a)[(i)] : (v)) #endif #define SET_ERRNO(e) \ do { \ parser->http_errno = (e); \ } while(0) #define CURRENT_STATE() p_state #define UPDATE_STATE(V) p_state = (enum state) (V); #define RETURN(V) \ do { \ parser->state = CURRENT_STATE(); \ return (V); \ } while (0); #define REEXECUTE() \ goto reexecute; \ #ifdef __GNUC__ # define LIKELY(X) __builtin_expect(!!(X), 1) # define UNLIKELY(X) __builtin_expect(!!(X), 0) #else # define LIKELY(X) (X) # define UNLIKELY(X) (X) #endif /* Run the notify callback FOR, returning ER if it fails */ #define CALLBACK_NOTIFY_(FOR, ER) \ do { \ assert(HTTP_PARSER_ERRNO(parser) == HPE_OK); \ \ if (LIKELY(settings->on_##FOR)) { \ parser->state = CURRENT_STATE(); \ if (UNLIKELY(0 != settings->on_##FOR(parser))) { \ SET_ERRNO(HPE_CB_##FOR); \ } \ UPDATE_STATE(parser->state); \ \ /* We either errored above or got paused; get out */ \ if (UNLIKELY(HTTP_PARSER_ERRNO(parser) != HPE_OK)) { \ return (ER); \ } \ } \ } while (0) /* Run the notify callback FOR and consume the current byte */ #define CALLBACK_NOTIFY(FOR) CALLBACK_NOTIFY_(FOR, p - data + 1) /* Run the notify callback FOR and don't consume the current byte */ #define CALLBACK_NOTIFY_NOADVANCE(FOR) CALLBACK_NOTIFY_(FOR, p - data) /* Run data callback FOR with LEN bytes, returning ER if it fails */ #define CALLBACK_DATA_(FOR, LEN, ER) \ do { \ assert(HTTP_PARSER_ERRNO(parser) == HPE_OK); \ \ if (FOR##_mark) { \ if (LIKELY(settings->on_##FOR)) { \ parser->state = CURRENT_STATE(); \ if (UNLIKELY(0 != \ settings->on_##FOR(parser, FOR##_mark, (LEN)))) { \ SET_ERRNO(HPE_CB_##FOR); \ } \ UPDATE_STATE(parser->state); \ \ /* We either errored above or got paused; get out */ \ if (UNLIKELY(HTTP_PARSER_ERRNO(parser) != HPE_OK)) { \ return (ER); \ } \ } \ FOR##_mark = NULL; \ } \ } while (0) /* Run the data callback FOR and consume the current byte */ #define CALLBACK_DATA(FOR) \ CALLBACK_DATA_(FOR, p - FOR##_mark, p - data + 1) /* Run the data callback FOR and don't consume the current byte */ #define CALLBACK_DATA_NOADVANCE(FOR) \ CALLBACK_DATA_(FOR, p - FOR##_mark, p - data) /* Set the mark FOR; non-destructive if mark is already set */ #define MARK(FOR) \ do { \ if (!FOR##_mark) { \ FOR##_mark = p; \ } \ } while (0) /* Don't allow the total size of the HTTP headers (including the status * line) to exceed HTTP_MAX_HEADER_SIZE. This check is here to protect * embedders against denial-of-service attacks where the attacker feeds * us a never-ending header that the embedder keeps buffering. * * This check is arguably the responsibility of embedders but we're doing * it on the embedder's behalf because most won't bother and this way we * make the web a little safer. HTTP_MAX_HEADER_SIZE is still far bigger * than any reasonable request or response so this should never affect * day-to-day operation. */ #define COUNT_HEADER_SIZE(V) \ do { \ parser->nread += (V); \ if (UNLIKELY(parser->nread > (HTTP_MAX_HEADER_SIZE))) { \ SET_ERRNO(HPE_HEADER_OVERFLOW); \ goto error; \ } \ } while (0) #define PROXY_CONNECTION "proxy-connection" #define CONNECTION "connection" #define CONTENT_LENGTH "content-length" #define TRANSFER_ENCODING "transfer-encoding" #define UPGRADE "upgrade" #define CHUNKED "chunked" #define KEEP_ALIVE "keep-alive" #define CLOSE "close" static const char *method_strings[] = { #define XX(num, name, string) #string, HTTP_METHOD_MAP(XX) #undef XX }; /* Tokens as defined by rfc 2616. Also lowercases them. * token = 1* * separators = "(" | ")" | "<" | ">" | "@" * | "," | ";" | ":" | "\" | <"> * | "/" | "[" | "]" | "?" | "=" * | "{" | "}" | SP | HT */ static const char tokens[256] = { /* 0 nul 1 soh 2 stx 3 etx 4 eot 5 enq 6 ack 7 bel */ 0, 0, 0, 0, 0, 0, 0, 0, /* 8 bs 9 ht 10 nl 11 vt 12 np 13 cr 14 so 15 si */ 0, 0, 0, 0, 0, 0, 0, 0, /* 16 dle 17 dc1 18 dc2 19 dc3 20 dc4 21 nak 22 syn 23 etb */ 0, 0, 0, 0, 0, 0, 0, 0, /* 24 can 25 em 26 sub 27 esc 28 fs 29 gs 30 rs 31 us */ 0, 0, 0, 0, 0, 0, 0, 0, /* 32 sp 33 ! 34 " 35 # 36 $ 37 % 38 & 39 ' */ 0, '!', 0, '#', '$', '%', '&', '\'', /* 40 ( 41 ) 42 * 43 + 44 , 45 - 46 . 47 / */ 0, 0, '*', '+', 0, '-', '.', 0, /* 48 0 49 1 50 2 51 3 52 4 53 5 54 6 55 7 */ '0', '1', '2', '3', '4', '5', '6', '7', /* 56 8 57 9 58 : 59 ; 60 < 61 = 62 > 63 ? */ '8', '9', 0, 0, 0, 0, 0, 0, /* 64 @ 65 A 66 B 67 C 68 D 69 E 70 F 71 G */ 0, 'a', 'b', 'c', 'd', 'e', 'f', 'g', /* 72 H 73 I 74 J 75 K 76 L 77 M 78 N 79 O */ 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', /* 80 P 81 Q 82 R 83 S 84 T 85 U 86 V 87 W */ 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', /* 88 X 89 Y 90 Z 91 [ 92 \ 93 ] 94 ^ 95 _ */ 'x', 'y', 'z', 0, 0, 0, '^', '_', /* 96 ` 97 a 98 b 99 c 100 d 101 e 102 f 103 g */ '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', /* 104 h 105 i 106 j 107 k 108 l 109 m 110 n 111 o */ 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', /* 112 p 113 q 114 r 115 s 116 t 117 u 118 v 119 w */ 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', /* 120 x 121 y 122 z 123 { 124 | 125 } 126 ~ 127 del */ 'x', 'y', 'z', 0, '|', 0, '~', 0 }; static const int8_t unhex[256] = {-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1 ,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1 ,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1 , 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,-1,-1,-1,-1,-1,-1 ,-1,10,11,12,13,14,15,-1,-1,-1,-1,-1,-1,-1,-1,-1 ,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1 ,-1,10,11,12,13,14,15,-1,-1,-1,-1,-1,-1,-1,-1,-1 ,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1 }; #if HTTP_PARSER_STRICT # define T(v) 0 #else # define T(v) v #endif static const uint8_t normal_url_char[32] = { /* 0 nul 1 soh 2 stx 3 etx 4 eot 5 enq 6 ack 7 bel */ 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0, /* 8 bs 9 ht 10 nl 11 vt 12 np 13 cr 14 so 15 si */ 0 | T(2) | 0 | 0 | T(16) | 0 | 0 | 0, /* 16 dle 17 dc1 18 dc2 19 dc3 20 dc4 21 nak 22 syn 23 etb */ 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0, /* 24 can 25 em 26 sub 27 esc 28 fs 29 gs 30 rs 31 us */ 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0, /* 32 sp 33 ! 34 " 35 # 36 $ 37 % 38 & 39 ' */ 0 | 2 | 4 | 0 | 16 | 32 | 64 | 128, /* 40 ( 41 ) 42 * 43 + 44 , 45 - 46 . 47 / */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 48 0 49 1 50 2 51 3 52 4 53 5 54 6 55 7 */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 56 8 57 9 58 : 59 ; 60 < 61 = 62 > 63 ? */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 0, /* 64 @ 65 A 66 B 67 C 68 D 69 E 70 F 71 G */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 72 H 73 I 74 J 75 K 76 L 77 M 78 N 79 O */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 80 P 81 Q 82 R 83 S 84 T 85 U 86 V 87 W */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 88 X 89 Y 90 Z 91 [ 92 \ 93 ] 94 ^ 95 _ */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 96 ` 97 a 98 b 99 c 100 d 101 e 102 f 103 g */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 104 h 105 i 106 j 107 k 108 l 109 m 110 n 111 o */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 112 p 113 q 114 r 115 s 116 t 117 u 118 v 119 w */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128, /* 120 x 121 y 122 z 123 { 124 | 125 } 126 ~ 127 del */ 1 | 2 | 4 | 8 | 16 | 32 | 64 | 0, }; #undef T enum state { s_dead = 1 /* important that this is > 0 */ , s_start_req_or_res , s_res_or_resp_H , s_start_res , s_res_H , s_res_HT , s_res_HTT , s_res_HTTP , s_res_first_http_major , s_res_http_major , s_res_first_http_minor , s_res_http_minor , s_res_first_status_code , s_res_status_code , s_res_status_start , s_res_status , s_res_line_almost_done , s_start_req , s_req_method , s_req_spaces_before_url , s_req_schema , s_req_schema_slash , s_req_schema_slash_slash , s_req_server_start , s_req_server , s_req_server_with_at , s_req_path , s_req_query_string_start , s_req_query_string , s_req_fragment_start , s_req_fragment , s_req_http_start , s_req_http_H , s_req_http_HT , s_req_http_HTT , s_req_http_HTTP , s_req_first_http_major , s_req_http_major , s_req_first_http_minor , s_req_http_minor , s_req_line_almost_done , s_header_field_start , s_header_field , s_header_value_discard_ws , s_header_value_discard_ws_almost_done , s_header_value_discard_lws , s_header_value_start , s_header_value , s_header_value_lws , s_header_almost_done , s_chunk_size_start , s_chunk_size , s_chunk_parameters , s_chunk_size_almost_done , s_headers_almost_done , s_headers_done /* Important: 's_headers_done' must be the last 'header' state. All * states beyond this must be 'body' states. It is used for overflow * checking. See the PARSING_HEADER() macro. */ , s_chunk_data , s_chunk_data_almost_done , s_chunk_data_done , s_body_identity , s_body_identity_eof , s_message_done }; #define PARSING_HEADER(state) (state <= s_headers_done) enum header_states { h_general = 0 , h_C , h_CO , h_CON , h_matching_connection , h_matching_proxy_connection , h_matching_content_length , h_matching_transfer_encoding , h_matching_upgrade , h_connection , h_content_length , h_transfer_encoding , h_upgrade , h_matching_transfer_encoding_chunked , h_matching_connection_token_start , h_matching_connection_keep_alive , h_matching_connection_close , h_matching_connection_upgrade , h_matching_connection_token , h_transfer_encoding_chunked , h_connection_keep_alive , h_connection_close , h_connection_upgrade }; enum http_host_state { s_http_host_dead = 1 , s_http_userinfo_start , s_http_userinfo , s_http_host_start , s_http_host_v6_start , s_http_host , s_http_host_v6 , s_http_host_v6_end , s_http_host_port_start , s_http_host_port }; /* Macros for character classes; depends on strict-mode */ #define CR '\r' #define LF '\n' #define LOWER(c) (unsigned char)(c | 0x20) #define IS_ALPHA(c) (LOWER(c) >= 'a' && LOWER(c) <= 'z') #define IS_NUM(c) ((c) >= '0' && (c) <= '9') #define IS_ALPHANUM(c) (IS_ALPHA(c) || IS_NUM(c)) #define IS_HEX(c) (IS_NUM(c) || (LOWER(c) >= 'a' && LOWER(c) <= 'f')) #define IS_MARK(c) ((c) == '-' || (c) == '_' || (c) == '.' || \ (c) == '!' || (c) == '~' || (c) == '*' || (c) == '\'' || (c) == '(' || \ (c) == ')') #define IS_USERINFO_CHAR(c) (IS_ALPHANUM(c) || IS_MARK(c) || (c) == '%' || \ (c) == ';' || (c) == ':' || (c) == '&' || (c) == '=' || (c) == '+' || \ (c) == '$' || (c) == ',') #define STRICT_TOKEN(c) (tokens[(unsigned char)c]) #if HTTP_PARSER_STRICT #define TOKEN(c) (tokens[(unsigned char)c]) #define IS_URL_CHAR(c) (BIT_AT(normal_url_char, (unsigned char)c)) #define IS_HOST_CHAR(c) (IS_ALPHANUM(c) || (c) == '.' || (c) == '-') #else #define TOKEN(c) ((c == ' ') ? ' ' : tokens[(unsigned char)c]) #define IS_URL_CHAR(c) \ (BIT_AT(normal_url_char, (unsigned char)c) || ((c) & 0x80)) #define IS_HOST_CHAR(c) \ (IS_ALPHANUM(c) || (c) == '.' || (c) == '-' || (c) == '_') #endif #define start_state (parser->type == HTTP_REQUEST ? s_start_req : s_start_res) #if HTTP_PARSER_STRICT # define STRICT_CHECK(cond) \ do { \ if (cond) { \ SET_ERRNO(HPE_STRICT); \ goto error; \ } \ } while (0) # define NEW_MESSAGE() (http_should_keep_alive(parser) ? start_state : s_dead) #else # define STRICT_CHECK(cond) # define NEW_MESSAGE() start_state #endif /* Map errno values to strings for human-readable output */ #define HTTP_STRERROR_GEN(n, s) { "HPE_" #n, s }, static struct { const char *name; const char *description; } http_strerror_tab[] = { HTTP_ERRNO_MAP(HTTP_STRERROR_GEN) }; #undef HTTP_STRERROR_GEN int http_message_needs_eof(const http_parser *parser); /* Our URL parser. * * This is designed to be shared by http_parser_execute() for URL validation, * hence it has a state transition + byte-for-byte interface. In addition, it * is meant to be embedded in http_parser_parse_url(), which does the dirty * work of turning state transitions URL components for its API. * * This function should only be invoked with non-space characters. It is * assumed that the caller cares about (and can detect) the transition between * URL and non-URL states by looking for these. */ static enum state parse_url_char(enum state s, const char ch) { if (ch == ' ' || ch == '\r' || ch == '\n') { return s_dead; } #if HTTP_PARSER_STRICT if (ch == '\t' || ch == '\f') { return s_dead; } #endif switch (s) { case s_req_spaces_before_url: /* Proxied requests are followed by scheme of an absolute URI (alpha). * All methods except CONNECT are followed by '/' or '*'. */ if (ch == '/' || ch == '*') { return s_req_path; } if (IS_ALPHA(ch)) { return s_req_schema; } break; case s_req_schema: if (IS_ALPHA(ch)) { return s; } if (ch == ':') { return s_req_schema_slash; } break; case s_req_schema_slash: if (ch == '/') { return s_req_schema_slash_slash; } break; case s_req_schema_slash_slash: if (ch == '/') { return s_req_server_start; } break; case s_req_server_with_at: if (ch == '@') { return s_dead; } /* FALLTHROUGH */ case s_req_server_start: case s_req_server: if (ch == '/') { return s_req_path; } if (ch == '?') { return s_req_query_string_start; } if (ch == '@') { return s_req_server_with_at; } if (IS_USERINFO_CHAR(ch) || ch == '[' || ch == ']') { return s_req_server; } break; case s_req_path: if (IS_URL_CHAR(ch)) { return s; } switch (ch) { case '?': return s_req_query_string_start; case '#': return s_req_fragment_start; } break; case s_req_query_string_start: case s_req_query_string: if (IS_URL_CHAR(ch)) { return s_req_query_string; } switch (ch) { case '?': /* allow extra '?' in query string */ return s_req_query_string; case '#': return s_req_fragment_start; } break; case s_req_fragment_start: if (IS_URL_CHAR(ch)) { return s_req_fragment; } switch (ch) { case '?': return s_req_fragment; case '#': return s; } break; case s_req_fragment: if (IS_URL_CHAR(ch)) { return s; } switch (ch) { case '?': case '#': return s; } break; default: break; } /* We should never fall out of the switch above unless there's an error */ return s_dead; } size_t http_parser_execute (http_parser *parser, const http_parser_settings *settings, const char *data, size_t len) { char c, ch; int8_t unhex_val; const char *p = data; const char *header_field_mark = 0; const char *header_value_mark = 0; const char *url_mark = 0; const char *body_mark = 0; const char *status_mark = 0; enum state p_state = (enum state) parser->state; /* We're in an error state. Don't bother doing anything. */ if (HTTP_PARSER_ERRNO(parser) != HPE_OK) { return 0; } if (len == 0) { switch (CURRENT_STATE()) { case s_body_identity_eof: /* Use of CALLBACK_NOTIFY() here would erroneously return 1 byte read if * we got paused. */ CALLBACK_NOTIFY_NOADVANCE(message_complete); return 0; case s_dead: case s_start_req_or_res: case s_start_res: case s_start_req: return 0; default: SET_ERRNO(HPE_INVALID_EOF_STATE); return 1; } } if (CURRENT_STATE() == s_header_field) header_field_mark = data; if (CURRENT_STATE() == s_header_value) header_value_mark = data; switch (CURRENT_STATE()) { case s_req_path: case s_req_schema: case s_req_schema_slash: case s_req_schema_slash_slash: case s_req_server_start: case s_req_server: case s_req_server_with_at: case s_req_query_string_start: case s_req_query_string: case s_req_fragment_start: case s_req_fragment: url_mark = data; break; case s_res_status: status_mark = data; break; default: break; } for (p=data; p != data + len; p++) { ch = *p; if (PARSING_HEADER(CURRENT_STATE())) COUNT_HEADER_SIZE(1); reexecute: switch (CURRENT_STATE()) { case s_dead: /* this state is used after a 'Connection: close' message * the parser will error out if it reads another message */ if (LIKELY(ch == CR || ch == LF)) break; SET_ERRNO(HPE_CLOSED_CONNECTION); goto error; case s_start_req_or_res: { if (ch == CR || ch == LF) break; parser->flags = 0; parser->content_length = ULLONG_MAX; if (ch == 'H') { UPDATE_STATE(s_res_or_resp_H); CALLBACK_NOTIFY(message_begin); } else { parser->type = HTTP_REQUEST; UPDATE_STATE(s_start_req); REEXECUTE(); } break; } case s_res_or_resp_H: if (ch == 'T') { parser->type = HTTP_RESPONSE; UPDATE_STATE(s_res_HT); } else { if (UNLIKELY(ch != 'E')) { SET_ERRNO(HPE_INVALID_CONSTANT); goto error; } parser->type = HTTP_REQUEST; parser->method = HTTP_HEAD; parser->index = 2; UPDATE_STATE(s_req_method); } break; case s_start_res: { parser->flags = 0; parser->content_length = ULLONG_MAX; switch (ch) { case 'H': UPDATE_STATE(s_res_H); break; case CR: case LF: break; default: SET_ERRNO(HPE_INVALID_CONSTANT); goto error; } CALLBACK_NOTIFY(message_begin); break; } case s_res_H: STRICT_CHECK(ch != 'T'); UPDATE_STATE(s_res_HT); break; case s_res_HT: STRICT_CHECK(ch != 'T'); UPDATE_STATE(s_res_HTT); break; case s_res_HTT: STRICT_CHECK(ch != 'P'); UPDATE_STATE(s_res_HTTP); break; case s_res_HTTP: STRICT_CHECK(ch != '/'); UPDATE_STATE(s_res_first_http_major); break; case s_res_first_http_major: if (UNLIKELY(ch < '0' || ch > '9')) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_major = ch - '0'; UPDATE_STATE(s_res_http_major); break; /* major HTTP version or dot */ case s_res_http_major: { if (ch == '.') { UPDATE_STATE(s_res_first_http_minor); break; } if (!IS_NUM(ch)) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_major *= 10; parser->http_major += ch - '0'; if (UNLIKELY(parser->http_major > 999)) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } break; } /* first digit of minor HTTP version */ case s_res_first_http_minor: if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_minor = ch - '0'; UPDATE_STATE(s_res_http_minor); break; /* minor HTTP version or end of request line */ case s_res_http_minor: { if (ch == ' ') { UPDATE_STATE(s_res_first_status_code); break; } if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_minor *= 10; parser->http_minor += ch - '0'; if (UNLIKELY(parser->http_minor > 999)) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } break; } case s_res_first_status_code: { if (!IS_NUM(ch)) { if (ch == ' ') { break; } SET_ERRNO(HPE_INVALID_STATUS); goto error; } parser->status_code = ch - '0'; UPDATE_STATE(s_res_status_code); break; } case s_res_status_code: { if (!IS_NUM(ch)) { switch (ch) { case ' ': UPDATE_STATE(s_res_status_start); break; case CR: UPDATE_STATE(s_res_line_almost_done); break; case LF: UPDATE_STATE(s_header_field_start); break; default: SET_ERRNO(HPE_INVALID_STATUS); goto error; } break; } parser->status_code *= 10; parser->status_code += ch - '0'; if (UNLIKELY(parser->status_code > 999)) { SET_ERRNO(HPE_INVALID_STATUS); goto error; } break; } case s_res_status_start: { if (ch == CR) { UPDATE_STATE(s_res_line_almost_done); break; } if (ch == LF) { UPDATE_STATE(s_header_field_start); break; } MARK(status); UPDATE_STATE(s_res_status); parser->index = 0; break; } case s_res_status: if (ch == CR) { UPDATE_STATE(s_res_line_almost_done); CALLBACK_DATA(status); break; } if (ch == LF) { UPDATE_STATE(s_header_field_start); CALLBACK_DATA(status); break; } break; case s_res_line_almost_done: STRICT_CHECK(ch != LF); UPDATE_STATE(s_header_field_start); break; case s_start_req: { if (ch == CR || ch == LF) break; parser->flags = 0; parser->content_length = ULLONG_MAX; if (UNLIKELY(!IS_ALPHA(ch))) { SET_ERRNO(HPE_INVALID_METHOD); goto error; } parser->method = (enum http_method) 0; parser->index = 1; switch (ch) { case 'C': parser->method = HTTP_CONNECT; /* or COPY, CHECKOUT */ break; case 'D': parser->method = HTTP_DELETE; break; case 'G': parser->method = HTTP_GET; break; case 'H': parser->method = HTTP_HEAD; break; case 'L': parser->method = HTTP_LOCK; break; case 'M': parser->method = HTTP_MKCOL; /* or MOVE, MKACTIVITY, MERGE, M-SEARCH, MKCALENDAR */ break; case 'N': parser->method = HTTP_NOTIFY; break; case 'O': parser->method = HTTP_OPTIONS; break; case 'P': parser->method = HTTP_POST; /* or PROPFIND|PROPPATCH|PUT|PATCH|PURGE */ break; case 'R': parser->method = HTTP_REPORT; break; case 'S': parser->method = HTTP_SUBSCRIBE; /* or SEARCH */ break; case 'T': parser->method = HTTP_TRACE; break; case 'U': parser->method = HTTP_UNLOCK; /* or UNSUBSCRIBE */ break; default: SET_ERRNO(HPE_INVALID_METHOD); goto error; } UPDATE_STATE(s_req_method); CALLBACK_NOTIFY(message_begin); break; } case s_req_method: { const char *matcher; if (UNLIKELY(ch == '\0')) { SET_ERRNO(HPE_INVALID_METHOD); goto error; } matcher = method_strings[parser->method]; if (ch == ' ' && matcher[parser->index] == '\0') { UPDATE_STATE(s_req_spaces_before_url); } else if (ch == matcher[parser->index]) { ; /* nada */ } else if (parser->method == HTTP_CONNECT) { if (parser->index == 1 && ch == 'H') { parser->method = HTTP_CHECKOUT; } else if (parser->index == 2 && ch == 'P') { parser->method = HTTP_COPY; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else if (parser->method == HTTP_MKCOL) { if (parser->index == 1 && ch == 'O') { parser->method = HTTP_MOVE; } else if (parser->index == 1 && ch == 'E') { parser->method = HTTP_MERGE; } else if (parser->index == 1 && ch == '-') { parser->method = HTTP_MSEARCH; } else if (parser->index == 2 && ch == 'A') { parser->method = HTTP_MKACTIVITY; } else if (parser->index == 3 && ch == 'A') { parser->method = HTTP_MKCALENDAR; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else if (parser->method == HTTP_SUBSCRIBE) { if (parser->index == 1 && ch == 'E') { parser->method = HTTP_SEARCH; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else if (parser->index == 1 && parser->method == HTTP_POST) { if (ch == 'R') { parser->method = HTTP_PROPFIND; /* or HTTP_PROPPATCH */ } else if (ch == 'U') { parser->method = HTTP_PUT; /* or HTTP_PURGE */ } else if (ch == 'A') { parser->method = HTTP_PATCH; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else if (parser->index == 2) { if (parser->method == HTTP_PUT) { if (ch == 'R') { parser->method = HTTP_PURGE; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else if (parser->method == HTTP_UNLOCK) { if (ch == 'S') { parser->method = HTTP_UNSUBSCRIBE; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } } else if (parser->index == 4 && parser->method == HTTP_PROPFIND && ch == 'P') { parser->method = HTTP_PROPPATCH; } else { SET_ERRNO(HPE_INVALID_METHOD); goto error; } ++parser->index; break; } case s_req_spaces_before_url: { if (ch == ' ') break; MARK(url); if (parser->method == HTTP_CONNECT) { UPDATE_STATE(s_req_server_start); } UPDATE_STATE(parse_url_char(CURRENT_STATE(), ch)); if (UNLIKELY(CURRENT_STATE() == s_dead)) { SET_ERRNO(HPE_INVALID_URL); goto error; } break; } case s_req_schema: case s_req_schema_slash: case s_req_schema_slash_slash: case s_req_server_start: { switch (ch) { /* No whitespace allowed here */ case ' ': case CR: case LF: SET_ERRNO(HPE_INVALID_URL); goto error; default: UPDATE_STATE(parse_url_char(CURRENT_STATE(), ch)); if (UNLIKELY(CURRENT_STATE() == s_dead)) { SET_ERRNO(HPE_INVALID_URL); goto error; } } break; } case s_req_server: case s_req_server_with_at: case s_req_path: case s_req_query_string_start: case s_req_query_string: case s_req_fragment_start: case s_req_fragment: { switch (ch) { case ' ': UPDATE_STATE(s_req_http_start); CALLBACK_DATA(url); break; case CR: case LF: parser->http_major = 0; parser->http_minor = 9; UPDATE_STATE((ch == CR) ? s_req_line_almost_done : s_header_field_start); CALLBACK_DATA(url); break; default: UPDATE_STATE(parse_url_char(CURRENT_STATE(), ch)); if (UNLIKELY(CURRENT_STATE() == s_dead)) { SET_ERRNO(HPE_INVALID_URL); goto error; } } break; } case s_req_http_start: switch (ch) { case 'H': UPDATE_STATE(s_req_http_H); break; case ' ': break; default: SET_ERRNO(HPE_INVALID_CONSTANT); goto error; } break; case s_req_http_H: STRICT_CHECK(ch != 'T'); UPDATE_STATE(s_req_http_HT); break; case s_req_http_HT: STRICT_CHECK(ch != 'T'); UPDATE_STATE(s_req_http_HTT); break; case s_req_http_HTT: STRICT_CHECK(ch != 'P'); UPDATE_STATE(s_req_http_HTTP); break; case s_req_http_HTTP: STRICT_CHECK(ch != '/'); UPDATE_STATE(s_req_first_http_major); break; /* first digit of major HTTP version */ case s_req_first_http_major: if (UNLIKELY(ch < '1' || ch > '9')) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_major = ch - '0'; UPDATE_STATE(s_req_http_major); break; /* major HTTP version or dot */ case s_req_http_major: { if (ch == '.') { UPDATE_STATE(s_req_first_http_minor); break; } if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_major *= 10; parser->http_major += ch - '0'; if (UNLIKELY(parser->http_major > 999)) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } break; } /* first digit of minor HTTP version */ case s_req_first_http_minor: if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_minor = ch - '0'; UPDATE_STATE(s_req_http_minor); break; /* minor HTTP version or end of request line */ case s_req_http_minor: { if (ch == CR) { UPDATE_STATE(s_req_line_almost_done); break; } if (ch == LF) { UPDATE_STATE(s_header_field_start); break; } /* XXX allow spaces after digit? */ if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } parser->http_minor *= 10; parser->http_minor += ch - '0'; if (UNLIKELY(parser->http_minor > 999)) { SET_ERRNO(HPE_INVALID_VERSION); goto error; } break; } /* end of request line */ case s_req_line_almost_done: { if (UNLIKELY(ch != LF)) { SET_ERRNO(HPE_LF_EXPECTED); goto error; } UPDATE_STATE(s_header_field_start); break; } case s_header_field_start: { if (ch == CR) { UPDATE_STATE(s_headers_almost_done); break; } if (ch == LF) { /* they might be just sending \n instead of \r\n so this would be * the second \n to denote the end of headers*/ UPDATE_STATE(s_headers_almost_done); REEXECUTE(); } c = TOKEN(ch); if (UNLIKELY(!c)) { SET_ERRNO(HPE_INVALID_HEADER_TOKEN); goto error; } MARK(header_field); parser->index = 0; UPDATE_STATE(s_header_field); switch (c) { case 'c': parser->header_state = h_C; break; case 'p': parser->header_state = h_matching_proxy_connection; break; case 't': parser->header_state = h_matching_transfer_encoding; break; case 'u': parser->header_state = h_matching_upgrade; break; default: parser->header_state = h_general; break; } break; } case s_header_field: { const char* start = p; for (; p != data + len; p++) { ch = *p; c = TOKEN(ch); if (!c) break; switch (parser->header_state) { case h_general: break; case h_C: parser->index++; parser->header_state = (c == 'o' ? h_CO : h_general); break; case h_CO: parser->index++; parser->header_state = (c == 'n' ? h_CON : h_general); break; case h_CON: parser->index++; switch (c) { case 'n': parser->header_state = h_matching_connection; break; case 't': parser->header_state = h_matching_content_length; break; default: parser->header_state = h_general; break; } break; /* connection */ case h_matching_connection: parser->index++; if (parser->index > sizeof(CONNECTION)-1 || c != CONNECTION[parser->index]) { parser->header_state = h_general; } else if (parser->index == sizeof(CONNECTION)-2) { parser->header_state = h_connection; } break; /* proxy-connection */ case h_matching_proxy_connection: parser->index++; if (parser->index > sizeof(PROXY_CONNECTION)-1 || c != PROXY_CONNECTION[parser->index]) { parser->header_state = h_general; } else if (parser->index == sizeof(PROXY_CONNECTION)-2) { parser->header_state = h_connection; } break; /* content-length */ case h_matching_content_length: parser->index++; if (parser->index > sizeof(CONTENT_LENGTH)-1 || c != CONTENT_LENGTH[parser->index]) { parser->header_state = h_general; } else if (parser->index == sizeof(CONTENT_LENGTH)-2) { parser->header_state = h_content_length; } break; /* transfer-encoding */ case h_matching_transfer_encoding: parser->index++; if (parser->index > sizeof(TRANSFER_ENCODING)-1 || c != TRANSFER_ENCODING[parser->index]) { parser->header_state = h_general; } else if (parser->index == sizeof(TRANSFER_ENCODING)-2) { parser->header_state = h_transfer_encoding; } break; /* upgrade */ case h_matching_upgrade: parser->index++; if (parser->index > sizeof(UPGRADE)-1 || c != UPGRADE[parser->index]) { parser->header_state = h_general; } else if (parser->index == sizeof(UPGRADE)-2) { parser->header_state = h_upgrade; } break; case h_connection: case h_content_length: case h_transfer_encoding: case h_upgrade: if (ch != ' ') parser->header_state = h_general; break; default: assert(0 && "Unknown header_state"); break; } } COUNT_HEADER_SIZE(p - start); if (p == data + len) { --p; break; } if (ch == ':') { UPDATE_STATE(s_header_value_discard_ws); CALLBACK_DATA(header_field); break; } SET_ERRNO(HPE_INVALID_HEADER_TOKEN); goto error; } case s_header_value_discard_ws: if (ch == ' ' || ch == '\t') break; if (ch == CR) { UPDATE_STATE(s_header_value_discard_ws_almost_done); break; } if (ch == LF) { UPDATE_STATE(s_header_value_discard_lws); break; } /* FALLTHROUGH */ case s_header_value_start: { MARK(header_value); UPDATE_STATE(s_header_value); parser->index = 0; c = LOWER(ch); switch (parser->header_state) { case h_upgrade: parser->flags |= F_UPGRADE; parser->header_state = h_general; break; case h_transfer_encoding: /* looking for 'Transfer-Encoding: chunked' */ if ('c' == c) { parser->header_state = h_matching_transfer_encoding_chunked; } else { parser->header_state = h_general; } break; case h_content_length: if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_CONTENT_LENGTH); goto error; } parser->content_length = ch - '0'; break; case h_connection: /* looking for 'Connection: keep-alive' */ if (c == 'k') { parser->header_state = h_matching_connection_keep_alive; /* looking for 'Connection: close' */ } else if (c == 'c') { parser->header_state = h_matching_connection_close; } else if (c == 'u') { parser->header_state = h_matching_connection_upgrade; } else { parser->header_state = h_matching_connection_token; } break; /* Multi-value `Connection` header */ case h_matching_connection_token_start: break; default: parser->header_state = h_general; break; } break; } case s_header_value: { const char* start = p; enum header_states h_state = (enum header_states) parser->header_state; for (; p != data + len; p++) { ch = *p; if (ch == CR) { UPDATE_STATE(s_header_almost_done); parser->header_state = h_state; CALLBACK_DATA(header_value); break; } if (ch == LF) { UPDATE_STATE(s_header_almost_done); COUNT_HEADER_SIZE(p - start); parser->header_state = h_state; CALLBACK_DATA_NOADVANCE(header_value); REEXECUTE(); } c = LOWER(ch); switch (h_state) { case h_general: { const char* p_cr; const char* p_lf; size_t limit = data + len - p; limit = MIN(limit, HTTP_MAX_HEADER_SIZE); p_cr = (const char*) memchr(p, CR, limit); p_lf = (const char*) memchr(p, LF, limit); if (p_cr != NULL) { if (p_lf != NULL && p_cr >= p_lf) p = p_lf; else p = p_cr; } else if (UNLIKELY(p_lf != NULL)) { p = p_lf; } else { p = data + len; } --p; break; } case h_connection: case h_transfer_encoding: assert(0 && "Shouldn't get here."); break; case h_content_length: { uint64_t t; if (ch == ' ') break; if (UNLIKELY(!IS_NUM(ch))) { SET_ERRNO(HPE_INVALID_CONTENT_LENGTH); parser->header_state = h_state; goto error; } t = parser->content_length; t *= 10; t += ch - '0'; /* Overflow? Test against a conservative limit for simplicity. */ if (UNLIKELY((ULLONG_MAX - 10) / 10 < parser->content_length)) { SET_ERRNO(HPE_INVALID_CONTENT_LENGTH); parser->header_state = h_state; goto error; } parser->content_length = t; break; } /* Transfer-Encoding: chunked */ case h_matching_transfer_encoding_chunked: parser->index++; if (parser->index > sizeof(CHUNKED)-1 || c != CHUNKED[parser->index]) { h_state = h_general; } else if (parser->index == sizeof(CHUNKED)-2) { h_state = h_transfer_encoding_chunked; } break; case h_matching_connection_token_start: /* looking for 'Connection: keep-alive' */ if (c == 'k') { h_state = h_matching_connection_keep_alive; /* looking for 'Connection: close' */ } else if (c == 'c') { h_state = h_matching_connection_close; } else if (c == 'u') { h_state = h_matching_connection_upgrade; } else if (STRICT_TOKEN(c)) { h_state = h_matching_connection_token; } else if (c == ' ' || c == '\t') { /* Skip lws */ } else { h_state = h_general; } break; /* looking for 'Connection: keep-alive' */ case h_matching_connection_keep_alive: parser->index++; if (parser->index > sizeof(KEEP_ALIVE)-1 || c != KEEP_ALIVE[parser->index]) { h_state = h_matching_connection_token; } else if (parser->index == sizeof(KEEP_ALIVE)-2) { h_state = h_connection_keep_alive; } break; /* looking for 'Connection: close' */ case h_matching_connection_close: parser->index++; if (parser->index > sizeof(CLOSE)-1 || c != CLOSE[parser->index]) { h_state = h_matching_connection_token; } else if (parser->index == sizeof(CLOSE)-2) { h_state = h_connection_close; } break; /* looking for 'Connection: upgrade' */ case h_matching_connection_upgrade: parser->index++; if (parser->index > sizeof(UPGRADE) - 1 || c != UPGRADE[parser->index]) { h_state = h_matching_connection_token; } else if (parser->index == sizeof(UPGRADE)-2) { h_state = h_connection_upgrade; } break; case h_matching_connection_token: if (ch == ',') { h_state = h_matching_connection_token_start; parser->index = 0; } break; case h_transfer_encoding_chunked: if (ch != ' ') h_state = h_general; break; case h_connection_keep_alive: case h_connection_close: case h_connection_upgrade: if (ch == ',') { if (h_state == h_connection_keep_alive) { parser->flags |= F_CONNECTION_KEEP_ALIVE; } else if (h_state == h_connection_close) { parser->flags |= F_CONNECTION_CLOSE; } else if (h_state == h_connection_upgrade) { parser->flags |= F_CONNECTION_UPGRADE; } h_state = h_matching_connection_token_start; parser->index = 0; } else if (ch != ' ') { h_state = h_matching_connection_token; } break; default: UPDATE_STATE(s_header_value); h_state = h_general; break; } } parser->header_state = h_state; COUNT_HEADER_SIZE(p - start); if (p == data + len) --p; break; } case s_header_almost_done: { STRICT_CHECK(ch != LF); UPDATE_STATE(s_header_value_lws); break; } case s_header_value_lws: { if (ch == ' ' || ch == '\t') { UPDATE_STATE(s_header_value_start); REEXECUTE(); } /* finished the header */ switch (parser->header_state) { case h_connection_keep_alive: parser->flags |= F_CONNECTION_KEEP_ALIVE; break; case h_connection_close: parser->flags |= F_CONNECTION_CLOSE; break; case h_transfer_encoding_chunked: parser->flags |= F_CHUNKED; break; case h_connection_upgrade: parser->flags |= F_CONNECTION_UPGRADE; break; default: break; } UPDATE_STATE(s_header_field_start); REEXECUTE(); } case s_header_value_discard_ws_almost_done: { STRICT_CHECK(ch != LF); UPDATE_STATE(s_header_value_discard_lws); break; } case s_header_value_discard_lws: { if (ch == ' ' || ch == '\t') { UPDATE_STATE(s_header_value_discard_ws); break; } else { switch (parser->header_state) { case h_connection_keep_alive: parser->flags |= F_CONNECTION_KEEP_ALIVE; break; case h_connection_close: parser->flags |= F_CONNECTION_CLOSE; break; case h_connection_upgrade: parser->flags |= F_CONNECTION_UPGRADE; break; case h_transfer_encoding_chunked: parser->flags |= F_CHUNKED; break; default: break; } /* header value was empty */ MARK(header_value); UPDATE_STATE(s_header_field_start); CALLBACK_DATA_NOADVANCE(header_value); REEXECUTE(); } } case s_headers_almost_done: { STRICT_CHECK(ch != LF); if (parser->flags & F_TRAILING) { /* End of a chunked request */ UPDATE_STATE(NEW_MESSAGE()); CALLBACK_NOTIFY(message_complete); break; } UPDATE_STATE(s_headers_done); /* Set this here so that on_headers_complete() callbacks can see it */ parser->upgrade = ((parser->flags & (F_UPGRADE | F_CONNECTION_UPGRADE)) == (F_UPGRADE | F_CONNECTION_UPGRADE) || parser->method == HTTP_CONNECT); /* Here we call the headers_complete callback. This is somewhat * different than other callbacks because if the user returns 1, we * will interpret that as saying that this message has no body. This * is needed for the annoying case of recieving a response to a HEAD * request. * * We'd like to use CALLBACK_NOTIFY_NOADVANCE() here but we cannot, so * we have to simulate it by handling a change in errno below. */ if (settings->on_headers_complete) { switch (settings->on_headers_complete(parser)) { case 0: break; case 1: parser->flags |= F_SKIPBODY; break; default: SET_ERRNO(HPE_CB_headers_complete); RETURN(p - data); /* Error */ } } if (HTTP_PARSER_ERRNO(parser) != HPE_OK) { RETURN(p - data); } REEXECUTE(); } case s_headers_done: { STRICT_CHECK(ch != LF); parser->nread = 0; /* Exit, the rest of the connect is in a different protocol. */ if (parser->upgrade) { UPDATE_STATE(NEW_MESSAGE()); CALLBACK_NOTIFY(message_complete); RETURN((p - data) + 1); } if (parser->flags & F_SKIPBODY) { UPDATE_STATE(NEW_MESSAGE()); CALLBACK_NOTIFY(message_complete); } else if (parser->flags & F_CHUNKED) { /* chunked encoding - ignore Content-Length header */ UPDATE_STATE(s_chunk_size_start); } else { if (parser->content_length == 0) { /* Content-Length header given but zero: Content-Length: 0\r\n */ UPDATE_STATE(NEW_MESSAGE()); CALLBACK_NOTIFY(message_complete); } else if (parser->content_length != ULLONG_MAX) { /* Content-Length header given and non-zero */ UPDATE_STATE(s_body_identity); } else { if (parser->type == HTTP_REQUEST || !http_message_needs_eof(parser)) { /* Assume content-length 0 - read the next */ UPDATE_STATE(NEW_MESSAGE()); CALLBACK_NOTIFY(message_complete); } else { /* Read body until EOF */ UPDATE_STATE(s_body_identity_eof); } } } break; } case s_body_identity: { uint64_t to_read = MIN(parser->content_length, (uint64_t) ((data + len) - p)); assert(parser->content_length != 0 && parser->content_length != ULLONG_MAX); /* The difference between advancing content_length and p is because * the latter will automaticaly advance on the next loop iteration. * Further, if content_length ends up at 0, we want to see the last * byte again for our message complete callback. */ MARK(body); parser->content_length -= to_read; p += to_read - 1; if (parser->content_length == 0) { UPDATE_STATE(s_message_done); /* Mimic CALLBACK_DATA_NOADVANCE() but with one extra byte. * * The alternative to doing this is to wait for the next byte to * trigger the data callback, just as in every other case. The * problem with this is that this makes it difficult for the test * harness to distinguish between complete-on-EOF and * complete-on-length. It's not clear that this distinction is * important for applications, but let's keep it for now. */ CALLBACK_DATA_(body, p - body_mark + 1, p - data); REEXECUTE(); } break; } /* read until EOF */ case s_body_identity_eof: MARK(body); p = data + len - 1; break; case s_message_done: UPDATE_STATE(NEW_MESSAGE()); CALLBACK_NOTIFY(message_complete); break; case s_chunk_size_start: { assert(parser->nread == 1); assert(parser->flags & F_CHUNKED); unhex_val = unhex[(unsigned char)ch]; if (UNLIKELY(unhex_val == -1)) { SET_ERRNO(HPE_INVALID_CHUNK_SIZE); goto error; } parser->content_length = unhex_val; UPDATE_STATE(s_chunk_size); break; } case s_chunk_size: { uint64_t t; assert(parser->flags & F_CHUNKED); if (ch == CR) { UPDATE_STATE(s_chunk_size_almost_done); break; } unhex_val = unhex[(unsigned char)ch]; if (unhex_val == -1) { if (ch == ';' || ch == ' ') { UPDATE_STATE(s_chunk_parameters); break; } SET_ERRNO(HPE_INVALID_CHUNK_SIZE); goto error; } t = parser->content_length; t *= 16; t += unhex_val; /* Overflow? Test against a conservative limit for simplicity. */ if (UNLIKELY((ULLONG_MAX - 16) / 16 < parser->content_length)) { SET_ERRNO(HPE_INVALID_CONTENT_LENGTH); goto error; } parser->content_length = t; break; } case s_chunk_parameters: { assert(parser->flags & F_CHUNKED); /* just ignore this shit. TODO check for overflow */ if (ch == CR) { UPDATE_STATE(s_chunk_size_almost_done); break; } break; } case s_chunk_size_almost_done: { assert(parser->flags & F_CHUNKED); STRICT_CHECK(ch != LF); parser->nread = 0; if (parser->content_length == 0) { parser->flags |= F_TRAILING; UPDATE_STATE(s_header_field_start); } else { UPDATE_STATE(s_chunk_data); } break; } case s_chunk_data: { uint64_t to_read = MIN(parser->content_length, (uint64_t) ((data + len) - p)); assert(parser->flags & F_CHUNKED); assert(parser->content_length != 0 && parser->content_length != ULLONG_MAX); /* See the explanation in s_body_identity for why the content * length and data pointers are managed this way. */ MARK(body); parser->content_length -= to_read; p += to_read - 1; if (parser->content_length == 0) { UPDATE_STATE(s_chunk_data_almost_done); } break; } case s_chunk_data_almost_done: assert(parser->flags & F_CHUNKED); assert(parser->content_length == 0); STRICT_CHECK(ch != CR); UPDATE_STATE(s_chunk_data_done); CALLBACK_DATA(body); break; case s_chunk_data_done: assert(parser->flags & F_CHUNKED); STRICT_CHECK(ch != LF); parser->nread = 0; UPDATE_STATE(s_chunk_size_start); break; default: assert(0 && "unhandled state"); SET_ERRNO(HPE_INVALID_INTERNAL_STATE); goto error; } } /* Run callbacks for any marks that we have leftover after we ran our of * bytes. There should be at most one of these set, so it's OK to invoke * them in series (unset marks will not result in callbacks). * * We use the NOADVANCE() variety of callbacks here because 'p' has already * overflowed 'data' and this allows us to correct for the off-by-one that * we'd otherwise have (since CALLBACK_DATA() is meant to be run with a 'p' * value that's in-bounds). */ assert(((header_field_mark ? 1 : 0) + (header_value_mark ? 1 : 0) + (url_mark ? 1 : 0) + (body_mark ? 1 : 0) + (status_mark ? 1 : 0)) <= 1); CALLBACK_DATA_NOADVANCE(header_field); CALLBACK_DATA_NOADVANCE(header_value); CALLBACK_DATA_NOADVANCE(url); CALLBACK_DATA_NOADVANCE(body); CALLBACK_DATA_NOADVANCE(status); RETURN(len); error: if (HTTP_PARSER_ERRNO(parser) == HPE_OK) { SET_ERRNO(HPE_UNKNOWN); } RETURN(p - data); } /* Does the parser need to see an EOF to find the end of the message? */ int http_message_needs_eof (const http_parser *parser) { if (parser->type == HTTP_REQUEST) { return 0; } /* See RFC 2616 section 4.4 */ if (parser->status_code / 100 == 1 || /* 1xx e.g. Continue */ parser->status_code == 204 || /* No Content */ parser->status_code == 304 || /* Not Modified */ parser->flags & F_SKIPBODY) { /* response to a HEAD request */ return 0; } if ((parser->flags & F_CHUNKED) || parser->content_length != ULLONG_MAX) { return 0; } return 1; } int http_should_keep_alive (const http_parser *parser) { if (parser->http_major > 0 && parser->http_minor > 0) { /* HTTP/1.1 */ if (parser->flags & F_CONNECTION_CLOSE) { return 0; } } else { /* HTTP/1.0 or earlier */ if (!(parser->flags & F_CONNECTION_KEEP_ALIVE)) { return 0; } } return !http_message_needs_eof(parser); } const char * http_method_str (enum http_method m) { return ELEM_AT(method_strings, m, ""); } void http_parser_init (http_parser *parser, enum http_parser_type t) { void *data = parser->data; /* preserve application data */ memset(parser, 0, sizeof(*parser)); parser->data = data; parser->type = t; parser->state = (t == HTTP_REQUEST ? s_start_req : (t == HTTP_RESPONSE ? s_start_res : s_start_req_or_res)); parser->http_errno = HPE_OK; } void http_parser_settings_init(http_parser_settings *settings) { memset(settings, 0, sizeof(*settings)); } const char * http_errno_name(enum http_errno err) { assert(err < (sizeof(http_strerror_tab)/sizeof(http_strerror_tab[0]))); return http_strerror_tab[err].name; } const char * http_errno_description(enum http_errno err) { assert(err < (sizeof(http_strerror_tab)/sizeof(http_strerror_tab[0]))); return http_strerror_tab[err].description; } static enum http_host_state http_parse_host_char(enum http_host_state s, const char ch) { switch(s) { case s_http_userinfo: case s_http_userinfo_start: if (ch == '@') { return s_http_host_start; } if (IS_USERINFO_CHAR(ch)) { return s_http_userinfo; } break; case s_http_host_start: if (ch == '[') { return s_http_host_v6_start; } if (IS_HOST_CHAR(ch)) { return s_http_host; } break; case s_http_host: if (IS_HOST_CHAR(ch)) { return s_http_host; } /* FALLTHROUGH */ case s_http_host_v6_end: if (ch == ':') { return s_http_host_port_start; } break; case s_http_host_v6: if (ch == ']') { return s_http_host_v6_end; } /* FALLTHROUGH */ case s_http_host_v6_start: if (IS_HEX(ch) || ch == ':' || ch == '.') { return s_http_host_v6; } break; case s_http_host_port: case s_http_host_port_start: if (IS_NUM(ch)) { return s_http_host_port; } break; default: break; } return s_http_host_dead; } static int http_parse_host(const char * buf, struct http_parser_url *u, int found_at) { enum http_host_state s; const char *p; size_t buflen = u->field_data[UF_HOST].off + u->field_data[UF_HOST].len; u->field_data[UF_HOST].len = 0; s = found_at ? s_http_userinfo_start : s_http_host_start; for (p = buf + u->field_data[UF_HOST].off; p < buf + buflen; p++) { enum http_host_state new_s = http_parse_host_char(s, *p); if (new_s == s_http_host_dead) { return 1; } switch(new_s) { case s_http_host: if (s != s_http_host) { u->field_data[UF_HOST].off = p - buf; } u->field_data[UF_HOST].len++; break; case s_http_host_v6: if (s != s_http_host_v6) { u->field_data[UF_HOST].off = p - buf; } u->field_data[UF_HOST].len++; break; case s_http_host_port: if (s != s_http_host_port) { u->field_data[UF_PORT].off = p - buf; u->field_data[UF_PORT].len = 0; u->field_set |= (1 << UF_PORT); } u->field_data[UF_PORT].len++; break; case s_http_userinfo: if (s != s_http_userinfo) { u->field_data[UF_USERINFO].off = p - buf ; u->field_data[UF_USERINFO].len = 0; u->field_set |= (1 << UF_USERINFO); } u->field_data[UF_USERINFO].len++; break; default: break; } s = new_s; } /* Make sure we don't end somewhere unexpected */ switch (s) { case s_http_host_start: case s_http_host_v6_start: case s_http_host_v6: case s_http_host_port_start: case s_http_userinfo: case s_http_userinfo_start: return 1; default: break; } return 0; } int http_parser_parse_url(const char *buf, size_t buflen, int is_connect, struct http_parser_url *u) { enum state s; const char *p; enum http_parser_url_fields uf, old_uf; int found_at = 0; u->port = u->field_set = 0; s = is_connect ? s_req_server_start : s_req_spaces_before_url; old_uf = UF_MAX; for (p = buf; p < buf + buflen; p++) { s = parse_url_char(s, *p); /* Figure out the next field that we're operating on */ switch (s) { case s_dead: return 1; /* Skip delimeters */ case s_req_schema_slash: case s_req_schema_slash_slash: case s_req_server_start: case s_req_query_string_start: case s_req_fragment_start: continue; case s_req_schema: uf = UF_SCHEMA; break; case s_req_server_with_at: found_at = 1; /* FALLTHRU */ case s_req_server: uf = UF_HOST; break; case s_req_path: uf = UF_PATH; break; case s_req_query_string: uf = UF_QUERY; break; case s_req_fragment: uf = UF_FRAGMENT; break; default: assert(!"Unexpected state"); return 1; } /* Nothing's changed; soldier on */ if (uf == old_uf) { u->field_data[uf].len++; continue; } u->field_data[uf].off = p - buf; u->field_data[uf].len = 1; u->field_set |= (1 << uf); old_uf = uf; } /* host must be present if there is a schema */ /* parsing http:///toto will fail */ if ((u->field_set & ((1 << UF_SCHEMA) | (1 << UF_HOST))) != 0) { if (http_parse_host(buf, u, found_at) != 0) { return 1; } } /* CONNECT requests can only contain "hostname:port" */ if (is_connect && u->field_set != ((1 << UF_HOST)|(1 << UF_PORT))) { return 1; } if (u->field_set & (1 << UF_PORT)) { /* Don't bother with endp; we've already validated the string */ unsigned long v = strtoul(buf + u->field_data[UF_PORT].off, NULL, 10); /* Ports have a max value of 2^16 */ if (v > 0xffff) { return 1; } u->port = (uint16_t) v; } return 0; } void http_parser_pause(http_parser *parser, int paused) { /* Users should only be pausing/unpausing a parser that is not in an error * state. In non-debug builds, there's not much that we can do about this * other than ignore it. */ if (HTTP_PARSER_ERRNO(parser) == HPE_OK || HTTP_PARSER_ERRNO(parser) == HPE_PAUSED) { SET_ERRNO((paused) ? HPE_PAUSED : HPE_OK); } else { assert(0 && "Attempting to pause parser in error state"); } } int http_body_is_final(const struct http_parser *parser) { return parser->state == s_message_done; } unsigned long http_parser_version(void) { return HTTP_PARSER_VERSION_MAJOR * 0x10000 | HTTP_PARSER_VERSION_MINOR * 0x00100 | HTTP_PARSER_VERSION_PATCH * 0x00001; } #endif ================================================ FILE: src/ngx_stream_http_parser.h ================================================ /* Copyright Joyent, Inc. and other Node contributors. All rights reserved. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to * deal in the Software without restriction, including without limitation the * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or * sell copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS * IN THE SOFTWARE. */ #ifndef http_parser_h #define http_parser_h #ifdef __cplusplus extern "C" { #endif /* Also update SONAME in the Makefile whenever you change these. */ #define HTTP_PARSER_VERSION_MAJOR 2 #define HTTP_PARSER_VERSION_MINOR 4 #define HTTP_PARSER_VERSION_PATCH 2 #include #if defined(_WIN32) && !defined(__MINGW32__) && (!defined(_MSC_VER) || _MSC_VER<1600) #include #include typedef __int8 int8_t; typedef unsigned __int8 uint8_t; typedef __int16 int16_t; typedef unsigned __int16 uint16_t; typedef __int32 int32_t; typedef unsigned __int32 uint32_t; typedef __int64 int64_t; typedef unsigned __int64 uint64_t; #else #include #endif /* Compile with -DHTTP_PARSER_STRICT=0 to make less checks, but run * faster */ #ifndef HTTP_PARSER_STRICT # define HTTP_PARSER_STRICT 1 #endif /* Maximium header size allowed. If the macro is not defined * before including this header then the default is used. To * change the maximum header size, define the macro in the build * environment (e.g. -DHTTP_MAX_HEADER_SIZE=). To remove * the effective limit on the size of the header, define the macro * to a very large number (e.g. -DHTTP_MAX_HEADER_SIZE=0x7fffffff) */ #ifndef HTTP_MAX_HEADER_SIZE # define HTTP_MAX_HEADER_SIZE (80*1024) #endif typedef struct http_parser http_parser; typedef struct http_parser_settings http_parser_settings; /* Callbacks should return non-zero to indicate an error. The parser will * then halt execution. * * The one exception is on_headers_complete. In a HTTP_RESPONSE parser * returning '1' from on_headers_complete will tell the parser that it * should not expect a body. This is used when receiving a response to a * HEAD request which may contain 'Content-Length' or 'Transfer-Encoding: * chunked' headers that indicate the presence of a body. * * http_data_cb does not return data chunks. It will be called arbitrarily * many times for each string. E.G. you might get 10 callbacks for "on_url" * each providing just a few characters more data. */ typedef int (*http_data_cb) (http_parser*, const char *at, size_t length); typedef int (*http_cb) (http_parser*); /* Request Methods */ #define HTTP_METHOD_MAP(XX) \ XX(0, DELETE, DELETE) \ XX(1, GET, GET) \ XX(2, HEAD, HEAD) \ XX(3, POST, POST) \ XX(4, PUT, PUT) \ /* pathological */ \ XX(5, CONNECT, CONNECT) \ XX(6, OPTIONS, OPTIONS) \ XX(7, TRACE, TRACE) \ /* webdav */ \ XX(8, COPY, COPY) \ XX(9, LOCK, LOCK) \ XX(10, MKCOL, MKCOL) \ XX(11, MOVE, MOVE) \ XX(12, PROPFIND, PROPFIND) \ XX(13, PROPPATCH, PROPPATCH) \ XX(14, SEARCH, SEARCH) \ XX(15, UNLOCK, UNLOCK) \ /* subversion */ \ XX(16, REPORT, REPORT) \ XX(17, MKACTIVITY, MKACTIVITY) \ XX(18, CHECKOUT, CHECKOUT) \ XX(19, MERGE, MERGE) \ /* upnp */ \ XX(20, MSEARCH, M-SEARCH) \ XX(21, NOTIFY, NOTIFY) \ XX(22, SUBSCRIBE, SUBSCRIBE) \ XX(23, UNSUBSCRIBE, UNSUBSCRIBE) \ /* RFC-5789 */ \ XX(24, PATCH, PATCH) \ XX(25, PURGE, PURGE) \ /* CalDAV */ \ XX(26, MKCALENDAR, MKCALENDAR) \ enum http_method { #define XX(num, name, string) HTTP_##name = num, HTTP_METHOD_MAP(XX) #undef XX }; enum http_parser_type { HTTP_REQUEST, HTTP_RESPONSE, HTTP_BOTH }; /* Flag values for http_parser.flags field */ enum flags { F_CHUNKED = 1 << 0 , F_CONNECTION_KEEP_ALIVE = 1 << 1 , F_CONNECTION_CLOSE = 1 << 2 , F_CONNECTION_UPGRADE = 1 << 3 , F_TRAILING = 1 << 4 , F_UPGRADE = 1 << 5 , F_SKIPBODY = 1 << 6 }; /* Map for errno-related constants * * The provided argument should be a macro that takes 2 arguments. */ #define HTTP_ERRNO_MAP(XX) \ /* No error */ \ XX(OK, "success") \ \ /* Callback-related errors */ \ XX(CB_message_begin, "the on_message_begin callback failed") \ XX(CB_url, "the on_url callback failed") \ XX(CB_header_field, "the on_header_field callback failed") \ XX(CB_header_value, "the on_header_value callback failed") \ XX(CB_headers_complete, "the on_headers_complete callback failed") \ XX(CB_body, "the on_body callback failed") \ XX(CB_message_complete, "the on_message_complete callback failed") \ XX(CB_status, "the on_status callback failed") \ \ /* Parsing-related errors */ \ XX(INVALID_EOF_STATE, "stream ended at an unexpected time") \ XX(HEADER_OVERFLOW, \ "too many header bytes seen; overflow detected") \ XX(CLOSED_CONNECTION, \ "data received after completed connection: close message") \ XX(INVALID_VERSION, "invalid HTTP version") \ XX(INVALID_STATUS, "invalid HTTP status code") \ XX(INVALID_METHOD, "invalid HTTP method") \ XX(INVALID_URL, "invalid URL") \ XX(INVALID_HOST, "invalid host") \ XX(INVALID_PORT, "invalid port") \ XX(INVALID_PATH, "invalid path") \ XX(INVALID_QUERY_STRING, "invalid query string") \ XX(INVALID_FRAGMENT, "invalid fragment") \ XX(LF_EXPECTED, "LF character expected") \ XX(INVALID_HEADER_TOKEN, "invalid character in header") \ XX(INVALID_CONTENT_LENGTH, \ "invalid character in content-length header") \ XX(INVALID_CHUNK_SIZE, \ "invalid character in chunk size header") \ XX(INVALID_CONSTANT, "invalid constant string") \ XX(INVALID_INTERNAL_STATE, "encountered unexpected internal state")\ XX(STRICT, "strict mode assertion failed") \ XX(PAUSED, "parser is paused") \ XX(UNKNOWN, "an unknown error occurred") /* Define HPE_* values for each errno value above */ #define HTTP_ERRNO_GEN(n, s) HPE_##n, enum http_errno { HTTP_ERRNO_MAP(HTTP_ERRNO_GEN) }; #undef HTTP_ERRNO_GEN /* Get an http_errno value from an http_parser */ #define HTTP_PARSER_ERRNO(p) ((enum http_errno) (p)->http_errno) struct http_parser { /** PRIVATE **/ unsigned int type : 2; /* enum http_parser_type */ unsigned int flags : 7; /* F_* values from 'flags' enum; semi-public */ unsigned int state : 7; /* enum state from http_parser.c */ unsigned int header_state : 8; /* enum header_state from http_parser.c */ unsigned int index : 8; /* index into current matcher */ uint32_t nread; /* # bytes read in various scenarios */ uint64_t content_length; /* # bytes in body (0 if no Content-Length header) */ /** READ-ONLY **/ unsigned short http_major; unsigned short http_minor; unsigned int status_code : 16; /* responses only */ unsigned int method : 8; /* requests only */ unsigned int http_errno : 7; /* 1 = Upgrade header was present and the parser has exited because of that. * 0 = No upgrade header present. * Should be checked when http_parser_execute() returns in addition to * error checking. */ unsigned int upgrade : 1; /** PUBLIC **/ void *data; /* A pointer to get hook to the "connection" or "socket" object */ }; struct http_parser_settings { http_cb on_message_begin; http_data_cb on_url; http_data_cb on_status; http_data_cb on_header_field; http_data_cb on_header_value; http_cb on_headers_complete; http_data_cb on_body; http_cb on_message_complete; }; enum http_parser_url_fields { UF_SCHEMA = 0 , UF_HOST = 1 , UF_PORT = 2 , UF_PATH = 3 , UF_QUERY = 4 , UF_FRAGMENT = 5 , UF_USERINFO = 6 , UF_MAX = 7 }; /* Result structure for http_parser_parse_url(). * * Callers should index into field_data[] with UF_* values iff field_set * has the relevant (1 << UF_*) bit set. As a courtesy to clients (and * because we probably have padding left over), we convert any port to * a uint16_t. */ struct http_parser_url { uint16_t field_set; /* Bitmask of (1 << UF_*) values */ uint16_t port; /* Converted UF_PORT string */ struct { uint16_t off; /* Offset into buffer in which field starts */ uint16_t len; /* Length of run in buffer */ } field_data[UF_MAX]; }; /* Returns the library version. Bits 16-23 contain the major version number, * bits 8-15 the minor version number and bits 0-7 the patch level. * Usage example: * * unsigned long version = http_parser_version(); * unsigned major = (version >> 16) & 255; * unsigned minor = (version >> 8) & 255; * unsigned patch = version & 255; * printf("http_parser v%u.%u.%u\n", major, minor, patch); */ unsigned long http_parser_version(void); void http_parser_init(http_parser *parser, enum http_parser_type type); /* Initialize http_parser_settings members to 0 */ void http_parser_settings_init(http_parser_settings *settings); /* Executes the parser. Returns number of parsed bytes. Sets * `parser->http_errno` on error. */ size_t http_parser_execute(http_parser *parser, const http_parser_settings *settings, const char *data, size_t len); /* If http_should_keep_alive() in the on_headers_complete or * on_message_complete callback returns 0, then this should be * the last message on the connection. * If you are the server, respond with the "Connection: close" header. * If you are the client, close the connection. */ int http_should_keep_alive(const http_parser *parser); /* Returns a string version of the HTTP method. */ const char *http_method_str(enum http_method m); /* Return a string name of the given error */ const char *http_errno_name(enum http_errno err); /* Return a string description of the given error */ const char *http_errno_description(enum http_errno err); /* Parse a URL; return nonzero on failure */ int http_parser_parse_url(const char *buf, size_t buflen, int is_connect, struct http_parser_url *u); /* Pause or un-pause the parser; a nonzero value pauses */ void http_parser_pause(http_parser *parser, int paused); /* Checks if this is the final chunk of the body. */ int http_body_is_final(const http_parser *parser); #ifdef __cplusplus } #endif #endif ================================================ FILE: src/ngx_stream_json.c ================================================ /* Copyright (c) 2009 Dave Gamble Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ /* cJSON */ /* JSON parser in C. */ #include #ifndef NGX_HTTP_UPSYNC #include #include #include #include #include #include #include #include "ngx_stream_json.h" static const char *ep; const char *cJSON_GetErrorPtr(void) {return ep;} static int cJSON_strcasecmp(const char *s1,const char *s2) { if (!s1) return (s1==s2)?0:1; if (!s2) return 1; for(; tolower(*s1) == tolower(*s2); ++s1, ++s2) if(*s1 == 0) return 0; return tolower(*(const unsigned char *)s1) - tolower(*(const unsigned char *)s2); } static void *(*cJSON_malloc)(size_t sz) = malloc; static void (*cJSON_free)(void *ptr) = free; static char* cJSON_strdup(const char* str) { size_t len; char* copy; len = strlen(str) + 1; if (!(copy = (char*)cJSON_malloc(len))) return 0; memcpy(copy,str,len); return copy; } void cJSON_InitHooks(cJSON_Hooks* hooks) { if (!hooks) { /* Reset hooks */ cJSON_malloc = malloc; cJSON_free = free; return; } cJSON_malloc = (hooks->malloc_fn)?hooks->malloc_fn:malloc; cJSON_free = (hooks->free_fn)?hooks->free_fn:free; } /* Internal constructor. */ static cJSON *cJSON_New_Item(void) { cJSON* node = (cJSON*)cJSON_malloc(sizeof(cJSON)); if (node) memset(node,0,sizeof(cJSON)); return node; } /* Delete a cJSON structure. */ void cJSON_Delete(cJSON *c) { cJSON *next; while (c) { next=c->next; if (!(c->type&cJSON_IsReference) && c->child) cJSON_Delete(c->child); if (!(c->type&cJSON_IsReference) && c->valuestring) cJSON_free(c->valuestring); if (c->string) cJSON_free(c->string); cJSON_free(c); c=next; } } /* Parse the input text to generate a number, and populate the result into item. */ static const char *parse_number(cJSON *item,const char *num) { double n=0,sign=1,scale=0;int subscale=0,signsubscale=1; /* Could use sscanf for this? */ if (*num=='-') sign=-1,num++; /* Has sign? */ if (*num=='0') num++; /* is zero */ if (*num>='1' && *num<='9') do n=(n*10.0)+(*num++ -'0'); while (*num>='0' && *num<='9'); /* Number? */ if (*num=='.' && num[1]>='0' && num[1]<='9') {num++; do n=(n*10.0)+(*num++ -'0'),scale--; while (*num>='0' && *num<='9');} /* Fractional part? */ if (*num=='e' || *num=='E') /* Exponent? */ { num++;if (*num=='+') num++; else if (*num=='-') signsubscale=-1,num++; /* With sign? */ while (*num>='0' && *num<='9') subscale=(subscale*10)+(*num++ - '0'); /* Number? */ } n=sign*n*pow(10.0,(scale+subscale*signsubscale)); /* number = +/- number.fraction * 10^+/- exponent */ item->valuedouble=n; item->valueint=(int)n; item->type=cJSON_Number; return num; } /* Render the number nicely from the given item into a string. */ static char *print_number(cJSON *item) { char *str; double d=item->valuedouble; if (fabs(((double)item->valueint)-d)<=DBL_EPSILON && d<=INT_MAX && d>=INT_MIN) { str=(char*)cJSON_malloc(21); /* 2^64+1 can be represented in 21 chars. */ if (str) sprintf(str,"%d",item->valueint); } else { str=(char*)cJSON_malloc(64); /* This is a nice tradeoff. */ if (str) { if (fabs(floor(d)-d)<=DBL_EPSILON && fabs(d)<1.0e60)sprintf(str,"%.0f",d); else if (fabs(d)<1.0e-6 || fabs(d)>1.0e9) sprintf(str,"%e",d); else sprintf(str,"%f",d); } } return str; } /* Parse the input text into an unescaped cstring, and populate item. */ static const unsigned char firstByteMark[7] = { 0x00, 0x00, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC }; static const char *parse_string(cJSON *item,const char *str) { const char *ptr=str+1;char *ptr2;char *out;int len=0;unsigned uc,uc2; if (*str!='\"') {ep=str;return 0;} /* not a string! */ while (*ptr!='\"' && *ptr && ++len) if (*ptr++ == '\\') ptr++; /* Skip escaped quotes. */ out=(char*)cJSON_malloc(len+1); /* This is how long we need for the string, roughly. */ if (!out) return 0; ptr=str+1;ptr2=out; while (*ptr!='\"' && *ptr) { if (*ptr!='\\') *ptr2++=*ptr++; else { ptr++; switch (*ptr) { case 'b': *ptr2++='\b'; break; case 'f': *ptr2++='\f'; break; case 'n': *ptr2++='\n'; break; case 'r': *ptr2++='\r'; break; case 't': *ptr2++='\t'; break; case 'u': /* transcode utf16 to utf8. */ sscanf(ptr+1,"%4x",&uc);ptr+=4; /* get the unicode char. */ if ((uc>=0xDC00 && uc<=0xDFFF) || uc==0) break; /* check for invalid. */ if (uc>=0xD800 && uc<=0xDBFF) /* UTF16 surrogate pairs. */ { if (ptr[1]!='\\' || ptr[2]!='u') break; /* missing second-half of surrogate. */ sscanf(ptr+3,"%4x",&uc2);ptr+=6; if (uc2<0xDC00 || uc2>0xDFFF) break; /* invalid second-half of surrogate. */ uc=0x10000 + (((uc&0x3FF)<<10) | (uc2&0x3FF)); } len=4;if (uc<0x80) len=1;else if (uc<0x800) len=2;else if (uc<0x10000) len=3; ptr2+=len; switch (len) { case 4: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; /* FALLTHRU */ case 3: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; /* FALLTHRU */ case 2: *--ptr2 =((uc | 0x80) & 0xBF); uc >>= 6; /* FALLTHRU */ case 1: *--ptr2 =(uc | firstByteMark[len]); } ptr2+=len; break; default: *ptr2++=*ptr; break; } ptr++; } } *ptr2=0; if (*ptr=='\"') ptr++; item->valuestring=out; item->type=cJSON_String; return ptr; } /* Render the cstring provided to an escaped version that can be printed. */ static char *print_string_ptr(const char *str) { const char *ptr;char *ptr2,*out;int len=0;unsigned char token; if (!str) return cJSON_strdup(""); ptr=str;while ((token=*ptr) && ++len) {if (strchr("\"\\\b\f\n\r\t",token)) len++; else if (token<32) len+=5;ptr++;} out=(char*)cJSON_malloc(len+3); if (!out) return 0; ptr2=out;ptr=str; *ptr2++='\"'; while (*ptr) { if ((unsigned char)*ptr>31 && *ptr!='\"' && *ptr!='\\') *ptr2++=*ptr++; else { *ptr2++='\\'; switch (token=*ptr++) { case '\\': *ptr2++='\\'; break; case '\"': *ptr2++='\"'; break; case '\b': *ptr2++='b'; break; case '\f': *ptr2++='f'; break; case '\n': *ptr2++='n'; break; case '\r': *ptr2++='r'; break; case '\t': *ptr2++='t'; break; default: sprintf(ptr2,"u%04x",token);ptr2+=5; break; /* escape and print */ } } } *ptr2++='\"';*ptr2++=0; return out; } /* Invote print_string_ptr (which is useful) on an item. */ static char *print_string(cJSON *item) {return print_string_ptr(item->valuestring);} /* Predeclare these prototypes. */ static const char *parse_value(cJSON *item,const char *value); static char *print_value(cJSON *item,int depth,int fmt); static const char *parse_array(cJSON *item,const char *value); static char *print_array(cJSON *item,int depth,int fmt); static const char *parse_object(cJSON *item,const char *value); static char *print_object(cJSON *item,int depth,int fmt); /* Utility to jump whitespace and cr/lf */ static const char *skip(const char *in) {while (in && *in && (unsigned char)*in<=32) in++; return in;} /* Parse an object - create a new root, and populate. */ cJSON *cJSON_ParseWithOpts(const char *value,const char **return_parse_end,int require_null_terminated) { const char *end=0; cJSON *c=cJSON_New_Item(); ep=0; if (!c) return 0; /* memory fail */ end=parse_value(c,skip(value)); if (!end) {cJSON_Delete(c);return 0;} /* parse failure. ep is set. */ /* if we require null-terminated JSON without appended garbage, skip and then check for a null terminator */ if (require_null_terminated) {end=skip(end);if (*end) {cJSON_Delete(c);ep=end;return 0;}} if (return_parse_end) *return_parse_end=end; return c; } /* Default options for cJSON_Parse */ cJSON *cJSON_Parse(const char *value) {return cJSON_ParseWithOpts(value,0,0);} /* Render a cJSON item/entity/structure to text. */ char *cJSON_Print(cJSON *item) {return print_value(item,0,1);} char *cJSON_PrintUnformatted(cJSON *item) {return print_value(item,0,0);} /* Parser core - when encountering text, process appropriately. */ static const char *parse_value(cJSON *item,const char *value) { if (!value) return 0; /* Fail on null. */ if (!strncmp(value,"null",4)) { item->type=cJSON_NULL; return value+4; } if (!strncmp(value,"false",5)) { item->type=cJSON_False; return value+5; } if (!strncmp(value,"true",4)) { item->type=cJSON_True; item->valueint=1; return value+4; } if (*value=='\"') { return parse_string(item,value); } if (*value=='-' || (*value>='0' && *value<='9')) { return parse_number(item,value); } if (*value=='[') { return parse_array(item,value); } if (*value=='{') { return parse_object(item,value); } ep=value;return 0; /* failure. */ } /* Render a value to text. */ static char *print_value(cJSON *item,int depth,int fmt) { char *out=0; if (!item) return 0; switch ((item->type)&255) { case cJSON_NULL: out=cJSON_strdup("null"); break; case cJSON_False: out=cJSON_strdup("false");break; case cJSON_True: out=cJSON_strdup("true"); break; case cJSON_Number: out=print_number(item);break; case cJSON_String: out=print_string(item);break; case cJSON_Array: out=print_array(item,depth,fmt);break; case cJSON_Object: out=print_object(item,depth,fmt);break; } return out; } /* Build an array from input text. */ static const char *parse_array(cJSON *item,const char *value) { cJSON *child; if (*value!='[') {ep=value;return 0;} /* not an array! */ item->type=cJSON_Array; value=skip(value+1); if (*value==']') return value+1; /* empty array. */ item->child=child=cJSON_New_Item(); if (!item->child) return 0; /* memory fail */ value=skip(parse_value(child,skip(value))); /* skip any spacing, get the value. */ if (!value) return 0; while (*value==',') { cJSON *new_item; if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ child->next=new_item;new_item->prev=child;child=new_item; value=skip(parse_value(child,skip(value+1))); if (!value) return 0; /* memory fail */ } if (*value==']') return value+1; /* end of array */ ep=value;return 0; /* malformed. */ } /* Render an array to text */ static char *print_array(cJSON *item,int depth,int fmt) { char **entries; char *out=0,*ptr,*ret;int len=5; cJSON *child=item->child; int numentries=0,i=0,fail=0; /* How many entries in the array? */ while (child) numentries++,child=child->next; /* Explicitly handle numentries==0 */ if (!numentries) { out=(char*)cJSON_malloc(3); if (out) strcpy(out,"[]"); return out; } /* Allocate an array to hold the values for each */ entries=(char**)cJSON_malloc(numentries*sizeof(char*)); if (!entries) return 0; memset(entries,0,numentries*sizeof(char*)); /* Retrieve all the results: */ child=item->child; while (child && !fail) { ret=print_value(child,depth+1,fmt); entries[i++]=ret; if (ret) len+=strlen(ret)+2+(fmt?1:0); else fail=1; child=child->next; } /* If we didn't fail, try to malloc the output string */ if (!fail) out=(char*)cJSON_malloc(len); /* If that fails, we fail. */ if (!out) fail=1; /* Handle failure. */ if (fail) { for (i=0;itype=cJSON_Object; value=skip(value+1); if (*value=='}') return value+1; /* empty array. */ item->child=child=cJSON_New_Item(); if (!item->child) return 0; value=skip(parse_string(child,skip(value))); if (!value) return 0; child->string=child->valuestring;child->valuestring=0; if (*value!=':') {ep=value;return 0;} /* fail! */ value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ if (!value) return 0; while (*value==',') { cJSON *new_item; if (!(new_item=cJSON_New_Item())) return 0; /* memory fail */ child->next=new_item;new_item->prev=child;child=new_item; value=skip(parse_string(child,skip(value+1))); if (!value) return 0; child->string=child->valuestring;child->valuestring=0; if (*value!=':') {ep=value;return 0;} /* fail! */ value=skip(parse_value(child,skip(value+1))); /* skip any spacing, get the value. */ if (!value) return 0; } if (*value=='}') return value+1; /* end of array */ ep=value;return 0; /* malformed. */ } /* Render an object to text. */ static char *print_object(cJSON *item,int depth,int fmt) { char **entries=0,**names=0; char *out=0,*ptr,*ret,*str;int len=7,i=0,j; cJSON *child=item->child; int numentries=0,fail=0; /* Count the number of entries. */ while (child) numentries++,child=child->next; /* Explicitly handle empty object case */ if (!numentries) { out=(char*)cJSON_malloc(fmt?depth+3:3); if (!out) return 0; ptr=out;*ptr++='{'; if (fmt) {*ptr++='\n';for (i=0;ichild;depth++;if (fmt) len+=depth; while (child) { names[i]=str=print_string_ptr(child->string); entries[i++]=ret=print_value(child,depth,fmt); if (str && ret) len+=strlen(ret)+strlen(str)+2+(fmt?2+depth:0); else fail=1; child=child->next; } /* Try to allocate the output string */ if (!fail) out=(char*)cJSON_malloc(len); if (!out) fail=1; /* Handle failure */ if (fail) { for (i=0;ichild;int i=0;while(c)i++,c=c->next;return i;} cJSON *cJSON_GetArrayItem(cJSON *array,int item) {cJSON *c=array->child; while (c && item>0) item--,c=c->next; return c;} cJSON *cJSON_GetObjectItem(cJSON *object,const char *string) {cJSON *c=object->child; while (c && cJSON_strcasecmp(c->string,string)) c=c->next; return c;} /* Utility for array list handling. */ static void suffix_object(cJSON *prev,cJSON *item) {prev->next=item;item->prev=prev;} /* Utility for handling references. */ static cJSON *create_reference(cJSON *item) {cJSON *ref=cJSON_New_Item();if (!ref) return 0;memcpy(ref,item,sizeof(cJSON));ref->string=0;ref->type|=cJSON_IsReference;ref->next=ref->prev=0;return ref;} /* Add item to array/object. */ void cJSON_AddItemToArray(cJSON *array, cJSON *item) {cJSON *c=array->child;if (!item) return; if (!c) {array->child=item;} else {while (c && c->next) c=c->next; suffix_object(c,item);}} void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item) {if (!item) return; if (item->string) cJSON_free(item->string);item->string=cJSON_strdup(string);cJSON_AddItemToArray(object,item);} void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item) {cJSON_AddItemToArray(array,create_reference(item));} void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item) {cJSON_AddItemToObject(object,string,create_reference(item));} cJSON *cJSON_DetachItemFromArray(cJSON *array,int which) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return 0; if (c->prev) c->prev->next=c->next; if (c->next) c->next->prev=c->prev; if (c==array->child) array->child=c->next; c->prev=c->next=0;return c;} void cJSON_DeleteItemFromArray(cJSON *array,int which) {cJSON_Delete(cJSON_DetachItemFromArray(array,which));} cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string) {int i=0;cJSON *c=object->child;while (c && cJSON_strcasecmp(c->string,string)) i++,c=c->next;if (c) return cJSON_DetachItemFromArray(object,i);return 0;} void cJSON_DeleteItemFromObject(cJSON *object,const char *string) {cJSON_Delete(cJSON_DetachItemFromObject(object,string));} /* Replace array/object items with new ones. */ void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem) {cJSON *c=array->child;while (c && which>0) c=c->next,which--;if (!c) return; newitem->next=c->next;newitem->prev=c->prev;if (newitem->next) newitem->next->prev=newitem; if (c==array->child) array->child=newitem; else newitem->prev->next=newitem;c->next=c->prev=0;cJSON_Delete(c);} void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem){int i=0;cJSON *c=object->child;while(c && cJSON_strcasecmp(c->string,string))i++,c=c->next;if(c){newitem->string=cJSON_strdup(string);cJSON_ReplaceItemInArray(object,i,newitem);}} /* Create basic types: */ cJSON *cJSON_CreateNull(void) {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_NULL;return item;} cJSON *cJSON_CreateTrue(void) {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_True;return item;} cJSON *cJSON_CreateFalse(void) {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_False;return item;} cJSON *cJSON_CreateBool(int b) {cJSON *item=cJSON_New_Item();if(item)item->type=b?cJSON_True:cJSON_False;return item;} cJSON *cJSON_CreateNumber(double num) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_Number;item->valuedouble=num;item->valueint=(int)num;}return item;} cJSON *cJSON_CreateString(const char *string) {cJSON *item=cJSON_New_Item();if(item){item->type=cJSON_String;item->valuestring=cJSON_strdup(string);}return item;} cJSON *cJSON_CreateArray(void) {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Array;return item;} cJSON *cJSON_CreateObject(void) {cJSON *item=cJSON_New_Item();if(item)item->type=cJSON_Object;return item;} /* Create Arrays: */ cJSON *cJSON_CreateIntArray(int *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} cJSON *cJSON_CreateFloatArray(float *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} cJSON *cJSON_CreateDoubleArray(double *numbers,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} cJSON *cJSON_CreateStringArray(const char **strings,int count) {int i;cJSON *n=0,*p=0,*a=cJSON_CreateArray();for(i=0;a && ichild=n;else suffix_object(p,n);p=n;}return a;} /* Duplication */ cJSON *cJSON_Duplicate(cJSON *item,int recurse) { cJSON *newitem,*cptr,*nptr=0,*newchild; /* Bail on bad ptr */ if (!item) return 0; /* Create new item */ newitem=cJSON_New_Item(); if (!newitem) return 0; /* Copy over all vars */ newitem->type=item->type&(~cJSON_IsReference),newitem->valueint=item->valueint,newitem->valuedouble=item->valuedouble; if (item->valuestring) {newitem->valuestring=cJSON_strdup(item->valuestring); if (!newitem->valuestring) {cJSON_Delete(newitem);return 0;}} if (item->string) {newitem->string=cJSON_strdup(item->string); if (!newitem->string) {cJSON_Delete(newitem);return 0;}} /* If non-recursive, then we're done! */ if (!recurse) return newitem; /* Walk the ->next chain for the child. */ cptr=item->child; while (cptr) { newchild=cJSON_Duplicate(cptr,1); /* Duplicate (with recurse) each item in the ->next chain */ if (!newchild) {cJSON_Delete(newitem);return 0;} if (nptr) {nptr->next=newchild,newchild->prev=nptr;nptr=newchild;} /* If newitem->child already set, then crosswire ->prev and ->next and move on */ else {newitem->child=newchild;nptr=newchild;} /* Set newitem->child and move to it */ cptr=cptr->next; } return newitem; } #endif ================================================ FILE: src/ngx_stream_json.h ================================================ /* Copyright (c) 2009 Dave Gamble Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ #ifndef cJSON__h #define cJSON__h #ifdef __cplusplus extern "C" { #endif /* cJSON Types: */ #define cJSON_False 0 #define cJSON_True 1 #define cJSON_NULL 2 #define cJSON_Number 3 #define cJSON_String 4 #define cJSON_Array 5 #define cJSON_Object 6 #define cJSON_IsReference 256 /* The cJSON structure: */ typedef struct cJSON { struct cJSON *next,*prev; /* next/prev allow you to walk array/object chains. Alternatively, use GetArraySize/GetArrayItem/GetObjectItem */ struct cJSON *child; /* An array or object item will have a child pointer pointing to a chain of the items in the array/object. */ int type; /* The type of the item, as above. */ char *valuestring; /* The item's string, if type==cJSON_String */ int valueint; /* The item's number, if type==cJSON_Number */ double valuedouble; /* The item's number, if type==cJSON_Number */ char *string; /* The item's name string, if this item is the child of, or is in the list of subitems of an object. */ } cJSON; typedef struct cJSON_Hooks { void *(*malloc_fn)(size_t sz); void (*free_fn)(void *ptr); } cJSON_Hooks; /* Supply malloc, realloc and free functions to cJSON */ extern void cJSON_InitHooks(cJSON_Hooks* hooks); /* Supply a block of JSON, and this returns a cJSON object you can interrogate. Call cJSON_Delete when finished. */ extern cJSON *cJSON_Parse(const char *value); /* Render a cJSON entity to text for transfer/storage. Free the char* when finished. */ extern char *cJSON_Print(cJSON *item); /* Render a cJSON entity to text for transfer/storage without any formatting. Free the char* when finished. */ extern char *cJSON_PrintUnformatted(cJSON *item); /* Delete a cJSON entity and all subentities. */ extern void cJSON_Delete(cJSON *c); /* Returns the number of items in an array (or object). */ extern int cJSON_GetArraySize(cJSON *array); /* Retrieve item number "item" from array "array". Returns NULL if unsuccessful. */ extern cJSON *cJSON_GetArrayItem(cJSON *array,int item); /* Get item "string" from object. Case insensitive. */ extern cJSON *cJSON_GetObjectItem(cJSON *object,const char *string); /* For analysing failed parses. This returns a pointer to the parse error. You'll probably need to look a few chars back to make sense of it. Defined when cJSON_Parse() returns 0. 0 when cJSON_Parse() succeeds. */ extern const char *cJSON_GetErrorPtr(void); /* These calls create a cJSON item of the appropriate type. */ extern cJSON *cJSON_CreateNull(void); extern cJSON *cJSON_CreateTrue(void); extern cJSON *cJSON_CreateFalse(void); extern cJSON *cJSON_CreateBool(int b); extern cJSON *cJSON_CreateNumber(double num); extern cJSON *cJSON_CreateString(const char *string); extern cJSON *cJSON_CreateArray(void); extern cJSON *cJSON_CreateObject(void); /* These utilities create an Array of count items. */ extern cJSON *cJSON_CreateIntArray(int *numbers,int count); extern cJSON *cJSON_CreateFloatArray(float *numbers,int count); extern cJSON *cJSON_CreateDoubleArray(double *numbers,int count); extern cJSON *cJSON_CreateStringArray(const char **strings,int count); /* Append item to the specified array/object. */ extern void cJSON_AddItemToArray(cJSON *array, cJSON *item); extern void cJSON_AddItemToObject(cJSON *object,const char *string,cJSON *item); /* Append reference to item to the specified array/object. Use this when you want to add an existing cJSON to a new cJSON, but don't want to corrupt your existing cJSON. */ extern void cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item); extern void cJSON_AddItemReferenceToObject(cJSON *object,const char *string,cJSON *item); /* Remove/Detatch items from Arrays/Objects. */ extern cJSON *cJSON_DetachItemFromArray(cJSON *array,int which); extern void cJSON_DeleteItemFromArray(cJSON *array,int which); extern cJSON *cJSON_DetachItemFromObject(cJSON *object,const char *string); extern void cJSON_DeleteItemFromObject(cJSON *object,const char *string); /* Update array items. */ extern void cJSON_ReplaceItemInArray(cJSON *array,int which,cJSON *newitem); extern void cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem); /* Duplicate a cJSON item */ extern cJSON *cJSON_Duplicate(cJSON *item,int recurse); /* Duplicate will create a new, identical cJSON item to the one you pass, in new memory that will need to be released. With recurse!=0, it will duplicate any children connected to the item. The item->next and ->prev pointers are always zero on return from Duplicate. */ /* ParseWithOpts allows you to require (and check) that the JSON is null terminated, and to retrieve the pointer to the final byte parsed. */ extern cJSON *cJSON_ParseWithOpts(const char *value,const char **return_parse_end,int require_null_terminated); /* Macros for creating things quickly. */ #define cJSON_AddNullToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateNull()) #define cJSON_AddTrueToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateTrue()) #define cJSON_AddFalseToObject(object,name) cJSON_AddItemToObject(object, name, cJSON_CreateFalse()) #define cJSON_AddBoolToObject(object,name,b) cJSON_AddItemToObject(object, name, cJSON_CreateBool(b)) #define cJSON_AddNumberToObject(object,name,n) cJSON_AddItemToObject(object, name, cJSON_CreateNumber(n)) #define cJSON_AddStringToObject(object,name,s) cJSON_AddItemToObject(object, name, cJSON_CreateString(s)) /* When assigning an integer value, it needs to be propagated to valuedouble too. */ #define cJSON_SetIntValue(object,val) ((object)?(object)->valueint=(object)->valuedouble=(val):(val)) #ifdef __cplusplus } #endif #endif ================================================ FILE: src/ngx_stream_upsync_module.c ================================================ /* * Copyright (C) 2015 Weibo Group Holding Limited * Copyright (C) 2015 Xiaokai Wang (xiaokai.wang@live.com) */ #include #include #include #include "ngx_stream_upsync_module.h" /* client for conf service */ typedef struct { ngx_int_t sd; ngx_int_t port; ngx_int_t connected; char ip[16]; struct sockaddr_in addr; } ngx_stream_conf_client; typedef struct { u_char sockaddr[NGX_SOCKADDRLEN]; ngx_int_t weight; ngx_uint_t max_fails; time_t fail_timeout; unsigned down:1; unsigned backup:1; } ngx_stream_upsync_conf_t; #define NGX_STREAM_UPSYNC_CONSUL 0x0001 #define NGX_STREAM_UPSYNC_CONSUL_SERVICES 0x0002 #define NGX_STREAM_UPSYNC_ETCD 0x0003 typedef ngx_int_t (*ngx_stream_upsync_packet_init_pt) (void *upsync_server); typedef ngx_int_t (*ngx_stream_upsync_packet_parse_pt) (void *upsync_server); typedef void (*ngx_stream_upsync_packet_clean_pt) (void *upsync_server); typedef struct { ngx_str_t name; ngx_uint_t upsync_type; ngx_event_handler_pt send_handler; ngx_event_handler_pt recv_handler; ngx_stream_upsync_packet_init_pt init; ngx_stream_upsync_packet_parse_pt parse; ngx_stream_upsync_packet_clean_pt clean; } ngx_upsync_conf_t; typedef struct { ngx_pool_t *pool; ngx_buf_t send; ngx_buf_t recv; ngx_buf_t body; ngx_array_t del_upstream; /* ngx_stream_upsync_conf_t */ ngx_array_t add_upstream; ngx_array_t upstream_conf; } ngx_stream_upsync_ctx_t; typedef struct { ngx_str_t upsync_host; ngx_int_t upsync_port; ngx_msec_t upsync_timeout; ngx_msec_t upsync_interval; ngx_int_t upsync_lb; ngx_uint_t strong_dependency; ngx_str_t upsync_send; ngx_str_t upsync_dump_path; ngx_open_file_t *conf_file; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upstream_server_t conf_server; /* conf server */ } ngx_stream_upsync_srv_conf_t; /* based on upstream conf, every unit upsync from consul */ typedef struct { ngx_str_t host; uint64_t index; uint64_t update_generation; ngx_event_t upsync_ev; ngx_event_t upsync_timeout_ev; ngx_queue_t delete_ev; ngx_shmtx_t upsync_accept_mutex; ngx_peer_connection_t pc; ngx_stream_upsync_ctx_t ctx; ngx_stream_upsync_srv_conf_t *upscf; ngx_stream_upstream_srv_conf_t *uscf; } ngx_stream_upsync_server_t; typedef struct { ngx_event_t delay_delete_ev; ngx_queue_t queue; time_t start_sec; ngx_msec_t start_msec; void *data; } ngx_delay_event_t; typedef struct { ngx_uint_t upstream_num; ngx_stream_upsync_server_t *upsync_server; } ngx_stream_upsync_main_conf_t; /* http parser state */ typedef struct { u_char status[3]; char headers[NGX_MAX_HEADERS][2][NGX_MAX_ELEMENT_SIZE]; ngx_uint_t num_headers; enum { NONE=0, FIELD, VALUE } last_header; u_char http_body[NGX_PAGE_SIZE * NGX_PAGE_NUMBER]; } ngx_stream_http_state; static ngx_upsync_conf_t *ngx_stream_upsync_get_type_conf(ngx_str_t *str); static char *ngx_stream_upsync_set_lb(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static char *ngx_stream_upsync_server(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static char *ngx_stream_upsync_set_conf_dump(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static void *ngx_stream_upsync_create_main_conf(ngx_conf_t *cf); static void *ngx_stream_upsync_create_srv_conf(ngx_conf_t *cf); static char *ngx_stream_upsync_init_main_conf(ngx_conf_t *cf, void *conf); static char *ngx_stream_upsync_init_srv_conf(ngx_conf_t *cf, void *conf, ngx_uint_t num); static void ngx_stream_upsync_process(ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_upsync_init_process(ngx_cycle_t *cycle); static ngx_int_t ngx_stream_upsync_init_module(ngx_cycle_t *cycle); static ngx_int_t ngx_stream_upsync_init_shm_mutex(ngx_cycle_t *cycle); static ngx_int_t ngx_stream_upsync_add_timers(ngx_cycle_t *cycle); static void ngx_stream_upsync_begin_handler(ngx_event_t *event); static void ngx_stream_upsync_connect_handler(ngx_event_t *event); static void ngx_stream_upsync_recv_handler(ngx_event_t *event); static void ngx_stream_upsync_send_handler(ngx_event_t *event); static void ngx_stream_upsync_recv_empty_handler(ngx_event_t *event); static void ngx_stream_upsync_send_empty_handler(ngx_event_t *event); static void ngx_stream_upsync_timeout_handler(ngx_event_t *event); static void ngx_stream_upsync_clean_event(void *upsync_server); static ngx_int_t ngx_stream_upsync_etcd_parse_init(void *upsync_server); static ngx_int_t ngx_stream_upsync_consul_parse_init(void *upsync_server); static ngx_int_t ngx_stream_upsync_dump_server( ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_upsync_init_server(ngx_event_t *event); static ngx_int_t ngx_stream_upsync_add_peers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_upsync_del_peers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_upsync_replace_peers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server); static void ngx_stream_upsync_update_peer(ngx_stream_upstream_rr_peers_t *peers, ngx_stream_upstream_rr_peer_t *peer, ngx_stream_upsync_conf_t *upstream_conf, ngx_uint_t *updated); static void ngx_stream_upsync_diff_filter(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server, ngx_uint_t *diff); static void ngx_stream_upsync_event_init(ngx_stream_upstream_rr_peer_t *peer, ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_http_parser_init(); static int ngx_stream_http_status(http_parser *p, const char *buf, size_t len); static int ngx_stream_http_header_field_cb(http_parser *p, const char *buf, size_t len); static int ngx_stream_http_header_value_cb(http_parser *p, const char *buf, size_t len); static int ngx_stream_http_body(http_parser *p, const char *buf, size_t len); static ngx_int_t ngx_stream_upsync_check_index( ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_upsync_consul_parse_json(void *upsync_server); static ngx_int_t ngx_stream_upsync_consul_services_parse_json( void *upsync_server); static ngx_int_t ngx_stream_upsync_etcd_parse_json(void *upsync_server); static ngx_int_t ngx_stream_upsync_check_key(u_char *key, ngx_str_t host); static void *ngx_stream_upsync_servers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server, ngx_flag_t flag); static void *ngx_stream_upsync_addrs(ngx_pool_t *pool, u_char *sockaddr); static void ngx_stream_upsync_del_delay_delete(ngx_event_t *event); static ngx_int_t ngx_stream_upsync_need_exit(); static void ngx_stream_upsync_clear_all_events(ngx_cycle_t *cycle); static ngx_int_t ngx_stream_upsync_get_upstream(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server, char **conf_value); static ngx_stream_conf_client *ngx_stream_create_client(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_client_conn(ngx_stream_conf_client *client); static void ngx_stream_client_destroy(ngx_stream_conf_client *client); static ngx_int_t ngx_stream_client_send(ngx_stream_conf_client *client, ngx_stream_upsync_server_t *upsync_server); static ngx_int_t ngx_stream_client_recv(ngx_stream_conf_client *client, char **data, int size); static char *ngx_stream_upsync_set(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static void ngx_stream_upsync_show(ngx_stream_session_t *r); static http_parser_settings settings = { .on_message_begin = 0, .on_header_field = ngx_stream_http_header_field_cb, .on_header_value = ngx_stream_http_header_value_cb, .on_url = 0, .on_status = ngx_stream_http_status, .on_body = ngx_stream_http_body, .on_headers_complete = 0, .on_message_complete = 0 }; ngx_atomic_t stream_upsync_shared_created0; ngx_atomic_t *stream_upsync_shared_created = &stream_upsync_shared_created0; static http_parser *parser = NULL; static ngx_stream_http_state state; static ngx_stream_upsync_main_conf_t *upsync_ctx = NULL; static ngx_command_t ngx_stream_upsync_commands[] = { { ngx_string("upsync"), NGX_STREAM_UPS_CONF|NGX_CONF_1MORE, ngx_stream_upsync_server, NGX_STREAM_SRV_CONF_OFFSET, 0, NULL }, { ngx_string("upsync_lb"), NGX_STREAM_UPS_CONF|NGX_CONF_TAKE1, ngx_stream_upsync_set_lb, 0, 0, NULL }, { ngx_string("upsync_dump_path"), NGX_STREAM_UPS_CONF|NGX_CONF_TAKE1, ngx_stream_upsync_set_conf_dump, NGX_STREAM_SRV_CONF_OFFSET, 0, NULL }, { ngx_string("upstream_show"), NGX_STREAM_SRV_CONF|NGX_CONF_NOARGS, ngx_stream_upsync_set, 0, 0, NULL }, ngx_null_command }; static ngx_stream_module_t ngx_stream_upsync_module_ctx = { NULL, /* preconfiguration */ NULL, /* postconfiguration */ ngx_stream_upsync_create_main_conf, /* create main configuration */ ngx_stream_upsync_init_main_conf, /* init main configuration */ ngx_stream_upsync_create_srv_conf, /* create server configuration */ NULL, /* merge server configuration */ }; ngx_module_t ngx_stream_upsync_module = { NGX_MODULE_V1, &ngx_stream_upsync_module_ctx, /* module context */ ngx_stream_upsync_commands, /* module directives */ NGX_STREAM_MODULE, /* module type */ NULL, /* init master */ ngx_stream_upsync_init_module, /* init module */ ngx_stream_upsync_init_process, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ ngx_stream_upsync_clear_all_events, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; static ngx_upsync_conf_t ngx_upsync_types[] = { { ngx_string("consul"), NGX_STREAM_UPSYNC_CONSUL, ngx_stream_upsync_send_handler, ngx_stream_upsync_recv_handler, ngx_stream_upsync_consul_parse_init, ngx_stream_upsync_consul_parse_json, ngx_stream_upsync_clean_event }, { ngx_string("consul_services"), NGX_STREAM_UPSYNC_CONSUL_SERVICES, ngx_stream_upsync_send_handler, ngx_stream_upsync_recv_handler, ngx_stream_upsync_consul_parse_init, ngx_stream_upsync_consul_services_parse_json, ngx_stream_upsync_clean_event }, { ngx_string("etcd"), NGX_STREAM_UPSYNC_ETCD, ngx_stream_upsync_send_handler, ngx_stream_upsync_recv_handler, ngx_stream_upsync_etcd_parse_init, ngx_stream_upsync_etcd_parse_json, ngx_stream_upsync_clean_event }, { ngx_null_string, 0, NULL, NULL, NULL, NULL, NULL } }; static char * ngx_stream_upsync_server(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { u_char *p = NULL; time_t upsync_timeout = 0, upsync_interval = 0; ngx_str_t *value, s; ngx_url_t u; ngx_uint_t i, strong_dependency = 0; ngx_stream_upstream_server_t *conf_server; ngx_stream_upsync_srv_conf_t *upscf; value = cf->args->elts; upscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_upsync_module); conf_server = &upscf->conf_server; for (i = 2; i < cf->args->nelts; i++) { if (ngx_strncmp(value[i].data, "upsync_timeout=", 15) == 0) { s.len = value[i].len - 15; s.data = &value[i].data[15]; upsync_timeout = ngx_parse_time(&s, 0); if (upsync_timeout == (time_t) NGX_ERROR) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: invalid parameter:\"%V\"", &value[i]); goto invalid; } continue; } if (ngx_strncmp(value[i].data, "upsync_interval=", 16) == 0) { s.len = value[i].len - 16; s.data = &value[i].data[16]; upsync_interval = ngx_parse_time(&s, 0); if (upsync_interval == (time_t) NGX_ERROR) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: invalid parameter: \"%V\"", &value[i]); goto invalid; } continue; } if (ngx_strncmp(value[i].data, "strong_dependency=", 18) == 0) { s.len = value[i].len - 18; s.data = value[i].data + 18; if (ngx_strcasecmp(s.data, (u_char *) "on") == 0) { strong_dependency = 1; } else if (ngx_strcasecmp(s.data, (u_char *) "off") == 0) { strong_dependency = 0; } else { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "invalid value \"%s\", " "it must be \"on\" or \"off\"", value[i].data); return NGX_CONF_ERROR; } continue; } if (ngx_strncmp(value[i].data, "upsync_type=", 12) == 0) { s.len = value[i].len - 12; s.data = value[i].data + 12; upscf->upsync_type_conf = ngx_stream_upsync_get_type_conf(&s); if (upscf->upsync_type_conf == NULL) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: upsync_type invalid para"); goto invalid; } continue; } goto invalid; } if (upsync_interval != 0) { upscf->upsync_interval = upsync_interval; } if (upsync_timeout != 0) { upscf->upsync_timeout = upsync_timeout; } if (strong_dependency != 0) { upscf->strong_dependency = strong_dependency; } if (upscf->upsync_type_conf == NGX_CONF_UNSET_PTR) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: upsync_type cannt be null"); goto invalid; } ngx_memzero(&u, sizeof(ngx_url_t)); p = (u_char *)ngx_strchr(value[1].data, '/'); if (p == NULL) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: " "please input conf_server upstream key in upstream"); return NGX_CONF_ERROR; } upscf->upsync_send.data = p; upscf->upsync_send.len = value[1].len - (p - value[1].data); u.url.data = value[1].data; u.url.len = p - value[1].data; p = (u_char *)ngx_strchr(value[1].data, ':'); if (p != NULL) { upscf->upsync_host.data = value[1].data; upscf->upsync_host.len = p - value[1].data; upscf->upsync_port = ngx_atoi(p + 1, upscf->upsync_send.data - p - 1); if (upscf->upsync_port < 1 || upscf->upsync_port > 65535) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: " "conf server port is invalid"); return NGX_CONF_ERROR; } } else { upscf->upsync_host.data = value[1].data; upscf->upsync_host.len = u.url.len; upscf->upsync_port = 80; } u.default_port = 80; if (ngx_parse_url(cf->pool, &u) != NGX_OK) { if (u.err) { ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "upsync_server: " "%s in upstream \"%V\"", u.err, &u.url); } return NGX_CONF_ERROR; } conf_server->name = u.url; conf_server->addrs = u.addrs; conf_server->naddrs = u.naddrs; conf_server->weight = 1; conf_server->max_fails = 1; conf_server->fail_timeout = 10; return NGX_CONF_OK; invalid: return NGX_CONF_ERROR; } static ngx_upsync_conf_t * ngx_stream_upsync_get_type_conf(ngx_str_t *str) { ngx_uint_t i; for (i = 0; /* void */ ; i++) { if (ngx_upsync_types[i].upsync_type == 0) { break; } if (str->len != ngx_upsync_types[i].name.len) { continue; } if (ngx_strncmp(str->data, ngx_upsync_types[i].name.data, str->len) == 0) { return &ngx_upsync_types[i]; } } return NULL; } static char * ngx_stream_upsync_set_lb(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { ngx_str_t *value, *str; ngx_stream_upsync_srv_conf_t *upscf; upscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_upsync_module); value = cf->args->elts; str = &value[1]; if (str->len == NGX_CONF_UNSET_SIZE) { upscf->upsync_lb = NGX_STREAM_LB_DEFAULT; return NGX_CONF_OK; } switch(str->len) { case 7: if (ngx_memcmp((char *)str->data, "ip_hash", 7) == 0) { upscf->upsync_lb = NGX_STREAM_LB_IP_HASH; return NGX_CONF_OK; } break; case 10: if (ngx_memcmp((char *)str->data, "roundrobin", 10) == 0) { upscf->upsync_lb = NGX_STREAM_LB_ROUNDROBIN; return NGX_CONF_OK; } if (ngx_memcmp((char *)str->data, "least_conn", 10) == 0) { upscf->upsync_lb = NGX_STREAM_LB_LEAST_CONN; return NGX_CONF_OK; } break; case 11: if (ngx_memcmp((char *)str->data, "hash_modula", 11) == 0) { upscf->upsync_lb = NGX_STREAM_LB_HASH_MODULA; return NGX_CONF_OK; } if (ngx_memcmp((char *)str->data, "hash_ketama", 11) == 0) { upscf->upsync_lb = NGX_STREAM_LB_HASH_KETAMA; return NGX_CONF_OK; } break; } return NGX_CONF_OK; } static char * ngx_stream_upsync_set_conf_dump(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { ngx_str_t *value; ngx_stream_upsync_srv_conf_t *upscf; upscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_upsync_module); value = cf->args->elts; upscf->upsync_dump_path = value[1]; if (upscf->upsync_dump_path.len == NGX_CONF_UNSET_SIZE) { return NGX_CONF_ERROR; } upscf->conf_file = ngx_conf_open_file(cf->cycle, &value[1]); if (upscf->conf_file == NULL) { return NGX_CONF_ERROR; } return NGX_CONF_OK; } static void ngx_stream_upsync_process(ngx_stream_upsync_server_t *upsync_server) { ngx_uint_t diff = 0; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_ctx_t *ctx; if (ngx_stream_upsync_need_exit()) { return; } ctx = &upsync_server->ctx; upsync_type_conf = upsync_server->upscf->upsync_type_conf; if (ngx_stream_upsync_check_index(upsync_server) == NGX_ERROR) { return; } if (upsync_type_conf->parse(upsync_server) == NGX_ERROR) { if (upsync_server->index != 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_process: parse json error"); } return; } ngx_log_debug0(NGX_LOG_DEBUG, ngx_cycle->log, 0, "upsync_process: parse json success"); ngx_stream_upsync_diff_filter((ngx_cycle_t *)ngx_cycle, upsync_server, &diff); if (ctx->add_upstream.nelts > 0) { if (upsync_server->update_generation != 0) { if (ngx_stream_upsync_add_peers((ngx_cycle_t *)ngx_cycle, upsync_server) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_process: upstream add peers failed"); return; } } else { if (ngx_stream_upsync_replace_peers((ngx_cycle_t *)ngx_cycle, upsync_server) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_process: upstream add/replace peers failed"); return; } } upsync_server->update_generation++; } if (ctx->del_upstream.nelts > 0) { if (upsync_server->update_generation != 0) { if (ngx_stream_upsync_del_peers((ngx_cycle_t *)ngx_cycle, upsync_server) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_process: upstream del peers failed"); return; } } else { if (ngx_stream_upsync_replace_peers((ngx_cycle_t *)ngx_cycle, upsync_server) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_process: upstream del/replace peers failed"); return; } } upsync_server->update_generation++; } if (diff) { if (ngx_shmtx_trylock(&upsync_server->upsync_accept_mutex)) { ngx_stream_upsync_dump_server(upsync_server); ngx_shmtx_unlock(&upsync_server->upsync_accept_mutex); } } return; } static ngx_int_t ngx_stream_upsync_check_index(ngx_stream_upsync_server_t *upsync_server) { char *p; ngx_uint_t i; uint64_t index = 0; ngx_upsync_conf_t *upsync_type_conf; upsync_type_conf = upsync_server->upscf->upsync_type_conf; if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL_SERVICES) { for (i = 0; i < state.num_headers; i++) { if (ngx_memcmp(state.headers[i][0], NGX_INDEX_HEADER, NGX_INDEX_HEADER_LEN) == 0) { p = ngx_strchr(state.headers[i][1], '\r'); *p = '\0'; index = ngx_strtoull((u_char *)state.headers[i][1], (char **)NULL, 10); break; } } if (index == upsync_server->index) { ngx_log_error(NGX_LOG_NOTICE, ngx_cycle->log, 0, "upsync_check_index: upstream index has not changed: %V", &upsync_server->upscf->upsync_dump_path); return NGX_ERROR; } else { upsync_server->index = index; } } if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_ETCD) { for (i = 0; i < state.num_headers; i++) { if (ngx_memcmp(state.headers[i][0], NGX_INDEX_ETCD_HEADER, NGX_INDEX_ETCD_HEADER_LEN) == 0) { p = ngx_strchr(state.headers[i][1], '\r'); *p = '\0'; index = ngx_strtoull((u_char *)state.headers[i][1], (char **)NULL, 10); break; } } upsync_server->index = index + 1; } return NGX_OK; } static ngx_int_t ngx_stream_upsync_add_peers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server) { ngx_uint_t i=0, n=0, w=0, len=0; ngx_array_t *servers; ngx_stream_upstream_server_t *server = NULL; ngx_stream_upstream_rr_peer_t *peer = NULL; ngx_stream_upstream_rr_peers_t *peers = NULL; ngx_stream_upstream_srv_conf_t *uscf; if (ngx_stream_upsync_need_exit()) { return NGX_OK; } u_char *namep = NULL; struct sockaddr *saddr = NULL; len = sizeof(struct sockaddr); uscf = upsync_server->uscf; servers = ngx_stream_upsync_servers(cycle, upsync_server, NGX_ADD); if (servers == NULL) { return NGX_ERROR; } if (servers->nelts < 1) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_add_peers: no servers to add \"%V\"", &uscf->host); return NGX_ERROR; } if (uscf->peer.data == NULL) { return NGX_ERROR; } peers = (ngx_stream_upstream_rr_peers_t *)uscf->peer.data; if (peers && servers->nelts >= 1) { n = peers->number + servers->nelts; for (i = 0; i < servers->nelts; i++) { server = (ngx_stream_upstream_server_t *)servers->elts + i; //if (server->backup) { // continue; //} // // FIXME: until backup is fully implemented this causes crashes // on startup with nodes set backup=1. Let them in for now peer = ngx_calloc(sizeof(ngx_stream_upstream_rr_peer_t), cycle->log); if (peer == NULL) { goto invalid; } if ((saddr = ngx_calloc(len, cycle->log)) == NULL) { goto invalid; } ngx_memcpy(saddr, server->addrs->sockaddr, len); peer->sockaddr = saddr; if ((namep = ngx_calloc(server->addrs->name.len, cycle->log)) == NULL) { goto invalid; } ngx_memcpy(namep, server->addrs->name.data, server->addrs->name.len); peer->name.data = namep; peer->socklen = server->addrs->socklen; peer->name.len = server->addrs->name.len; peer->max_fails = server->max_fails; peer->fail_timeout = server->fail_timeout; peer->down = server->down; peer->weight = server->weight; peer->effective_weight = server->weight; peer->current_weight = 0; peer->conns = 0; peer->next = peers->peer; peers->peer = peer; w += server->weight; } w += peers->total_weight; peers->single = (n == 1); peers->number = n; peers->weighted = (w != n); peers->total_weight = w; if (upsync_server->upscf->upsync_lb == NGX_STREAM_LB_HASH_KETAMA) { ngx_stream_upsync_chash_init(uscf, peers); } } return NGX_OK; invalid: ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_add_peers: calloc peer failed \"%V\"", &uscf->host); if (peer != NULL) { if (peer->sockaddr != NULL) { ngx_free(peer->sockaddr); } ngx_free(peer); peer = NULL; } return NGX_ERROR; } static void ngx_stream_upsync_update_peer(ngx_stream_upstream_rr_peers_t *peers, ngx_stream_upstream_rr_peer_t *peer, ngx_stream_upsync_conf_t *upstream_conf, ngx_uint_t *updated) { ngx_uint_t w = peers->total_weight, pw = 0; *updated = 0; if (peer->max_fails == upstream_conf->max_fails && peer->fail_timeout == upstream_conf->fail_timeout && peer->down == upstream_conf->down && peer->weight == upstream_conf->weight) { return; } pw = peer->weight; peer->max_fails = upstream_conf->max_fails; peer->fail_timeout = upstream_conf->fail_timeout; peer->down = upstream_conf->down; peer->weight = upstream_conf->weight; peer->effective_weight = upstream_conf->weight; peer->current_weight = 0; w = w + upstream_conf->weight - pw; peers->weighted = (w != peers->number); peers->total_weight = w; *updated = 1; return; } static void ngx_stream_upsync_diff_filter(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server, ngx_uint_t *diff) { ngx_uint_t i, j, len, updated; ngx_uint_t *flags = NULL; ngx_array_t flag_array; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_conf_t *upstream_conf; ngx_stream_upsync_conf_t *add_upstream, *del_upstream; ngx_stream_upstream_rr_peer_t *peer = NULL; ngx_stream_upstream_rr_peers_t *peers = NULL; ngx_stream_upstream_srv_conf_t *uscf; *diff = 0; ctx = &upsync_server->ctx; if (ngx_stream_upsync_need_exit()) { return; } if (ngx_array_init(&ctx->add_upstream, ctx->pool, 16, sizeof(*add_upstream)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_diff_filter_add: alloc error"); return; } if (ngx_array_init(&ctx->del_upstream, ctx->pool, 16, sizeof(*del_upstream)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_diff_filter_del: alloc error"); return; } uscf = upsync_server->uscf; if (uscf->peer.data == NULL) { return; } peers = (ngx_stream_upstream_rr_peers_t *)uscf->peer.data; if (peers->number != 0) { if (ngx_array_init(&flag_array, ctx->pool, peers->number, sizeof(*flags)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_diff_filter: alloc error"); return; } ngx_memzero(flag_array.elts, sizeof(ngx_uint_t) * flag_array.nalloc); flags = (ngx_uint_t*)flag_array.elts; } len = ctx->upstream_conf.nelts; for (i = 0; i < len; i++) { upstream_conf = (ngx_stream_upsync_conf_t *)ctx->upstream_conf.elts + i; for (peer = peers->peer, j = 0; peer; peer = peer->next, j++) { if (*(flags + j) == 1) { continue; } if (ngx_memn2cmp(peer->name.data, upstream_conf->sockaddr, peer->name.len, ngx_strlen(upstream_conf->sockaddr)) == 0) { // update peer ngx_stream_upsync_update_peer(peers, peer, upstream_conf, &updated); *diff |= updated; // set flag, not to be deleted *(flags + j) = 1; break; } } // add_upstream if (j == peers->number) { add_upstream = ngx_array_push(&ctx->add_upstream); ngx_memcpy(add_upstream, upstream_conf, sizeof(*upstream_conf)); } } // del_upstream for (peer = peers->peer, j = 0; peer; peer = peer->next, j++) { if (*(flags + j) == 1) { continue; } del_upstream = ngx_array_push(&ctx->del_upstream); ngx_memzero(del_upstream, sizeof(*del_upstream)); ngx_memcpy(&del_upstream->sockaddr, peer->name.data, peer->name.len); } *diff |= (ctx->add_upstream.nelts > 0); *diff |= (ctx->del_upstream.nelts > 0); return; } static ngx_int_t ngx_stream_upsync_del_peers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server) { ngx_uint_t i, n=0, w=0, len=0; ngx_array_t *servers; ngx_stream_upstream_server_t *server = NULL; ngx_stream_upstream_rr_peer_t *peer = NULL, *pre_peer = NULL; ngx_stream_upstream_rr_peer_t *del_peer = NULL, *tmp_del_peer = NULL; ngx_stream_upstream_rr_peers_t *peers = NULL; ngx_stream_upstream_srv_conf_t *uscf; len = sizeof(struct sockaddr); uscf = upsync_server->uscf; if (ngx_stream_upsync_need_exit()) { return NGX_OK; } servers = ngx_stream_upsync_servers(cycle, upsync_server, NGX_DEL); if (servers == NULL) { return NGX_ERROR; } if (servers->nelts < 1) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_del_peers: no servers to delete \"%V\"", &uscf->host); return NGX_ERROR; } if (uscf->peer.data == NULL) { return NGX_ERROR; } peers = (ngx_stream_upstream_rr_peers_t *)uscf->peer.data; if (peers->number <= servers->nelts) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_del_peer: upstream \"%V\" cannot delete all peers", &uscf->host); return NGX_ERROR; } n = peers->number - servers->nelts; w = peers->total_weight; pre_peer = peers->peer; for (peer = peers->peer; peer; peer = peer->next) { for (i = 0; i < servers->nelts; i++) { server = (ngx_stream_upstream_server_t *)servers->elts + i; if (ngx_memn2cmp((u_char *) peer->sockaddr, (u_char *) server->addrs->sockaddr, len, len) == 0) { if (del_peer == NULL) { del_peer = peer; tmp_del_peer = peer; } else { tmp_del_peer->next = peer; tmp_del_peer = peer; } if (pre_peer == peer) { peers->peer = peer->next; pre_peer = peer->next; } else { pre_peer->next = peer->next; } w -= peer->weight; break; } } if (i == servers->nelts) { pre_peer = peer; } } if (tmp_del_peer) { tmp_del_peer->next = NULL; } peers->single = (n == 1); peers->number = n; peers->weighted = (w != n); peers->total_weight = w; if (upsync_server->upscf->upsync_lb == NGX_STREAM_LB_HASH_KETAMA) { ngx_stream_upsync_del_chash_peer(uscf); } ngx_stream_upsync_event_init(del_peer, upsync_server); return NGX_OK; } static ngx_int_t ngx_stream_upsync_replace_peers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server) { ngx_uint_t i, len, n=0, w=0; ngx_array_t *servers; ngx_stream_upstream_server_t *server = NULL; ngx_stream_upstream_rr_peer_t *peer = NULL; ngx_stream_upstream_rr_peers_t *peers = NULL; ngx_stream_upstream_srv_conf_t *uscf; uscf = upsync_server->uscf; u_char *namep = NULL; struct sockaddr *saddr = NULL; len = sizeof(struct sockaddr); if (uscf->peer.data == NULL) { return NGX_ERROR; } peers = (ngx_stream_upstream_rr_peers_t *)uscf->peer.data; servers = ngx_stream_upsync_servers(cycle, upsync_server, NGX_ALL); if (servers == NULL) { return NGX_ERROR; } if (servers->nelts < 1) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_replace_peers: no servers to replace \"%V\"", &uscf->host); return NGX_ERROR; } //tmp_peer = peers->peer; if (peers && servers->nelts >= 1) { n = servers->nelts; for (i = 0; i < servers->nelts; i++) { server = (ngx_stream_upstream_server_t *)servers->elts + i; peer = ngx_calloc(sizeof(ngx_stream_upstream_rr_peer_t), cycle->log); if (peer == NULL) { goto invalid; } if ((saddr = ngx_calloc(len, cycle->log)) == NULL) { goto invalid; } ngx_memcpy(saddr, server->addrs->sockaddr, len); peer->sockaddr = saddr; if ((namep = ngx_calloc(server->addrs->name.len, cycle->log)) == NULL) { goto invalid; } ngx_memcpy(namep, server->addrs->name.data, server->addrs->name.len); peer->name.data = namep; peer->socklen = server->addrs->socklen; peer->name.len = server->addrs->name.len; peer->max_fails = server->max_fails; peer->fail_timeout = server->fail_timeout; peer->down = server->down; peer->weight = server->weight; peer->effective_weight = server->weight; peer->current_weight = 0; peer->conns = 0; peer->next = peers->peer; peers->peer = peer; w += server->weight; if(i == 0) { peer->next = NULL; } } peers->single = (n == 1); peers->number = n; peers->weighted = (w != n); peers->total_weight = w; if (upsync_server->upscf->upsync_lb == NGX_STREAM_LB_HASH_KETAMA) { ngx_stream_upsync_chash_init(uscf, NULL); } //ngx_pfree(cycle->pool, tmp_peer); not free for caused address invalid. } return NGX_OK; invalid: ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_init_peers: copy failed \"%V\"", &uscf->host); return NGX_ERROR; } static ngx_int_t ngx_stream_upsync_consul_parse_json(void *data) { u_char *p; ngx_buf_t *buf; ngx_int_t max_fails=2, backup=0, down=0; ngx_str_t src, dst; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_conf_t *upstream_conf = NULL; ngx_stream_upsync_server_t *upsync_server = data; ctx = &upsync_server->ctx; buf = &ctx->body; src.len = 0, src.data = NULL; dst.len = 0, dst.data = NULL; cJSON *root = cJSON_Parse((char *)buf->pos); if (root == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: root error"); return NGX_ERROR; } if (ngx_array_init(&ctx->upstream_conf, ctx->pool, 16, sizeof(*upstream_conf)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: array init error"); cJSON_Delete(root); return NGX_ERROR; } cJSON *server_next; for (server_next = root->child; server_next != NULL; server_next = server_next->next) { cJSON *temp1 = cJSON_GetObjectItem(server_next, "Key"); if (temp1 != NULL && temp1->valuestring != NULL) { if (ngx_stream_upsync_check_key((u_char *)temp1->valuestring, upsync_server->host) != NGX_OK) { continue; } p = (u_char *)ngx_strrchr(temp1->valuestring, '/'); upstream_conf = ngx_array_push(&ctx->upstream_conf); ngx_memzero(upstream_conf, sizeof(*upstream_conf)); ngx_sprintf(upstream_conf->sockaddr, "%*s", ngx_strlen(p + 1), p + 1); } temp1 = NULL; if (upstream_conf == NULL) { continue; } temp1 = cJSON_GetObjectItem(server_next, "Value"); if (temp1 != NULL && temp1->valuestring != NULL) { src.data = (u_char *)temp1->valuestring; src.len = ngx_strlen(temp1->valuestring); if (dst.data == NULL) { dst.data = ngx_pcalloc(ctx->pool, 1024); } else { ngx_memzero(dst.data, 1024); } dst.len = 0; ngx_decode_base64(&dst, &src); } temp1 = NULL; /* default value, server attribute */ upstream_conf->weight = 1; upstream_conf->max_fails = 2; upstream_conf->fail_timeout = 10; upstream_conf->down = 0; upstream_conf->backup = 0; p = NULL; if (dst.data != NULL && dst.len != 0) { p = dst.data; cJSON *sub_root = cJSON_Parse((char *)p); if (sub_root == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: parse \'%s\' failed", p); continue; } cJSON *sub_attribute = sub_root; cJSON *temp1 = cJSON_GetObjectItem(sub_attribute, "weight"); if (temp1 != NULL) { if (temp1->valuestring != NULL) { upstream_conf->weight = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } else if (temp1->valueint >= 0) { upstream_conf->weight = temp1->valueint; } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "max_fails"); if (temp1 != NULL) { if (temp1->valuestring != NULL) { max_fails = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } else if (temp1->valueint >= 0) { max_fails = temp1->valueint; } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "fail_timeout"); if (temp1 != NULL){ if (temp1->valuestring != NULL) { upstream_conf->fail_timeout = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } else if (temp1->valueint >= 0) { upstream_conf->fail_timeout = temp1->valueint; } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "down"); if (temp1 != NULL) { if (temp1->valueint != 0) { down = temp1->valueint; } else if (temp1->valuestring != NULL) { down = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "backup"); if (temp1 != NULL) { if (temp1->valueint != 0) { backup = temp1->valueint; } else if (temp1->valuestring != NULL) { backup = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } } temp1 = NULL; dst.len = 0; cJSON_Delete(sub_root); } if (upstream_conf->weight <= 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: \"weight\" value is " "invalid, setting default value 1"); upstream_conf->weight = 1; } if (max_fails < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: \"max_fails\" value is " "invalid, setting default value 2"); } else { upstream_conf->max_fails = (ngx_uint_t)max_fails; } if (upstream_conf->fail_timeout < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: \"fail_timeout\" value is " "invalid, setting default value 10"); upstream_conf->fail_timeout = 10; } if (down != 1 && down != 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "consul_upsync_parse_json: \"down\" value is invalid," "setting default value 0"); } else { upstream_conf->down = (ngx_uint_t)down; } if (backup != 1 && backup != 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: \"backup\" value is invalid," "setting default value 0"); } else { upstream_conf->backup = (ngx_uint_t)backup; } max_fails=2, backup=0, down=0; } cJSON_Delete(root); return NGX_OK; } static ngx_int_t ngx_stream_upsync_consul_services_parse_json(void *data) { ngx_buf_t *buf; ngx_int_t attr_value; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_conf_t *upstream_conf = NULL; ngx_stream_upsync_server_t *upsync_server = data; ctx = &upsync_server->ctx; buf = &ctx->body; cJSON *root = cJSON_Parse((char *)buf->pos); if (root == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: root error"); return NGX_ERROR; } if (ngx_array_init(&ctx->upstream_conf, ctx->pool, 16, sizeof(*upstream_conf)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: array init error"); cJSON_Delete(root); return NGX_ERROR; } cJSON *server_next; for (server_next = root->child; server_next != NULL; server_next = server_next->next) { cJSON *addr, *port, *tags, *tag_next; size_t addr_len, port_len; u_char port_buf[8]; addr = cJSON_GetObjectItem(server_next, "ServiceAddress"); if (addr == NULL || addr->valuestring == NULL || addr->valuestring[0] == '\0') { addr = cJSON_GetObjectItem(server_next, "Address"); if (addr == NULL || addr->valuestring == NULL) { continue; } } port = cJSON_GetObjectItem(server_next, "ServicePort"); if (port == NULL || port->valueint < 1 || port->valueint > 65535) { continue; } ngx_memzero(port_buf, 8); ngx_sprintf(port_buf, "%d", port->valueint); addr_len = ngx_strlen(addr->valuestring); port_len = ngx_strlen(port_buf); if (addr_len + port_len + 2 > NGX_SOCKADDRLEN) { continue; } upstream_conf = ngx_array_push(&ctx->upstream_conf); if (upstream_conf == NULL) { cJSON_Delete(root); return NGX_ERROR; } ngx_memzero(upstream_conf, sizeof(*upstream_conf)); ngx_memcpy(upstream_conf->sockaddr, addr->valuestring, addr_len); ngx_memcpy(upstream_conf->sockaddr + addr_len, ":", 1); ngx_memcpy(upstream_conf->sockaddr + addr_len + 1, port_buf, port_len); /* default value, server attribute */ upstream_conf->weight = 1; upstream_conf->max_fails = 2; upstream_conf->fail_timeout = 10; upstream_conf->down = 0; upstream_conf->backup = 0; tags = cJSON_GetObjectItem(server_next, "ServiceTags"); if (tags == NULL) { continue; } for (tag_next = tags->child; tag_next != NULL; tag_next = tag_next->next) { u_char *tag = (u_char *) tag_next->valuestring; if (tag == NULL) { continue; } if (ngx_strncmp(tag, "weight=", 7) == 0) { attr_value = ngx_atoi(tag + 7, (size_t)ngx_strlen(tag) - 7); if (attr_value == NGX_ERROR || attr_value <= 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: \"weight\" value is " "invalid, setting default value 1"); continue; } else { upstream_conf->weight = attr_value; } } if (ngx_strncmp(tag, "max_fails=", 10) == 0) { attr_value = ngx_atoi(tag + 10, (size_t)ngx_strlen(tag) - 10); if (attr_value == NGX_ERROR || attr_value < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: \"max_fails\" value is " "invalid, setting default value 2"); continue; } else { upstream_conf->max_fails = attr_value; } } if (ngx_strncmp(tag, "fail_timeout=", 13) == 0) { ngx_str_t value = {ngx_strlen(tag) - 13, tag + 13}; attr_value = ngx_parse_time(&value, 1); if (attr_value == NGX_ERROR || attr_value < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: \"fail_timeout\" value is " "invalid, setting default value 10"); continue; } else { upstream_conf->fail_timeout = attr_value; } } if (ngx_strncmp(tag, "down", 4) == 0 && tag[4] == '\0') { upstream_conf->down = 1; } if (ngx_strncmp(tag, "backup", 6) == 0 && tag[6] == '\0') { upstream_conf->backup = 1; } } } cJSON_Delete(root); return NGX_OK; } static ngx_int_t ngx_stream_upsync_etcd_parse_json(void *data) { u_char *p; ngx_buf_t *buf; ngx_int_t max_fails=2, backup=0, down=0; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_conf_t *upstream_conf = NULL; ngx_stream_upsync_server_t *upsync_server = data; ctx = &upsync_server->ctx; buf = &ctx->body; cJSON *root = cJSON_Parse((char *)buf->pos); if (root == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: root error"); return NGX_ERROR; } cJSON *errorCode = cJSON_GetObjectItem(root, "errorCode"); if (errorCode != NULL) { if (errorCode->valueint == 401) { // trigger reload, we've gone too far with index upsync_server->index = 0; ngx_del_timer(&upsync_server->upsync_timeout_ev); ngx_add_timer(&upsync_server->upsync_ev, 0); } cJSON_Delete(root); return NGX_ERROR; } cJSON *action = cJSON_GetObjectItem(root, "action"); if (action != NULL) { if (action->valuestring != NULL) { if (ngx_memcmp(action->valuestring, "get", 3) != 0) { upsync_server->index = 0; ngx_del_timer(&upsync_server->upsync_timeout_ev); ngx_add_timer(&upsync_server->upsync_ev, 0); cJSON_Delete(root); return NGX_ERROR; } } } action = NULL; if (ngx_array_init(&ctx->upstream_conf, ctx->pool, 16, sizeof(*upstream_conf)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: array init error"); cJSON_Delete(root); return NGX_ERROR; } cJSON *node = cJSON_GetObjectItem(root, "node"); if (node == NULL) { cJSON_Delete(root); return NGX_ERROR; } cJSON *nodes = cJSON_GetObjectItem(node, "nodes"); if (nodes == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: nodes is null, no servers"); cJSON_Delete(root); return NGX_ERROR; } cJSON *server_next; for (server_next = nodes->child; server_next != NULL; server_next = server_next->next) { cJSON *temp0 = cJSON_GetObjectItem(server_next, "key"); if (temp0 != NULL && temp0->valuestring != NULL) { if (ngx_stream_upsync_check_key((u_char *)temp0->valuestring, upsync_server->host) != NGX_OK) { continue; } p = (u_char *)ngx_strrchr(temp0->valuestring, '/'); upstream_conf = ngx_array_push(&ctx->upstream_conf); ngx_memzero(upstream_conf, sizeof(*upstream_conf)); ngx_sprintf(upstream_conf->sockaddr, "%*s", ngx_strlen(p + 1), p + 1); } temp0 = NULL; /* default value, server attribute */ upstream_conf->weight = 1; upstream_conf->max_fails = 2; upstream_conf->fail_timeout = 10; upstream_conf->down = 0; upstream_conf->backup = 0; temp0 = cJSON_GetObjectItem(server_next, "value"); if (temp0 != NULL && ngx_strlen(temp0->valuestring) != 0) { cJSON *sub_attribute = cJSON_Parse((char *)temp0->valuestring); if (sub_attribute == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: \'%s\' is invalid", temp0->valuestring); continue; } cJSON *temp1 = cJSON_GetObjectItem(sub_attribute, "weight"); if (temp1 != NULL) { if (temp1->valuestring != NULL) { upstream_conf->weight = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } else if (temp1->valueint >= 0) { upstream_conf->weight = temp1->valueint; } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "max_fails"); if (temp1 != NULL) { if (temp1->valuestring != NULL) { max_fails = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } else if (temp1->valueint >= 0) { max_fails = temp1->valueint; } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "fail_timeout"); if (temp1 != NULL){ if (temp1->valuestring != NULL) { upstream_conf->fail_timeout = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } else if (temp1->valueint >= 0) { upstream_conf->fail_timeout = temp1->valueint; } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "down"); if (temp1 != NULL) { if (temp1->valueint != 0) { down = temp1->valueint; } else if (temp1->valuestring != NULL) { down = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } } temp1 = NULL; temp1 = cJSON_GetObjectItem(sub_attribute, "backup"); if (temp1 != NULL) { if (temp1->valueint != 0) { backup = temp1->valueint; } else if (temp1->valuestring != NULL) { backup = ngx_atoi((u_char *)temp1->valuestring, (size_t)ngx_strlen(temp1->valuestring)); } } temp1 = NULL; } else { continue; } if (upstream_conf->weight <= 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: \"weight\" value is invalid," " setting default value 1"); upstream_conf->weight = 1; } if (max_fails < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: \"max_fails\" value is invalid," " setting default value 2"); } else { upstream_conf->max_fails = (ngx_uint_t)max_fails; } if (upstream_conf->fail_timeout < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: \"fail_timeout\" value is " "invalid, setting default value 10"); upstream_conf->fail_timeout = 10; } if (down != 1 && down != 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: \"down\" value is invalid" ", setting default value 0"); } else { upstream_conf->down = (ngx_uint_t)down; } if (backup != 1 && backup != 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "etcd_upsync_parse_json: \"backup\" value is invalid" ", setting default value 0"); } else { upstream_conf->backup = (ngx_uint_t)backup; } max_fails=2, backup=0, down=0; } cJSON_Delete(root); return NGX_OK; } static ngx_int_t ngx_stream_upsync_check_key(u_char *key, ngx_str_t host) { u_char *last, *ip_p, *port_p, *s_p; // *u_p; ngx_int_t port; /* u_p = (u_char *)ngx_strstr(key, host.data); if (u_p == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: %s is illegal, " "dont contains subkey %V", key, &host); return NGX_ERROR; } if (*(u_p + host.len) != '/' || *(u_p - 1) != '/') { return NGX_ERROR; } */ s_p = (u_char *)ngx_strrchr(key, '/'); if (s_p == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_parse_json: %s key format is illegal, " "contains no slash ('/')", key); return NGX_ERROR; } port_p = (u_char *)ngx_strchr(s_p, ':'); if (port_p == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_check_key: has no port in %s", s_p); return NGX_ERROR; } ip_p = s_p + 1; if (ngx_inet_addr(ip_p, port_p - ip_p) == INADDR_NONE) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_check_key: invalid ip in %s", s_p); return NGX_ERROR; } last = ip_p + ngx_strlen(ip_p); port = ngx_atoi(port_p + 1, last - port_p - 1); if (port < 1 || port > 65535) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_check_key: invalid port in %s", s_p); return NGX_ERROR; } return NGX_OK; } static void * ngx_stream_upsync_servers(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server, ngx_flag_t flag) { ngx_uint_t i; ngx_addr_t *addrs; ngx_array_t *servers; /* ngx_stream_upstream_server_t */ ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_conf_t *conf; ngx_stream_upstream_server_t *server; ctx = &upsync_server->ctx; servers = ngx_pcalloc(ctx->pool, sizeof(ngx_array_t)); if (servers == NULL) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_servers: alloc error"); return NULL; } if (ngx_array_init(servers, ctx->pool, 16, sizeof(*server)) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_servers: alloc error"); return NULL; } if (flag == NGX_ADD) { for (i = 0; i < ctx->add_upstream.nelts; i++) { conf = (ngx_stream_upsync_conf_t *)ctx->add_upstream.elts + i; addrs = ngx_stream_upsync_addrs(ctx->pool, conf->sockaddr); if (addrs == NULL) { continue; } server = ngx_array_push(servers); ngx_memzero(server, sizeof(ngx_stream_upstream_server_t)); server->addrs = addrs; server->naddrs = 1; server->down = conf->down; server->backup = conf->backup; server->weight = conf->weight; server->max_fails = conf->max_fails; server->fail_timeout = conf->fail_timeout; } } else if (flag == NGX_DEL) { for (i = 0; i < ctx->del_upstream.nelts; i++) { conf = (ngx_stream_upsync_conf_t *)ctx->del_upstream.elts + i; addrs = ngx_stream_upsync_addrs(ctx->pool, conf->sockaddr); if (addrs == NULL) { continue; } server = ngx_array_push(servers); ngx_memzero(server, sizeof(ngx_stream_upstream_server_t)); server->addrs = addrs; server->naddrs = 1; server->down = conf->down; server->backup = conf->backup; server->weight = conf->weight; server->max_fails = conf->max_fails; server->fail_timeout = conf->fail_timeout; } } else { for (i = 0; i < ctx->upstream_conf.nelts; i++) { conf = (ngx_stream_upsync_conf_t *)ctx->upstream_conf.elts + i; addrs = ngx_stream_upsync_addrs(ctx->pool, conf->sockaddr); if (addrs == NULL) { continue; } server = ngx_array_push(servers); ngx_memzero(server, sizeof(ngx_stream_upstream_server_t)); server->addrs = addrs; server->naddrs = 1; server->down = conf->down; server->backup = conf->backup; server->weight = conf->weight; server->max_fails = conf->max_fails; server->fail_timeout = conf->fail_timeout; } } return servers; } static void * ngx_stream_upsync_addrs(ngx_pool_t *pool, u_char *sockaddr) { u_char *port_p, *p, *last, *pp; ngx_int_t port; ngx_addr_t *addrs; struct sockaddr_in *sin; p = sockaddr; last = p + ngx_strlen(p); port_p = ngx_strlchr(p, last, ':'); if (port_p == NULL) { ngx_log_error(NGX_LOG_ERR, pool->log, 0, "upsync_addrs: has no port in %s", p); return NULL; } port = ngx_atoi(port_p + 1, last - port_p - 1); if (port < 1 || port > 65535) { ngx_log_error(NGX_LOG_ERR, pool->log, 0, "upsync_addrs: invalid port in %s", p); return NULL; } sin = ngx_pcalloc(pool, sizeof(struct sockaddr_in)); if (sin == NULL) { return NULL; } sin->sin_family = AF_INET; sin->sin_port = htons((in_port_t) port); sin->sin_addr.s_addr = ngx_inet_addr(p, port_p - p); if (sin->sin_addr.s_addr == INADDR_NONE) { ngx_log_error(NGX_LOG_ERR, pool->log, 0, "upsync_addrs: invalid ip in %s", p); return NULL; } addrs = ngx_pcalloc(pool, sizeof(ngx_addr_t)); if (addrs == NULL) { return NULL; } addrs->sockaddr = (struct sockaddr *) sin; addrs->socklen = sizeof(struct sockaddr_in); pp = ngx_pcalloc(pool, last - p); if (pp == NULL) { return NULL; } addrs->name.len = ngx_sprintf(pp, "%s", p) - pp; addrs->name.data = pp; return addrs; } static void * ngx_stream_upsync_create_main_conf(ngx_conf_t *cf) { ngx_stream_upsync_main_conf_t *upmcf; upmcf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_upsync_main_conf_t)); if (upmcf == NULL) { return NULL; } upmcf->upstream_num = NGX_CONF_UNSET_UINT; upmcf->upsync_server = NGX_CONF_UNSET_PTR; return upmcf; } static char * ngx_stream_upsync_init_main_conf(ngx_conf_t *cf, void *conf) { ngx_uint_t i; ngx_stream_upsync_main_conf_t *upmcf = conf; ngx_stream_upstream_srv_conf_t **uscfp; ngx_stream_upstream_main_conf_t *umcf; umcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_upstream_module); upmcf->upsync_server = ngx_pcalloc(cf->pool, umcf->upstreams.nelts * sizeof(ngx_stream_upsync_server_t)); if (upmcf->upsync_server == NULL) { return NGX_CONF_ERROR; } upmcf->upstream_num = 0; upsync_ctx = upmcf; uscfp = umcf->upstreams.elts; for (i = 0; i < umcf->upstreams.nelts; i++) { if (ngx_stream_upsync_init_srv_conf(cf, uscfp[i], i) != NGX_OK) { return NGX_CONF_ERROR; } } return NGX_CONF_OK; } static void * ngx_stream_upsync_create_srv_conf(ngx_conf_t *cf) { ngx_stream_upsync_srv_conf_t *upscf; upscf = ngx_pcalloc(cf->pool, sizeof(ngx_stream_upsync_srv_conf_t)); if (upscf == NULL) { return NULL; } upscf->upsync_host.len = NGX_CONF_UNSET_SIZE; upscf->upsync_host.data = NGX_CONF_UNSET_PTR; upscf->upsync_port = NGX_CONF_UNSET; upscf->upsync_dump_path.len = NGX_CONF_UNSET_SIZE; upscf->upsync_dump_path.data = NGX_CONF_UNSET_PTR; upscf->upsync_timeout = NGX_CONF_UNSET_MSEC; upscf->upsync_interval = NGX_CONF_UNSET_MSEC; upscf->upsync_lb = NGX_CONF_UNSET; upscf->strong_dependency = NGX_CONF_UNSET_UINT; upscf->conf_file = NGX_CONF_UNSET_PTR; upscf->upsync_type_conf = NGX_CONF_UNSET_PTR; ngx_memzero(&upscf->conf_server, sizeof(upscf->conf_server)); return upscf; } static char * ngx_stream_upsync_init_srv_conf(ngx_conf_t *cf, void *conf, ngx_uint_t num) { u_char *buf; ngx_stream_upsync_server_t *upsync_server; ngx_stream_upsync_srv_conf_t *upscf; ngx_stream_upstream_srv_conf_t *uscf = conf; if (uscf->srv_conf == NULL) { return NGX_CONF_OK; } upscf = ngx_stream_conf_upstream_srv_conf(uscf, ngx_stream_upsync_module); if (upscf->upsync_host.data == NGX_CONF_UNSET_PTR && upscf->upsync_host.len == NGX_CONF_UNSET_SIZE) { return NGX_CONF_OK; } upsync_ctx->upstream_num++; upsync_server = &upsync_ctx->upsync_server[upsync_ctx->upstream_num - 1]; if (upsync_server == NULL) { return NGX_CONF_ERROR; } if (upscf->upsync_timeout == NGX_CONF_UNSET_MSEC) { upscf->upsync_timeout = 1000 * 60 * 6; } if (upscf->upsync_interval == NGX_CONF_UNSET_MSEC) { upscf->upsync_interval = 1000 * 5; } if (upscf->upsync_lb == NGX_CONF_UNSET) { upscf->upsync_lb = NGX_STREAM_LB_DEFAULT; } if (upscf->strong_dependency == NGX_CONF_UNSET_UINT) { upscf->strong_dependency = 0; } if (upscf->upsync_dump_path.len == NGX_CONF_UNSET_SIZE) { buf = ngx_pcalloc(cf->pool, ngx_strlen("/tmp/servers_.conf") + uscf->host.len + 1); ngx_sprintf(buf, "/tmp/servers_%V.conf", &uscf->host); upscf->upsync_dump_path.data = buf; upscf->upsync_dump_path.len = ngx_strlen("/tmp/servers_.conf") + uscf->host.len; } upscf->conf_file = ngx_pcalloc(cf->pool, sizeof(ngx_open_file_t)); if (upscf->conf_file == NULL) { return NGX_CONF_ERROR; } upscf->conf_file->fd = NGX_INVALID_FILE; upscf->conf_file->name = upscf->upsync_dump_path; upscf->conf_file->flush = NULL; upscf->conf_file->data = NULL; upsync_server->index = 0; upsync_server->update_generation = 0; upsync_server->upscf = upscf; upsync_server->uscf = uscf; upsync_server->host.len = uscf->host.len; upsync_server->host.data = uscf->host.data; return NGX_CONF_OK; } static ngx_int_t ngx_stream_upsync_init_module(ngx_cycle_t *cycle) { ngx_uint_t i; ngx_stream_upsync_server_t *upsync_server; ngx_stream_upsync_srv_conf_t *upscf; // no stream {} block found if (upsync_ctx == NULL) { return NGX_OK; } upsync_server = upsync_ctx->upsync_server; if (ngx_stream_upsync_init_shm_mutex(cycle) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_init_module:" " init shm mutex failed"); return NGX_ERROR; } for (i = 0; i < upsync_ctx->upstream_num; i++) { upscf = upsync_server[i].upscf; if (upscf->conf_file->fd != NGX_INVALID_FILE) { ngx_close_file(upscf->conf_file->fd); upscf->conf_file->fd = NGX_INVALID_FILE; } ngx_change_file_access(upscf->upsync_dump_path.data, S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH|S_IWOTH); } return NGX_OK; } static ngx_int_t ngx_stream_upsync_init_shm_mutex(ngx_cycle_t *cycle) { u_char *shared, *file; size_t size, cl; ngx_shm_t shm; ngx_uint_t i; ngx_stream_upsync_server_t *upsync_server; upsync_server = upsync_ctx->upsync_server; if (*stream_upsync_shared_created) { shm.size = 128 * (*stream_upsync_shared_created); shm.log = cycle->log; shm.addr = (u_char *)(stream_upsync_shared_created); shm.name.len = sizeof("ngx_upsync_shared_zone"); shm.name.data = (u_char *)"ngx_upsync_shared_zone"; ngx_shm_free(&shm); } /* cl should be equal to or greater than cache line size shared created flag upsync_accept_mutex for every upstream */ cl = 128; size = cl + cl * upsync_ctx->upstream_num; shm.size = size; shm.log = cycle->log; shm.name.len = sizeof("ngx_upsync_shared_zone"); shm.name.data = (u_char *)"ngx_upsync_shared_zone"; if (ngx_shm_alloc(&shm) != NGX_OK) { return NGX_ERROR; } shared = shm.addr; stream_upsync_shared_created = (ngx_atomic_t *)shared; for (i = 0; i < upsync_ctx->upstream_num; i++) { #if (NGX_HAVE_ATOMIC_OPS) file = NULL; #else file = ngx_pcalloc(cycle->pool, cycle->lock_file.len + ngx_strlen("upsync") + 3); if (file == NULL) { return NGX_ERROR; } (void) ngx_sprintf(file, "%V%s%d%Z", &ngx_cycle->lock_file, "upsync", i); #endif if (ngx_shmtx_create(&upsync_server[i].upsync_accept_mutex, (ngx_shmtx_sh_t *)(shared + (i + 1) * cl), file) != NGX_OK) { return NGX_ERROR; } } ngx_atomic_cmp_set(stream_upsync_shared_created, *stream_upsync_shared_created, upsync_ctx->upstream_num); return NGX_OK; } static ngx_int_t ngx_stream_upsync_init_process(ngx_cycle_t *cycle) { char *conf_value = NULL; ngx_int_t status = 0; ngx_uint_t i, j; ngx_pool_t *pool; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server; // no stream {} block found if (upsync_ctx == NULL) { return NGX_OK; } upsync_server = upsync_ctx->upsync_server; for (i = 0; i < upsync_ctx->upstream_num; i++) { ngx_queue_init(&upsync_server[i].delete_ev); if (upsync_server[i].upscf->strong_dependency == 0) { continue; } ctx = &upsync_server[i].ctx; ngx_memzero(ctx, sizeof(*ctx)); upsync_type_conf = upsync_server[i].upscf->upsync_type_conf; pool = ngx_create_pool(NGX_DEFAULT_POOL_SIZE, ngx_cycle->log); if (pool == NULL) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_init_process: recv error, " "server no enough memory"); return NGX_ERROR; } ctx->pool = pool; for (j = 0; j < NGX_STREAM_RETRY_TIMES; j++) { status = ngx_stream_upsync_get_upstream(cycle, &upsync_server[i], &conf_value); if (status == NGX_OK) { break; } } if (status != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_init_process: pull upstream \"%V\" conf failed", &upsync_server->host); return NGX_ERROR; } ctx->recv.pos = (u_char *)conf_value; ctx->recv.last = (u_char *)(conf_value + ngx_strlen(conf_value)); ctx->recv.end = ctx->recv.last; if (upsync_type_conf->init(&upsync_server[i]) == NGX_ERROR) { ngx_free(conf_value); conf_value = NULL; ngx_destroy_pool(pool); ctx->pool = NULL; continue; } ngx_stream_upsync_process(&upsync_server[i]); ngx_free(conf_value); conf_value = NULL; ngx_destroy_pool(pool); ctx->pool = NULL; } ngx_stream_upsync_add_timers(cycle); return NGX_OK; } static ngx_int_t ngx_stream_upsync_add_timers(ngx_cycle_t *cycle) { ngx_msec_t t, tmp; ngx_uint_t i; ngx_stream_upsync_server_t *upsync_server; ngx_stream_upsync_srv_conf_t *upscf; upsync_server = upsync_ctx->upsync_server; if (upsync_server == NULL) { return NGX_OK; } ngx_log_debug0(NGX_LOG_DEBUG_HTTP, cycle->log, 0, "upsync_add_timers"); srandom(ngx_pid); for (i = 0; i < upsync_ctx->upstream_num; i++) { upsync_server[i].upsync_ev.handler = ngx_stream_upsync_begin_handler; upsync_server[i].upsync_ev.log = cycle->log; upsync_server[i].upsync_ev.data = &upsync_server[i]; upsync_server[i].upsync_ev.timer_set = 0; upsync_server[i].upsync_timeout_ev.handler = ngx_stream_upsync_timeout_handler; upsync_server[i].upsync_timeout_ev.log = cycle->log; upsync_server[i].upsync_timeout_ev.data = &upsync_server[i]; upsync_server[i].upsync_timeout_ev.timer_set = 0; /* * We add a random start time here, since we don't want to trigger * the check events too close to each other at the beginning. */ upscf = upsync_server[i].upscf; tmp = upscf->upsync_interval; t = ngx_random() % 1000 + tmp; ngx_add_timer(&upsync_server[i].upsync_ev, t); } return NGX_OK; } static void ngx_stream_upsync_begin_handler(ngx_event_t *event) { ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server; if (ngx_stream_upsync_need_exit()) { return; } upsync_server = event->data; if (upsync_server == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_upsync_begin_handler: upsync_server is null"); return; } ctx = &upsync_server->ctx; if (ctx->pool != NULL) { ngx_destroy_pool(ctx->pool); } ctx->pool = NULL; ngx_memzero(ctx, sizeof(*ctx)); if (parser != NULL) { ngx_free(parser); parser = NULL; } if (upsync_server->upsync_ev.timer_set) { ngx_del_timer(&upsync_server->upsync_ev); } ngx_stream_upsync_connect_handler(event); } static void ngx_stream_upsync_connect_handler(ngx_event_t *event) { ngx_int_t rc; ngx_connection_t *c; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_server_t *upsync_server; ngx_stream_upsync_srv_conf_t *upscf; if (ngx_stream_upsync_need_exit()) { return; } if (ngx_stream_upsync_init_server(event) != NGX_OK) { return; } upsync_server = event->data; upscf = upsync_server->upscf; upsync_type_conf = upscf->upsync_type_conf; ngx_add_timer(&upsync_server->upsync_timeout_ev, upscf->upsync_timeout); rc = ngx_event_connect_peer(&upsync_server->pc); if (rc == NGX_ERROR || rc == NGX_DECLINED) { ngx_log_error(NGX_LOG_ERR, event->log, 0, "upsync_connect_handler: cannot connect to upsync_server: %V ", upsync_server->pc.name); ngx_del_timer(&upsync_server->upsync_timeout_ev); ngx_add_timer(&upsync_server->upsync_ev, 0); return; } /* NGX_OK or NGX_AGAIN */ c = upsync_server->pc.connection; c->data = upsync_server; c->log = upsync_server->pc.log; c->sendfile = 0; c->read->log = c->log; c->write->log = c->log; c->idle = 1; //for quick exit. c->write->handler = upsync_type_conf->send_handler; c->read->handler = upsync_type_conf->recv_handler; /* The kqueue's loop interface needs it. */ if (rc == NGX_OK) { c->write->handler(c->write); } } static void ngx_stream_upsync_send_handler(ngx_event_t *event) { ssize_t size; ngx_connection_t *c; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server; ngx_stream_upsync_srv_conf_t *upscf; if (ngx_stream_upsync_need_exit()) { return; } c = event->data; upsync_server = c->data; upscf = upsync_server->upscf; upsync_type_conf = upscf->upsync_type_conf; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "upsync_send"); ctx = &upsync_server->ctx; u_char request[ngx_pagesize]; ngx_memzero(request, ngx_pagesize); if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL_SERVICES) { ngx_sprintf(request, "GET %V?recurse&index=%uL HTTP/1.0\r\nHost: %V\r\n" "Accept: */*\r\n\r\n", &upscf->upsync_send, upsync_server->index, &upscf->upsync_host); } if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_ETCD) { if (upsync_server->index != 0) { ngx_sprintf(request, "GET %V?wait=true&recursive=true&waitIndex=%uL" " HTTP/1.0\r\nHost: %V\r\nAccept: */*\r\n\r\n", &upscf->upsync_send, upsync_server->index, &upscf->upsync_host); } else { ngx_sprintf(request, "GET %V?" " HTTP/1.0\r\nHost: %V\r\nAccept: */*\r\n\r\n", &upscf->upsync_send, &upscf->upsync_host); } } ctx->send.pos = request; ctx->send.last = ctx->send.pos + ngx_strlen(request); while (ctx->send.pos < ctx->send.last) { size = c->send(c, ctx->send.pos, ctx->send.last - ctx->send.pos); #if (NGX_DEBUG) { ngx_err_t err; err = (size >=0) ? 0 : ngx_socket_errno; ngx_log_debug2(NGX_LOG_DEBUG_HTTP, c->log, err, "upsync_send: send size: %z, total: %z", size, ctx->send.last - ctx->send.pos); } #endif if (size > 0) { ctx->send.pos += size; } else if (size == 0 || size == NGX_AGAIN) { return; } else { c->error = 1; goto upsync_send_fail; } } if (ctx->send.pos == ctx->send.last) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "upsync_send: send done."); } c->write->handler = ngx_stream_upsync_send_empty_handler; return; upsync_send_fail: ngx_log_error(NGX_LOG_ERR, event->log, 0, "upsync_send: send error with upsync_server: %V", upsync_server->pc.name); ngx_stream_upsync_clean_event(upsync_server); } static void ngx_stream_upsync_recv_handler(ngx_event_t *event) { u_char *new_buf; ssize_t size, n; ngx_pool_t *pool; ngx_connection_t *c; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server; if (ngx_stream_upsync_need_exit()) { return; } c = event->data; upsync_server = c->data; upsync_type_conf = upsync_server->upscf->upsync_type_conf; ctx = &upsync_server->ctx; if (ctx->pool == NULL) { pool = ngx_create_pool(NGX_DEFAULT_POOL_SIZE, ngx_cycle->log); if (pool == NULL) { ngx_log_error(NGX_LOG_ERR, event->log, 0, "upsync_recv: recv not enough memory"); return; } ctx->pool = pool; } else { pool = ctx->pool; } if (ctx->recv.start == NULL) { /* 1 of the page_size, is it enough? */ ctx->recv.start = ngx_pcalloc(pool, ngx_pagesize); if (ctx->recv.start == NULL) { goto upsync_recv_fail; } ctx->recv.last = ctx->recv.pos = ctx->recv.start; ctx->recv.end = ctx->recv.start + ngx_pagesize; } while (1) { n = ctx->recv.end - ctx->recv.last; /* buffer not big enough? enlarge it by twice */ if (n == 0) { size = ctx->recv.end - ctx->recv.start; new_buf = ngx_pcalloc(pool, size * 2); if (new_buf == NULL) { goto upsync_recv_fail; } ngx_memcpy(new_buf, ctx->recv.start, size); ctx->recv.pos = ctx->recv.start = new_buf; ctx->recv.last = new_buf + size; ctx->recv.end = new_buf + size * 2; n = ctx->recv.end - ctx->recv.last; } size = c->recv(c, ctx->recv.last, n); #if (NGX_DEBUG) { ngx_err_t err; err = (size >= 0) ? 0 : ngx_socket_errno; ngx_log_debug2(NGX_LOG_DEBUG_HTTP, c->log, err, "upsync_recv: recv size: %z, upsync_server: %V ", size, upsync_server->pc.name); } #endif if (size > 0) { ctx->recv.last += size; continue; } else if (size == 0) { break; } else if (size == NGX_AGAIN) { return; } else { c->error = 1; goto upsync_recv_fail; } } if (upsync_type_conf->init(upsync_server) == NGX_OK) { ngx_stream_upsync_process(upsync_server); c->read->handler = ngx_stream_upsync_recv_empty_handler; } upsync_type_conf->clean(upsync_server); return; upsync_recv_fail: ngx_log_error(NGX_LOG_ERR, event->log, 0, "upsync_recv: recv error with upstream: \"%V\"", &upsync_server->host); ngx_stream_upsync_clean_event(upsync_server); } static void ngx_stream_upsync_send_empty_handler(ngx_event_t *event) { /* void */ } static void ngx_stream_upsync_recv_empty_handler(ngx_event_t *event) { /* void */ } static ngx_int_t ngx_stream_upsync_consul_parse_init(void *data) { char *buf; size_t parsed; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server = data; ctx = &upsync_server->ctx; if (ngx_stream_http_parser_init() == NGX_ERROR) { return NGX_ERROR; } buf = (char *)ctx->recv.pos; ctx->body.pos = ctx->body.last = NULL; parsed = http_parser_execute(parser, &settings, buf, ngx_strlen(buf)); if (parsed != ngx_strlen(buf)) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_consul_parse_init: parsed upstream \"%V\" wrong", &upsync_server->host); if (parser != NULL) { ngx_free(parser); parser = NULL; } return NGX_ERROR; } if (ngx_strncmp(state.status, "OK", 2) == 0) { if (ngx_strlen(state.http_body) != 0) { ctx->body.pos = state.http_body; ctx->body.last = state.http_body + ngx_strlen(state.http_body); *(ctx->body.last + 1) = '\0'; } } else { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_consul_parse_init: recv upstream \"%V\" error; " "http_status: %d", &upsync_server->host, parser->status_code); if (parser != NULL) { ngx_free(parser); parser = NULL; } return NGX_ERROR; } if (parser != NULL) { ngx_free(parser); parser = NULL; } return NGX_OK; } static ngx_int_t ngx_stream_upsync_etcd_parse_init(void *data) { char *buf; size_t parsed; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server = data; ctx = &upsync_server->ctx; if (ngx_stream_http_parser_init() == NGX_ERROR) { return NGX_ERROR; } buf = (char *)ctx->recv.pos; ctx->body.pos = ctx->body.last = NULL; parsed = http_parser_execute(parser, &settings, buf, ngx_strlen(buf)); if (parsed != ngx_strlen(buf)) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_etcd_parse_init: parsed upstream \"%V\" wrong", &upsync_server->host); if (parser != NULL) { ngx_free(parser); parser = NULL; } return NGX_ERROR; } if (ngx_strncmp(state.status, "OK", 2) == 0 || ngx_strncmp(state.status, "Bad", 3) == 0) { if (ngx_strlen(state.http_body) != 0) { ctx->body.pos = state.http_body; ctx->body.last = state.http_body + ngx_strlen(state.http_body); *(ctx->body.last + 1) = '\0'; } } else { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_consul_parse_init: recv upstream \"%V\" error; " "http_status: %d", &upsync_server->host, parser->status_code); if (parser != NULL) { ngx_free(parser); parser = NULL; } return NGX_ERROR; } if (parser != NULL) { ngx_free(parser); parser = NULL; } return NGX_OK; } static ngx_int_t ngx_stream_upsync_dump_server(ngx_stream_upsync_server_t *upsync_server) { ngx_buf_t *b=NULL; ngx_stream_upsync_srv_conf_t *upscf = NULL; ngx_stream_upstream_rr_peer_t *peer = NULL; ngx_stream_upstream_rr_peers_t *peers = NULL; ngx_stream_upstream_srv_conf_t *uscf = NULL; uscf = upsync_server->uscf; if (uscf->peer.data != NULL) { peers = (ngx_stream_upstream_rr_peers_t *)uscf->peer.data; } else { ngx_log_error(NGX_LOG_ERR, upsync_server->ctx.pool->log, 0, "upsync_dump_server: no peers"); return NGX_ERROR; } if (peers->number == 0) { ngx_log_error(NGX_LOG_ERR, upsync_server->ctx.pool->log, 0, "upsync_dump_server: there are no peers to dump"); return NGX_ERROR; } b = ngx_create_temp_buf(upsync_server->ctx.pool, NGX_PAGE_SIZE * NGX_PAGE_NUMBER); if (b == NULL) { ngx_log_error(NGX_LOG_ERR, upsync_server->ctx.pool->log, 0, "upsync_dump_server: dump failed %V", &uscf->host); return NGX_ERROR; } for (peer = peers->peer; peer; peer = peer->next) { b->last = ngx_snprintf(b->last, b->end - b->last, "server %V", &peer->name); b->last = ngx_snprintf(b->last, b->end - b->last, " weight=%d", peer->weight); b->last = ngx_snprintf(b->last, b->end - b->last, " max_fails=%d", peer->max_fails); b->last = ngx_snprintf(b->last, b->end - b->last, " fail_timeout=%ds", peer->fail_timeout); if (peer->down) { b->last = ngx_snprintf(b->last, b->end - b->last, " down"); } b->last = ngx_snprintf(b->last, b->end - b->last, ";\n"); } upscf = upsync_server->upscf; upscf->conf_file->fd = ngx_open_file(upscf->upsync_dump_path.data, NGX_FILE_TRUNCATE, NGX_FILE_WRONLY, NGX_FILE_DEFAULT_ACCESS); if (upscf->conf_file->fd == NGX_INVALID_FILE) { ngx_log_error(NGX_LOG_ERR, upsync_server->ctx.pool->log, 0, "upsync_dump_server: open dump file \"%V\" failed", &upscf->upsync_dump_path); return NGX_ERROR; } ngx_lseek(upscf->conf_file->fd, 0, SEEK_SET); if (ngx_write_fd(upscf->conf_file->fd, b->start, b->last - b->start) == NGX_ERROR) { ngx_log_error(NGX_LOG_ERR, upsync_server->ctx.pool->log, 0, "upsync_dump_server: write file failed %V", &upscf->upsync_dump_path); ngx_close_file(upscf->conf_file->fd); return NGX_ERROR; } if (ngx_ftruncate(upscf->conf_file->fd, b->last - b->start) != NGX_OK) { ngx_log_error(NGX_LOG_ERR, upsync_server->ctx.pool->log, 0, "upsync_dump_server: truncate file failed %V", &upscf->upsync_dump_path); ngx_close_file(upscf->conf_file->fd); return NGX_ERROR; } ngx_close_file(upscf->conf_file->fd); upscf->conf_file->fd = NGX_INVALID_FILE; ngx_log_error(NGX_LOG_NOTICE, upsync_server->ctx.pool->log, 0, "upsync_dump_server: dump conf file %V succeeded, number of servers is %d", &upscf->upsync_dump_path, peers->number); return NGX_OK; } static ngx_int_t ngx_stream_upsync_init_server(ngx_event_t *event) { ngx_uint_t n = 0, r = 0, cur = 0; ngx_pool_t *pool; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server; ngx_stream_upsync_srv_conf_t *upscf; ngx_stream_upstream_server_t *conf_server; u_char *p, *host = NULL; size_t len; ngx_str_t *name; struct addrinfo hints, *res = NULL, *rp = NULL; struct sockaddr_in *sin; upsync_server = event->data; upscf = upsync_server->upscf; conf_server = &upscf->conf_server; ctx = &upsync_server->ctx; if (ctx->pool == NULL) { pool = ngx_create_pool(NGX_DEFAULT_POOL_SIZE, ngx_cycle->log); if (pool == NULL) { ngx_log_error(NGX_LOG_ERR, event->log, 0, "upsync_init_consul: cannot create pool, not enough memory"); return NGX_ERROR; } ctx->pool = pool; } ngx_memzero(&upsync_server->pc, sizeof(ngx_peer_connection_t)); upsync_server->pc.get = ngx_event_get_peer; upsync_server->pc.log = event->log; upsync_server->pc.log_error = NGX_ERROR_ERR; upsync_server->pc.cached = 0; upsync_server->pc.connection = NULL; if (ngx_inet_addr(upscf->upsync_host.data, upscf->upsync_host.len) == INADDR_NONE) { host = ngx_pcalloc(ctx->pool, upscf->upsync_host.len + 1); if (host == NULL) { return NGX_ERROR; } (void) ngx_cpystrn(host, upscf->upsync_host.data, upscf->upsync_host.len + 1); ngx_memzero(&hints, sizeof(struct addrinfo)); hints.ai_family = AF_UNSPEC; hints.ai_socktype = SOCK_STREAM; #ifdef AI_ADDRCONFIG hints.ai_flags = AI_ADDRCONFIG; #endif if (getaddrinfo((char *) host, NULL, &hints, &res) != 0) { res = NULL; goto valid; } /* bad method: for get random server*/ for (rp = res; rp != NULL; rp = rp->ai_next) { if (rp->ai_family != AF_INET) { continue; } n++; } r = ngx_random() % n; for (rp = res; rp != NULL; rp = rp->ai_next) { if (rp->ai_family != AF_INET) { continue; } if (cur != r) { cur++; continue; } sin = ngx_pcalloc(ctx->pool, rp->ai_addrlen); if (sin == NULL) { goto valid; } ngx_memcpy(sin, rp->ai_addr, rp->ai_addrlen); sin->sin_port = htons((in_port_t) upscf->upsync_port); upsync_server->pc.sockaddr = (struct sockaddr *) sin; upsync_server->pc.socklen = rp->ai_addrlen; len = NGX_INET_ADDRSTRLEN + sizeof(":65535") - 1; p = ngx_pcalloc(ctx->pool, len); if (p == NULL) { goto valid; } len = ngx_sock_ntop((struct sockaddr *) sin, rp->ai_addrlen, p, len, 1); name = ngx_pcalloc(ctx->pool, sizeof(*name)); if (name == NULL) { goto valid; } name->len = len; name->data = p; upsync_server->pc.name = name; freeaddrinfo(res); return NGX_OK; } } valid: upsync_server->pc.sockaddr = conf_server->addrs[0].sockaddr; upsync_server->pc.socklen = conf_server->addrs[0].socklen; upsync_server->pc.name = &conf_server->addrs[0].name; if (res != NULL) { freeaddrinfo(res); } return NGX_OK; } static void ngx_stream_upsync_event_init(ngx_stream_upstream_rr_peer_t *peer, ngx_stream_upsync_server_t *upsync_server) { ngx_time_t *tp; ngx_delay_event_t *delay_event; delay_event = ngx_calloc(sizeof(*delay_event), ngx_cycle->log); if (delay_event == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "upsync_event_init: calloc failed"); return; } tp = ngx_timeofday(); delay_event->start_sec = tp->sec; delay_event->start_msec = tp->msec; delay_event->delay_delete_ev.handler = ngx_stream_upsync_del_delay_delete; delay_event->delay_delete_ev.log = ngx_cycle->log; delay_event->delay_delete_ev.data = delay_event; delay_event->delay_delete_ev.timer_set = 0; ngx_queue_insert_head(&upsync_server->delete_ev, &delay_event->queue); delay_event->data = peer; ngx_add_timer(&delay_event->delay_delete_ev, NGX_DELAY_DELETE); return; } static void ngx_stream_upsync_del_delay_delete(ngx_event_t *event) { ngx_msec_t t; ngx_uint_t i, conn_interval; ngx_connection_t *c; ngx_delay_event_t *delay_event; ngx_stream_session_t *s = NULL; ngx_stream_upstream_rr_peer_t *peer = NULL, *pre_peer = NULL; u_char *namep = NULL; struct sockaddr *saddr = NULL; delay_event = event->data; if (delay_event == NULL) { return; } peer = delay_event->data; c = ngx_cycle->connections; conn_interval = ngx_cycle->connection_n / 30; for (i = 0; i < ngx_cycle->connection_n; i += conn_interval) { if (c[i].fd == (ngx_socket_t) -1) { continue; } else { if (c[i].log->data != NULL) { s = c[i].log->data; } } if (s) { if (s->upstream->start_sec < delay_event->start_sec) { t = ngx_random() % NGX_DELAY_DELETE + NGX_DELAY_DELETE; ngx_add_timer(&delay_event->delay_delete_ev, t); return; } } } while (peer != NULL) { saddr = peer->sockaddr; if (saddr != NULL) { ngx_free(saddr); saddr = NULL; } namep = peer->name.data; if (namep != NULL) { ngx_free(namep); namep = NULL; } pre_peer = peer; peer = peer->next; ngx_free(pre_peer); } ngx_queue_remove(&delay_event->queue); ngx_free(delay_event); delay_event = NULL; return; } static size_t ngx_stream_upsync_strnlen(const char *s, size_t maxlen) { const char *p; p = ngx_strchr(s, '\0'); if (p == NULL) { return maxlen; } return p - s; } static size_t ngx_strlncat(char *dst, size_t len, const char *src, size_t n) { size_t slen; size_t dlen; size_t rlen; size_t ncpy; slen = ngx_stream_upsync_strnlen(src, n); dlen = ngx_stream_upsync_strnlen(dst, len); if (dlen < len) { rlen = len - dlen; ncpy = slen < rlen ? slen : (rlen - 1); ngx_memcpy(dst + dlen, src, ncpy); dst[dlen + ncpy] = '\0'; } return slen + dlen; } static ngx_int_t ngx_stream_http_parser_init() { ngx_memzero(state.status, 3); ngx_memzero(state.http_body, NGX_PAGE_SIZE * NGX_PAGE_NUMBER); ngx_memzero(state.headers, NGX_MAX_HEADERS * 2 * NGX_MAX_ELEMENT_SIZE); state.num_headers = 0; state.last_header = NONE; parser = ngx_calloc(sizeof(http_parser), ngx_cycle->log); if (parser == NULL) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_http_parser_init: ngx_calloc failed"); return NGX_ERROR; } http_parser_init(parser, HTTP_RESPONSE); return NGX_OK; } static int ngx_stream_http_status(http_parser *p, const char *buf, size_t len) { if (p != parser) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_http_status: parser argument is wrong"); return NGX_ERROR; } ngx_memcpy(state.status, buf, len); return 0; } static int ngx_stream_http_header_field_cb(http_parser *p, const char *buf, size_t len) { if (p != parser) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_http_header_field_cb: parser argument is wrong"); return NGX_ERROR; } if (state.last_header != FIELD) { state.num_headers++; } ngx_strlncat(state.headers[state.num_headers-1][0], sizeof(state.headers[state.num_headers-1][0]), buf, len); state.last_header = FIELD; return NGX_OK; } static int ngx_stream_http_header_value_cb(http_parser *p, const char *buf, size_t len) { if (p != parser) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_http_header_field_cb: parser argument is wrong"); return NGX_ERROR; } ngx_strlncat(state.headers[state.num_headers-1][1], sizeof(state.headers[state.num_headers-1][1]), buf, len); state.last_header = VALUE; return NGX_OK; } static int ngx_stream_http_body(http_parser *p, const char *buf, size_t len) { char *tmp_buf; tmp_buf = (char *)state.http_body; if (p != parser) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_http_body: parser argument is wrong"); return NGX_ERROR; } ngx_memcpy(tmp_buf, buf, len); tmp_buf += len; return NGX_OK; } static void ngx_stream_upsync_timeout_handler(ngx_event_t *event) { ngx_stream_upsync_server_t *upsync_server; if (ngx_stream_upsync_need_exit()) { return; } upsync_server = event->data; ngx_log_error(NGX_LOG_ERR, event->log, 0, "[WARN] upsync_timeout: timed out reading upsync_server: %V ", upsync_server->pc.name); ngx_stream_upsync_clean_event(upsync_server); } static void ngx_stream_upsync_clean_event(void *data) { ngx_msec_t t, tmp; ngx_pool_t *pool; ngx_connection_t *c; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_ctx_t *ctx; ngx_stream_upsync_server_t *upsync_server = data; ngx_stream_upsync_srv_conf_t *upscf; upscf = upsync_server->upscf; upsync_type_conf = upscf->upsync_type_conf; ctx = &upsync_server->ctx; pool = ctx->pool; c = upsync_server->pc.connection; if (c) { ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0, "upsync_clean_event: clean event: fd: %d", c->fd); ngx_close_connection(c); upsync_server->pc.connection = NULL; } if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL_SERVICES || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_ETCD) { if (parser != NULL) { ngx_free(parser); parser = NULL; } } if (pool != NULL) { ngx_destroy_pool(pool); } ctx->pool = NULL; if (!upsync_server->upsync_ev.timer_set) { tmp = upscf->upsync_interval; t = ngx_random() % 1000 + tmp; ngx_add_timer(&upsync_server->upsync_ev, t); } if (upsync_server->upsync_timeout_ev.timer_set) { ngx_del_timer(&upsync_server->upsync_timeout_ev); } return; } static ngx_int_t ngx_stream_upsync_need_exit() { if (ngx_terminate || ngx_exiting || ngx_quit) { ngx_stream_upsync_clear_all_events((ngx_cycle_t *)ngx_cycle); return 1; } return NGX_OK; } static void ngx_stream_upsync_clear_all_events(ngx_cycle_t *cycle) { ngx_uint_t i; ngx_queue_t *head, *next; ngx_connection_t *c; ngx_delay_event_t *queue_event; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_server_t *upsync_server; static ngx_flag_t has_cleared = 0; if (has_cleared || upsync_ctx == NULL || upsync_ctx->upstream_num == 0) { return; } ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "[WARN]:" "upsync_clear_all_events: on %P ", ngx_pid); has_cleared = 1; upsync_server = upsync_ctx->upsync_server; upsync_type_conf = upsync_server->upscf->upsync_type_conf; for (i = 0; i < upsync_ctx->upstream_num; i++) { if (upsync_server[i].upsync_ev.timer_set) { ngx_del_timer(&upsync_server[i].upsync_ev); } if (upsync_server[i].upsync_timeout_ev.timer_set) { c = upsync_server[i].pc.connection; if (c) { ngx_close_connection(c); upsync_server->pc.connection = NULL; } ngx_del_timer(&upsync_server[i].upsync_timeout_ev); } head = &upsync_server[i].delete_ev; for (next = ngx_queue_head(head); next != ngx_queue_sentinel(head); next = ngx_queue_next(next)) { queue_event = ngx_queue_data(next, ngx_delay_event_t, queue); if (queue_event->delay_delete_ev.timer_set) { ngx_del_timer(&queue_event->delay_delete_ev); } } } if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL_SERVICES || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_ETCD) { if (parser != NULL) { ngx_free(parser); parser = NULL; } } return; } static ngx_int_t ngx_stream_upsync_get_upstream(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server, char **conf_value) { ngx_stream_conf_client *client = ngx_stream_create_client(cycle, upsync_server); if (client == NULL) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_get_upstream: http client create error"); return NGX_ERROR; } ngx_int_t status = ngx_stream_client_conn(client); if (status != NGX_OK) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_get_upstream: http client conn error"); ngx_stream_client_destroy(client); return NGX_ERROR; } char *response = NULL; ngx_stream_client_send(client, upsync_server); if (ngx_stream_client_recv(client, &response, 0) <= 0) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "upsync_get_upstream: http client recv fail"); if (response != NULL) { ngx_free(response); response = NULL; } ngx_stream_client_destroy(client); return NGX_ERROR; } ngx_stream_client_destroy(client); if (ngx_stream_http_parser_init() == NGX_ERROR) { return NGX_ERROR; } http_parser_execute(parser, &settings, response, ngx_strlen(response)); if (parser != NULL) { ngx_free(parser); parser = NULL; } if (ngx_strncmp(state.status, "OK", 2) != 0) { return NGX_ERROR; } *conf_value = response; return NGX_OK; } static ngx_stream_conf_client * ngx_stream_create_client(ngx_cycle_t *cycle, ngx_stream_upsync_server_t *upsync_server) { ngx_stream_conf_client *client = NULL; ngx_stream_upstream_server_t *conf_server; ngx_stream_upsync_srv_conf_t *upscf; upscf = upsync_server->upscf; conf_server = &upscf->conf_server; client = ngx_calloc(sizeof(ngx_stream_conf_client), cycle->log); if (client == NULL) { return NULL; } client->sd = -1; client->connected = 0; client->addr = *(struct sockaddr_in *)conf_server->addrs[0].sockaddr; if((client->sd = socket(AF_INET,SOCK_STREAM, 0)) == NGX_ERROR) { ngx_free(client); client = NULL; return NULL; } struct timeval tv_timeout; tv_timeout.tv_sec = NGX_STREAM_SOCKET_TIMEOUT; tv_timeout.tv_usec = 0; if (setsockopt(client->sd, SOL_SOCKET, SO_SNDTIMEO, (void *) &tv_timeout, sizeof(struct timeval)) < 0) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "ngx_stream_create_client: setsockopt SO_SNDTIMEO error"); ngx_stream_client_destroy(client); return NULL; } if (setsockopt(client->sd, SOL_SOCKET, SO_RCVTIMEO, (void *) &tv_timeout, sizeof(struct timeval)) < 0) { ngx_log_error(NGX_LOG_ERR, cycle->log, 0, "ngx_stream_create_client: setsockopt SO_RCVTIMEO error"); ngx_stream_client_destroy(client); return NULL; } return client; } static ngx_int_t ngx_stream_client_conn(ngx_stream_conf_client *client) { if (connect(client->sd, (struct sockaddr *)&(client->addr), sizeof(struct sockaddr)) == NGX_ERROR) { return NGX_ERROR; } client->connected = 1; return NGX_OK; } static void ngx_stream_client_destroy(ngx_stream_conf_client *client) { close(client->sd); ngx_free(client); client = NULL; } static ngx_int_t ngx_stream_client_send(ngx_stream_conf_client *client, ngx_stream_upsync_server_t *upsync_server) { size_t size = 0; ngx_int_t tmp_send = 0; ngx_uint_t send_num = 0; ngx_upsync_conf_t *upsync_type_conf; ngx_stream_upsync_srv_conf_t *upscf; upscf = upsync_server->upscf; upsync_type_conf = upscf->upsync_type_conf; u_char request[ngx_pagesize]; ngx_memzero(request, ngx_pagesize); if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL || upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_CONSUL_SERVICES) { ngx_sprintf(request, "GET %V?recurse&index=%uL HTTP/1.0\r\nHost: %V\r\n" "Accept: */*\r\n\r\n", &upscf->upsync_send, upsync_server->index, &upscf->conf_server.name); } if (upsync_type_conf->upsync_type == NGX_STREAM_UPSYNC_ETCD) { ngx_sprintf(request, "GET %V? HTTP/1.0\r\nHost: %V\r\n" "Accept: */*\r\n\r\n", &upscf->upsync_send, &upscf->conf_server.name); } size = ngx_strlen(request); while(send_num < size) { tmp_send = send(client->sd, request + send_num, size - send_num, 0); /* TODO if tmp send is 0? */ if (tmp_send < 0) { ngx_log_error(NGX_LOG_ERR, ngx_cycle->log, 0, "ngx_stream_client_send: send byte %d", tmp_send); return NGX_ERROR; } send_num += tmp_send; } return send_num; } static ngx_int_t ngx_stream_client_recv(ngx_stream_conf_client *client, char **data, int size) { ssize_t recv_num = 0, tmp_recv = 0; char buff[ngx_pagesize]; char *tmp_data; ngx_int_t page_count = 0; *data = NULL; while(recv_num < size || size == 0) { tmp_recv = recv(client->sd, buff, ngx_pagesize, 0); if (tmp_recv <= 0) { break; } recv_num += tmp_recv; if (*data == NULL) { *data = (char *) ngx_calloc(ngx_pagesize, ngx_cycle->log); if (*data == NULL) { return NGX_ERROR; } page_count++; } if (recv_num >= (ssize_t)(page_count * ngx_pagesize)) { tmp_data = *data; page_count++; *data = (char *) ngx_calloc(page_count * ngx_pagesize, ngx_cycle->log); if (*data == NULL) { ngx_free(tmp_data); return NGX_ERROR; } ngx_memcpy(*data, tmp_data, recv_num - tmp_recv); ngx_free(tmp_data); } ngx_memcpy(*data + recv_num - tmp_recv, buff, tmp_recv); } if (*data != NULL) { *(*data + recv_num) = '\0'; } return recv_num; } static char * ngx_stream_upsync_set(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { ngx_stream_core_srv_conf_t *cscf; cscf = ngx_stream_conf_get_module_srv_conf(cf, ngx_stream_core_module); cscf->handler = ngx_stream_upsync_show; return NGX_CONF_OK; } static void ngx_stream_upsync_show_send(ngx_stream_session_t *s, ngx_buf_t *b) { u_char *data; ssize_t temp_send = 0, send_num = 0, len; ngx_connection_t *c; c = s->connection; data = b->pos; len = b->last - b->pos; ngx_log_debug0(NGX_LOG_DEBUG, c->log, 0, "upstream_show_send"); while (send_num < len) { temp_send = c->send(c, data + temp_send, len - send_num); #if (NGX_DEBUG) { ngx_err_t err; err = (temp_send >=0) ? 0 : ngx_socket_errno; ngx_log_debug2(NGX_LOG_DEBUG, c->log, err, "upsync show send size: %z, total: %z", temp_send, len); if (temp_send > 0) { ngx_log_debug2(NGX_LOG_DEBUG, c->log, err, "upsync show send content: %*s ", temp_send, data); } } #endif if (temp_send > 0) { send_num += temp_send; } else if (temp_send == 0 || temp_send == NGX_AGAIN) { continue; } else { c->error = 1; break; } } if (send_num == len) { ngx_log_debug0(NGX_LOG_DEBUG, c->log, 0, "upsync_show_send done."); } return; } static void ngx_stream_upsync_show_upstream(ngx_stream_upstream_srv_conf_t *uscf, ngx_buf_t *b) { ngx_str_t *host; ngx_stream_upstream_rr_peer_t *peer = NULL; ngx_stream_upstream_rr_peers_t *peers = NULL; host = &(uscf->host); //HTTP Body b->last = ngx_snprintf(b->last, b->end - b->last, "Upstream name: %V; ", host); if (uscf->peer.data == NULL) { b->last = ngx_snprintf(b->last, b->end - b->last, "Backend server count: %d\n", 0); return; } else { peers = (ngx_stream_upstream_rr_peers_t *) uscf->peer.data; b->last = ngx_snprintf(b->last, b->end - b->last, "Backend server count: %d\n", peers->number); } for (peer = peers->peer; peer; peer = peer->next) { b->last = ngx_snprintf(b->last, b->end - b->last, " server %V", &peer->name); b->last = ngx_snprintf(b->last, b->end - b->last, " weight=%d", peer->weight); b->last = ngx_snprintf(b->last, b->end - b->last, " max_fails=%d", peer->max_fails); b->last = ngx_snprintf(b->last, b->end - b->last, " fail_timeout=%ds", peer->fail_timeout); if (peer->down) { b->last = ngx_snprintf(b->last, b->end - b->last, " down"); } b->last = ngx_snprintf(b->last, b->end - b->last, ";\n"); } } static void ngx_stream_upsync_show(ngx_stream_session_t *s) { ngx_buf_t *b_header, *b_body; ngx_uint_t i; ngx_stream_upstream_srv_conf_t **uscfp = NULL; ngx_stream_upstream_main_conf_t *umcf; umcf = ngx_stream_cycle_get_module_main_conf(ngx_cycle, ngx_stream_upstream_module); uscfp = umcf->upstreams.elts; b_header = ngx_create_temp_buf(s->connection->pool, NGX_PAGE_SIZE); b_body = ngx_create_temp_buf(s->connection->pool, NGX_PAGE_SIZE * NGX_PAGE_NUMBER); if (b_header == NULL || b_body == NULL) { ngx_log_error(NGX_LOG_ERR, s->connection->log, 0, "alloc upsync_show buf failed"); ngx_stream_finalize_session(s, NGX_OK); return; } if (umcf->upstreams.nelts == 0) { b_body->last = ngx_snprintf(b_body->last, b_body->end - b_body->last, "No upstreams defined\n"); } for (i = 0; i < umcf->upstreams.nelts; i++) { ngx_stream_upsync_show_upstream(uscfp[i], b_body); b_body->last = ngx_snprintf(b_body->last, b_body->end - b_body->last, "\n"); } //HTTP Header b_header->last = ngx_snprintf(b_header->last, b_header->end - b_header->last, "HTTP/1.0 200 OK\r\n"); b_header->last = ngx_snprintf(b_header->last, b_header->end - b_header->last, "Server: nginx\r\n"); b_header->last = ngx_snprintf(b_header->last, b_header->end - b_header->last, "Content-Type: text/plain\r\n"); b_header->last = ngx_snprintf(b_header->last, b_header->end - b_header->last, "Content-Length: %d\r\n", b_body->last - b_body->pos); b_header->last = ngx_snprintf(b_header->last, b_header->end - b_header->last, "Connection: close\r\n\r\n"); ngx_stream_upsync_show_send(s, b_header); //send header ngx_stream_upsync_show_send(s, b_body); //send body ngx_stream_finalize_session(s, NGX_OK); return; } ================================================ FILE: src/ngx_stream_upsync_module.h ================================================ #ifndef _NGX_HTTP_UPSYNC_MODELE_H_INCLUDED_ #define _NGX_HTTP_UPSYNC_MODELE_H_INCLUDED_ #include #include #include #include "ngx_stream_json.h" #include "ngx_stream_http_parser.h" #define ngx_strrchr(s1, c) strrchr((const char *) s1, (int) c) #define ngx_ftruncate(fd, offset) ftruncate(fd, offset) #define ngx_lseek(fd, offset, whence) lseek(fd, offset, whence) #define ngx_fgets(fp, offset, whence) fgets(fp, offset, whence) #define ngx_fopen(path, mode) fopen(path, mode) #define ngx_fclose(fp) fclose(fp) #define ngx_strtoull(nptr, endptr, base) strtoull((const char *) nptr, \ (char **) endptr, (int) base) #define NGX_INDEX_HEADER "X-Consul-Index" #define NGX_INDEX_HEADER_LEN 14 #define NGX_INDEX_ETCD_HEADER "X-Etcd-Index" #define NGX_INDEX_ETCD_HEADER_LEN 12 #define NGX_MAX_HEADERS 20 #define NGX_MAX_ELEMENT_SIZE 512 #define NGX_DELAY_DELETE 30 * 60 * 1000 //75 * 1000 #define NGX_ADD 0 #define NGX_DEL 1 #define NGX_ALL 2 #define NGX_PAGE_SIZE 4 * 1024 #define NGX_PAGE_NUMBER 1024 #define NGX_STREAM_RETRY_TIMES 3 #define NGX_STREAM_SOCKET_TIMEOUT 1 #define NGX_STREAM_LB_DEFAULT 0 #define NGX_STREAM_LB_ROUNDROBIN 1 #define NGX_STREAM_LB_IP_HASH 2 #define NGX_STREAM_LB_LEAST_CONN 4 #define NGX_STREAM_LB_HASH_MODULA 8 #define NGX_STREAM_LB_HASH_KETAMA 16 /******************************hash*********************************/ extern ngx_module_t ngx_stream_upstream_hash_module; typedef struct { uint32_t hash; ngx_str_t *server; } ngx_stream_upstream_chash_point_t; typedef struct { ngx_uint_t number; ngx_stream_upstream_chash_point_t point[1]; } ngx_stream_upstream_chash_points_t; typedef struct { ngx_stream_complex_value_t key; ngx_stream_upstream_chash_points_t *points; } ngx_stream_upstream_hash_srv_conf_t; /****************************hash_end*******************************/ static int ngx_libc_cdecl ngx_stream_upsync_chash_cmp_points(const void *one, const void *two); static ngx_int_t ngx_stream_upsync_chash_init(ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_rr_peers_t *tmp_peers); static ngx_int_t ngx_stream_upsync_del_chash_peer( ngx_stream_upstream_srv_conf_t *uscf); static int ngx_libc_cdecl ngx_stream_upsync_chash_cmp_points(const void *one, const void *two) { ngx_stream_upstream_chash_point_t *first = (ngx_stream_upstream_chash_point_t *) one; ngx_stream_upstream_chash_point_t *second = (ngx_stream_upstream_chash_point_t *) two; if (first->hash < second->hash) { return -1; } else if (first->hash > second->hash) { return 1; } else { return 0; } } static ngx_int_t ngx_stream_upsync_chash_init(ngx_stream_upstream_srv_conf_t *uscf, ngx_stream_upstream_rr_peers_t *tmp_peers) { size_t new_size; size_t host_len, port_len; u_char *host, *port, c; uint32_t hash, base_hash; ngx_str_t *server; ngx_uint_t npoints, new_npoints; ngx_uint_t i, j; ngx_stream_upstream_rr_peer_t *peer; ngx_stream_upstream_rr_peers_t *peers; ngx_stream_upstream_chash_points_t *points; ngx_stream_upstream_hash_srv_conf_t *hcf; union { uint32_t value; u_char byte[4]; } prev_hash; hcf = ngx_stream_conf_upstream_srv_conf(uscf, ngx_stream_upstream_hash_module); if(hcf->points == NULL) { return 0; } peers = uscf->peer.data; if (tmp_peers != NULL) { new_npoints = peers->total_weight * 160; new_size = sizeof(ngx_stream_upstream_chash_points_t) + sizeof(ngx_stream_upstream_chash_point_t) * (new_npoints - 1); points = ngx_calloc(new_size, ngx_cycle->log); if (points == NULL ) { return NGX_ERROR; } ngx_free(hcf->points); /* free old points */ hcf->points = points; for (peer = peers->peer; peer; peer = peer->next) { server = &peer->server; /* * Hash expression is compatible with Cache::Memcached::Fast: * crc32(HOST \0 PORT PREV_HASH). */ if (server->len >= 5 && ngx_strncasecmp(server->data, (u_char *) "unix:", 5) == 0) { host = server->data + 5; host_len = server->len - 5; port = NULL; port_len = 0; goto done; } for (j = 0; j < server->len; j++) { c = server->data[server->len - j - 1]; if (c == ':') { host = server->data; host_len = server->len - j - 1; port = server->data + server->len - j; port_len = j; goto done; } if (c < '0' || c > '9') { break; } } host = server->data; host_len = server->len; port = NULL; port_len = 0; done: ngx_crc32_init(base_hash); ngx_crc32_update(&base_hash, host, host_len); ngx_crc32_update(&base_hash, (u_char *) "", 1); ngx_crc32_update(&base_hash, port, port_len); prev_hash.value = 0; npoints = peer->weight * 160; for (j = 0; j < npoints; j++) { hash = base_hash; ngx_crc32_update(&hash, prev_hash.byte, 4); ngx_crc32_final(hash); points->point[points->number].hash = hash; points->point[points->number].server = server; points->number++; #if (NGX_HAVE_LITTLE_ENDIAN) prev_hash.value = hash; #else prev_hash.byte[0] = (u_char) (hash & 0xff); prev_hash.byte[1] = (u_char) ((hash >> 8) & 0xff); prev_hash.byte[2] = (u_char) ((hash >> 16) & 0xff); prev_hash.byte[3] = (u_char) ((hash >> 24) & 0xff); #endif } } } else { new_npoints = peers->total_weight * 160; new_size = sizeof(ngx_stream_upstream_chash_points_t) + sizeof(ngx_stream_upstream_chash_point_t) * (new_npoints - 1); points = ngx_calloc(new_size, ngx_cycle->log); if (points == NULL ) { return NGX_ERROR; } ngx_memcpy(points, hcf->points, new_size); ngx_pfree(ngx_cycle->pool, hcf->points); hcf->points = points; return NGX_OK; } ngx_qsort(points->point, points->number, sizeof(ngx_stream_upstream_chash_point_t), ngx_stream_upsync_chash_cmp_points); for (i = 0, j = 1; j < points->number; j++) { if (points->point[i].hash != points->point[j].hash) { points->point[++i] = points->point[j]; } } points->number = i + 1; return NGX_OK; } static ngx_int_t ngx_stream_upsync_del_chash_peer(ngx_stream_upstream_srv_conf_t *uscf) { size_t host_len, port_len; u_char *host, *port, c; uint32_t hash, base_hash; ngx_str_t *server; ngx_uint_t npoints, i, j; ngx_stream_upstream_rr_peer_t *peer; ngx_stream_upstream_rr_peers_t *peers; ngx_stream_upstream_chash_points_t *points; ngx_stream_upstream_hash_srv_conf_t *hcf; union { uint32_t value; u_char byte[4]; } prev_hash; hcf = ngx_stream_conf_upstream_srv_conf(uscf, ngx_stream_upstream_hash_module); if(hcf->points == NULL) { return 0; } peers = uscf->peer.data; points = hcf->points; points->number = 0; for (peer = peers->peer; peer; peer = peer->next) { server = &peer->server; /* * Hash expression is compatible with Cache::Memcached::Fast: * crc32(HOST \0 PORT PREV_HASH). */ if (server->len >= 5 && ngx_strncasecmp(server->data, (u_char *) "unix:", 5) == 0) { host = server->data + 5; host_len = server->len - 5; port = NULL; port_len = 0; goto done; } for (j = 0; j < server->len; j++) { c = server->data[server->len - j - 1]; if (c == ':') { host = server->data; host_len = server->len - j - 1; port = server->data + server->len - j; port_len = j; goto done; } if (c < '0' || c > '9') { break; } } host = server->data; host_len = server->len; port = NULL; port_len = 0; done: ngx_crc32_init(base_hash); ngx_crc32_update(&base_hash, host, host_len); ngx_crc32_update(&base_hash, (u_char *) "", 1); ngx_crc32_update(&base_hash, port, port_len); prev_hash.value = 0; npoints = peer->weight * 160; for (j = 0; j < npoints; j++) { hash = base_hash; ngx_crc32_update(&hash, prev_hash.byte, 4); ngx_crc32_final(hash); points->point[points->number].hash = hash; points->point[points->number].server = server; points->number++; #if (NGX_HAVE_LITTLE_ENDIAN) prev_hash.value = hash; #else prev_hash.byte[0] = (u_char) (hash & 0xff); prev_hash.byte[1] = (u_char) ((hash >> 8) & 0xff); prev_hash.byte[2] = (u_char) ((hash >> 16) & 0xff); prev_hash.byte[3] = (u_char) ((hash >> 24) & 0xff); #endif } } ngx_qsort(points->point, points->number, sizeof(ngx_stream_upstream_chash_point_t), ngx_stream_upsync_chash_cmp_points); for (i = 0, j = 1; j < points->number; j++) { if (points->point[i].hash != points->point[j].hash) { points->point[++i] = points->point[j]; } } points->number = i + 1; return NGX_OK; } #endif //_NGX_HTTP_UPSYNC_MODELE_H_INCLUDED_ ================================================ FILE: test/README ================================================ NAME Test::Nginx - Testing modules for Nginx C module development DESCRIPTION This distribution provides two testing modules for Nginx C module development: * Test::Nginx::LWP * Test::Nginx::Socket All of them are based on Test::Base. Usually, Test::Nginx::Socket is preferred because it works on a much lower level and not that fault tolerant like Test::Nginx::LWP. Also, a lot of connection hang issues (like wrong "r->main->count" value in nginx 0.8.x) can only be captured by Test::Nginx::Socket because Perl's LWP::UserAgent client will close the connection itself which will conceal such issues from the testers. Test::Nginx automatically starts an nginx instance (from the "PATH" env) rooted at t/servroot/ and the default config template makes this nginx instance listen on the port 1984 by default. One can specify a different port number by setting his port number to the "TEST_NGINX_PORT" environment, as in export TEST_NGINX_PORT=1989 etcproxy integration The default settings in etcproxy (https://github.com/chaoslawful/etcproxy) makes this small TCP proxy split the TCP packets into bytes and introduce 1 ms latency among them. There's usually various TCP chains that we can put etcproxy into, for example Test::Nginx <=> nginx $ ./etcproxy 1234 1984 Here we tell etcproxy to listen on port 1234 and to delegate all the TCP traffic to the port 1984, the default port that Test::Nginx makes nginx listen to. And then we tell Test::Nginx to test against the port 1234, where etcproxy listens on, rather than the port 1984 that nginx directly listens on: $ TEST_NGINX_CLIENT_PORT=1234 prove -r t/ Then the TCP chain now looks like this: Test::Nginx <=> etcproxy (1234) <=> nginx (1984) So etcproxy can effectively emulate extreme network conditions and exercise "unusual" code paths in your nginx server by your tests. In practice, *tons* of weird bugs can be captured by this setting. Even ourselves didn't expect that this simple approach is so effective. nginx <=> memcached We first start the memcached server daemon on port 11211: memcached -p 11211 -vv and then we another etcproxy instance to listen on port 11984 like this $ ./etcproxy 11984 11211 Then we tell our t/foo.t test script to connect to 11984 rather than 11211: # foo.t use Test::Nginx::Socket; repeat_each(1); plan tests => 2 * repeat_each() * blocks(); $ENV{TEST_NGINX_MEMCACHED_PORT} ||= 11211; # make this env take a default value run_tests(); __DATA__ === TEST 1: sanity --- config location /foo { set $memc_cmd set; set $memc_key foo; set $memc_value bar; memc_pass 127.0.0.1:$TEST_NGINX_MEMCACHED_PORT; } --- request GET /foo --- response_body_like: STORED The Test::Nginx library will automatically expand the special macro $TEST_NGINX_MEMCACHED_PORT to the environment with the same name. You can define your own $TEST_NGINX_BLAH_BLAH_PORT macros as long as its prefix is "TEST_NGINX_" and all in upper case letters. And now we can run your test script against the etcproxy port 11984: TEST_NGINX_MEMCACHED_PORT=11984 prove t/foo.t Then the TCP chains look like this: Test::Nginx <=> nginx (1984) <=> etcproxy (11984) <=> memcached (11211) If "TEST_NGINX_MEMCACHED_PORT" is not set, then it will take the default value 11211, which is what we want when there's no etcproxy configured: Test::Nginx <=> nginx (1984) <=> memcached (11211) This approach also works for proxied mysql and postgres traffic. Please see the live test suite of ngx_drizzle and ngx_postgres for more details. Usually we set both "TEST_NGINX_CLIENT_PORT" and "TEST_NGINX_MEMCACHED_PORT" (and etc) at the same time, effectively yielding the following chain: Test::Nginx <=> etcproxy (1234) <=> nginx (1984) <=> etcproxy (11984) <=> memcached (11211) as long as you run two separate etcproxy instances in two separate terminals. It's easy to verify if the traffic actually goes through your etcproxy server. Just check if the terminal running etcproxy emits outputs. By default, etcproxy always dump out the incoming and outgoing data to stdout/stderr. valgrind integration Test::Nginx has integrated support for valgrind () even though by default it does not bother running it with the tests because valgrind will significantly slow down the test sutie. First ensure that your valgrind executable visible in your PATH env. And then run your test suite with the "TEST_NGINX_USE_VALGRIND" env set to true: TEST_NGINX_USE_VALGRIND=1 prove -r t If you see false alarms, you do have a chance to skip them by defining a ./valgrind.suppress file at the root of your module source tree, as in This is the suppression file for ngx_drizzle. Test::Nginx will automatically use it to start nginx with valgrind memcheck if this file does exist at the expected location. If you do see a lot of "Connection refused" errors while running the tests this way, then you probably have a slow machine (or a very busy one) that the default waiting time is not sufficient for valgrind to start. You can define the sleep time to a larger value by setting the "TEST_NGINX_SLEEP" env: TEST_NGINX_SLEEP=1 prove -r t The time unit used here is "second". The default sleep setting just fits my ThinkPad ("Core2Duo T9600"). Applying the no-pool patch to your nginx core is recommended while running nginx with valgrind: The nginx memory pool can prevent valgrind from spotting lots of invalid memory reads/writes as well as certain double-free errors. We did find a lot more memory issues in many of our modules when we first introduced the no-pool patch in practice ;) There's also more advanced features in Test::Nginx that have never documented. I'd like to write more about them in the near future ;) Nginx C modules that use Test::Nginx to drive their test suites ngx_echo ngx_headers_more ngx_chunkin ngx_memc ngx_drizzle ngx_rds_json ngx_rds_csv ngx_xss ngx_srcache ngx_lua ngx_set_misc ngx_array_var ngx_form_input ngx_iconv ngx_set_cconv ngx_postgres ngx_coolkit Naxsi SOURCE REPOSITORY This module has a Git repository on Github, which has access for all. http://github.com/agentzh/test-nginx If you want a commit bit, feel free to drop me a line. AUTHORS agentzh (章亦春) "" Antoine BONAVITA "" COPYRIGHT & LICENSE Copyright (c) 2009-2012, agentzh "". Copyright (c) 2011-2012, Antoine Bonavita "". This module is licensed under the terms of the BSD license. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the authors nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. SEE ALSO Test::Nginx::LWP, Test::Nginx::Socket, Test::Base. ================================================ FILE: test/conf-server.sh ================================================ #!/bin/sh /usr/local/bin/consul agent -server -bootstrap -data-dir=/tmp/consul #/usr/local/bin/etcd --peers 127.0.0.1:8500 ================================================ FILE: test/t/lib/Test/Nginx/IMAP.pm ================================================ package Test::Nginx::IMAP; # (C) Maxim Dounin # Module for nginx imap tests. ############################################################################### use warnings; use strict; use Test::More qw//; use IO::Socket; use Socket qw/ CRLF /; use Test::Nginx; use base qw/ IO::Socket::INET /; sub new { my $class = shift; my $self = return $class->SUPER::new( Proto => "tcp", PeerAddr => "127.0.0.1:8143", @_ ) or die "Can't connect to nginx: $!\n"; $self->autoflush(1); return $self; } sub send { my ($self, $cmd) = @_; log_out($cmd); $self->print($cmd . CRLF); } sub read { my ($self) = @_; eval { local $SIG{ALRM} = sub { die "timeout\n" }; alarm(3); while (<$self>) { log_in($_); # XXX next if m/^\d\d\d-/; last; } alarm(0); }; alarm(0); if ($@) { log_in("died: $@"); return undef; } return $_; } sub check { my ($self, $regex, $name) = @_; Test::More->builder->like($self->read(), $regex, $name); } sub ok { my $self = shift; Test::More->builder->like($self->read(), qr/^\S+ OK/, @_); } ############################################################################### sub imap_test_daemon { my $server = IO::Socket::INET->new( Proto => 'tcp', LocalAddr => '127.0.0.1:8144', Listen => 5, Reuse => 1 ) or die "Can't create listening socket: $!\n"; while (my $client = $server->accept()) { $client->autoflush(1); print $client "* OK fake imap server ready" . CRLF; while (<$client>) { my $tag = ''; $tag = $1 if m/^(\S+)/; s/^(\S+)\s+//; if (/^logout/i) { print $client $tag . ' OK logout ok' . CRLF; } elsif (/^login /i) { print $client $tag . ' OK login ok' . CRLF; } else { print $client $tag . ' ERR unknown command' . CRLF; } } close $client; } } ############################################################################### 1; ############################################################################### ================================================ FILE: test/t/lib/Test/Nginx/POP3.pm ================================================ package Test::Nginx::POP3; # (C) Maxim Dounin # Module for nginx pop3 tests. ############################################################################### use warnings; use strict; use Test::More qw//; use IO::Socket; use Socket qw/ CRLF /; use Test::Nginx; use base qw/ IO::Socket::INET /; sub new { my $class = shift; my $self = return $class->SUPER::new( Proto => "tcp", PeerAddr => "127.0.0.1:8110", @_ ) or die "Can't connect to nginx: $!\n"; $self->autoflush(1); return $self; } sub send { my ($self, $cmd) = @_; log_out($cmd); $self->print($cmd . CRLF); } sub read { my ($self) = @_; eval { local $SIG{ALRM} = sub { die "timeout\n" }; alarm(3); while (<$self>) { log_in($_); # XXX next if m/^\d\d\d-/; last; } alarm(0); }; alarm(0); if ($@) { log_in("died: $@"); return undef; } return $_; } sub check { my ($self, $regex, $name) = @_; Test::More->builder->like($self->read(), $regex, $name); } sub ok { my $self = shift; Test::More->builder->like($self->read(), qr/^\+OK/, @_); } ############################################################################### sub pop3_test_daemon { my $server = IO::Socket::INET->new( Proto => 'tcp', LocalAddr => '127.0.0.1:8111', Listen => 5, Reuse => 1 ) or die "Can't create listening socket: $!\n"; while (my $client = $server->accept()) { $client->autoflush(1); print $client "+OK fake pop3 server ready" . CRLF; while (<$client>) { if (/^quit/i) { print $client '+OK quit ok' . CRLF; } elsif (/^user test\@example.com/i) { print $client '+OK user ok' . CRLF; } elsif (/^pass secret/i) { print $client '+OK pass ok' . CRLF; } else { print $client "-ERR unknown command" . CRLF; } } close $client; } } ############################################################################### 1; ############################################################################### ================================================ FILE: test/t/lib/Test/Nginx/SMTP.pm ================================================ package Test::Nginx::SMTP; # (C) Maxim Dounin # Module for nginx smtp tests. ############################################################################### use warnings; use strict; use Test::More qw//; use IO::Socket; use Socket qw/ CRLF /; use Test::Nginx; use base qw/ IO::Socket::INET /; sub new { my $class = shift; my $self = return $class->SUPER::new( Proto => "tcp", PeerAddr => "127.0.0.1:8025", @_ ) or die "Can't connect to nginx: $!\n"; $self->autoflush(1); return $self; } sub send { my ($self, $cmd) = @_; log_out($cmd); $self->print($cmd . CRLF); } sub read { my ($self) = @_; eval { local $SIG{ALRM} = sub { die "timeout\n" }; alarm(3); while (<$self>) { log_in($_); next if m/^\d\d\d-/; last; } alarm(0); }; alarm(0); if ($@) { log_in("died: $@"); return undef; } return $_; } sub check { my ($self, $regex, $name) = @_; Test::More->builder->like($self->read(), $regex, $name); } sub ok { my $self = shift; Test::More->builder->like($self->read(), qr/^2\d\d /, @_); } sub authok { my $self = shift; Test::More->builder->like($self->read(), qr/^235 /, @_); } ############################################################################### sub smtp_test_daemon { my $server = IO::Socket::INET->new( Proto => 'tcp', LocalAddr => '127.0.0.1:8026', Listen => 5, Reuse => 1 ) or die "Can't create listening socket: $!\n"; while (my $client = $server->accept()) { $client->autoflush(1); print $client "220 fake esmtp server ready" . CRLF; while (<$client>) { Test::Nginx::log_core('||', $_); if (/^quit/i) { print $client '221 quit ok' . CRLF; } elsif (/^(ehlo|helo)/i) { print $client '250 hello ok' . CRLF; } elsif (/^rset/i) { print $client '250 rset ok' . CRLF; } elsif (/^mail from:[^@]+$/i) { print $client '500 mail from error' . CRLF; } elsif (/^mail from:/i) { print $client '250 mail from ok' . CRLF; } elsif (/^rcpt to:[^@]+$/i) { print $client '500 rcpt to error' . CRLF; } elsif (/^rcpt to:/i) { print $client '250 rcpt to ok' . CRLF; } elsif (/^xclient/i) { print $client '220 xclient ok' . CRLF; } else { print $client "500 unknown command" . CRLF; } } close $client; } } ############################################################################### 1; ############################################################################### ================================================ FILE: test/t/lib/Test/Nginx.pm ================================================ package Test::Nginx; # (C) Maxim Dounin # Generic module for nginx tests. ############################################################################### use warnings; use strict; use base qw/ Exporter /; our @EXPORT = qw/ log_in log_out http http_get http_head /; our @EXPORT_OK = qw/ http_gzip_request http_gzip_like http_start http_end /; our %EXPORT_TAGS = ( gzip => [ qw/ http_gzip_request http_gzip_like / ] ); ############################################################################### use File::Path qw/ rmtree /; use File::Temp qw/ tempdir /; use IO::Socket; use POSIX qw/ waitpid WNOHANG /; use Socket qw/ CRLF /; use Test::More qw//; ############################################################################### our $NGINX = defined $ENV{TEST_NGINX_BINARY} ? $ENV{TEST_NGINX_BINARY} : '../nginx/objs/nginx'; sub new { my $self = {}; bless $self; $self->{_pid} = $$; $self->{_testdir} = tempdir( 'nginx-test-XXXXXXXXXX', TMPDIR => 1, CLEANUP => not $ENV{TEST_NGINX_LEAVE} ) or die "Can't create temp directory: $!\n"; $self->{_testdir} =~ s!\\!/!g if $^O eq 'MSWin32'; $self->{_dso_module} = (); mkdir "$self->{_testdir}/logs" or die "Can't create logs directory: $!\n"; return $self; } sub DESTROY { my ($self) = @_; local $?; return if $self->{_pid} != $$; $self->stop(); $self->stop_daemons(); if ($ENV{TEST_NGINX_CATLOG}) { system("cat $self->{_testdir}/error.log"); } if (not $ENV{TEST_NGINX_LEAVE}) { eval { rmtree($self->{_testdir}); }; } } sub has($;) { my ($self, @features) = @_; foreach my $feature (@features) { Test::More::plan(skip_all => "$feature not compiled in") unless $self->has_module($feature); } return $self; } sub set_dso($;) { my ($self, $module_name, $module_path) = @_; $self->{_dso_module}{$module_name} = $module_path; } sub has_module($) { my ($self, $feature) = @_; my %regex = ( sni => 'TLS SNI support enabled', mail => '--with-mail(?!\S)', flv => '--with-http_flv_module', perl => '--with-http_perl_module', auth_request => '--with-http_auth_request_module', charset => '(?s)^(?!.*--without-http_charset_module)', gzip => '(?s)^(?!.*--without-http_gzip_module)', ssi => '(?s)^(?!.*--without-http_ssi_module)', userid => '(?s)^(?!.*--without-http_userid_module)', access => '(?s)^(?!.*--without-http_access_module)', auth_basic => '(?s)^(?!.*--without-http_auth_basic_module)', autoindex => '(?s)^(?!.*--without-http_autoindex_module)', geo => '(?s)^(?!.*--without-http_geo_module)', map => '(?s)^(?!.*--without-http_map_module)', referer => '(?s)^(?!.*--without-http_referer_module)', rewrite => '(?s)^(?!.*--without-http_rewrite_module)', proxy => '(?s)^(?!.*--without-http_proxy_module)', fastcgi => '(?s)^(?!.*--without-http_fastcgi_module)', uwsgi => '(?s)^(?!.*--without-http_uwsgi_module)', scgi => '(?s)^(?!.*--without-http_scgi_module)', memcached => '(?s)^(?!.*--without-http_memcached_module)', limit_conn => '(?s)^(?!.*--without-http_limit_conn_module)', limit_req => '(?s)^(?!.*--without-http_limit_req_module)', empty_gif => '(?s)^(?!.*--without-http_empty_gif_module)', browser => '(?s)^(?!.*--without-http_browser_module)', upstream_hash => '(?s)^(?!.*--without-http_upstream_hash_module)', upstream_ip_hash => '(?s)^(?!.*--without-http_upstream_ip_hash_module)', reqstat => '(?s)^(?!.*--without-http_reqstat_module)', upstream_least_conn => '(?s)^(?!.*--without-http_upstream_least_conn_mod)', upstream_keepalive => '(?s)^(?!.*--without-http_upstream_keepalive_modu)', http => '(?s)^(?!.*--without-http(?!\S))', cache => '(?s)^(?!.*--without-http-cache)', pop3 => '(?s)^(?!.*--without-mail_pop3_module)', imap => '(?s)^(?!.*--without-mail_imap_module)', smtp => '(?s)^(?!.*--without-mail_smtp_module)', pcre => '(?s)^(?!.*--without-pcre)', split_clients => '(?s)^(?!.*--without-http_split_clients_module)', ); my $re = $regex{$feature}; $re = $feature if !defined $re; $self->{_configure_args} = `$NGINX -V 2>&1` if !defined $self->{_configure_args}; return ($self->{_configure_args} =~ $re or $self->{_configure_args} =~ '--enable-mods-static=all') ? 1 : 0; } sub has_version($) { my ($self, $need) = @_; $self->{_configure_args} = `$NGINX -V 2>&1` if !defined $self->{_configure_args}; $self->{_configure_args} =~ m!nginx/([0-9.]+)!; my @v = split(/\./, $1); my ($n, $v); for $n (split(/\./, $need)) { $v = shift @v || 0; return 0 if $n > $v; return 1 if $v > $n; } return 1; } sub has_daemon($) { my ($self, $daemon) = @_; if ($^O eq 'MSWin32') { Test::More::plan(skip_all => "win32"); return $self; } if ($^O eq 'solaris') { Test::More::plan(skip_all => "$daemon not found") unless `command -v $daemon`; return $self; } Test::More::plan(skip_all => "$daemon not found") unless `which $daemon`; return $self; } sub try_run($$) { my ($self, $message) = @_; $self->run(); eval { open OLDERR, ">&", \*STDERR; close STDERR; $self->run(); open STDERR, ">&", \*OLDERR; }; Test::More::plan(skip_all => $message) if $@; return $self; } sub plan($) { my ($self, $plan) = @_; Test::More::plan(tests => $plan); return $self; } sub run(;$) { my ($self, $conf) = @_; my $testdir = $self->{_testdir}; if (defined $conf) { my $c = `cat $conf`; $self->write_file_expand('nginx.conf', $c); } my $pid = fork(); die "Unable to fork(): $!\n" unless defined $pid; if ($pid == 0) { my @globals = $self->{_test_globals} ? () : ('-g', "pid $testdir/nginx.pid; " . "error_log $testdir/error.log debug;"); exec($NGINX, '-p', $testdir, '-c', 'nginx.conf', @globals), or die "Unable to exec(): $!\n"; } # wait for nginx to start $self->waitforfile("$testdir/nginx.pid", $pid) or die "Can't start nginx"; $self->{_started} = 1; return $self; } sub waitforfile($;$) { my ($self, $file, $pid) = @_; my $exited; # wait for file to appear # or specified process to exit for (1 .. 30) { return 1 if -e $file; return 0 if $exited; $exited = waitpid($pid, WNOHANG) != 0 if $pid; select undef, undef, undef, 0.1; } return undef; } sub waitforsocket($) { my ($self, $peer) = @_; # wait for socket to accept connections for (1 .. 30) { my $s = IO::Socket::INET->new( Proto => 'tcp', PeerAddr => $peer ); return 1 if defined $s; select undef, undef, undef, 0.1; } return undef; } sub reload() { my ($self) = @_; return $self unless $self->{_started}; local $/; open F, '<' . $self->{_testdir} . '/nginx.pid' or die "Can't open nginx.pid: $!"; my $pid = ; close F; if ($^O eq 'MSWin32') { my $testdir = $self->{_testdir}; my @globals = $self->{_test_globals} ? () : ('-g', "pid $testdir/nginx.pid; " . "error_log $testdir/error.log debug;"); system($NGINX, '-c', "$testdir/nginx.conf", '-s', 'reload', @globals) == 0 or die "system() failed: $?\n"; } else { kill 'HUP', $pid; } sleep(1); return $self; } sub stop() { my ($self) = @_; return $self unless $self->{_started}; local $/; open F, '<' . $self->{_testdir} . '/nginx.pid' or die "Can't open nginx.pid: $!"; my $pid = ; close F; if ($^O eq 'MSWin32') { my $testdir = $self->{_testdir}; my @globals = $self->{_test_globals} ? () : ('-g', "pid $testdir/nginx.pid; " . "error_log $testdir/error.log debug;"); system($NGINX, '-p', $testdir, '-c', "nginx.conf", '-s', 'stop', @globals) == 0 or die "system() failed: $?\n"; } else { kill 'QUIT', $pid; } waitpid($pid, 0); $self->{_started} = 0; return $self; } sub stop_daemons() { my ($self) = @_; while ($self->{_daemons} && scalar @{$self->{_daemons}}) { my $p = shift @{$self->{_daemons}}; kill $^O eq 'MSWin32' ? 9 : 'TERM', $p; waitpid($p, 0); } return $self; } sub read_file($) { my ($self, $name) = @_; local $/; open F, '<', $self->{_testdir} . '/' . $name or die "Can't open $name: $!"; my $content = ; close F; return $content; } sub write_file($$) { my ($self, $name, $content) = @_; open F, '>' . $self->{_testdir} . '/' . $name or die "Can't create $name: $!"; print F $content; close F; return $self; } sub write_file_expand($$) { my ($self, $name, $content) = @_; $content =~ s/%%TEST_GLOBALS%%/$self->test_globals()/gmse; $content =~ s/%%TEST_GLOBALS_DSO%%/$self->test_globals_dso()/gmse; $content =~ s/%%TEST_GLOBALS_HTTP%%/$self->test_globals_http()/gmse; $content =~ s/%%TESTDIR%%/$self->{_testdir}/gms; return $self->write_file($name, $content); } sub run_daemon($;@) { my ($self, $code, @args) = @_; my $pid = fork(); die "Can't fork daemon: $!\n" unless defined $pid; if ($pid == 0) { if (ref($code) eq 'CODE') { $code->(@args); exit 0; } else { exec($code, @args); exit 0; } } $self->{_daemons} = [] unless defined $self->{_daemons}; push @{$self->{_daemons}}, $pid; return $self; } sub testdir() { my ($self) = @_; return $self->{_testdir}; } sub test_globals() { my ($self) = @_; return $self->{_test_globals} if defined $self->{_test_globals}; my $s = ''; $s .= "pid $self->{_testdir}/nginx.pid;\n"; $s .= "error_log $self->{_testdir}/error.log debug;\n"; $s .= $ENV{TEST_NGINX_GLOBALS} if $ENV{TEST_NGINX_GLOBALS}; $self->{_test_globals} = $s; } sub test_globals_dso() { my ($self) = @_; return "" unless defined $ENV{TEST_NGINX_DSO}; return $self->{_test_globals_dso} if defined $self->{_test_globals_dso}; my $s = ''; $s .= "dso {\n"; if (defined $ENV{NGINX_DSO_PATH}) { $s .= "path $ENV{NGINX_DSO_PATH};\n"; } while ( my ($key, $value) = each(%{$self->{_dso_module}}) ) { $s .= "load $key $value;\n"; } $s .= "}\n"; $self->{_test_globals_dso} = $s; } sub test_globals_http() { my ($self) = @_; return $self->{_test_globals_http} if defined $self->{_test_globals_http}; my $s = ''; $s .= "root $self->{_testdir};\n"; $s .= "access_log $self->{_testdir}/access.log;\n"; $s .= "client_body_temp_path $self->{_testdir}/client_body_temp;\n"; $s .= "fastcgi_temp_path $self->{_testdir}/fastcgi_temp;\n" if $self->has_module('fastcgi'); $s .= "proxy_temp_path $self->{_testdir}/proxy_temp;\n" if $self->has_module('proxy'); $s .= "uwsgi_temp_path $self->{_testdir}/uwsgi_temp;\n" if $self->has_module('uwsgi'); $s .= "scgi_temp_path $self->{_testdir}/scgi_temp;\n" if $self->has_module('scgi'); $s .= $ENV{TEST_NGINX_GLOBALS_HTTP} if $ENV{TEST_NGINX_GLOBALS_HTTP}; $self->{_test_globals_http} = $s; } ############################################################################### sub log_core { return unless $ENV{TEST_NGINX_VERBOSE}; my ($prefix, $msg) = @_; ($prefix, $msg) = ('', $prefix) unless defined $msg; $prefix .= ' ' if length($prefix) > 0; if (length($msg) > 2048) { $msg = substr($msg, 0, 2048) . "(...logged only 2048 of " . length($msg) . " bytes)"; } $msg =~ s/^/# $prefix/gm; $msg =~ s/([^\x20-\x7e])/sprintf('\\x%02x', ord($1)) . (($1 eq "\n") ? "\n" : '')/gmxe; $msg .= "\n" unless $msg =~ /\n\Z/; print $msg; } sub log_out { log_core('>>', @_); } sub log_in { log_core('<<', @_); } ############################################################################### sub http_get($;%) { my ($url, %extra) = @_; return http(<new( Proto => 'tcp', PeerAddr => '127.0.0.1:8080' ) or die "Can't connect to nginx: $!\n"; log_out($request); $s->print($request); select undef, undef, undef, $extra{sleep} if $extra{sleep}; return '' if $extra{aborted}; if ($extra{body}) { log_out($extra{body}); $s->print($extra{body}); } alarm(0); }; alarm(0); if ($@) { log_in("died: $@"); return undef; } return $s; } sub http_end($;%) { my ($s) = @_; my $reply; eval { local $SIG{ALRM} = sub { die "timeout\n" }; local $SIG{PIPE} = sub { die "sigpipe\n" }; alarm(5); local $/; $reply = $s->getline(); alarm(0); }; alarm(0); if ($@) { log_in("died: $@"); return undef; } log_in($reply); return $reply; } ############################################################################### sub http_gzip_request { my ($url) = @_; my $r = http(<builder->skip( "IO::Uncompress::Gunzip not installed", 1) if $@; my $in = http_content($text); my $out; IO::Uncompress::Gunzip::gunzip(\$in => \$out); Test::More->builder->like($out, $re, $name); } } ############################################################################### 1; ############################################################################### ================================================ FILE: test/t/lib/Time/Parse.pm ================================================ # Date::Parse # # Copyright (c) 1995 Graham Barr. All rights reserved. This program is free # software; you can redistribute it and/or modify it under the same terms # as Perl itself. package Time::Parse; =head1 NAME Date::Parse - Parse date strings into time values =head1 SYNOPSIS use Date::Parse; $time = str2time($date); ($ss,$mm,$hh,$day,$month,$year,$zone) = strptime($date); =head1 DESCRIPTION C provides two routines for parsing date strings into time values. =over 4 =item str2time(DATE [, ZONE]) C parses C and returns a unix time value, or undef upon failure. C, if given, specifies the timezone to assume when parsing if the date string does not specify a timezome. =item strptime(DATE [, ZONE]) C takes the same arguments as str2time but returns an array of values C<($ss,$mm,$hh,$day,$month,$year,$zone)>. Elements are only defined if they could be extracted from the date string. The C<$zone> element is the timezone offset in seconds from GMT. An empty array is returned upon failure. =head1 MULTI-LANGUAGE SUPPORT Date::Parse is capable of parsing dates in several languages, these are English, French, German and Italian. Changing the language is done via a static method call, for example Date::Parse->language('German'); will cause Date::Parse to attempt to parse any subsequent dates in German. This is only a first pass, I am considering changing this to be $lang = Date::Language->new('German'); $lang->str2time("25 Jun 1996 21:09:55 +0100"); I am open to suggestions on this. =head1 AUTHOR Graham Barr =head1 REVISION $Revision: 2.6 $ =head1 COPYRIGHT Copyright (c) 1995 Graham Barr. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. =cut require 5.000; use strict; use vars qw($VERSION @ISA @EXPORT); use Time::Local; use Carp; use Time::Zone; use Exporter; @ISA = qw(Exporter); @EXPORT = qw(&strtotime &str2time &strptime); $VERSION = sprintf("%d.%02d", q$Revision: 2.6 $ =~ m#(\d+)\.(\d+)#); my %month = ( january => 0, february => 1, march => 2, april => 3, may => 4, june => 5, july => 6, august => 7, september => 8, sept => 8, october => 9, november => 10, december => 11, ); my %day = ( sunday => 0, monday => 1, tuesday => 2, tues => 2, wednesday => 3, wednes => 3, thursday => 4, thur => 4, thurs => 4, friday => 5, saturday => 6, ); my @suf = (qw(th st nd rd th th th th th th)) x 3; @suf[11,12,13] = qw(th th th); #Abbreviations map { $month{substr($_,0,3)} = $month{$_} } keys %month; map { $day{substr($_,0,3)} = $day{$_} } keys %day; my $strptime = <<'ESQ'; my %month = map { lc $_ } %$mon_ref; my $daypat = join("|", map { lc $_ } keys %$day_ref); my $monpat = join("|", keys %month); my $sufpat = join("|", map { lc $_ } @$suf_ref); my %ampm = ( am => 0, pm => 12 ); # allow map am +. a.m. map { my($z) = $_; $z =~ s#(\w)#$1\.#g; $ampm{$z} = $ampm{$_} } keys %ampm; my($AM, $PM) = (0,12); sub { my $dtstr = lc shift; my $merid = 24; my($year,$month,$day,$hh,$mm,$ss,$zone) = (undef) x 7; $zone = tz_offset(shift) if(@_); while(1) { last unless($dtstr =~ s#\([^\(\)]*\)# #o) } $dtstr =~ s#(\A|\n|\Z)# #sog; # ignore day names $dtstr =~ s#([\d\w\s])[\.\,]\s#$1 #sog; $dtstr =~ s#($daypat)\s*(den\s)?# #o; # Time: 12:00 or 12:00:00 with optional am/pm if($dtstr =~ s#[:\s](\d\d?):(\d\d)(:(\d\d)(?:\.\d+)?)?\s*([ap]\.?m\.?)?\s# #o) { ($hh,$mm,$ss) = ($1,$2,$4 || 0); $merid = $ampm{$5} if($5); } # Time: 12 am elsif($dtstr =~ s#\s(\d\d?)\s*([ap]\.?m\.?)\s# #o) { ($hh,$mm,$ss) = ($1,0,0); $merid = $ampm{$2}; } # Date: 12-June-96 (using - . or /) if($dtstr =~ s#\s(\d\d?)([\-\./])($monpat)(\2(\d\d+))?\s# #o) { ($month,$day) = ($month{$3},$1); $year = $5 if($5); } # Date: 12-12-96 (using '-', '.' or '/' ) elsif($dtstr =~ s#\s(\d\d*)([\-\./])(\d\d?)(\2(\d\d+))?\s# #o) { ($month,$day) = ($1 - 1,$3); if($5) { $year = $5; # Possible match for 1995-01-24 (short mainframe date format); ($year,$month,$day) = ($1, $3 - 1, $5) if($month > 12); } } elsif($dtstr =~ s#\s(\d+)\s*($sufpat)?\s*($monpat)# #o) { ($month,$day) = ($month{$3},$1); } elsif($dtstr =~ s#($monpat)\s*(\d+)\s*($sufpat)?\s# #o) { ($month,$day) = ($month{$1},$2); } # Date: 961212 elsif($dtstr =~ s#\s(\d\d)(\d\d)(\d\d)\s# #o) { ($year,$month,$day) = ($1,$2-1,$3); } $year = $1 if(!defined($year) && $dtstr =~ s#\s(\d{2}(\d{2})?)[\s\.,]# #o); # Zone if($dtstr =~ s#\s"?(\w{3,})\s# #o) { $zone = tz_offset($1); return () unless(defined $zone); } elsif($dtstr =~ s#\s(([\-\+])\d\d?)(\d\d)\s# #o) { my $m = $2 . $3; $zone = 60 * ($m + (60 * $1)); } return () if($dtstr =~ /\S/o); $hh += 12 if(defined $hh && $merid == $PM); $year -= 1900 if(defined $year && $year > 1900); return ($ss,$mm,$hh,$day,$month,$year,$zone); } ESQ use vars qw($day_ref $mon_ref $suf_ref $obj); sub gen_parser { local($day_ref,$mon_ref,$suf_ref,$obj) = @_; if($obj) { my $obj_strptime = $strptime; substr($obj_strptime,index($strptime,"sub")+6,0) = <<'ESQ'; shift; # package ESQ return eval "$obj_strptime"; } eval "$strptime"; } *strptime = gen_parser(\%day,\%month,\@suf); sub str2time { my @t = strptime(@_); return undef unless @t; my($ss,$mm,$hh,$day,$month,$year,$zone) = @t; my @lt = localtime(time); $hh ||= 0; $mm ||= 0; $ss ||= 0; $month = $lt[4] unless(defined $month); $day = $lt[3] unless(defined $day); $year = ($month > $lt[4]) ? ($lt[5] - 1) : $lt[5] unless(defined $year); return defined $zone ? timegm($ss,$mm,$hh,$day,$month,$year) - $zone : timelocal($ss,$mm,$hh,$day,$month,$year); } 1; ================================================ FILE: test/t/lib/Time/Zone.pm ================================================ package Time::Zone; =head1 NAME Time::Zone -- miscellaneous timezone manipulations routines =head1 SYNOPSIS use Time::Zone; print tz2zone(); print tz2zone($ENV{'TZ'}); print tz2zone($ENV{'TZ'}, time()); print tz2zone($ENV{'TZ'}, undef, $isdst); $offset = tz_local_offset(); $offset = tz_offset($TZ); =head1 DESCRIPTION This is a collection of miscellaneous timezone manipulation routines. C parses the TZ environment variable and returns a timezone string suitable for inclusion in L-like output. It opionally takes a timezone string, a time, and a is-dst flag. C determins the offset from GMT time in seconds. It only does the calculation once. C determines the offset from GMT in seconds of a specified timezone. C determines the name of the timezone based on its offset =head1 AUTHORS Graham Barr David Muir Sharnoff Paul Foley =cut require 5.002; require Exporter; use Carp; use strict; use vars qw(@ISA @EXPORT $VERSION @tz_local); @ISA = qw(Exporter); @EXPORT = qw(tz2zone tz_local_offset tz_offset tz_name); $VERSION = "2.04"; # Parts stolen from code by Paul Foley sub tz2zone (;$$$) { my($TZ, $time, $isdst) = @_; use vars qw(%tzn_cache); $TZ = defined($ENV{'TZ'}) ? ( $ENV{'TZ'} ? $ENV{'TZ'} : 'GMT' ) : '' unless $TZ; # Hack to deal with 'PST8PDT' format of TZ # Note that this can't deal with all the esoteric forms, but it # does recognize the most common: [:]STDoff[DST[off][,rule]] if (! defined $isdst) { my $j; $time = time() unless $time; ($j, $j, $j, $j, $j, $j, $j, $j, $isdst) = localtime($time); } if (defined $tzn_cache{$TZ}->[$isdst]) { return $tzn_cache{$TZ}->[$isdst]; } if ($TZ =~ /^ ( [^:\d+\-,] {3,} ) ( [+-] ? \d {1,2} ( : \d {1,2} ) {0,2} ) ( [^\d+\-,] {3,} )? /x ) { $TZ = $isdst ? $4 : $1; $tzn_cache{$TZ} = [ $1, $4 ]; } else { $tzn_cache{$TZ} = [ $TZ, $TZ ]; } return $TZ; } sub tz_local_offset (;$) { my ($time) = @_; $time = time() unless $time; my (@l) = localtime($time); my $isdst = $l[8]; if (defined($tz_local[$isdst])) { return $tz_local[$isdst]; } $tz_local[$isdst] = &calc_off($time); return $tz_local[$isdst]; } sub calc_off { my ($time) = @_; my (@l) = localtime($time); my (@g) = gmtime($time); my $off; $off = $l[0] - $g[0] + ($l[1] - $g[1]) * 60 + ($l[2] - $g[2]) * 3600; # subscript 7 is yday. if ($l[7] == $g[7]) { # done } elsif ($l[7] == $g[7] + 1) { $off += 86400; } elsif ($l[7] == $g[7] - 1) { $off -= 86400; } elsif ($l[7] < $g[7]) { # crossed over a year boundry! # localtime is beginning of year, gmt is end # therefore local is ahead $off += 86400; } else { $off -= 86400; } return $off; } # constants CONFIG: { use vars qw(%dstZone %zoneOff %dstZoneOff %Zone); %dstZone = ( # "ndt" => -2*3600-1800, # Newfoundland Daylight "adt" => -3*3600, # Atlantic Daylight "edt" => -4*3600, # Eastern Daylight "cdt" => -5*3600, # Central Daylight "mdt" => -6*3600, # Mountain Daylight "pdt" => -7*3600, # Pacific Daylight "ydt" => -8*3600, # Yukon Daylight "hdt" => -9*3600, # Hawaii Daylight "bst" => +1*3600, # British Summer "mest" => +2*3600, # Middle European Summer "sst" => +2*3600, # Swedish Summer "fst" => +2*3600, # French Summer "wadt" => +8*3600, # West Australian Daylight # "cadt" => +10*3600+1800, # Central Australian Daylight "eadt" => +11*3600, # Eastern Australian Daylight "nzdt" => +13*3600, # New Zealand Daylight ); %Zone = ( "gmt" => 0, # Greenwich Mean "ut" => 0, # Universal (Coordinated) "utc" => 0, "wet" => 0, # Western European "wat" => -1*3600, # West Africa "at" => -2*3600, # Azores # For completeness. BST is also British Summer, and GST is also Guam Standard. # "bst" => -3*3600, # Brazil Standard # "gst" => -3*3600, # Greenland Standard # "nft" => -3*3600-1800,# Newfoundland # "nst" => -3*3600-1800,# Newfoundland Standard "ast" => -4*3600, # Atlantic Standard "est" => -5*3600, # Eastern Standard "cst" => -6*3600, # Central Standard "mst" => -7*3600, # Mountain Standard "pst" => -8*3600, # Pacific Standard "yst" => -9*3600, # Yukon Standard "hst" => -10*3600, # Hawaii Standard "cat" => -10*3600, # Central Alaska "ahst" => -10*3600, # Alaska-Hawaii Standard "nt" => -11*3600, # Nome "idlw" => -12*3600, # International Date Line West "cet" => +1*3600, # Central European "met" => +1*3600, # Middle European "mewt" => +1*3600, # Middle European Winter "swt" => +1*3600, # Swedish Winter "fwt" => +1*3600, # French Winter "eet" => +2*3600, # Eastern Europe, USSR Zone 1 "bt" => +3*3600, # Baghdad, USSR Zone 2 # "it" => +3*3600+1800,# Iran "zp4" => +4*3600, # USSR Zone 3 "zp5" => +5*3600, # USSR Zone 4 # "ist" => +5*3600+1800,# Indian Standard "zp6" => +6*3600, # USSR Zone 5 # For completeness. NST is also Newfoundland Stanard, and SST is also Swedish Summer. # "nst" => +6*3600+1800,# North Sumatra # "sst" => +7*3600, # South Sumatra, USSR Zone 6 "wast" => +7*3600, # West Australian Standard # "jt" => +7*3600+1800,# Java (3pm in Cronusland!) "cct" => +8*3600, # China Coast, USSR Zone 7 "jst" => +9*3600, # Japan Standard, USSR Zone 8 # "cast" => +9*3600+1800,# Central Australian Standard "east" => +10*3600, # Eastern Australian Standard "gst" => +10*3600, # Guam Standard, USSR Zone 9 "nzt" => +12*3600, # New Zealand "nzst" => +12*3600, # New Zealand Standard "idle" => +12*3600, # International Date Line East ); %zoneOff = reverse(%Zone); %dstZoneOff = reverse(%dstZone); # Preferences $zoneOff{0} = 'gmt'; $dstZoneOff{3600} = 'bst'; } sub tz_offset (;$$) { my ($zone, $time) = @_; return &tz_local_offset() unless($zone); $time = time() unless $time; my(@l) = localtime($time); my $dst = $l[8]; $zone = lc $zone; if($zone =~ /^(([\-\+])\d\d?)(\d\d)$/) { my $v = $2 . $3; return $1 * 3600 + $v * 60; } elsif (exists $dstZone{$zone} && ($dst || !exists $Zone{$zone})) { return $dstZone{$zone}; } elsif(exists $Zone{$zone}) { return $Zone{$zone}; } undef; } sub tz_name (;$$) { my ($off, $dst) = @_; $off = tz_offset() unless(defined $off); $dst = (localtime(time))[8] unless(defined $dst); if (exists $dstZoneOff{$off} && ($dst || !exists $zoneOff{$off})) { return $dstZoneOff{$off}; } elsif (exists $zoneOff{$off}) { return $zoneOff{$off}; } sprintf("%+05d", int($off / 60) * 100 + $off % 60); } 1; ================================================ FILE: test/t/upsync.t ================================================ use warnings; use strict; use Test::More; BEGIN { use FindBin; chdir($FindBin::Bin); } use lib 'lib'; use File::Path; use Test::Nginx; my $NGINX = defined $ENV{TEST_NGINX_BINARY} ? $ENV{TEST_NGINX_BINARY} : '../../nginx/objs/nginx'; my $t = Test::Nginx->new()->plan(40); sub mhttp_get($;$;$;%) { my ($url, $host, $port, %extra) = @_; return mhttp(< }; close $fd; } else { $content = 'The file could not be opened.'; } return $content; } sub mrun($;$) { my ($self, $conf) = @_; my $testdir = $self->{_testdir}; if (defined $conf) { my $c = `cat $conf`; $self->write_file_expand('nginx.conf', $c); } my $pid = fork(); die "Unable to fork(): $!\n" unless defined $pid; if ($pid == 0) { my @globals = $self->{_test_globals} ? () : ('-g', "pid $testdir/nginx.pid; " . "error_log $testdir/error.log debug;"); exec($NGINX, '-c', "$testdir/nginx.conf", '-p', "$testdir", @globals) or die "Unable to exec(): $!\n"; } # wait for nginx to start $self->waitforfile("$testdir/nginx.pid") or die "Can't start nginx"; $self->{_started} = 1; return $self; } ############################################################################### select STDERR; $| = 1; select STDOUT; $| = 1; warn "your test dir is ".$t->testdir(); $t->write_file_expand('nginx.conf', <<'EOF'); %%TEST_GLOBALS%% daemon off; worker_processes 1; events { accept_mutex off; } stream { upstream test { upsync 127.0.0.1:8500/v1/kv/upstreams/test upsync_interval=50ms upsync_timeout=6m upsync_type=consul; upsync_dump_path /tmp/servers_test.conf; server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; } upstream backend { server 127.0.0.1:8090 weight=10 max_fails=3 fail_timeout=10; } server { listen 8080; upstream_show; } server { listen 8081; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass test; } server { listen 8082; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass backend; } } EOF mrun($t); ############################################################################### my $rep; my $dump; like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8089', 8500), qr/true/m, '2015-12-27 18:21:37'); like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8088', 8500), qr/true/m, '2015-12-27 18:20:37'); sleep(1); like(mhttp_delete('/v1/kv/upstreams/test?recurse', 8500), qr/true/m, '2015-12-27 18:22:37'); like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8089', '', 8500), qr/true/m, '2015-12-27 17:50:35'); $rep = qr/ Upstream name: test;Backend server counts: 1 server 127.0.0.1:8089 weight=1 max_fails=2 fail_timeout=10; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2016-04-17 17:43:30'); ############################################################################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":10,"max_fails":3,"fail_timeout":10}', 8500), qr/true/m, '2015-12-27 17:42:35'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8089 weight=1 max_fails=2 fail_timeout=10; server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 18:17:53'); ######################### like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8089', 8500), qr/true/m, '2015-12-27 18:24:37'); $rep = qr/ Upstream name: test;Backend server counts: 1 server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2016-04-17 18:30:40'); ######################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8089', '{"weight":20,"max_fails":0,"fail_timeout":30}', 8500), qr/true/m, '2015-12-27 18:31:39'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2016-04-17 18:32:53'); ######################## like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8088', 8500), qr/true/m, '2015-12-27 18:33:37'); $rep = qr/ Upstream name: test;Backend server counts: 1 server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 18:34:40'); ####################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":20,"max_fails":0,"fail_timeout":30}', 8500), qr/true/m, '2015-12-27 18:35:35'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; server 127.0.0.1:8088 weight=20 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 18:34:40'); ####################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":40}', 8500), qr/true/m, '2015-12-27 18:42:35'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; server 127.0.0.1:8088 weight=40 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 18:43:40'); ####################### like(mhttp_delete('/v1/kv/upstreams/test?recurse', 8500), qr/true/m, '2015-12-27 18:22:33'); sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2016-04-14 18:44:51'); ####################### $dump = qr/server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30s; server 127.0.0.1:8088 weight=40 max_fails=0 fail_timeout=30s; /m; like(get_dump_content('/tmp/servers_test.conf'), $dump, '2015-12-27 18:51:35'); ####################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":40,"max_fails":3,"fail_timeout":20, "down":1}', 8500), qr/true/m, '2016-03-15 18:35:35'); sleep(1); $dump = qr/server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30s; server 127.0.0.1:8088 weight=40 max_fails=3 fail_timeout=20s down; /m; like(get_dump_content('/tmp/servers_test.conf'), $dump, '2016-03-15 18:51:35'); ####################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":40,"max_fails":0,"fail_timeout":30, "down":0}', 8500), qr/true/m, '2016-03-15 17:35:35'); sleep(1); $dump = qr/server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30s; server 127.0.0.1:8088 weight=40 max_fails=0 fail_timeout=30s; /m; like(get_dump_content('/tmp/servers_test.conf'), $dump, '2016-03-15 17:51:35'); $t->stop(); ############################################################################## $t->write_file_expand('nginx.conf', <<'EOF'); %%TEST_GLOBALS%% daemon off; worker_processes auto; events { accept_mutex off; } http { upstream test { upsync 127.0.0.1:8500/v1/kv/upstreams/test upsync_interval=50ms upsync_timeout=6m upsync_type=consul; upsync_dump_path /tmp/servers_test.conf; server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; } upstream backend { server 127.0.0.1:8090 weight=10 max_fails=3 fail_timeout=10; } server { listen 8080; upstream_show; } server { listen 8081; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass test; } server { listen 8082; proxy_connect_timeout 1s; proxy_timeout 3s; proxy_pass backend; } } EOF mrun($t); ############################################################################### like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8088', 8500), qr/true/m, '2015-12-27 19:20:37'); like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8089', 8500), qr/true/m, '2015-12-27 19:21:37'); sleep(1); like(mhttp_delete('/v1/kv/upstreams/test?recurse', 8500), qr/true/m, '2015-12-27 19:49:37'); like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8089', '', 8500), qr/true/m, '2015-12-27 19:25:35'); $rep = qr/ Upstream name: test;Backend server counts: 1 server 127.0.0.1:8089 weight=1 max_fails=2 fail_timeout=10; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 19:20:30'); ############################################################################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":10,"max_fails":3,"fail_timeout":10}', 8500), qr/true/m, '2015-12-27 17:50:35'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8089 weight=1 max_fails=2 fail_timeout=10; server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 19:26:53'); ########################### like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8089', 8500), qr/true/m, '2015-12-27 19:28:37'); $rep = qr/ Upstream name: test;Backend server counts: 1 server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 19:30:40'); ########################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8089', '{"weight":20,"max_fails":0,"fail_timeout":30}', 8500), qr/true/m, '2015-12-27 19:31:39'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8088 weight=10 max_fails=3 fail_timeout=10; server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; /m; like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 19:32:53'); ########################## like(mhttp_delete('/v1/kv/upstreams/test/127.0.0.1:8088', 8500), qr/true/m, '2015-12-27 19:35:37'); $rep = qr/ Upstream name: test;Backend server counts: 1 server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2016-04-17 19:37:40'); ########################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":20,"max_fails":0,"fail_timeout":30}', 8500), qr/true/m, '2015-12-27 19:39:35'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; server 127.0.0.1:8088 weight=20 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2016-04-17 19:47:40'); ########################### like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":40}', 8500), qr/true/m, '2015-12-27 19:48:35'); $rep = qr/ Upstream name: test;Backend server counts: 2 server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30; server 127.0.0.1:8088 weight=40 max_fails=0 fail_timeout=30; /m; sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 19:49:40'); ########################### like(mhttp_delete('/v1/kv/upstreams/test?recurse', 8500), qr/true/m, '2015-12-27 19:50:37'); sleep(1); like(mhttp_get('/upstream_list', 'localhost', 8080), $rep, '2015-04-14 19:51:40'); ########################## $dump = qr/server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30s; server 127.0.0.1:8088 weight=40 max_fails=0 fail_timeout=30s; /m; like(get_dump_content('/tmp/servers_test.conf'), $dump, '2015-12-27 19:53:35'); ########################## like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":40,"max_fails":3,"fail_timeout":20, "down":1}', 8500), qr/true/m, '2016-03-15 19:39:35'); sleep(1); $dump = qr/server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30s; server 127.0.0.1:8088 weight=40 max_fails=3 fail_timeout=20s down; /m; like(get_dump_content('/tmp/servers_test.conf'), $dump, '2016-03-15 19:53:35'); ########################## like(mhttp_put('/v1/kv/upstreams/test/127.0.0.1:8088', '{"weight":40,"max_fails":3,"fail_timeout":20, "down":0}', 8500), qr/true/m, '2016-03-15 20:39:35'); sleep(1); $dump = qr/server 127.0.0.1:8089 weight=20 max_fails=0 fail_timeout=30s; server 127.0.0.1:8088 weight=40 max_fails=3 fail_timeout=20s; /m; like(get_dump_content('/tmp/servers_test.conf'), $dump, '2016-03-15 20:53:35'); $t->stop(); ############################################################################### sub mhttp($;$;%) { my ($request, $port, %extra) = @_; my $reply; eval { local $SIG{ALRM} = sub { die "timeout\n" }; local $SIG{PIPE} = sub { die "sigpipe\n" }; alarm(2); my $s = IO::Socket::INET->new( Proto => "tcp", PeerAddr => "127.0.0.1:$port" ); log_out($request); $s->print($request); local $/; select undef, undef, undef, $extra{sleep} if $extra{sleep}; return '' if $extra{aborted}; $reply = $s->getline(); alarm(0); }; alarm(0); if ($@) { log_in("died: $@"); return undef; } log_in($reply); return $reply; } ================================================ FILE: test/test.sh ================================================ #!/bin/sh TEST_NGINX_USE_HUP=1 TEST_NGINX_BINARY=/usr/local/nginx/sbin/nginx prove -r t