Repository: trimstray/nginx-admins-handbook Branch: master Commit: 5aea1f81a2fd Files: 83 Total size: 1.0 MB Directory structure: gitextract_b_tfkvj5/ ├── .github/ │ ├── CODE_OF_CONDUCT.md │ ├── CONTRIBUTING.md │ └── FUNDING.yml ├── LICENSE.md ├── README.md ├── doc/ │ ├── EXAMPLES.md │ ├── HELPERS.md │ ├── HTTP_BASICS.md │ ├── NGINX_BASICS.md │ ├── RULES.md │ └── SSL_TLS_BASICS.md ├── lib/ │ ├── nginx/ │ │ ├── dhparam_4096-with-ds.pem │ │ ├── dhparam_4096.pem │ │ ├── html/ │ │ │ ├── 50x.html │ │ │ └── index.html │ │ ├── master/ │ │ │ ├── _acls/ │ │ │ │ ├── external.geo.acl │ │ │ │ ├── external.map.acl │ │ │ │ ├── internal.geo.acl │ │ │ │ └── internal.map.acl │ │ │ ├── _basic/ │ │ │ │ ├── logging.conf │ │ │ │ ├── main.conf │ │ │ │ ├── proxy-params.conf │ │ │ │ ├── rate-limiting.conf │ │ │ │ └── redirects-map.conf │ │ │ ├── _listen/ │ │ │ │ ├── 192.168.250.2/ │ │ │ │ │ ├── http.conf │ │ │ │ │ └── https.conf │ │ │ │ └── localhost/ │ │ │ │ ├── http.conf │ │ │ │ └── https.conf │ │ │ ├── _server/ │ │ │ │ ├── _helpers/ │ │ │ │ │ └── global.conf │ │ │ │ ├── blkcipher.info/ │ │ │ │ │ ├── acls/ │ │ │ │ │ │ └── demo.conf │ │ │ │ │ ├── backends.conf │ │ │ │ │ ├── certs/ │ │ │ │ │ │ ├── blkcipher.info.conf │ │ │ │ │ │ ├── blkcipher.info.key │ │ │ │ │ │ └── nginx_blkcipher.info_bundle.crt │ │ │ │ │ ├── credentials/ │ │ │ │ │ │ └── demo.txt │ │ │ │ │ └── servers.conf │ │ │ │ ├── defaults/ │ │ │ │ │ ├── backends.conf │ │ │ │ │ ├── certs/ │ │ │ │ │ │ ├── defaults.conf │ │ │ │ │ │ ├── defaults.key │ │ │ │ │ │ └── nginx_defaults_bundle.crt │ │ │ │ │ └── servers.conf │ │ │ │ └── localhost/ │ │ │ │ ├── backends.conf │ │ │ │ ├── certs/ │ │ │ │ │ ├── localhost.conf │ │ │ │ │ ├── localhost.key │ │ │ │ │ └── nginx_localhost_bundle.crt │ │ │ │ └── servers.conf │ │ │ └── _static/ │ │ │ └── errors.conf │ │ ├── mime.types │ │ ├── modules.conf │ │ ├── nginx.conf │ │ ├── snippets/ │ │ │ ├── gdb/ │ │ │ │ └── nginx-config.gdb │ │ │ ├── http-error-pages/ │ │ │ │ ├── README.md │ │ │ │ ├── httpgen │ │ │ │ ├── sites/ │ │ │ │ │ └── .gitkeep │ │ │ │ ├── src/ │ │ │ │ │ ├── 4xx.json │ │ │ │ │ ├── 5xx.json │ │ │ │ │ ├── index.html │ │ │ │ │ ├── main.css │ │ │ │ │ └── other.json │ │ │ │ └── templates/ │ │ │ │ ├── _template.html │ │ │ │ └── nginx/ │ │ │ │ └── errors.conf │ │ │ ├── logrotate.d/ │ │ │ │ ├── nginx.bsd │ │ │ │ └── nginx.linux │ │ │ ├── scripts/ │ │ │ │ ├── git-status.sh │ │ │ │ └── show-memory.sh │ │ │ ├── server-name-parser/ │ │ │ │ ├── check-server-name.sh │ │ │ │ └── server-name-parser.py │ │ │ ├── skel/ │ │ │ │ ├── .bashrc-bsd │ │ │ │ ├── .bashrc-linux │ │ │ │ ├── .cshrc-bsd │ │ │ │ ├── .exrc │ │ │ │ ├── .goprofile │ │ │ │ ├── .profile-bsd │ │ │ │ ├── .vimrc │ │ │ │ ├── global-aliases.bash │ │ │ │ └── global-aliases.csh │ │ │ └── systemd/ │ │ │ └── nginx.service │ │ └── win-utf │ ├── ngx_installer.conf │ ├── ngx_installer.sh │ └── ngx_installer.vars └── static/ └── img/ └── cheatsheets/ ├── nginx-hardening-cheatsheet-tls12-100p.xcf └── nginx-hardening-cheatsheet-tls13.xcf ================================================ FILE CONTENTS ================================================ ================================================ FILE: .github/CODE_OF_CONDUCT.md ================================================ # Contributor Covenant Code of Conduct ## Our Pledge In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. ## Our Standards Examples of behavior that contributes to creating a positive environment include: * Using welcoming and inclusive language * Being respectful of differing viewpoints and experiences * Gracefully accepting constructive criticism * Focusing on what is best for the community * Showing empathy towards other community members Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances * Trolling, insulting/derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or electronic address, without explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Our Responsibilities Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. ## Scope This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at trimstray@gmail.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] [homepage]: http://contributor-covenant.org [version]: http://contributor-covenant.org/version/1/4/ ================================================ FILE: .github/CONTRIBUTING.md ================================================ # Contributing > _A real community, however, exists only when its members interact in a meaningful way that deepens their understanding of each other and leads to learning._ If you would like to support this project, have an interesting idea how to improve the operation of this tool, or if you found some errors - fork this, add your fixes, and add a pull request of your branch to the **master branch**. ## Using the issue tracker The [issue tracker](https://github.com/trimstray/nginx-admins-handbook/issues) is the preferred channel for bug reports, features requests and submitting pull requests, but please respect the following restrictions: * Please **do not** use the issue tracker for personal support requests (use [Stack Overflow](https://stackoverflow.com) or IRC) * Please **do not** derail or troll issues. Keep the discussion on topic and respect the opinions of others ## Signature of commit Moving forward all commits to this project must include a "signed-off-by" line indicating the name and email address of the contributor signing off on the change. To enable signatures add the following lines to `.git/hooks/prepare-commit-msg` : ``` SOB=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/- signed-off-by: \1/p') grep -qs "^$SOB" "$1" || echo "$SOB" >> "$1" ``` ## Pull requests When creating a pull request, please heed the following: - Base your code on the latest master branch to avoid manual merges - Code review may ensue in order to help shape your proposal - Explain the problem and your proposed solution ================================================ FILE: .github/FUNDING.yml ================================================ open_collective: trimstray github: trimstray ================================================ FILE: LICENSE.md ================================================ MIT License Copyright (c) 2017 trimstray Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================

Nginx Admin's Handbook

My notes on NGINX administration basics, tips & tricks, caveats, and gotchas.

Meme


Hi-diddle-diddle, he played on his
fiddle and danced with lady pigs.
Number three said, "Nicks on tricks!
I'll build my house with EN-jin-EKS!".
The Three Little Pigs: Who's Afraid of the Big Bad Wolf?


Pull Requests MIT License


**** # Table of Contents - **[Introduction](#introduction)** * [Prologue](#prologue) * [Why I created this handbook](#why-i-created-this-handbook) * [Who this handbook is for](#who-this-handbook-is-for) * [Before you start](#before-you-start) * [Contributing & Support](#contributing--support) * [RSS Feed & Updates](#rss-feed--updates) * [Checklist to rule them all](#checklist-to-rule-them-all) - **[Bonus Stuff](#bonus-stuff)** * [Configuration reports](#configuration-reports) * [SSL Labs](#ssl-labs) * [Mozilla Observatory](#mozilla-observatory) * [Printable hardening cheatsheets](#printable-hardening-cheatsheets) * [Fully automatic installation](#fully-automatic-installation) * [Static error pages generator](#static-error-pages-generator) * [Server names parser](#server-names-parser) - **[Books](#books)** * [Nginx Essentials](#nginx-essentials) * [Nginx Cookbook](#nginx-cookbook) * [Nginx HTTP Server](#nginx-http-server) * [Nginx High Performance](#nginx-high-performance) * [Mastering Nginx](#mastering-nginx) * [ModSecurity 3.0 and NGINX: Quick Start Guide](#modsecurity-30-and-nginx-quick-start-guide) * [Cisco ACE to NGINX: Migration Guide](#cisco-ace-to-nginx-migration-guide) - **[External Resources](#external-resources)** * [Nginx official](#nginx-official) * [Nginx distributions](#nginx-distributions) * [Comparison reviews](#comparison-reviews) * [Cheatsheets & References](#cheatsheets--references) * [Performance & Hardening](#performance--hardening) * [Presentations & Videos](#presentations--videos) * [Playgrounds](#playgrounds) * [Config generators](#config-generators) * [Config parsers](#config-parsers) * [Config managers](#config-managers) * [Static analyzers](#static-analyzers) * [Log analyzers](#log-analyzers) * [Performance analyzers](#performance-analyzers) * [Builder tools](#builder-tools) * [Benchmarking tools](#benchmarking-tools) * [Debugging tools](#debugging-tools) * [Security & Web testing tools](#security--web-testing-tools) * [Development](#development) * [Online & Web tools](#online--web-tools) * [Other stuff](#other-stuff) - **[What's next?](#whats-next)**
Other chapters
- **[HTTP Basics](doc/HTTP_BASICS.md#http-basics)** * [Introduction](doc/HTTP_BASICS.md#introduction-1) * [Features and architecture](doc/HTTP_BASICS.md#features-and-architecture) * [HTTP/2](doc/HTTP_BASICS.md#http2) * [How to debug HTTP/2?](doc/HTTP_BASICS.md#how-to-debug-http2) * [HTTP/3](doc/HTTP_BASICS.md#http3) * [URI vs URL](doc/HTTP_BASICS.md#uri-vs-url) * [Connection vs request](doc/HTTP_BASICS.md#connection-vs-request) * [HTTP Headers](doc/HTTP_BASICS.md#http-headers) * [Header compression](#header-compression) * [HTTP Methods](doc/HTTP_BASICS.md#http-methods) * [Request](doc/HTTP_BASICS.md#request) * [Request line](doc/HTTP_BASICS.md#request-line) * [Methods](doc/HTTP_BASICS.md#methods) * [Request URI](doc/HTTP_BASICS.md#request-uri) * [HTTP version](doc/HTTP_BASICS.md#http-version) * [Request header fields](doc/HTTP_BASICS.md#request-header-fields) * [Message body](doc/HTTP_BASICS.md#message-body) * [Generate requests](doc/HTTP_BASICS.md#generate-requests) * [Response](doc/HTTP_BASICS.md#response) * [Status line](doc/HTTP_BASICS.md#status-line) * [HTTP version](doc/HTTP_BASICS.md#http-version-1) * [Status codes and reason phrase](doc/HTTP_BASICS.md#status-codes-and-reason-phrase) * [Response header fields](doc/HTTP_BASICS.md#response-header-fields) * [Message body](doc/HTTP_BASICS.md#message-body-1) * [HTTP client](doc/HTTP_BASICS.md#http-client) * [IP address shortcuts](doc/HTTP_BASICS.md#ip-address-shortcuts) * [Back-End web architecture](doc/HTTP_BASICS.md#back-end-web-architecture) * [Useful video resources](doc/HTTP_BASICS.md#useful-video-resources) - **[SSL/TLS Basics](doc/SSL_TLS_BASICS.md#ssltls-basics)** * [Introduction](doc/SSL_TLS_BASICS.md#introduction-2) * [TLS versions](doc/SSL_TLS_BASICS.md#tls-versions) * [TLS handshake](doc/SSL_TLS_BASICS.md#tls-handshake) * [In which layer is TLS situated within the TCP/IP stack?](doc/SSL_TLS_BASICS.md#in-which-layer-is-tls-situated-within-the-tcpip-stack) * [RSA and ECC keys/certificates](doc/SSL_TLS_BASICS.md#rsa-and-ecc-keyscertificates) * [Cipher suites](doc/SSL_TLS_BASICS.md#cipher-suites) * [Authenticated encryption (AEAD) cipher suites](doc/SSL_TLS_BASICS.md#authenticated-encryption-aead-cipher-suites) * [Why cipher suites are important?](doc/SSL_TLS_BASICS.md#why-cipher-suites-are-important) * [What does insecure, weak, secure and recommended mean?](doc/SSL_TLS_BASICS.md#what-does-insecure-weak-secure-and-recommended-mean) * [NGINX and TLS 1.3 Cipher Suites](doc/SSL_TLS_BASICS.md#nginx-and-tls-13-cipher-suites) * [Diffie-Hellman key exchange](doc/SSL_TLS_BASICS.md#diffie-hellman-key-exchange) * [What exactly is the purpose of these DH Parameters?](doc/SSL_TLS_BASICS.md#what-exactly-is-the-purpose-of-these-dh-parameters) * [Certificates](doc/SSL_TLS_BASICS.md#certificates) * [Chain of Trust](doc/SSL_TLS_BASICS.md#chain-of-trust) * [What is the main purpose of the Intermediate CA?](doc/SSL_TLS_BASICS.md#what-is-the-main-purpose-of-the-intermediate-ca) * [Single-domain](doc/SSL_TLS_BASICS.md#single-domain) * [Multi-domain](doc/SSL_TLS_BASICS.md#multi-domain) * [Wildcard](doc/SSL_TLS_BASICS.md#wildcard) * [Wildcard SSL doesn't handle root domain?](doc/SSL_TLS_BASICS.md#wildcard-ssl-doesnt-handle-root-domain) * [HTTPS with self-signed certificate vs HTTP](doc/SSL_TLS_BASICS.md#https-with-self-signed-certificate-vs-http) * [TLS Server Name Indication](doc/SSL_TLS_BASICS.md#tls-server-name-indication) * [Verify your SSL, TLS & Ciphers implementation](doc/SSL_TLS_BASICS.md#verify-your-ssl-tls--ciphers-implementation) * [Useful video resources](doc/SSL_TLS_BASICS.md#useful-video-resources) - **[NGINX Basics](doc/NGINX_BASICS.md#nginx-basics)** * [Directories and files](doc/NGINX_BASICS.md#directories-and-files) * [Commands](doc/NGINX_BASICS.md#commands) * [Processes](doc/NGINX_BASICS.md#processes) * [CPU pinning](doc/NGINX_BASICS.md#cpu-pinning) * [Shutdown of worker processes](doc/NGINX_BASICS.md#shutdown-of-worker-processes) * [Configuration syntax](doc/NGINX_BASICS.md#configuration-syntax) * [Comments](doc/NGINX_BASICS.md#comments) * [End of lines](doc/NGINX_BASICS.md#end-of-lines) * [Variables, Strings, and Quotes](doc/NGINX_BASICS.md#variables-strings-and-quotes) * [Directives, Blocks, and Contexts](doc/NGINX_BASICS.md#directives-blocks-and-contexts) * [External files](doc/NGINX_BASICS.md#external-files) * [Measurement units](doc/NGINX_BASICS.md#measurement-units) * [Regular expressions with PCRE](doc/NGINX_BASICS.md#regular-expressions-with-pcre) * [Enable syntax highlighting](doc/NGINX_BASICS.md#enable-syntax-highlighting) * [Connection processing](doc/NGINX_BASICS.md#connection-processing) * [Event-Driven architecture](doc/NGINX_BASICS.md#event-driven-architecture) * [Multiple processes](doc/NGINX_BASICS.md#multiple-processes) * [Simultaneous connections](doc/NGINX_BASICS.md#simultaneous-connections) * [HTTP Keep-Alive connections](doc/NGINX_BASICS.md#http-keep-alive-connections) * [sendfile, tcp_nodelay, and tcp_nopush](doc/NGINX_BASICS.md#sendfile-tcp_nodelay-and-tcp_nopush) * [Request processing stages](doc/NGINX_BASICS.md#request-processing-stages) * [Server blocks logic](doc/NGINX_BASICS.md#server-blocks-logic) * [Handle incoming connections](doc/NGINX_BASICS.md#handle-incoming-connections) * [Matching location](doc/NGINX_BASICS.md#matching-location) * [rewrite vs return](doc/NGINX_BASICS.md#rewrite-vs-return) * [URL redirections](doc/NGINX_BASICS.md#url-redirections) * [try_files directive](doc/NGINX_BASICS.md#try_files-directive) * [if, break, and set](doc/NGINX_BASICS.md#if-break-and-set) * [root vs alias](doc/NGINX_BASICS.md#root-vs-alias) * [internal directive](doc/NGINX_BASICS.md#internal-directive) * [External and internal redirects](doc/NGINX_BASICS.md#external-and-internal-redirects) * [allow and deny](doc/NGINX_BASICS.md#allow-and-deny) * [uri vs request_uri](doc/NGINX_BASICS.md#uri-vs-request_uri) * [Compression and decompression](doc/NGINX_BASICS.md#compression-and-decompression) * [What is the best NGINX compression gzip level?](doc/NGINX_BASICS.md#what-is-the-best-nginx-compression-gzip-level) * [Hash tables](doc/NGINX_BASICS.md#hash-tables) * [Server names hash table](doc/NGINX_BASICS.md#server-names-hash-table) * [Log files](doc/NGINX_BASICS.md#log-files) * [Conditional logging](doc/NGINX_BASICS.md#conditional-logging) * [Manually log rotation](doc/NGINX_BASICS.md#manually-log-rotation) * [Error log severity levels](doc/NGINX_BASICS.md#error-log-severity-levels) * [How to log the start time of a request?](doc/NGINX_BASICS.md#how-to-log-the-start-time-of-a-request) * [How to log the HTTP request body?](doc/NGINX_BASICS.md#how-to-log-the-http-request-body) * [NGINX upstream variables returns 2 values](doc/NGINX_BASICS.md#nginx-upstream-variables-returns-2-values) * [Reverse proxy](doc/NGINX_BASICS.md#reverse-proxy) * [Passing requests](doc/NGINX_BASICS.md#passing-requests) * [Trailing slashes](doc/NGINX_BASICS.md#trailing-slashes) * [Passing headers to the backend](doc/NGINX_BASICS.md#passing-headers-to-the-backend) * [Importance of the Host header](doc/NGINX_BASICS.md#importance-of-the-host-header) * [Redirects and X-Forwarded-Proto](doc/NGINX_BASICS.md#redirects-and-x-forwarded-proto) * [A warning about the X-Forwarded-For](doc/NGINX_BASICS.md#a-warning-about-the-x-forwarded-for) * [Improve extensibility with Forwarded](doc/NGINX_BASICS.md#improve-extensibility-with-forwarded) * [Response headers](doc/NGINX_BASICS.md#response-headers) * [Load balancing algorithms](doc/NGINX_BASICS.md#load-balancing-algorithms) * [Backend parameters](doc/NGINX_BASICS.md#backend-parameters) * [Upstream servers with SSL](doc/NGINX_BASICS.md#upstream-servers-with-ssl) * [Round Robin](doc/NGINX_BASICS.md#round-robin) * [Weighted Round Robin](doc/NGINX_BASICS.md#weighted-round-robin) * [Least Connections](doc/NGINX_BASICS.md#least-connections) * [Weighted Least Connections](doc/NGINX_BASICS.md#weighted-least-connections) * [IP Hash](doc/NGINX_BASICS.md#ip-hash) * [Generic Hash](doc/NGINX_BASICS.md#generic-hash) * [Other methods](doc/NGINX_BASICS.md#other-methods) * [Rate limiting](doc/NGINX_BASICS.md#rate-limiting) * [Variables](doc/NGINX_BASICS.md#variables) * [Directives, keys, and zones](doc/NGINX_BASICS.md#directives-keys-and-zones) * [Burst and nodelay parameters](doc/NGINX_BASICS.md#burst-and-nodelay-parameters) * [NAXSI Web Application Firewall](doc/NGINX_BASICS.md#naxsi-web-application-firewall) * [OWASP ModSecurity Core Rule Set (CRS)](doc/NGINX_BASICS.md#owasp-modsecurity-core-rule-set-crs) * [Core modules](doc/NGINX_BASICS.md#core-modules) * [ngx_http_geo_module](doc/NGINX_BASICS.md#ngx_http_geo_module) * [3rd party modules](doc/NGINX_BASICS.md#3rd-party-modules) * [ngx_set_misc](doc/NGINX_BASICS.md#ngx_set_misc) * [ngx_http_geoip_module](doc/NGINX_BASICS.md#ngx_http_geoip_module) - **[Helpers](doc/HELPERS.md#helpers)** * [Installing from prebuilt packages](doc/HELPERS.md#installing-from-prebuilt-packages) * [RHEL7 or CentOS 7](doc/HELPERS.md#rhel7-or-centos-7) * [Debian or Ubuntu](doc/HELPERS.md#debian-or-ubuntu) * [FreeBSD](doc/HELPERS.md#freebsd) * [Installing from source](doc/HELPERS.md#installing-from-source) * [Automatic installation on RHEL/Debian/BSD](doc/HELPERS.md#automatic-installation-on-rheldebianbsd) * [Nginx package](doc/HELPERS.md#nginx-package) * [Dependencies](doc/HELPERS.md#dependencies) * [Patches](doc/HELPERS.md#patches) * [3rd party modules](doc/HELPERS.md#3rd-party-modules) * [Configure options](doc/HELPERS.md#cconfigure-options) * [Compiler and linker](doc/HELPERS.md#compiler-and-linker) * [Debugging Symbols](doc/HELPERS.md#debugging-symbols) * [SystemTap](doc/HELPERS.md#systemtap) * [stapxx](doc/HELPERS.md#stapxx) * [Installation Nginx on CentOS 7](doc/HELPERS.md#installation-nginx-on-centos-7) * [Pre installation tasks](doc/HELPERS.md#pre-installation-tasks) * [Dependencies](doc/HELPERS.md#dependencies) * [Get Nginx sources](doc/HELPERS.md#get-nginx-sources) * [Download 3rd party modules](doc/HELPERS.md#download-3rd-party-modules) * [Build Nginx](doc/HELPERS.md#build-nginx) * [Post installation tasks](doc/HELPERS.md#post-installation-tasks) * [Installation OpenResty on CentOS 7](doc/HELPERS.md#installation-openresty-on-centos-7) * [Installation Tengine on Ubuntu 18.04](doc/HELPERS.md#installation-tengine-on-ubuntu-1804) * [Installation Nginx on FreeBSD 11.3](doc/HELPERS.md#installation-nginx-on-freebsd-113) * [Installation Nginx on FreeBSD 11.3 (from ports)](doc/HELPERS.md#installation-nginx-on-freebsd-113-from-ports) * [Analyse configuration](doc/HELPERS.md#analyse-configuration) * [Monitoring](doc/HELPERS.md#monitoring) * [GoAccess](doc/HELPERS.md#goaccess) * [Build and install](doc/HELPERS.md#build-and-install) * [Analyse log file and enable all recorded statistics](doc/HELPERS.md#analyse-log-file-and-enable-all-recorded-statistics) * [Analyse compressed log file](doc/HELPERS.md#analyse-compressed-log-file) * [Analyse log file remotely](doc/HELPERS.md#analyse-log-file-remotely) * [Analyse log file and generate html report](doc/HELPERS.md#analyse-log-file-and-generate-html-report) * [Ngxtop](doc/HELPERS.md#ngxtop) * [Analyse log file](doc/HELPERS.md#analyse-log-file) * [Analyse log file and print requests with 4xx and 5xx](doc/HELPERS.md#analyse-log-file-and-print-requests-with-4xx-and-5xx) * [Analyse log file remotely](doc/HELPERS.md#analyse-log-file-remotely-1) * [Testing](doc/HELPERS.md#testing) * [Build OpenSSL 1.0.2-chacha version](doc/HELPERS.md#build-openssl-102-chacha-version) * [Send request and show response headers](doc/HELPERS.md#send-request-and-show-response-headers) * [Send request with http method, user-agent, follow redirects and show response headers](doc/HELPERS.md#send-request-with-http-method-user-agent-follow-redirects-and-show-response-headers) * [Send multiple requests](doc/HELPERS.md#send-multiple-requests) * [Testing SSL connection](doc/HELPERS.md#testing-ssl-connection) * [Testing SSL connection (debug mode)](doc/HELPERS.md#testing-ssl-connection-debug-mode) * [Testing SSL connection with SNI support](doc/HELPERS.md#testing-ssl-connection-with-sni-support) * [Testing SSL connection with specific SSL version](doc/HELPERS.md#testing-ssl-connection-with-specific-ssl-version) * [Testing SSL connection with specific cipher](doc/HELPERS.md#testing-ssl-connection-with-specific-cipher) * [Testing OCSP Stapling](doc/HELPERS.md#testing-ocsp-stapling) * [Verify 0-RTT](doc/HELPERS.md#verify-0-rtt) * [Testing SCSV](doc/HELPERS.md#testing-scsv) * [Load testing with ApacheBench (ab)](doc/HELPERS.md#load-testing-with-apachebench-ab) * [Standard test](doc/HELPERS.md#standard-test) * [Test with Keep-Alive header](doc/HELPERS.md#test-with-keep-alive-header) * [Load testing with wrk2](doc/HELPERS.md#load-testing-with-wrk2) * [Standard scenarios](doc/HELPERS.md#standard-scenarios) * [POST call (with Lua)](doc/HELPERS.md#post-call-with-lua) * [Random paths (with Lua)](doc/HELPERS.md#random-paths-with-lua) * [Multiple paths (with Lua)](doc/HELPERS.md#multiple-paths-with-lua) * [Random server address to each thread (with Lua)](doc/HELPERS.md#random-server-address-to-each-thread-with-lua) * [Multiple json requests (with Lua)](doc/HELPERS.md#multiple-json-requests-with-lua) * [Debug mode (with Lua)](doc/HELPERS.md#debug-mode-with-lua) * [Analyse data pass to and from the threads](doc/HELPERS.md#analyse-data-pass-to-and-from-the-threads) * [Parsing wrk result and generate report](doc/HELPERS.md#parsing-wrk-result-and-generate-report) * [Load testing with locust](doc/HELPERS.md#load-testing-with-locust) * [Multiple paths](doc/HELPERS.md#multiple-paths) * [Multiple paths with different user sessions](doc/HELPERS.md#multiple-paths-with-different-user-sessions) * [TCP SYN flood Denial of Service attack](doc/HELPERS.md#tcp-syn-flood-denial-of-service-attack) * [HTTP Denial of Service attack](doc/HELPERS.md#tcp-syn-flood-denial-of-service-attack) * [Debugging](doc/HELPERS.md#debugging) * [Show information about processes](doc/HELPERS.md#show-information-about-nginx-processes) * [Check memory usage](doc/HELPERS.md#check-memoryusage) * [Show open files](doc/HELPERS.md#show-open-files) * [Check segmentation fault messages](doc/HELPERS.md#check-segmentation-fault-messages) * [Dump configuration](doc/HELPERS.md#dump-configuration) * [Get the list of configure arguments](doc/HELPERS.md#get-the-list-of-configure-arguments) * [Check if the module has been compiled](doc/HELPERS.md#check-if-the-module-has-been-compiled) * [Show the most accessed IP addresses](doc/HELPERS.md#show-the-most-accessed-ip-addresses) * [Show the most accessed IP addresses (ip and url)](doc/HELPERS.md#show-the-most-accessed-ip-addresses-ip-and-url) * [Show the most accessed IP addresses (method, code, ip, and url)](doc/HELPERS.md#show-the-most-accessed-ip-addresses-method-code-ip-and-url) * [Show the top 5 visitors (IP addresses)](doc/HELPERS.md#show-the-top-5-visitors-ip-addresses) * [Show the most requested urls](doc/HELPERS.md#show-the-most-requested-urls) * [Show the most requested urls containing 'string'](doc/HELPERS.md#show-the-most-requested-urls-containing-string) * [Show the most requested urls with http methods](doc/HELPERS.md#show-the-most-requested-urls-with-http-methods) * [Show the most accessed response codes](doc/HELPERS.md#show-the-most-accessed-response-codes) * [Analyse web server log and show only 2xx http codes](doc/HELPERS.md#analyse-web-server-log-and-show-only-2xx-http-codes) * [Analyse web server log and show only 5xx http codes](doc/HELPERS.md#analyse-web-server-log-and-show-only-5xx-http-codes) * [Show requests which result 502 and sort them by number per requests by url](doc/HELPERS.md#show-requests-which-result-502-and-sort-them-by-number-per-requests-by-url) * [Show requests which result 404 for php files and sort them by number per requests by url](doc/HELPERS.md#show-requests-which-result-404-for-php-files-and-sort-them-by-number-per-requests-by-url) * [Calculating amount of http response codes](doc/HELPERS.md#calculating-amount-of-http-response-codes) * [Calculating requests per second](doc/HELPERS.md#calculating-requests-per-second) * [Calculating requests per second with IP addresses](doc/HELPERS.md#calculating-requests-per-second-with-ip-addresses) * [Calculating requests per second with IP addresses and urls](doc/HELPERS.md#calculating-requests-per-second-with-ip-addresses-and-urls) * [Get entries within last n hours](doc/HELPERS.md#get-entries-within-last-n-hours) * [Get entries between two timestamps (range of dates)](doc/HELPERS.md#get-entries-between-two-timestamps-range-of-dates) * [Get line rates from web server log](doc/HELPERS.md#get-line-rates-from-web-server-log) * [Trace network traffic for all processes](doc/HELPERS.md#trace-network-traffic-for-all-nginx-processes) * [List all files accessed by a NGINX](doc/HELPERS.md#list-all-files-accessed-by-a-nginx) * [Check that the gzip_static module is working](doc/HELPERS.md#check-that-the-gzip_static-module-is-working) * [Which worker processing current request](doc/HELPERS.md#which-worker-processing-current-request) * [Capture only http packets](doc/HELPERS.md#capture-only-http-packets) * [Extract User Agent from the http packets](doc/HELPERS.md#extract-user-agent-from-the-http-packets) * [Capture only http GET and POST packets](doc/HELPERS.md#capture-only-http-get-and-post-packets) * [Capture requests and filter by source ip and destination port](doc/HELPERS.md#capture-requests-and-filter-by-source-ip-and-destination-port) * [Capture HTTP requests/responses in real time, filter by GET, HEAD and save to a file](doc/HELPERS.md#capture-http-requests--responses-in-real-time-filter-by-get-head-and-save-to-a-file) * [Dump a process's memory](doc/HELPERS.md#dump-a-processs-memory) * [GNU Debugger (gdb)](doc/HELPERS.md#gnu-debugger-gdb) * [Dump configuration from a running process](doc/HELPERS.md#dump-configuration-from-a-running-process) * [Show debug log in memory](doc/HELPERS.md#show-debug-log-in-memory) * [Core dump backtrace](doc/HELPERS.md#core-dump-backtrace) * [Debugging socket leaks](doc/HELPERS.md#debugging-socket-leaks) * [Shell aliases](doc/HELPERS.md#shell-aliases) * [Configuration snippets](doc/HELPERS.md#configuration-snippets) * [Nginx server header removal](doc/HELPERS.md#nginx-server-header-removal) * [Custom log formats](doc/HELPERS.md#custom-log-formats) * [Log only 4xx/5xx](doc/HELPERS.md#log-only-4xx5xx) * [Restricting access with basic authentication](doc/HELPERS.md#restricting-access-with-basic-authentication) * [Restricting access with client certificate](doc/HELPERS.md#restricting-access-with-client-certificate) * [Restricting access by geographical location](doc/HELPERS.md#restricting-access-by-geographical-location) * [GeoIP 2 database](doc/HELPERS.md#geoip-2-database) * [Dynamic error pages with SSI](doc/HELPERS.md#dynamic-error-pages-with-ssi) * [Blocking/allowing IP addresses](doc/HELPERS.md#blockingallowing-ip-addresses) * [Blocking referrer spam](doc/HELPERS.md#blocking-referrer-spam) * [Limiting referrer spam](doc/HELPERS.md#limiting-referrer-spam) * [Blocking User-Agent](doc/HELPERS.md#blocking-user-agent) * [Limiting User-Agent](doc/HELPERS.md#limiting-user-agent) * [Limiting the rate of requests with burst mode](doc/HELPERS.md#limiting-the-rate-of-requests-with-burst-mode) * [Limiting the rate of requests with burst mode and nodelay](doc/HELPERS.md#limiting-the-rate-of-requests-with-burst-mode-and-nodelay) * [Limiting the rate of requests per IP with geo and map](doc/HELPERS.md#limiting-the-rate-of-requests-per-ip-with-geo-and-map) * [Limiting the number of connections](doc/HELPERS.md#limiting-the-number-of-connections) * [Using trailing slashes](doc/HELPERS.md#using-trailing-slashes) * [Properly redirect all HTTP requests to HTTPS](doc/HELPERS.md#properly-redirect-all-http-requests-to-https) * [Adding and removing the www prefix](doc/HELPERS.md#adding-and-removing-the-www-prefix) * [Proxy/rewrite and keep the original URL](doc/HELPERS.md#proxyrewrite-and-keep-the-original-url) * [Proxy/rewrite and keep the part of original URL](doc/HELPERS.md#proxyrewrite-and-keep-the-part-of-original-url) * [Proxy/rewrite without changing the original URL (in browser)](doc/HELPERS.md#proxyrewrite-without-changing-the-original-url-in-browser) * [Modify 301/302 response body](doc/HELPERS.md#modify-301302-response-body) * [Redirect POST request with payload to external endpoint](doc/HELPERS.md#redirect-post-request-with-payload-to-external-endpoint) * [Route to different backends based on HTTP method](doc/HELPERS.md#route-to-different-backends-based-on-HTTP-method) * [Allow multiple cross-domains using the CORS headers](doc/HELPERS.md#allow-multiple-cross-domains-using-the-cors-headers) * [Set correct scheme passed in X-Forwarded-Proto](doc/HELPERS.md#set-correct-scheme-passed-in-x-forwarded-proto) * [Other snippets](doc/HELPERS.md#other-snippets) * [Recreate base directory](doc/HELPERS.md#recreate-base-directory) * [Create a temporary static backend](doc/HELPERS.md#create-a-temporary-static-backend) * [Create a temporary static backend with SSL support](doc/HELPERS.md#create-a-temporary-static-backend-with-ssl-support) * [Generate password file with htpasswd command](doc/HELPERS.md#generate-password-file-with-htpasswd-command) * [Generate private key without passphrase](doc/HELPERS.md#generate-private-key-without-passphrase) * [Generate private key with passphrase](doc/HELPERS.md#generate-private-key-with-passphrase) * [Remove passphrase from private key](doc/HELPERS.md#remove-passphrase-from-private-key) * [Encrypt existing private key with a passphrase](doc/HELPERS.md#encrypt-existing-private-key-with-a-passphrase) * [Generate CSR](doc/HELPERS.md#generate-csr) * [Generate CSR (metadata from existing certificate)](doc/HELPERS.md#generate-csr-metadata-from-existing-certificate) * [Generate CSR with -config param](doc/HELPERS.md#generate-csr-with--config-param) * [Generate private key and CSR](doc/HELPERS.md#generate-private-key-and-csr) * [List available EC curves](doc/HELPERS.md#list-available-ec-curves) * [Print ECDSA private and public keys](doc/HELPERS.md#print-ecdsa-private-and-public-keys) * [Generate ECDSA private key](doc/HELPERS.md#generate-ecdsa-private-key) * [Generate private key and CSR (ECC)](doc/HELPERS.md#generate-private-key-and-csr-ecc) * [Generate self-signed certificate](doc/HELPERS.md#generate-self-signed-certificate) * [Generate self-signed certificate from existing private key](doc/HELPERS.md#generate-self-signed-certificate-from-existing-private-key) * [Generate self-signed certificate from existing private key and csr](doc/HELPERS.md#generate-self-signed-certificate-from-existing-private-key-and-csr) * [Generate multidomain certificate (Certbot)](doc/HELPERS.md#generate-multidomain-certificate-certbot) * [Generate wildcard certificate (Certbot)](doc/HELPERS.md#generate-wildcard-certificate-certbot) * [Generate certificate with 4096 bit private key (Certbot)](doc/HELPERS.md#generate-certificate-with-4096-bit-private-key-certbot) * [Generate DH public parameters](doc/HELPERS.md#generate-dh-public-parameters) * [Display DH public parameters](doc/HELPERS.md#display-dh-public-parameters) * [Extract private key from pfx](doc/HELPERS.md#extract-private-key-from-pfx) * [Extract private key and certs from pfx](doc/HELPERS.md#extract-private-key-and-certs-from-pfx) * [Extract certs from p7b](doc/HELPERS.md#extract-certs-from-p7b) * [Convert DER to PEM](doc/HELPERS.md#convert-der-to-pem) * [Convert PEM to DER](doc/HELPERS.md#convert-pem-to-der) * [Verification of the certificate's supported purposes](doc/HELPERS.md#verification-of-the-certificates-supported-purposes) * [Check private key](doc/HELPERS.md#check-private-key) * [Verification of the private key](doc/HELPERS.md#verification-of-the-private-key) * [Get public key from private key](doc/HELPERS.md#get-public-key-from-private-key) * [Verification of the public key](doc/HELPERS.md#verification-of-the-public-key) * [Verification of the certificate](doc/HELPERS.md#verification-of-the-certificate) * [Verification of the CSR](doc/HELPERS.md#verification-of-the-csr) * [Check the private key and the certificate are match](doc/HELPERS.md#check-the-private-key-and-the-certificate-are-match) * [Check the private key and the CSR are match](doc/HELPERS.md#check-the-private-key-and-the-csr-are-match) [TLSv1.3 and CCM ciphers](doc/HELPERS.md#tlsv13-and-ccm-ciphers) - **[Base Rules (16)](doc/RULES.md#base-rules)** * [Organising Nginx configuration](doc/RULES.md#beginner-organising-nginx-configuration) * [Format, prettify and indent your Nginx code](doc/RULES.md#beginner-format-prettify-and-indent-your-nginx-code) * [Use reload option to change configurations on the fly](doc/RULES.md#beginner-use-reload-option-to-change-configurations-on-the-fly) * [Separate listen directives for 80 and 443 ports](doc/RULES.md#beginner-separate-listen-directives-for-80-and-443-ports) * [Define the listen directives with address:port pair](doc/RULES.md#beginner-define-the-listen-directives-with-addressport-pair) * [Prevent processing requests with undefined server names](doc/RULES.md#beginner-prevent-processing-requests-with-undefined-server-names) * [Never use a hostname in a listen or upstream directives](doc/RULES.md#beginner-never-use-a-hostname-in-a-listen-or-upstream-directives) * [Set the HTTP headers with add_header and proxy_*_header directives properly](doc/RULES.md#beginner-set-the-http-headers-with-add_header-and-proxy__header-directives-properly) * [Use only one SSL config for the listen directive](doc/RULES.md#beginner-use-only-one-ssl-config-for-the-listen-directive) * [Use geo/map modules instead of allow/deny](doc/RULES.md#beginner-use-geomap-modules-instead-of-allowdeny) * [Map all the things...](doc/RULES.md#beginner-map-all-the-things) * [Set global root directory for unmatched locations](doc/RULES.md#beginner-set-global-root-directory-for-unmatched-locations) * [Use return directive for URL redirection (301, 302)](doc/RULES.md#beginner-use-return-directive-for-url-redirection-301-302) * [Configure log rotation policy](doc/RULES.md#beginner-configure-log-rotation-policy) * [Use simple custom error pages](doc/RULES.md#beginner-use-simple-custom-error-pages) * [Don't duplicate index directive, use it only in the http block](doc/RULES.md#beginner-dont-duplicate-index-directive-use-it-only-in-the-http-block) - **[Debugging (5)](doc/RULES.md#debugging)** * [Use custom log formats](doc/RULES.md#beginner-use-custom-log-formats) * [Use debug mode to track down unexpected behaviour](doc/RULES.md#beginner-use-debug-mode-to-track-down-unexpected-behaviour) * [Improve debugging by disable daemon, master process, and all workers except one](doc/RULES.md#beginner-improve-debugging-by-disable-daemon-master-process-and-all-workers-except-one) * [Use core dumps to figure out why NGINX keep crashing](doc/RULES.md#beginner-use-core-dumps-to-figure-out-why-nginx-keep-crashing) * [Use mirror module to copy requests to another backend](doc/RULES.md#beginner-use-mirror-module-to-copy-requests-to-another-backend) - **[Performance (13)](doc/RULES.md#performance)** * [Adjust worker processes](doc/RULES.md#beginner-adjust-worker-processes) * [Use HTTP/2](doc/RULES.md#beginner-use-http2) * [Maintaining SSL sessions](doc/RULES.md#beginner-maintaining-ssl-sessions) * [Enable OCSP Stapling](doc/RULES.md#beginner-enable-ocsp-stapling) * [Use exact names in a server_name directive if possible](doc/RULES.md#beginner-use-exact-names-in-a-server_name-directive-if-possible) * [Avoid checks server_name with if directive](doc/RULES.md#beginner-avoid-checks-server_name-with-if-directive) * [Use $request_uri to avoid using regular expressions](doc/RULES.md#beginner-use-request_uri-to-avoid-using-regular-expressions) * [Use try_files directive to ensure a file exists](doc/RULES.md#beginner-use-try_files-directive-to-ensure-a-file-exists) * [Use return directive instead of rewrite for redirects](doc/RULES.md#beginner-use-return-directive-instead-of-rewrite-for-redirects) * [Enable PCRE JIT to speed up processing of regular expressions](doc/RULES.md#beginner-enable-pcre-jit-to-speed-up-processing-of-regular-expressions) * [Activate the cache for connections to upstream servers](doc/RULES.md#beginner-activate-the-cache-for-connections-to-upstream-servers) * [Make an exact location match to speed up the selection process](doc/RULES.md#beginner-make-an-exact-location-match-to-speed-up-the-selection-process) * [Use limit_conn to improve limiting the download speed](doc/RULES.md#beginner-use-limit_conn-to-improve-limiting-the-download-speed) - **[Hardening (31)](doc/RULES.md#hardening)** * [Always keep NGINX up-to-date](doc/RULES.md#beginner-always-keep-nginx-up-to-date) * [Run as an unprivileged user](doc/RULES.md#beginner-run-as-an-unprivileged-user) * [Disable unnecessary modules](doc/RULES.md#beginner-disable-unnecessary-modules) * [Protect sensitive resources](doc/RULES.md#beginner-protect-sensitive-resources) * [Take care about your ACL rules](doc/RULES.md#beginner-take-care-about-your-acl-rules) * [Hide Nginx version number](doc/RULES.md#beginner-hide-nginx-version-number) * [Hide Nginx server signature](doc/RULES.md#beginner-hide-nginx-server-signature) * [Hide upstream proxy headers](doc/RULES.md#beginner-hide-upstream-proxy-headers) * [Remove support for legacy and risky HTTP request headers](doc/RULES.md#beginner-remove-support-for-legacy-and-risky-http-request-headers) * [Use only the latest supported OpenSSL version](doc/RULES.md#beginner-use-only-the-latest-supported-openssl-version) * [Force all connections over TLS](doc/RULES.md#beginner-force-all-connections-over-tls) * [Use min. 2048-bit for RSA and 256-bit for ECC](doc/RULES.md#beginner-use-min-2048-bit-for-rsa-and-256-bit-for-ecc) * [Keep only TLS 1.3 and TLS 1.2](doc/RULES.md#beginner-keep-only-tls-13-and-tls-12) * [Use only strong ciphers](doc/RULES.md#beginner-use-only-strong-ciphers) * [Use more secure ECDH Curve](doc/RULES.md#beginner-use-more-secure-ecdh-curve) * [Use strong Key Exchange with Perfect Forward Secrecy](doc/RULES.md#beginner-use-strong-key-exchange-with-perfect-forward-secrecy) * [Prevent Replay Attacks on Zero Round-Trip Time](doc/RULES.md#beginner-prevent-replay-attacks-on-zero-round-trip-time) * [Defend against the BEAST attack](doc/RULES.md#beginner-defend-against-the-beast-attack) * [Mitigation of CRIME/BREACH attacks](doc/RULES.md#beginner-mitigation-of-crimebreach-attacks) * [Enable HTTP Strict Transport Security](doc/RULES.md#beginner-enable-http-strict-transport-security) * [Reduce XSS risks (Content-Security-Policy)](doc/RULES.md#beginner-reduce-xss-risks-content-security-policy) * [Control the behaviour of the Referer header (Referrer-Policy)](doc/RULES.md#beginner-control-the-behaviour-of-the-referer-header-referrer-policy) * [Provide clickjacking protection (X-Frame-Options)](doc/RULES.md#beginner-provide-clickjacking-protection-x-frame-options) * [Prevent some categories of XSS attacks (X-XSS-Protection)](doc/RULES.md#beginner-prevent-some-categories-of-xss-attacks-x-xss-protection) * [Prevent Sniff Mimetype middleware (X-Content-Type-Options)](doc/RULES.md#beginner-prevent-sniff-mimetype-middleware-x-content-type-options) * [Deny the use of browser features (Feature-Policy)](doc/RULES.md#beginner-deny-the-use-of-browser-features-feature-policy) * [Reject unsafe HTTP methods](doc/RULES.md#beginner-reject-unsafe-http-methods) * [Prevent caching of sensitive data](doc/RULES.md#beginner-prevent-caching-of-sensitive-data) * [Limit concurrent connections](doc/RULES.md#beginner-limit-concurrent-connections) * [Control Buffer Overflow attacks](doc/RULES.md#beginner-control-buffer-overflow-attacks) * [Mitigating Slow HTTP DoS attacks (Closing Slow Connections)](doc/RULES.md#beginner-mitigating-slow-http-dos-attacks-closing-slow-connections) - **[Reverse Proxy (8)](doc/RULES.md#reverse-proxy)** * [Use pass directive compatible with backend protocol](doc/RULES.md#beginner-use-pass-directive-compatible-with-backend-protocol) * [Be careful with trailing slashes in proxy_pass directive](doc/RULES.md#beginner-be-careful-with-trailing-slashes-in-proxy_pass-directive) * [Set and pass Host header only with $host variable](doc/RULES.md#beginner-set-and-pass-host-header-only-with-host-variable) * [Set properly values of the X-Forwarded-For header](doc/RULES.md#beginner-set-properly-values-of-the-x-forwarded-for-header) * [Don't use X-Forwarded-Proto with $scheme behind reverse proxy](doc/RULES.md#beginner-dont-use-x-forwarded-proto-with-scheme-behind-reverse-proxy) * [Always pass Host, X-Real-IP, and X-Forwarded headers to the backend](doc/RULES.md#beginner-always-pass-host-x-real-ip-and-x-forwarded-headers-to-the-backend) * [Use custom headers without X- prefix](doc/RULES.md#beginner-use-custom-headers-without-x--prefix) * [Always use $request_uri instead of $uri in proxy_pass](doc/RULES.md#beginner-always-use-request_uri-instead-of-uri-in-proxy_pass) - **[Load Balancing (2)](doc/RULES.md#load-balancing)** * [Tweak passive health checks](doc/RULES.md#beginner-tweak-passive-health-checks) * [Don't disable backends by comments, use down parameter](doc/RULES.md#beginner-dont-disable-backends-by-comments-use-down-parameter) - **[Others (4)](doc/RULES.md#others)** * [Set the certificate chain correctly](doc/RULES.md#beginner-set-the-certificate-chain-correctly) * [Enable DNS CAA Policy](doc/RULES.md#beginner-enable-dns-caa-policy) * [Define security policies with security.txt](doc/RULES.md#beginner-define-security-policies-with-securitytxt) * [Use tcpdump to diagnose and troubleshoot the HTTP issues](doc/RULES.md#beginner-use-tcpdump-to-monitor-http-traffic) - **[Configuration Examples](doc/EXAMPLES.md#configuration-examples)** * [Reverse Proxy](doc/EXAMPLES.md#reverse-proxy) * [Installation](doc/EXAMPLES.md#installation) * [Configuration](doc/EXAMPLES.md#configuration) * [Import configuration](doc/EXAMPLES.md#import-configuration) * [Set bind IP address](doc/EXAMPLES.md#set-bind-ip-address) * [Set your domain name](doc/EXAMPLES.md#set-your-domain-name) * [Regenerate private keys and certs](doc/EXAMPLES.md#regenerate-private-keys-and-certs) * [Update modules list](doc/EXAMPLES.md#update-modules-list) * [Generating the necessary error pages](doc/EXAMPLES.md#generating-the-necessary-error-pages) * [Add new domain](doc/EXAMPLES.md#add-new-domain) * [Test your configuration](doc/EXAMPLES.md#test-your-configuration)
# Introduction


> Before you start playing with NGINX please read an official **[Beginner’s Guide](http://nginx.org/en/docs/beginners_guide.html)**. It's a great introduction for everyone. **Nginx** (_/ˌɛndʒɪnˈɛks/ EN-jin-EKS_, stylized as NGINX or nginx) is an open source HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server with a strong focus on high concurrency, performance and low memory usage. It is originally written by [Igor Sysoev](http://sysoev.ru/en/). For a long time, it has been running on many heavily loaded Russian sites including Yandex, Mail.Ru, VK, and Rambler. At this moment some high-profile companies using NGINX include Cisco, DuckDuckGo, Facebook, GitLab, Google, Twitter, Apple, Intel, and many more. In the September 2019 it was the most commonly used HTTP server (see [Netcraft survey](https://news.netcraft.com/archives/category/web-server-survey/)). NGINX is a fast, light-weight and powerful web server that can also be used as a: - fast HTTP reverse proxy - reliable load balancer - high performance caching server - full-fledged web platform So, to be brief, it provides the core of complete web stacks and is designed to help build scalable web applications. When it comes to performance, NGINX can easily handle a huge amount of traffic. The other main advantage of the NGINX is that allows you to do the same thing in different ways. Unlike traditional HTTP servers, NGINX doesn't rely on threads to handle requests and it was written with a different architecture in mind - one which is much more suitable for nonlinear scalability in both the number of simultaneous connections and requests per second. NGINX is also known as a _Apache Killer_ (mainly because of its lightness and much less RAM consumption). It is event-based, so it does not follow Apache's style of spawning new processes or threads for each web page request. Generally, it was created to solve the [C10K problem](http://www.kegel.com/c10k.html). For me, it is a one of the best and most important service that I used in my SysAdmin career. ---- These essential documents should be the main source of knowledge for you: - **[Getting Started](https://www.nginx.com/resources/wiki/start/)** - **[NGINX Documentation](https://nginx.org/en/docs/)** - **[Development guide](http://nginx.org/en/docs/dev/development_guide.html)** - **[Security Controls](https://docs.nginx.com/nginx/admin-guide/security-controls/)** In addition, I would like to recommend three great docs focuses on the concept of the HTTP protocol: - **[HTTP Made Really Easy](https://www.jmarshall.com/easy/http/)** - **[Hypertext Transfer Protocol Specification](https://www.w3.org/Protocols/)** - **[Web technology for developers - HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP)** If you love security keep your eye on this one: [Cryptology ePrint Archive](https://eprint.iacr.org/). It provides access to recent research in cryptology and explores many subjects of security (e.g. Ciphers, Algorithms, SSL/TLS protocols). A great introduction that covers core concepts of cryptography is [Practical Cryptography for Developers](https://cryptobook.nakov.com/). I also recommend to read the [Bulletproof SSL and TLS](https://www.feistyduck.com/books/bulletproof-ssl-and-tls/). Yep, it's definitely the most comprehensive book about deploying TLS for me. An obligatory source of knowledge is also the [OWASP Cheat Sheet Series](https://cheatsheetseries.owasp.org/). You should ought treat it as an excellent security guidance. [Burp Scanner - Issue Definitions](https://portswigger.net/kb/issues) introduces you to the web apps and security vulnerabilities. Finally, [The Web Security Academy](https://portswigger.net/web-security) is a free online training center for web application security with high-quality reading materials and interactive labs of varying levels of difficulty. All are really good source to start learning about web application security. And, of course, always browse official [Nginx Security Advisories](http://nginx.org/en/security_advisories.html) and CVE databases like [CVE Details](https://www.cvedetails.com/vendor/10048/Nginx.html) or [CVE - The MITRE Corporation](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=NGINX) - to stay Up-to-Date on NGINX vulnerabilities. ## Prologue When I was studying architecture of HTTP servers I became interested in NGINX. As I was going through research, I kept notes. I found a lot of information about it, e.g. forum posts on the web about every conceivable problem was great. However, I've never found one guide that covers the most important things in a suitable form. I was a little disappointed. I was interested in everything: NGINX internals, functions, security best practices, performance optimisations, tips & tricks, hacks and rules, but for me some of the documents treated the subject lightly. Of course, [NGINX Official Documentation](https://nginx.org/en/docs/) is the best place but I know that we also have other great resources: - [agentzh's Nginx Tutorials](https://openresty.org/download/agentzh-nginx-tutorials-en.html) - [Nginx Guts](http://www.nginxguts.com/) - [Nginx discovery journey](http://www.nginx-discovery.com/) - [Nginx Secure Web Server](https://calomel.org/nginx.html) - [Emiller’s Guide To Nginx Module Development](https://www.evanmiller.org/nginx-modules-guide.html) - [Emiller’s Advanced Topics In Nginx Module Development](https://www.evanmiller.org/nginx-modules-guide-advanced.html) These are definitely the best assets for us and in the first place you should seek help there. Moreover, in order to improve your knowledge, please see [Books](#books) chapter - it contains top literature on NGINX. ## Why I created this handbook For me, however, there hasn't been a truly in-depth and reasonably simple cheatsheet which describe a variety of configurations and important cross-cutting topics for HTTP servers. Configuration of the NGINX can be tricky sometimes and you really need to get into the syntax and concepts to get an understanding tricks, loopholes, and mechanisms. The documentation isn't as pretty as other projects and should certainly include more robust examples. > This handbook is a set of rules and recommendations for the NGINX Open Source HTTP server. It also contains the best practices, notes, and helpers with countless examples. Many of them refer to external resources. There are a lot of things you can do to improve in your NGINX instance and this guide will attempt to cover as many of them as possible. For the most part, it contains the most important things about NGINX for me. I think the configuration you provided should work without any talisman. That's why I created this repository. With this handbook you will explore the many features and capabilities of the NGINX. You'll find out, for example, how to testing the performance or how to resolve debugging problems. You will learn configuration guidelines, security design patterns, ways to handle common issues and how to stay out of them. I explained here a few best tips to avoid pitfalls and configuration mistakes. I added set of guidelines and examples has also been produced to help you administer of the NGINX. They give us insight into NGINX internals also. Mostly, I apply the rules presented here on the NGINX working as a reverse proxy. However, does not to prevent them being implemented for NGINX as a standalone server. ## Who this handbook is for If you do not have the time to read hundreds of articles (just like me) this multipurpose handbook may be useful. I created it in the hope that it will be useful especially for System Administrators and Experts of Web-based applications. This handbook does not get into all aspects of NGINX. What's more, some of the things described in this guide may be rather basic because most of us do not configure NGINX every day and it is easy to forget about basic/trivial things. On the other hand, also discusses heavyweight topics so there is something for advanced users. I tried to put external resources in many places in this handbook in order to dispel any suspicion that may exist. I did my best to make this handbook a single and consistent (but now I know that is really hard). It's organized in an order that makes logical sense to me. I think it can also be a good complement to official documentation and other great documents. Many of the topics described here can certainly be done better or different. Of course, I still have a lot [to improve and to do](#contributing--support). I hope you enjoy and have fun with it. Do not treat this handbook and notes written here as revealed knowledge. You should take a scientific approach when reading this document. If you have any doubts and disagree with me, please point out my mistakes. You should to discover cause and effect relationships by asking questions, carefully gathering and examining the evidence, and seeing if all the available information can be combined in to a logical answer. I create this handbook for one more reason. Rather than starting from scratch in, I putting together a plan for answering your questions to help you find the best way to do things and ensure that you don't repeat my mistakes from the past. So, what's most important: - ask a questions about something that you observe - do background research - do tests with an experiments - analyze and draw conclusions - communicate results (for us!) Finally, you should know I'm not a NGINX expert but I love to know how stuff works and why work the way they do. [I’m not a crypto expert... but I do know the term "elliptic curve"](https://twitter.com/ErikVoorhees/status/1004313761224757248) (I really like this quote!). Don't need to be an expert to figure out the reason just got to have used this and not this or why something works this way and not another. It feels good to understand the recommendations and nuances of a topic you’re passionate about. ## Before you start Remember about the following most important things: > **`Blindly deploying of the rules described here can damage your web application!`** > **`Do not follow guides just to get 100% of something. Think about what you actually do at your server!`** > **`Copy-and-paste is not the best way to learn. Think twice before adopting rules from this handbook.`** > **`There are no settings that are perfect for everyone.`** > **`Always think about what is better and more important for you: security vs usability/compatibility.`** > **`Security mainly refers to minimise the risk.`** > **`Change one thing may open a whole new set of problems.`** > **`Read about how things work and what values are considered secure enough (and for what purposes).`** > **`The only correct approach is to understand your exposure, measure and tune.`** ```diff + Security is important for ethical reasons. Compliance is important for legal reasons. + The key to workplace contentment is understanding they are unrelated to each other. + Both are important, but one does not lead to the other (compliance != security). author: unknown + Security is always needed, no matter what type of website it is. It can be static HTML + or fully dynamic, an attacker can still inject hostile content into the page in transit + to attack the user. author: Scott Helme + Don’t enable older deprecated protocols just because Karen in Florida is still using + a PC that she bought back in 2001. author: thisinterestsmeblog ``` I think, in the age of phishing, cyber attacks, ransomware, etc., you should take care of security of your infrastructure as hard as possible but don't ever forget about this one...

Lastly, I would like to quote two very important comments found on the web about compliance with the standards and regulations, and essence of a human factor in security: > _Regulations that make sense are often not descriptive - capturing the intent and scope of a rule often requires technical expertise. More than that, it's the type of expertise most organisations do not have. And instead of improving themselves, these companies, who may form the grand majority of the industry, petition the regulators to provide a safe checklist of technical mitigations that can be implemented to remain compliant. [...] Instead of doing the right thing and meeting the planned intent, companies are instead ticking nonsensical boxes that the regulators and their auditors demand. Blindly. Mindlessly. Divorced from reality._ - by [bostik](https://news.ycombinator.com/user?id=bostik) > _Whenever considering security, the human factor is nearly always as important or more important than just the technical aspects. Policy and procedures need to consider the human element and try to ensure that these policies and procedures are structured in such a way as to help enable staff to do the right thing, even when they may not fully understand why they need to do it._ - by [Tim X](https://security.stackexchange.com/users/13958/tim-x) ## Contributing & Support > _A real community, however, exists only when its members interact in a meaningful way that deepens their understanding of each other and leads to learning._ If you find something which doesn't make sense, or something doesn't seem right, please make a pull request and please add valid and well-reasoned explanations about your changes or comments. Before adding a pull request, please see the **[contributing guidelines](.github/CONTRIBUTING.md)**. ## Code Contributors This project exists thanks to all the people who contribute. ### ToDo What needs to be done? Look at the following ToDo list: New chapters: - [x] **Bonus Stuff** - [x] **HTTP Basics** - [x] **SSL/TLS Basics** - [x] **Reverse Proxy** - [ ] **Caching** - [x] **Core modules** - [x] **3rd party modules** - [ ] **Web Application Firewall** - [ ] **ModSecurity** - [x] **Debugging** Existing chapters:
Introduction
- [x] _Prologue_ - [x] _Why I created this handbook_ - [x] _Who this handbook is for_ - [x] _Before you start_ - [x] _Contributing & Support_ - [x] _RSS Feed & Updates - [x] _Checklist to rule them all_
Bonus Stuff
- [x] _Fully automatic installation_ - [x] _Static error pages generator_ - [x] _Server names parser_
Books
- [x] _ModSecurity 3.0 and NGINX: Quick Start Guide_ - [x] _Cisco ACE to NGINX: Migration Guide_
External Resources
- _Nginx official_ - [x] _Nginx Forum_ - [x] _Nginx Mailing List_ - [x] _NGINX-Demos_ - _Presentations & Videos_ - [x] _NGINX: Basics and Best Practices_ - [x] _NGINX Installation and Tuning_ - [x] _Nginx Internals (by Joshua Zhu)_ - [x] _Nginx internals (by Liqiang Xu)_ - [x] _How to secure your web applications with NGINX_ - [x] _Tuning TCP and NGINX on EC2_ - [x] _Extending functionality in nginx, with modules!_ - [x] _Nginx - Tips and Tricks._ - [x] _Nginx Scripting - Extending Nginx Functionalities with Lua_ - [x] _How to handle over 1,200,000 HTTPS Reqs/Min_ - [x] _Using ngx_lua / lua-nginx-module in pixiv_ - _Cheatsheets & References_ - [x] _Nginx configurations for most popular CMS/CMF/Frameworks based on PHP_ - _Performance & Hardening_ - [x] _Memorable site for testing clients against bad SSL configs_ - _Config parsers_ - [x] _Quick and reliable way to convert NGINX configurations into JSON and back_ - [x] _Parses nginx configuration with Pyparsing_ - _Config managers_ - [x] _Ansible role to install and manage nginx configuration_ - [x] _Ansible Role - Nginx_ - [x] _Ansible role for NGINX_ - [x] _Puppet Module to manage NGINX on various UNIXes_ - _Static analyzers_ - [x] _nginx-minify-conf_ - _Comparison reviews_ - [x] _NGINX vs. Apache (Pro/Con Review, Uses, & Hosting for Each)_ - [x] _Web cache server performance benchmark: nuster vs nginx vs varnish vs squid_ - _Builder tools_ - [x] _Nginx-builder_ - _Benchmarking tools_ - [x] _wrk2_ - [x] _httperf_ - [x] _slowloris_ - [x] _slowhttptest_ - [x] _GoldenEye_ - _Debugging tools_ - [x] _strace_ - [x] _GDB_ - [x] _SystemTap_ - [x] _stapxx_ - [x] _htrace.sh_ - _Security & Web testing tools_ - [x] _Burp Suite_ - [x] _w3af_ - [x] _nikto_ - [x] _ssllabs-scan_ - [x] _http-observatory_ - [x] _testssl.sh_ - [x] _sslyze_ - [x] _cipherscan_ - [x] _O-Saft_ - [x] _Nghttp2_ - [x] _h2spec_ - [x] _http2fuzz_ - [x] _Arjun_ - [x] _Corsy_ - [x] _XSStrike_ - _Online & Web tools_ - [x] _ssltools_ - _Other stuff_ - [x] _OWASP Cheat Sheet Series_ - [x] _Mozilla Web Security_ - [x] _Application Security Wiki_ - [x] _OWASP ASVS 4.0_ - [x] _The System Design Primer_ - [x] _awesome-scalability_ - [x] _Web Architecture 101_
HTTP Basics
- [x] _Features and architecture_ - [x] _HTTP/2_ - [x] _How to debug HTTP/2?_ - [x] _HTTP/3_ - [x] _URI vs URL_ - [x] _Connection vs request_ - [x] _HTTP Headers_ - [x] _Header compression_ - [x] _HTTP Methods_ - [x] _Request_ - [x] _Request line_ - [x] _Methods_ - [x] _Request URI_ - [x] _HTTP version_ - [x] _Request header fields_ - [x] _Message body_ - [x] _Generate requests_ - [x] _Response_ - [x] _Status line_ - [x] _HTTP version_ - [x] _Status codes and reason phrase_ - [x] _Response header fields_ - [x] _Message body_ - [x] _HTTP client_ - [x] _IP address shortcuts_ - [x] _Back-End web architecture_ - [x] _Useful video resources_
SSL/TLS Basics
- [x] _TLS versions_ - [x] _TLS handshake_ - [x] _In which layer is TLS situated within the TCP/IP stack?_ - [x] _RSA and ECC keys/certificates_ - [x] _Cipher suites_ - [x] _Authenticated encryption (AEAD) cipher suites_ - [x] _Why cipher suites are important?_ - [x] _NGINX and TLS 1.3 Cipher Suites_ - [x] _Diffie-Hellman key exchange_ - [x] _Certificates_ - [x] _Chain of Trust_ - [x] _What is the main purpose of the Intermediate CA?_ - [x] _Single-domain_ - [x] _Multi-domain_ - [x] _Wildcard_ - [x] _Wildcard SSL doesn't handle root domain?_ - [x] _TLS Server Name Indication_ - [x] _Verify your SSL, TLS & Ciphers implementation_ - [x] _Useful video resources_
NGINX Basics
- _Processes_ - [x] _CPU pinning_ - [x] _Shutdown of worker processes_ - _Configuration syntax_ - [x] _Comments_ - [x] _End of lines_ - [x] _Variables, Strings, and Quotes_ - [x] _Directives, Blocks, and Contexts_ - [x] _External files_ - [x] _Measurement units_ - [x] _Regular expressions with PCRE_ - [x] _Enable syntax highlighting_ - _Connection processing_ - [x] _Event-Driven architecture_ - [x] _Multiple processes_ - [x] _Simultaneous connections_ - [x] _HTTP Keep-Alive connections_ - [x] _sendfile, tcp_nodelay, and tcp_nopush_ - _Server blocks logic_ - [x] _Matching location_ - [ ] _if in location_ - [ ] _Nested locations_ - [x] _rewrite vs return_ - [x] _try_files directive_ - [x] _if, break and set_ - [x] _root vs alias_ - [x] _internal directive_ - [x] _External and internal redirects_ - [x] _allow and deny_ - [x] _uri vs request_uri_ - _Compression and decompression_ - [x] _What is the best NGINX compression gzip level?_ - _Hash tables_ - [x] _Server names hash table_ - _Log files_ - [x] _Conditional logging_ - [x] _Manually log rotation_ - [x] _NGINX upstream variables returns 2 values_ - _Reverse proxy_ - [x] _Passing requests_ - [x] _Trailing slashes_ - [ ] _Processing headers_ - [x] _Passing headers_ - [x] _Importance of the Host header_ - [x] _Redirects and X-Forwarded-Proto_ - [x] _A warning about the X-Forwarded-For_ - [x] _Improve extensibility with Forwarded_ - [x] _Response headers_ - _Load balancing algorithms_ - [x] _Backend parameters_ - [x] _Upstream servers with SSL_ - [x] _Round Robin_ - [x] _Weighted Round Robin_ - [x] _Least Connections_ - [x] _Weighted Least Connections_ - [x] _IP Hash_ - [x] _Generic Hash_ - [ ] _Fair module_ - [x] _Other methods_ - _Rate Limiting_ - [x] _Variables_ - [x] _Directives, keys, and zones_ - [x] _Burst and nodelay parameters_ - _NAXSI Web Application Firewall_ - _OWASP ModSecurity Core Rule Set (CRS)_ - _Other subjects_ - [ ] _Secure Distribution of SSL Private Keys with NGINX_ - _Core modules_ - [x] _ngx_http_geo_module_ - _3rd party modules_ - [x] _ngx_set_misc_ - [x] _ngx_http_geoip_module_
Helpers
- _Installing from source_ - [x] _Automatic installation on RHEL/Debian/BSD_ - [x] _Compiler and linker_ - [x] _Debugging Symbols_ - [x] _SystemTap_ - [x] _stapxx_ - [x] _Separation and improvement of installation methods_ - [x] _Installation Nginx on CentOS 7_ - [x] _Installation OpenResty on CentOS 7_ - [x] _Installation Tengine on Ubuntu 18.04_ - [x] _Installation Nginx on FreeBSD 11.3_ - [x] _Installation Nginx on FreeBSD 11.3 (from ports)_ - _Monitoring_ - [ ] _CollectD, Prometheus, and Grafana_ - [ ] _nginx-vts-exporter_ - [ ] _CollectD, InfluxDB, and Grafana_ - [ ] _Telegraf, InfluxDB, and Grafana_ - _Testing_ - [x] _Build OpenSSL 1.0.2-chacha version_ - [x] _Send request and show response headers_ - [x] _Send request with http method, user-agent, follow redirects and show response headers_ - [x] _Send multiple requests_ - [x] _Testing SSL connection_ - [x] _Testing SSL connection (debug mode)_ - [x] _Testing SSL connection with SNI support_ - [x] _Testing SSL connection with specific SSL version_ - [x] _Testing SSL connection with specific cipher_ - [x] _Verify 0-RTT_ - [x] _Testing SCSV_ - _Load testing with ApacheBench (ab)_ - [x] _Standard test_ - [x] _Test with Keep-Alive header_ - _Load testing with wrk2_ - [x] _Standard scenarios_ - [x] _POST call (with Lua)_ - [x] _Random paths (with Lua)_ - [x] _Multiple paths (with Lua)_ - [x] _Random server address to each thread (with Lua)_ - [x] _Multiple json requests (with Lua)_ - [x] _Debug mode (with Lua)_ - [x] _Analyse data pass to and from the threads_ - [x] _Parsing wrk result and generate report_ - _Load testing with locust_ - [x] _Multiple paths_ - [x] _Multiple paths with different user sessions_ - [x] _TCP SYN flood Denial of Service attack_ - [x] _HTTP Denial of Service attack_ - _Debugging_ - [x] _Show information about processes_ - [x] _Check memory usage_ - [x] _Show open files_ - [x] _Check segmentation fault messages_ - [x] _Dump configuration_ - [x] _Get the list of configure arguments_ - [x] _Check if the module has been compiled_ - [x] _Show the most accessed IP addresses (ip and url)_ - [x] _Show the most requested urls with http methods_ - [x] _Show the most accessed response codes_ - [x] _Calculating requests per second with IP addresses and urls_ - [x] _Check that the gzip_static module is working_ - [x] _Which worker processing current request_ - [x] _Capture only http packets_ - [x] _Extract User Agent from the http packets_ - [x] _Capture only http GET and POST packets_ - [x] _Capture requests and filter by source ip and destination port_ - [x] _Capture HTTP requests/responses in real time, filter by GET, HEAD and save to a file_ - [ ] _Server Side Include (SSI) debugging_ - [x] _Dump a process's memory_ - _GNU Debugger (gdb)_ - [x] _Dump configuration from a running process_ - [x] _Show debug log in memory_ - [x] _Core dump backtrace_ - [x] _Debugging socket leaks_ - _SystemTap cheatsheet_ - [x] _stapxx_ - _Errors & Issues_ - [ ] _Common errors_ - _Configuration snippets_ - [x] _Nginx server header removal_ - [x] _Custom log formats_ - [x] _Log only 4xx/5xx_ - [x] _Restricting access with client certificate_ - [x] _Restricting access by geographical location_ - [x] _GeoIP 2 database_ - [ ] _Custom error pages_ - [x] _Dynamic error pages with SSI_ - [x] _Limiting the rate of requests per IP with geo and map_ - [x] _Using trailing slashes_ - [x] _Properly redirect all HTTP requests to HTTPS_ - [x] _Adding and removing the www prefix_ - [x] _Proxy/rewrite and keep the original URL_ - [x] _Proxy/rewrite and keep the part of original URL_ - [x] _Proxy/rewrite without changing the original URL (in browser)_ - [x] _Modify 301/302 response body_ - [x] _Redirect POST request with payload to external endpoint_ - [x] _Route to different backends based on HTTP method_ - [ ] _Redirect users with certain IP to special location_ - [x] _Allow multiple cross-domains using the CORS headers_ - [x] _Set correct scheme passed in X-Forwarded-Proto_ - [ ] _Securing URLs with the Secure Link Module_ - [ ] _Tips and methods for high load traffic testing (cheatsheet)_ - [ ] _Location matching examples_ - [ ] _Passing requests to the backend_ - [ ] _The HTTP backend server_ - [ ] _The uWSGI backend server_ - [ ] _The FastCGI backend server_ - [ ] _The memcached backend server_ - [ ] _The Redis backend server_ - [ ] _HTTPS traffic to upstream servers_ - [ ] _TCP and UDP load balancing_ - [ ] _Lua snippets_ - [ ] _nginscripts snippets_ - _Other snippets_ - [x] _Recreate base directory_ - [x] _Create a temporary static backend_ - [x] _Create a temporary static backend with SSL support_ - [x] _Generate password file with htpasswd command_ - [x] _Generate private key without passphrase_ - [x] _Generate private key with passphrase_ - [x] _Remove passphrase from private key_ - [x] _Encrypt existing private key with a passphrase_ - [x] _Generate CSR_ - [x] _Generate CSR (metadata from existing certificate)_ - [x] _Generate CSR with -config param_ - [x] _Generate private key and CSR_ - [x] _List available EC curves_ - [x] _Generate ECDSA private key_ - [x] _Generate private key and CSR (ECC)_ - [x] _Generate self-signed certificate_ - [x] _Generate self-signed certificate from existing private key_ - [x] _Generate self-signed certificate from existing private key and csr_ - [x] _Generate multidomain certificate (Certbot)_ - [x] _Generate wildcard certificate (Certbot)_ - [x] _Generate certificate with 4096 bit private key (Certbot)_ - [x] _Generate DH public parameters_ - [x] _Display DH public parameters_ - [x] _Extract certs from p7b_ - [x] _Convert DER to PEM_ - [x] _Convert PEM to DER_ - [x] _Verification of the certificate's supported purposes_ - [x] _Verification of the private key_ - [x] _Check private key_ - [x] _Get public key from private key_ - [x] _Verification of the public key_ - [x] _Verification of the certificate_ - [x] _Verification of the CSR_ - [x] _Check the private key and the certificate are match_ - [x] _TLSv1.3 and CCM ciphers_
Base Rules
- [x] _Format, prettify and indent your Nginx code_ - [x] _Never use a hostname in a listen or upstream directives_ - [x] _Set the HTTP headers with add_header and proxy_*_header directives properly_ - [ ] _Making a rewrite absolute (with scheme)_ - [x] _Use return directive for URL redirection (301, 302)_ - [x] _Use simple custom error pages_ - [x] _Configure log rotation policy_ - [x] _Don't duplicate index directive, use it only in the http block_
Debugging
- [x] _Improve debugging by disable daemon, master process, and all workers except one_ - [x] _Use core dumps to figure out why NGINX keep crashing_ - [x] _Use mirror module to copy requests to another backend_ - [ ] _Dynamic debugging with echo module_ - [ ] _Dynamic debugging with SSI_
Performance
- [x] _Enable OCSP Stapling_ - [ ] _Avoid multiple index directives_ - [x] _Use $request_uri to avoid using regular expressions_ - [x] _Use try_files directive to ensure a file exists_ - [ ] _Don't pass all requests to the backend - use try_files_ - [x] _Use return directive instead of rewrite for redirects_ - [x] _Enable PCRE JIT to speed up processing of regular expressions_ - [ ] _Set proxy timeouts for normal load and under heavy load_ - [ ] _Configure kernel parameters for high load traffic_ - [x] _Activate the cache for connections to upstream servers_
Hardening
- [x] _Keep NGINX up-to-date_ - [x] _Take care about your ACL rules_ - [x] _Use only the latest supported OpenSSL version_ - [x] _Remove support for legacy and risky HTTP request headers_ - [x] _Prevent Replay Attacks on Zero Round-Trip Time_ - [x] _Prevent caching of sensitive data_ - [x] _Limit concurrent connections_ - [ ] _Set properly files and directories permissions (also with acls) on a paths_ - [ ] _Implement HTTPOnly and secure attributes on cookies_
Reverse Proxy
- [x] _Use pass directive compatible with backend protocol_ - [x] _Be careful with trailing slashes in proxy_pass directive_ - [x] _Set and pass Host header only with $host variable_ - [x] _Set properly values of the X-Forwarded-For header_ - [x] _Don't use X-Forwarded-Proto with $scheme behind reverse proxy_ - [x] _Always pass Host, X-Real-IP, and X-Forwarded headers to the backend_ - [x] _Use custom headers without X- prefix_ - [x] _Always use $request_uri instead of $uri in proxy_pass_ - [ ] _Set proxy buffers and timeouts_
Others
- [x] _Set the certificate chain correctly_ - [x] _Define security policies with security.txt_ - [x] _Use tcpdump to diagnose and troubleshoot the HTTP issues_
If you have any idea, send it back to me or add a pull request. ## RSS Feed & Updates GitHub exposes an [RSS/Atom](https://github.com/trimstray/nginx-admins-handbook/commits.atom) feed of the commits, which may also be useful if you want to be kept informed about all changes. ## Checklist to rule them all This checklist was the primary aim of the _nginx-admins-handbook_. It contains a set of best practices and recommendations on how to configure and maintain the NGINX properly. > This checklist contains [all rules (79)](doc/RULES.md) from this handbook. Generally, I think that each of these principles is important and should be considered. I separated them into four levels of priority to help guide your decision. | PRIORITY | NAME | AMOUNT | DESCRIPTION | | :---: | :--- | :---: | :--- | | ![high](static/img/priorities/high.png) | critical | 33 | definitely use this rule, otherwise it will introduce high risks of your NGINX security, performance, and other | | ![medium](static/img/priorities/medium.png) | major | 26 | it's also very important but not critical, and should still be addressed at the earliest possible opportunity | | ![low](static/img/priorities/low.png) | normal | 12 | there is no need to implement but it is worth considering because it can improve the NGINX working and functions | | ![info](static/img/priorities/info.png) | minor | 8 | as an option to implement or use (not required) | Remember, these are only guidelines. My point of view may be different from yours so if you feel these priority levels do not reflect your configurations commitment to security, performance or whatever else, you should adjust them as you see fit. | RULE | CHAPTER | PRIORITY | | :--- | :--- | :---: | | [Define the listen directives with address:port pair](doc/RULES.md#beginner-define-the-listen-directives-with-addressport-pair)
Prevents soft mistakes which may be difficult to debug. | Base Rules | ![high](static/img/priorities/high.png) | | [Prevent processing requests with undefined server names](doc/RULES.md#beginner-prevent-processing-requests-with-undefined-server-names)
It protects against configuration errors, e.g. traffic forwarding to incorrect backends. | Base Rules | ![high](static/img/priorities/high.png) | | [Never use a hostname in a listen or upstream directives](doc/RULES.md#beginner-never-use-a-hostname-in-a-listen-or-upstream-directives)
While this may work, it will comes with a large number of issues. | Base Rules | ![high](static/img/priorities/high.png) | | [Set the HTTP headers with add_header and proxy_*_header directives properly](doc/RULES.md#beginner-set-the-http-headers-with-add_header-and-proxy__header-directives-properly)
Set the right security headers for all contexts. | Base Rules | ![high](static/img/priorities/high.png) | | [Configure log rotation policy](doc/RULES.md#beginner-configure-log-rotation-policy)
Save yourself trouble with your web server: configure appropriate logging policy. | Base Rules | ![high](static/img/priorities/high.png) | | [Use simple custom error pages](doc/RULES.md#beginner-use-simple-custom-error-pages)
Default error pages reveals information which leads to information leakage vulnerability. | Base Rules | ![high](static/img/priorities/high.png) | | [Use HTTP/2](doc/RULES.md#beginner-use-http2)
HTTP/2 will make our applications faster, simpler, and more robust. | Performance | ![high](static/img/priorities/high.png) | | [Always keep NGINX up-to-date](doc/RULES.md#beginner-always-keep-nginx-up-to-date)
Use newest NGINX package to fix vulnerabilities, bugs, and to use new features. | Hardening | ![high](static/img/priorities/high.png) | | [Run as an unprivileged user](doc/RULES.md#beginner-run-as-an-unprivileged-user)
Use the principle of least privilege. This way only master process runs as root. | Hardening | ![high](static/img/priorities/high.png) | | [Protect sensitive resources](doc/RULES.md#beginner-protect-sensitive-resources)
Hidden directories and files should never be web accessible. | Hardening | ![high](static/img/priorities/high.png) | | [Take care about your ACL rules](doc/RULES.md#beginner-take-care-about-your-acl-rules)
Test your access-control lists and to stay secure. | Hardening | ![high](static/img/priorities/high.png) | | [Hide upstream proxy headers](doc/RULES.md#beginner-hide-upstream-proxy-headers)
Don't expose what version of software is running on the server. | Hardening | ![high](static/img/priorities/high.png) | | [Remove support for legacy and risky HTTP request headers](doc/RULES.md#beginner-remove-support-for-legacy-and-risky-http-request-headers)
Supports for the offending headers should be removed. | Hardening | ![high](static/img/priorities/high.png) | | [Force all connections over TLS](doc/RULES.md#beginner-force-all-connections-over-tls)
Protects your website for handle sensitive communications. | Hardening | ![high](static/img/priorities/high.png) | | [Use min. 2048-bit for RSA and 256-bit for ECC](doc/RULES.md#beginner-use-min-2048-bit-for-rsa-and-256-bit-for-ecc)
2048 bit (RSA) or 256 bit (ECC) keys are sufficient for commercial use. | Hardening | ![high](static/img/priorities/high.png) | | [Keep only TLS 1.3 and TLS 1.2](doc/RULES.md#beginner-keep-only-tls-13-and-tls-12)
Use TLS with modern cryptographic algorithms and without protocol weaknesses. | Hardening | ![high](static/img/priorities/high.png) | | [Use only strong ciphers](doc/RULES.md#beginner-use-only-strong-ciphers)
Use only strong and not vulnerable cipher suites. | Hardening | ![high](static/img/priorities/high.png) | | [Use more secure ECDH Curve](doc/RULES.md#beginner-use-more-secure-ecdh-curve)
Use ECDH Curves with according to NIST recommendations. | Hardening | ![high](static/img/priorities/high.png) | | [Use strong Key Exchange with Perfect Forward Secrecy](doc/RULES.md#beginner-use-strong-key-exchange-with-perfect-forward-secrecy)
Establishes a shared secret between two parties that can be used for secret communication. | Hardening | ![high](static/img/priorities/high.png) | | [Defend against the BEAST attack](doc/RULES.md#beginner-defend-against-the-beast-attack)
The server ciphers should be preferred over the client ciphers. | Hardening | ![high](static/img/priorities/high.png) | | [Enable HTTP Strict Transport Security](doc/RULES.md#beginner-enable-http-strict-transport-security)
Tells browsers that it should only be accessed using HTTPS, instead of using HTTP. | Hardening | ![high](static/img/priorities/high.png) | | [Reduce XSS risks (Content-Security-Policy)](doc/RULES.md#beginner-reduce-xss-risks-content-security-policy)
CSP is best used as defence-in-depth. It reduces the harm that a malicious injection can cause. | Hardening | ![high](static/img/priorities/high.png) | | [Control the behaviour of the Referer header (Referrer-Policy)](doc/RULES.md#beginner-control-the-behaviour-of-the-referer-header-referrer-policy)
The default behaviour of referrer leaking puts websites at risk of privacy and security breaches. | Hardening | ![high](static/img/priorities/high.png) | | [Provide clickjacking protection (X-Frame-Options)](doc/RULES.md#beginner-provide-clickjacking-protection-x-frame-options)
Defends against clickjacking attack. | Hardening | ![high](static/img/priorities/high.png) | | [Prevent some categories of XSS attacks (X-XSS-Protection)](doc/RULES.md#beginner-prevent-some-categories-of-xss-attacks-x-xss-protection)
Prevents to render pages if a potential XSS reflection attack is detected. | Hardening | ![high](static/img/priorities/high.png) | | [Prevent Sniff Mimetype middleware (X-Content-Type-Options)](doc/RULES.md#beginner-prevent-sniff-mimetype-middleware-x-content-type-options)
Tells browsers not to sniff MIME types. | Hardening | ![high](static/img/priorities/high.png) | | [Reject unsafe HTTP methods](doc/RULES.md#beginner-reject-unsafe-http-methods)
Only allow the HTTP methods for which you, in fact, provide services. | Hardening | ![high](static/img/priorities/high.png) | | [Prevent caching of sensitive data](doc/RULES.md#beginner-prevent-caching-of-sensitive-data)
It helps to prevent critical data (e.g. credit card details, or username) leaked. | Hardening | ![high](static/img/priorities/high.png) | | [Limit concurrent connections](doc/RULES.md#beginner-limit-concurrent-connections)
Limit concurrent connections to prevent a rogue guys from repeatedly connecting to and monopolizing NGINX. | Hardening | ![high](static/img/priorities/high.png) | | [Use pass directive compatible with backend protocol](doc/RULES.md#beginner-use-pass-directive-compatible-with-backend-protocol)
Set pass directive only to working with compatible backend layer protocol. | Reverse Proxy | ![high](static/img/priorities/high.png) | | [Set properly values of the X-Forwarded-For header](doc/RULES.md#beginner-set-properly-values-of-the-x-forwarded-for-header)
Identify clients communicating with servers located behind the proxy. | Reverse Proxy | ![high](static/img/priorities/high.png) | | [Don't use X-Forwarded-Proto with $scheme behind reverse proxy](doc/RULES.md#beginner-dont-use-x-forwarded-proto-with-scheme-behind-reverse-proxy)
Prevent pass incorrect value of this header. | Reverse Proxy | ![high](static/img/priorities/high.png) | | [Always use $request_uri instead of $uri in proxy_pass](doc/RULES.md#beginner-always-use-request_uri-instead-of-uri-in-proxy_pass)
You should always pass unchanged URI to the backend layer. | Reverse Proxy | ![high](static/img/priorities/high.png) | | [Organising Nginx configuration](doc/RULES.md#beginner-organising-nginx-configuration)
Well organised code is easier to understand and maintain. | Base Rules | ![medium](static/img/priorities/medium.png) | | [Format, prettify and indent your Nginx code](doc/RULES.md#beginner-format-prettify-and-indent-your-nginx-code)
Formatted code is easier to maintain, debug, and can be read and understood in a short amount of time. | Base Rules | ![medium](static/img/priorities/medium.png) | | [Use reload option to change configurations on the fly](doc/RULES.md#beginner-use-reload-option-to-change-configurations-on-the-fly)
Graceful reload of the configuration without stopping the server and dropping any packets. | Base Rules | ![medium](static/img/priorities/medium.png) | | [Use return directive for URL redirection (301, 302)](doc/RULES.md#beginner-use-return-directive-for-url-redirection-301-302)
The by far simplest and fastest because there is no regexp that has to be evaluated. | Base Rules | ![medium](static/img/priorities/medium.png) | | [Maintaining SSL sessions](doc/RULES.md#beginner-maintaining-ssl-sessions)
Improves performance from the clients’ perspective. | Performance | ![medium](static/img/priorities/medium.png) | | [Enable OCSP Stapling](doc/RULES.md#beginner-enable-ocsp-stapling)
Enable to reduce the cost of an OCSP validation. | Performance | ![medium](static/img/priorities/medium.png) | | [Use exact names in a server_name directive if possible](doc/RULES.md#beginner-use-exact-names-in-a-server_name-directive-if-possible)
Helps speed up searching using exact names. | Performance | ![medium](static/img/priorities/medium.png) | | [Avoid checks server_name with if directive](doc/RULES.md#beginner-avoid-checks-server_name-with-if-directive)
It decreases NGINX processing requirements. | Performance | ![medium](static/img/priorities/medium.png) | | [Use $request_uri to avoid using regular expressions](doc/RULES.md#beginner-use-request_uri-to-avoid-using-regular-expressions)
By default, the regex is costly and will slow down the performance. | Performance | ![medium](static/img/priorities/medium.png) | | [Use try_files directive to ensure a file exists](doc/RULES.md#beginner-use-try_files-directive-to-ensure-a-file-exists)
Use it if you need to search for a file, it saving duplication of code also. | Performance | ![medium](static/img/priorities/medium.png) | | [Use return directive instead of rewrite for redirects](doc/RULES.md#beginner-use-return-directive-instead-of-rewrite-for-redirects)
Use return directive to more speedy response than rewrite. | Performance | ![medium](static/img/priorities/medium.png) | | [Enable PCRE JIT to speed up processing of regular expressions](doc/RULES.md#beginner-enable-pcre-jit-to-speed-up-processing-of-regular-expressions)
NGINX with PCRE JIT is much faster than without it. | Performance | ![medium](static/img/priorities/medium.png) | | [Activate the cache for connections to upstream servers](doc/RULES.md#beginner-activate-the-cache-for-connections-to-upstream-servers)
Nginx can now reuse its existing connections (keepalive) per upstream. | Performance | ![medium](static/img/priorities/medium.png) | | [Disable unnecessary modules](doc/RULES.md#beginner-disable-unnecessary-modules)
Limits vulnerabilities, improve performance and memory efficiency. | Hardening | ![medium](static/img/priorities/medium.png) | | [Hide Nginx version number](doc/RULES.md#beginner-hide-nginx-version-number)
Don't disclose sensitive information about NGINX. | Hardening | ![medium](static/img/priorities/medium.png) | | [Hide Nginx server signature](doc/RULES.md#beginner-hide-nginx-server-signature)
Don't disclose sensitive information about NGINX. | Hardening | ![medium](static/img/priorities/medium.png) | | [Use only the latest supported OpenSSL version](doc/RULES.md#beginner-use-only-the-latest-supported-openssl-version)
Stay protected from SSL security threats and don't miss out of new features. | Hardening | ![medium](static/img/priorities/medium.png) | | [Prevent Replay Attacks on Zero Round-Trip Time](doc/RULES.md#beginner-prevent-replay-attacks-on-zero-round-trip-time)
0-RTT is disabled by default but you should know that enabling this option creates a significant security risks. | Hardening | ![medium](static/img/priorities/medium.png) | | [Mitigation of CRIME/BREACH attacks](doc/RULES.md#beginner-mitigation-of-crimebreach-attacks)
Disable HTTP compression or compress only zero sensitive content. | Hardening | ![medium](static/img/priorities/medium.png) | | [Deny the use of browser features (Feature-Policy)](doc/RULES.md#beginner-deny-the-use-of-browser-features-feature-policy)
A mechanism to allow and deny the use of browser features. | Hardening | ![medium](static/img/priorities/medium.png) | | [Control Buffer Overflow attacks](doc/RULES.md#beginner-control-buffer-overflow-attacks)
Prevents errors are characterised by the overwriting of memory fragments of the NGINX process. | Hardening | ![medium](static/img/priorities/medium.png) | | [Mitigating Slow HTTP DoS attacks (Closing Slow Connections)](doc/RULES.md#beginner-mitigating-slow-http-dos-attack-closing-slow-connections)
Prevents attacks in which the attacker sends HTTP requests in pieces slowly. | Hardening | ![medium](static/img/priorities/medium.png) | | [Set and pass Host header only with $host variable](doc/RULES.md#beginner-set-and-pass-host-header-only-with-host-variable)
Use of the $host is the only one guaranteed to have something sensible. | Reverse Proxy | ![medium](static/img/priorities/medium.png) | | [Always pass Host, X-Real-IP, and X-Forwarded headers to the backend](doc/RULES.md#beginner-always-pass-host-x-real-ip-and-x-forwarded-headers-to-the-backend)
It gives you more control of forwarded headers. | Reverse Proxy | ![medium](static/img/priorities/medium.png) | | [Set the certificate chain correctly](doc/RULES.md#beginner-set-the-certificate-chain-correctly)
Send the complete chain to the client. | Others | ![medium](static/img/priorities/medium.png) | | [Enable DNS CAA Policy](doc/RULES.md#beginner-enable-dns-caa-policy)
Allows domain name holders to indicate to CA whether they are authorized to issue digital certificates. | Others | ![medium](static/img/priorities/medium.png) | | [Separate listen directives for 80 and 443 ports](doc/RULES.md#beginner-separate-listen-directives-for-80-and-443-ports)
Help you maintain and modify your configuration. | Base Rules | ![low](static/img/priorities/low.png) | | [Use only one SSL config for the listen directive](doc/RULES.md#beginner-use-only-one-ssl-config-for-the-listen-directive)
Prevents multiple configurations on the same listening address. | Base Rules | ![low](static/img/priorities/low.png) | | [Use geo/map modules instead of allow/deny](doc/RULES.md#beginner-use-geomap-modules-instead-of-allowdeny)
Provides the perfect way to block invalid visitors. | Base Rules | ![low](static/img/priorities/low.png) | | [Set global root directory for unmatched locations](doc/RULES.md#beginner-set-global-root-directory-for-unmatched-locations)
Specifies the root directory for an undefined locations. | Base Rules | ![low](static/img/priorities/low.png) | | [Don't duplicate index directive, use it only in the http block](doc/RULES.md#beginner-dont-duplicate-index-directive-use-it-only-in-the-http-block)
Watch out for duplicating the same rules. | Base Rules | ![low](static/img/priorities/low.png) | | [Adjust worker processes](doc/RULES.md#beginner-adjust-worker-processes)
You can adjust this value to maximum throughput under high concurrency. | Performance | ![low](static/img/priorities/low.png) | | [Make an exact location match to speed up the selection process](doc/RULES.md#beginner-make-an-exact-location-match-to-speed-up-the-selection-process)
Exact location matches are often used to speed up the selection process. | Performance | ![low](static/img/priorities/low.png) | | [Use limit_conn to improve limiting the download speed](doc/RULES.md#beginner-use-limit_conn-to-improve-limiting-the-download-speed)
Limits NGINX download speed per connection. | Performance | ![low](static/img/priorities/low.png) | | [Be careful with trailing slashes in proxy_pass directive](doc/RULES.md#beginner-be-careful-with-trailing-slashes-in-proxy_pass-directive)
Incorrect setting could end up with some strange url. | Reverse Proxy | ![low](static/img/priorities/low.png) | | [Use custom headers without X- prefix](doc/RULES.md#beginner-use-custom-headers-without-x--prefix)
The use of custom headers with X- prefix is discouraged. | Reverse Proxy | ![low](static/img/priorities/low.png) | | [Tweak passive health checks](doc/RULES.md#beginner-tweak-passive-health-checks)
Improve behaviour of the passive health checks. | Load Balancing | ![low](static/img/priorities/low.png) | | [Define security policies with security.txt](doc/RULES.md#beginner-define-security-policies-with-securitytxt)
Helps make things easier for companies and security researchers. | Others | ![low](static/img/priorities/low.png) | | [Map all the things...](doc/RULES.md#beginner-map-all-the-things)
Map module provides a more elegant solution for clearly parsing a big list of regexes. | Base Rules | ![info](static/img/priorities/info.png) | | [Use custom log formats](doc/RULES.md#beginner-use-custom-log-formats)
This is extremely helpful for debugging specific location directives. | Debugging | ![info](static/img/priorities/info.png) | | [Use debug mode to track down unexpected behaviour](doc/RULES.md#beginner-use-debug-mode-to-track-down-unexpected-behaviour)
There's probably more detail than you want, but that can sometimes be a lifesaver. | Debugging | ![info](static/img/priorities/info.png) | | [Improve debugging by disable daemon, master process, and all workers except one](doc/RULES.md#beginner-improve-debugging-by-disable-daemon-master-process-and-all-workers-except-one)
This simplifies the debugging and lets test configurations rapidly. | Debugging | ![info](static/img/priorities/info.png) | | [Use core dumps to figure out why NGINX keep crashing](doc/RULES.md#beginner-use-core-dumps-to-figure-out-why-nginx-keep-crashing)
Enable core dumps when your NGINX instance receive an unexpected error or when it crashed. | Debugging | ![info](static/img/priorities/info.png) | | [Use mirror module to copy requests to another backend](doc/RULES.md#beginner-use-mirror-module-to-copy-requests-to-another-backend)
Use mirroring for investigation and debugging of any original request. | Debugging | ![info](static/img/priorities/info.png) | | [Don't disable backends by comments, use down parameter](doc/RULES.md#beginner-dont-disable-backends-by-comments-use-down-parameter)
Is a good solution to marks the server as permanently unavailable. | Load Balancing | ![info](static/img/priorities/info.png) | | [Use tcpdump to diagnose and troubleshoot the HTTP issues](doc/RULES.md#beginner-use-tcpdump-to-diagnose-and-troubleshoot-the-http-issues)
Use tcpdump to monitor HTTP. | Others | ![info](static/img/priorities/info.png) | # Bonus Stuff You can find here a few of the different things I've worked and included to this repository. I hope that these extras will be useful. ## Configuration reports Many of these recipes have been applied to the configuration of my old private website. > An example configuration is in the [configuration examples](#configuration-examples) chapter. It's also based on [this](https://github.com/trimstray/nginx-admins-handbook/blob/master/static/img/cheatsheets/nginx-hardening-cheatsheet-tls13.png) version of printable high-res hardening cheatsheets. ### SSL Labs > Read about SSL Labs grading [here](https://community.qualys.com/docs/DOC-6321-ssl-labs-grading-2018) (SSL Labs Grading 2018). Short SSL Labs grades explanation: > _A+ is clearly the desired grade, both A and B grades are acceptable and result in adequate commercial security. The B grade, in particular, may be applied to configurations designed to support very wide audiences (for old clients)_. I finally got **A+** grade and following scores: - Certificate = **100%** - Protocol Support = **100%** - Key Exchange = **90%** - Cipher Strength = **90%** Look also at the following recommendations. I believe the right configuration of NGINX should give the following SSL Labs scores and provides the best security for the most cases: - **Recommended** - A/A+ - Certificate: 100/100 - Protocol Support: 95/100 - Key Exchange: 90/100 - Cipher Strength: 90/100 - **Perfect but restrictive** - A+ - Certificate: 100/100 - Protocol Support: 100/100 - Key Exchange: 100/100 - Cipher Strength: 100/100

blkcipher_ssllabs_preview

Something about SSL Labs grading mechanism (that's an interesting point of view): > _The whole grading mechanism is more propaganda and public relations than actual security. If you want good security, then you must mind the details and understand how things work internally. If you want a good grade then you should do whatever it takes to have a good grade. An "A+" from SSL Labs is a very nifty thing to add at the end of a report, but it does not really equate with having rock solid security. Having an "A+" equates with being able to say "I have an A+"._ - from [this](https://security.stackexchange.com/a/112539) answer by [Tom Leek](https://security.stackexchange.com/users/5411/tom-leek). ### Mozilla Observatory > Read about Mozilla Observatory [here](https://observatory.mozilla.org/faq/) and about [Observatory Scoring Methodology](https://github.com/mozilla/http-observatory/blob/master/httpobs/docs/scoring.md). I also got the highest summary note (**A+**) on the Observatory with a very high test score (120/100, max. 135/100):

blkcipher_mozilla_observatory_preview

## Printable hardening cheatsheets I created two versions of printable posters with hardening cheatsheets (High-Res 5000x8800) based on recipes from this handbook: > For `xcf` and `pdf` formats please see [this](https://github.com/trimstray/nginx-admins-handbook/tree/master/static/img) directory. - **A+** with all **100%’s** on @ssllabs and **120/100** on @mozilla observatory: > It provides the highest scores of the SSL Labs test. Setup is very restrictive with 4096-bit private key, only TLS 1.2, and also modern strict TLS cipher suites (non 128-bits). Think carefully about its use (no TLS 1.3, restrictive cipher suites), in my opinion, it is only suitable for obtaining the highest possible rating and seems a little impractical.

nginx-hardening-cheatsheet-100p

- **A+** on @ssllabs and **120/100** on @mozilla observatory with TLS 1.3 support: > It provides less restrictive setup with 2048-bit key for `RSA` or 256-bit key for `ECC`, TLS 1.3 and 1.2, modern strict TLS cipher suites (128/256-bits), and 2048-bit predefined `DH` groups recommended by Mozilla. The final grade is also in line with the industry standards and guidance. Recommend using this, for me, it is very reasonable configuration.

nginx-hardening-cheatsheet-tls13

## Fully automatic installation I created a set of scripts for unattended installation of NGINX from the raw, uncompiled code. It allows you to easily install, create a setup for dependencies (like `zlib` or `openssl`), and customized with installation parameters. For more information please see [Installing from source - Automatic installation](https://github.com/trimstray/nginx-admins-handbook/tree/master/lib) chapter which describes the installation of NGINX on systems/distros such as Ubuntu, Debian, CentOS, and FreeBSD. ## Static error pages generator I created a simple to use generator for static pages to replace the default error pages that comes with any web server like NGINX. For more information please see [HTTP Static Error Pages Generator](https://github.com/trimstray/nginx-admins-handbook/tree/master/lib/nginx/snippets/http-error-pages#http-static-error-pages-generator). ## Server names parser I added scripts for fast multiple domain searching in the configuration. These tools get specific `server_name` matches and print them on the screen as a `server { ... }` blocks. Both are very helpful if you really have tons of domains or if you want to list specific vhosts from file or the active configuration. You must follow one important rule to be able to use it. Your server block must have the following structure: ```nginx server { server_name example.com example.org; ... # other directives } ``` Example of use: ``` ./snippets/server-name-parser/check-server-name.sh example.com Searching 'example.com' in '/usr/local/etc/nginx' (from disk) /usr/local/etc/nginx/domains/example.com/servers.conf:79: return 301 https://example.com$request_uri; /usr/local/etc/nginx/domains/example.com/servers.conf:252: return 301 https://example.com$request_uri; /usr/local/etc/nginx/domains/example.com/servers.conf:3825: server_name example.com; Searching 'example.com' in server contexts (from a running process) >>>>>>>>>> BEG >>>>>>>>>> server { include listen/192.168.252.10/https.example.com.conf; server_name example.com; location / { return 204 "RFC 792"; } access_log /var/log/nginx/example.com/access.log standard; error_log /var/log/nginx/example.com/error.log warn; } <<<<<<<<<< END <<<<<<<<<< ``` For more information please see [snippets/server-name-parser](https://github.com/trimstray/nginx-admins-handbook/tree/master/lib/nginx/snippets/server-name-parser) directory. # Books #### [Nginx Essentials](https://www.amazon.com/Nginx-Essentials-Valery-Kholodkov/dp/1785289535) Authors: **Valery Kholodkov** _Excel in Nginx quickly by learning to use its most essential features in real-life applications._ - _Learn how to set up, configure, and operate an Nginx installation for day-to-day use_ - _Explore the vast features of Nginx to manage it like a pro, and use them successfully to run your website_ - _Example-based guide to get the best out of Nginx to reduce resource usage footprint_ This short review comes from this book or the store. #### [Nginx Cookbook](https://www.oreilly.com/library/view/nginx-cookbook/9781492049098/) Authors: **Derek DeJonghe** _You’ll find recipes for:_ - _Traffic management and A/B testing_ - _Managing programmability and automation with dynamic templating and the NGINX Plus API_ - _Securing access through encrypted traffic, secure links, HTTP authentication subrequests, and more_ - _Deploying NGINX to AWS, Azure, and Google cloud-computing services_ - _Using Docker to deploy containers and microservices_ - _Debugging and troubleshooting, performance tuning, and practical ops tips_ This short review comes from this book or the store. #### [Nginx HTTP Server](https://www.amazon.com/Nginx-HTTP-Server-Harness-infrastructure/dp/178862355X) Authors: **Martin Fjordvald**, **Clement Nedelcu** _Harness the power of Nginx to make the most of your infrastructure and serve pages faster than ever._ - _Discover possible interactions between Nginx and Apache to get the best of both worlds_ - _Learn to exploit the features offered by Nginx for your web applications_ - _Get your hands on the most updated version of Nginx (1.13.2) to support all your web administration requirements_ This short review comes from this book or the store. #### [Nginx High Performance](https://www.amazon.com/Nginx-High-Performance-Rahul-Sharma/dp/1785281836) Authors: **Rahul Sharma** _Optimize NGINX for high-performance, scalable web applications._ - _Configure Nginx for best performance, with configuration examples and explanations_ - _Step-by-step tutorials for performance testing using open source software_ - _Tune the TCP stack to make the most of the available infrastructure_ This short review comes from this book or the store. #### [Mastering Nginx](https://www.amazon.com/Mastering-Nginx-Dimitri-Aivaliotis/dp/1849517444) Authors: **Dimitri Aivaliotis** _Written for experienced systems administrators and engineers, this book teaches you from scratch how to configure Nginx for any situation. Step-by-step instructions and real-world code snippets clarify even the most complex areas._ This short review comes from this book or the store. #### [ModSecurity 3.0 and NGINX: Quick Start Guide](https://www.nginx.com/resources/library/modsecurity-3-nginx-quick-start-guide/) Authors: **Faisal Memon**, **Owen Garrett**, **Michael Pleshakov** _Learn in this ebook how to get started with ModSecurity, the world’s most widely deployed web application firewall (WAF), now available for NGINX and NGINX Plus._ This short review comes from this book or the store. #### [Cisco ACE to NGINX: Migration Guide](https://www.nginx.com/resources/library/cisco-ace-nginx-migration-guide/) Authors: **Faisal Memon** _This ebook provides step-by-step instructions on replacing Cisco ACE with NGINX and off-the-shelf servers. NGINX helps you cut costs and modernize._ _In this ebook you will learn:_ - _How to migrate Cisco ACE configuration to NGINX, with detailed examples_ - _Why you should go with a software load balancer, and not hardware_ This short review comes from this book or the store. # External Resources ##### Nginx official

  :black_small_square: Nginx Project
  :black_small_square: Nginx Documentation
  :black_small_square: Nginx Wiki
  :black_small_square: Nginx Admin's Guide
  :black_small_square: Nginx Pitfalls and Common Mistakes
  :black_small_square: Development Guide
  :black_small_square: Nginx Forum
  :black_small_square: Nginx Security Advisories
  :black_small_square: Nginx Security Controls
  :black_small_square: Nginx Mailing List
  :black_small_square: Nginx Read-only Mirror
  :black_small_square: NGINX-Demos
  :black_small_square: Thread Pools in NGINX Boost Performance 9x!

##### Nginx distributions

  :black_small_square: OpenResty
  :black_small_square: The Tengine Web Server

##### Comparison reviews

  :black_small_square: NGINX vs. Apache (Pro/Con Review, Uses, & Hosting for Each)
  :black_small_square: Web cache server performance benchmark: nuster vs nginx vs varnish vs squid

##### Cheatsheets & References

  :black_small_square: agentzh's Nginx Tutorials
  :black_small_square: Introduction to nginx.conf scripting
  :black_small_square: Nginx discovery journey
  :black_small_square: Nginx Guts
  :black_small_square: Nginx Cheatsheet
  :black_small_square: Nginx Tutorials, Linux Sysadmin Configuration & Optimizing Tips and Tricks
  :black_small_square: Nginx boilerplate configs
  :black_small_square: Awesome Nginx configuration template
  :black_small_square: Nginx Quick Reference
  :black_small_square: A collection of resources covering Nginx and more
  :black_small_square: A collection of useful Nginx configuration snippets
  :black_small_square: Nginx configurations for most popular CMS/CMF/Frameworks based on PHP
  :black_small_square: Boilerplate configuration for nginx and certbot with docker-compose

##### Performance & Hardening

  :black_small_square: Nginx Tuning For Best Performance by Denji
  :black_small_square: Nginx Optimization: understanding sendfile, tcp_nodelay and tcp_nopush
  :black_small_square: How we scaled nginx and saved the world 54 years every day
  :black_small_square: TLS has exactly one performance problem: it is not used widely enough
  :black_small_square: SSL/TLS Deployment Best Practices
  :black_small_square: SSL Server Rating Guide
  :black_small_square: SSL Pulse
  :black_small_square: How to Build a Tough NGINX Server in 15 Steps
  :black_small_square: Top 25 Nginx Web Server Best Security Practices
  :black_small_square: Nginx Secure Web Server
  :black_small_square: Strong SSL Security on Nginx
  :black_small_square: Enable cross-origin resource sharing (CORS)
  :black_small_square: NAXSI - WAF for Nginx
  :black_small_square: ModSecurity for Nginx

##### Presentations & Videos

  :black_small_square: NGINX: Basics and Best Practices
  :black_small_square: NGINX Installation and Tuning
  :black_small_square: Nginx Internals (by Joshua Zhu)
  :black_small_square: Nginx internals (by Liqiang Xu)
  :black_small_square: How to secure your web applications with NGINX
  :black_small_square: Tuning TCP and NGINX on EC2
  :black_small_square: Extending functionality in nginx, with modules!
  :black_small_square: Nginx - Tips and Tricks.
  :black_small_square: Nginx Scripting - Extending Nginx Functionalities with Lua
  :black_small_square: How to handle over 1,200,000 HTTPS Reqs/Min
  :black_small_square: Using ngx_lua / lua-nginx-module in pixiv
  :black_small_square: Reading nginx CHANGES together
  :black_small_square: Dynamic modules:how it works
  :black_small_square: NGINX Conf 2014
  :black_small_square: NGINX Conf 2015
  :black_small_square: NGINX Conf 2016
  :black_small_square: NGINX Conf 2017
  :black_small_square: NGINX Conf 2018 | Deep Dive Track
  :black_small_square: NGINX Conf 2018 | Keynotes and Sessions
  :black_small_square: Making HTTPS Fast(er): Ilya Grigorik @ nginx.conf 2014

##### Playgrounds

  :black_small_square: NGINX Rate Limit, Burst and nodelay sandbox

##### Config generators

  :black_small_square: nginxconfig - Nginx config generator on steroids.
  :black_small_square: ssl-config-generator - Mozilla SSL Configuration Generator.
  :black_small_square: nginx-config-builder - is a python library for building nginx configuration files programatically.

##### Config parsers

  :black_small_square: crossplane - quick and reliable way to convert NGINX configurations into JSON and back.
  :black_small_square: nginxparser - parses nginx configuration with Pyparsing.

##### Config managers

  :black_small_square: ansible-role-nginx - asible role to install and manage nginx configuration.
  :black_small_square: ansible-role-nginx - installs and configures the latest version of Nginx.
  :black_small_square: ansible-role-nginx - installs NGINX, NGINX Plus, the NGINX Amplify agent, and more.
  :black_small_square: puppet-nginx - puppet module to manage NGINX on various UNIXes.

##### Static analyzers

  :black_small_square: gixy - is a tool to analyze Nginx configuration to prevent security misconfiguration and automate flaw detection.
  :black_small_square: nginx-config-formatter - Nginx config file formatter/beautifier written in Python.
  :black_small_square: nginxbeautifier - format and beautify Nginx config files.
  :black_small_square: nginx-minify-conf - creates a minified version of a Nginx configuration.

##### Log analyzers

  :black_small_square: GoAccess - is a fast, terminal-based log analyzer (quickly analyze and view web server statistics in real time).
  :black_small_square: Graylog - is a leading centralized log management for capturing, storing, and enabling real-time analysis.
  :black_small_square: Logstash - is an open source, server-side data processing pipeline.

##### Performance analyzers

  :black_small_square: ngxtop - parses your Nginx access log and outputs useful, top-like, metrics of your Nginx server.

##### Builder tools

  :black_small_square: Nginx-builder - is a tool for building deb or rpm package NGINX from the source code.

##### Benchmarking tools

  :black_small_square: ab - is a single-threaded command line tool for measuring the performance of HTTP web servers.
  :black_small_square: siege - is an http load testing and benchmarking utility.
  :black_small_square: wrk - is a modern HTTP benchmarking tool capable of generating significant load.
  :black_small_square: wrk2 - is a constant throughput, correct latency recording variant of wrk.
  :black_small_square: vegeta - HTTP load testing tool and library.
  :black_small_square: bombardier - is a HTTP(S) benchmarking tool.
  :black_small_square: gobench - is a HTTP/HTTPS load testing and benchmarking tool.
  :black_small_square: hey - is a HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom.
  :black_small_square: boom - is a script you can use to quickly smoke-test your web app deployment.
  :black_small_square: httperf - the httperf HTTP load generator.
  :black_small_square: JMeter™ - is designed to load test functional behavior and measure performance.
  :black_small_square: Gatling - is a powerful open-source load and performance testing tool for web applications.
  :black_small_square: locust - is an easy-to-use, distributed, user load testing tool.
  :black_small_square: slowloris - low bandwidth DoS tool. Slowloris rewrite in Python.
  :black_small_square: slowhttptest - application layer DoS attack simulator.
  :black_small_square: GoldenEye - GoldenEye Layer 7 (KeepAlive+NoCache) DoS test tool.

##### Debugging tools

  :black_small_square: strace - is a diagnostic, debugging and instructional userspace utility (linux syscall tracer) for Linux.
  :black_small_square: GDB - allows you to see what is going on `inside' another program while it executes.
  :black_small_square: SystemTap - provides infrastructure to simplify the gathering of information about the running Linux system.
  :black_small_square: stapxx - simple macro language extensions to SystemTap.
  :black_small_square: htrace.sh - is a simple Swiss Army knife for http/https troubleshooting and profiling.

##### Security & Web testing tools

  :black_small_square: Burp Suite - is a graphical tool for testing Web application security.
  :black_small_square: w3af - is a Web Application Attack and Audit Framework.
  :black_small_square: nikto - web server scanner which performs comprehensive tests.
  :black_small_square: ssllabs-scan - client for SSL Labs APIs, designed for automated and/or bulk testing.
  :black_small_square: http-observatory - Mozilla HTTP Observatory.
  :black_small_square: testssl.sh - checks a server's service on any port for the support of TLS/SSL ciphers.
  :black_small_square: sslyze - is a fast and powerful SSL/TLS server scanning library.
  :black_small_square: cipherscan - is a very simple way to find out which SSL ciphersuites are supported by a target.
  :black_small_square: O-Saft - OWASP SSL advanced forensic tool.
  :black_small_square: Nghttp2 - is an implementation of HTTP/2 and its header compression algorithm HPACK in C.
  :black_small_square: h2spec - is a conformance testing tool for HTTP/2 implementation.
  :black_small_square: h2t - is a simple tool to help sysadmins to hardening their websites.
  :black_small_square: http2fuzz - HTTP/2 fuzzer written in Golang.
  :black_small_square: Arjun - HTTP parameter discovery suite.
  :black_small_square: Corsy - CORS misconfiguration scanner.
  :black_small_square: XSStrike - most advanced XSS scanner.

##### Development

  :black_small_square: Sample ebook generated from NGINX source code.
  :black_small_square: Programming in Lua (first edition)
  :black_small_square: Scripting Nginx with Lua
  :black_small_square: Emiller’s Guide To Nginx Module Development
  :black_small_square: Emiller’s Advanced Topics In Nginx Module Development
  :black_small_square: NGINX Tutorial: Developing Modules
  :black_small_square: An Introduction To OpenResty (nginx + lua) - Part 1
  :black_small_square: An Introduction To OpenResty - Part 2 - Concepts
  :black_small_square: An Introduction To OpenResty - Part 3
  :black_small_square: OpenResty (Nginx) with dynamically generated certificates
  :black_small_square: Programming OpenResty

##### Online & Web tools

  :black_small_square: SSL Server Test by SSL Labs
  :black_small_square: Test SSL/TLS (PCI DSS, HIPAA and NIST)
  :black_small_square: SSL analyzer and certificate checker
  :black_small_square: Tools for testing SSL configuration
  :black_small_square: Test your TLS server configuration (e.g. ciphers)
  :black_small_square: Scan your website for non-secure content
  :black_small_square: Analyze website security
  :black_small_square: TLS Cipher Suite Search
  :black_small_square: SSL/TLS Capabilities of Your Browser
  :black_small_square: SSL-Client Info's
  :black_small_square: Public Diffie-Hellman Parameter Service/Tool
  :black_small_square: Analyse the HTTP response headers by Security Headers
  :black_small_square: Analyze your website by Mozilla Observatory
  :black_small_square: CAA Record Helper
  :black_small_square: Linting tool that will help you with your site's accessibility, speed, security and more
  :black_small_square: Service to scan and analyse websites
  :black_small_square: Tool from above to either encode or decode a string of text
  :black_small_square: Online translator for search queries on log data
  :black_small_square: Online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript
  :black_small_square: Online tool to learn, build, & test Regular Expressions
  :black_small_square: Online Regex Tester & Debugger
  :black_small_square: Tool for testing regular expressions directly within an NGINX configuration
  :black_small_square: A web app for encryption, encoding, compression and data analysis
  :black_small_square: Nginx location match tester
  :black_small_square: Nginx location match visible

##### Other stuff

  :black_small_square: Web technology for developers
  :black_small_square: Mozilla Web Security
  :black_small_square: Application Security Wiki
  :black_small_square: OWASP ASVS 3.0.1
  :black_small_square: OWASP ASVS 3.0.1 Web App
  :black_small_square: OWASP ASVS 4.0
  :black_small_square: OWASP Top 10 Proactive Controls 2018.
  :black_small_square: OWASP Testing Guide v4
  :black_small_square: OWASP Dev Guide
  :black_small_square: Transport Layer Protection Cheat Sheet by OWASP
  :black_small_square: OWASP WSTG
  :black_small_square: Security/Server Side TLS by Mozilla
  :black_small_square: Applied Crypto Hardening
  :black_small_square: Browser support tables for modern web technologies
  :black_small_square: Memorable site for testing clients against bad SSL configs
  :black_small_square: The HTTPS-Only Standard
  :black_small_square: The Web Security Academy
  :black_small_square: Burp Scanner - Issue Definitions
  :black_small_square: Web application security: what to do when...
  :black_small_square: Transport Layer Security (TLS) Parameters
  :black_small_square: TLS Redirection (and Virtual Host Confusion)
  :black_small_square: TLS Security 6: Examples of TLS Vulnerabilities and Attacks
  :black_small_square: Guidelines for Setting Security Headers
  :black_small_square: Mozilla Guidelines - Web Security
  :black_small_square: Secure your web application with these HTTP headers
  :black_small_square: Security HTTP Headers
  :black_small_square: Analysis of various reverse proxies, cache proxies, load balancers, etc.
  :black_small_square: How HTTPS works ...in a comic!
  :black_small_square: Regular-Expressions
  :black_small_square: Regexp Security Cheatsheet
  :black_small_square: HTTPS on Stack Overflow: The End of a Long Road
  :black_small_square: The Architecture of Open Source Applications - Nginx
  :black_small_square: BBC Digital Media Distribution: How we improved throughput by 4x
  :black_small_square: The C10K problem by Dan Kegel
  :black_small_square: The Secret To 10 Million Concurrent Connections
  :black_small_square: High Performance Browser Networking
  :black_small_square: The System Design Primer
  :black_small_square: awesome-scalability
  :black_small_square: Web Architecture 101
  :black_small_square: Learn where some of the network sysctl variables fit into the Linux/Kernel network flow
  :black_small_square: jemalloc vs tcmalloc vs dlmalloc
  :black_small_square: On the Impact of Memory Allocation on High-Performance Query Processing
  :black_small_square: GLB: GitHub’s open source load balancer

# What's next? Go back to the [Table of Contents](#table-of-contents) or read the next chapters: - **[HTTP Basics](doc/HTTP_BASICS.md#http-basics)** > Introduction to HTTP. - **[SSL/TLS Basics](doc/SSL_TLS_BASICS.md#ssltls-basics)** > Introduction to SSL/TLS. - **[NGINX Basics](doc/NGINX_BASICS.md#nginx-basics)** > Introduction and explanation of the NGINX mechanisms. - **[Helpers](doc/HELPERS.md#helpers)** > One-liners, commands, utilities for building NGINX, and more. - **[Base Rules (16)](doc/RULES.md#base-rules)** > The basic set of rules to keep NGINX in a good condition. - **[Debugging (5)](doc/RULES.md#debugging)** > A few things for troubleshooting configuration problems. - **[Performance (13)](doc/RULES.md#performance)** > Many methods to make sure the NGINX as fast as possible. - **[Hardening (31)](doc/RULES.md#hardening)** > Security and hardening methods in line with best practices. - **[Reverse Proxy (8)](doc/RULES.md#reverse-proxy)** > A few rules about the NGINX proxy server. - **[Load Balancing (2)](doc/RULES.md#load-balancing)** > Some rules to improve NGINX as a load balancer. - **[Others (4)](doc/RULES.md#others)** > Other interesting rules, not necessarily linked to NGINX. - **[Configuration Examples](doc/EXAMPLES.md#configuration-examples)** > Here are some configuration examples. ----

Meme

================================================ FILE: doc/EXAMPLES.md ================================================ # Configuration Examples Go back to the **[⬆ Table of Contents](https://github.com/trimstray/nginx-admins-handbook#table-of-contents)** or **[⬆ What's next?](https://github.com/trimstray/nginx-admins-handbook#whats-next)** section. - **[≡ Configuration Examples](#examples)** * [Reverse Proxy](#reverse-proxy) * [Installation](#installation) * [Configuration](#configuration) * [Import configuration](#import-configuration) * [Set bind IP address](#set-bind-ip-address) * [Set your domain name](#set-your-domain-name) * [Regenerate private keys and certs](#regenerate-private-keys-and-certs) * [Update modules list](#update-modules-list) * [Generating the necessary error pages](#generating-the-necessary-error-pages) * [Add new domain](#add-new-domain) * [Test your configuration](#test-your-configuration) > Remember to make a copy of the current configuration and all files/directories. This chapter is still work in progress. ## Installation I used step-by-step tutorial from this handbook [Installing from source](HELPERS.md#installing-from-source). ## Configuration I used Google Cloud instance with following parameters: | ITEM | VALUE | COMMENT | | :--- | :--- | :--- | | VM | Google Cloud Platform | | | vCPU | 2x | | | Memory | 4096MB | | | HTTP | Varnish on port 80 | | | HTTPS | NGINX on port 443 | | ## Reverse Proxy This chapter describes the basic configuration of my proxy server (for [blkcipher.info](https://blkcipher.info) domain). > Configuration is based on the [installation from source](HELPERS.md#installing-from-source) chapter. If you go through the installation process step by step you can use the following configuration (minor adjustments may be required). #### Import configuration It's very simple - clone the repo, backup your current configuration and perform full directory sync: ```bash git clone https://github.com/trimstray/nginx-admins-handbook tar czvfp ~/nginx.etc.tgz /etc/nginx && mv /etc/nginx /etc/nginx.old rsync -avur lib/nginx/ /etc/nginx/ ``` > If you compiled NGINX from source you should also update/refresh modules. All compiled modules are stored in `/usr/local/src/nginx-${ngx_version}/master/objs` and installed in accordance with the value of the `--modules-path` variable. #### Set bind IP address ###### Find and replace 192.168.252.2 string in directory and file names ```bash cd /etc/nginx find . -depth -not -path '*/\.git*' -name '*192.168.252.2*' -execdir bash -c 'mv -v "$1" "${1//192.168.252.2/xxx.xxx.xxx.xxx}"' _ {} \; ``` ###### Find and replace 192.168.252.2 string in configuration files ```bash cd /etc/nginx find . -not -path '*/\.git*' -type f -print0 | xargs -0 sed -i 's/192.168.252.2/xxx.xxx.xxx.xxx/g' ``` #### Set your domain name ###### Find and replace blkcipher.info string in directory and file names ```bash cd /etc/nginx find . -not -path '*/\.git*' -depth -name '*blkcipher.info*' -execdir bash -c 'mv -v "$1" "${1//blkcipher.info/example.com}"' _ {} \; ``` ###### Find and replace blkcipher.info string in configuration files ```bash cd /etc/nginx find . -not -path '*/\.git*' -type f -print0 | xargs -0 sed -i 's/blkcipher_info/example_com/g' find . -not -path '*/\.git*' -type f -print0 | xargs -0 sed -i 's/blkcipher.info/example.com/g' ``` #### Regenerate private keys and certs ###### For localhost ```bash cd /etc/nginx/master/_server/localhost/certs # Private key + Self-signed certificate: ( _fd="localhost.key" ; _fd_crt="nginx_localhost_bundle.crt" ; \ openssl req -x509 -newkey rsa:2048 -keyout ${_fd} -out ${_fd_crt} -days 365 -nodes \ -subj "/C=X0/ST=localhost/L=localhost/O=localhost/OU=X00/CN=localhost" ) ``` ###### For `default_server` ```bash cd /etc/nginx/master/_server/defaults/certs # Private key + Self-signed certificate: ( _fd="defaults.key" ; _fd_crt="nginx_defaults_bundle.crt" ; \ openssl req -x509 -newkey rsa:2048 -keyout ${_fd} -out ${_fd_crt} -days 365 -nodes \ -subj "/C=X1/ST=default/L=default/O=default/OU=X11/CN=default_server" ) ``` ###### For your domain (e.g. Let's Encrypt) ```bash cd /etc/nginx/master/_server/example.com/certs # For multidomain: certbot certonly -d example.com -d www.example.com --rsa-key-size 2048 # For wildcard: certbot certonly --manual --preferred-challenges=dns -d example.com -d *.example.com --rsa-key-size 2048 # Copy private key and chain: cp /etc/letsencrypt/live/example.com/fullchain.pem nginx_example.com_bundle.crt cp /etc/letsencrypt/live/example.com/privkey.pem example.com.key ``` #### Update modules list Update modules list and include `modules.conf` to your configuration: ```bash _mod_dir="/etc/nginx/modules" :>"${_mod_dir}.conf" for _module in $(ls "${_mod_dir}/") ; do echo -en "load_module\t\t${_mod_dir}/$_module;\n" >> "${_mod_dir}.conf" ; done ``` #### Generating the necessary error pages > In the example (`lib/nginx`) error pages are included from `lib/nginx/master/_static/errors.conf` file. - default location: `/etc/nginx/html`: ``` 50x.html index.html ``` - custom location: `/usr/share/www`: ```bash cd /etc/nginx/snippets/http-error-pages ./httpgen # You can also sync sites/ directory with /etc/nginx/html: # rsync -var sites/ /etc/nginx/html/ rsync -var sites/ /usr/share/www/ ``` #### Add new domain ###### Updated `nginx.conf` ```nginx # At the end of the file (in 'IPS/DOMAINS' section): include /etc/nginx/master/_server/domain.com/servers.conf; include /etc/nginx/master/_server/domain.com/backends.conf; ``` ###### Init domain directory ```bash cd /etc/nginx/master/_server cp -R example.com domain.com cd domain.com find . -not -path '*/\.git*' -depth -name '*example.com*' -execdir bash -c 'mv -v "$1" "${1//example.com/domain.com}"' _ {} \; find . -not -path '*/\.git*' -type f -print0 | xargs -0 sed -i 's/example_com/domain_com/g' find . -not -path '*/\.git*' -type f -print0 | xargs -0 sed -i 's/example.com/domain.com/g' ``` #### Create log directories ```bash mkdir -p /var/log/nginx/localhost mkdir -p /var/log/nginx/defaults mkdir -p /var/log/nginx/others mkdir -p /var/log/nginx/domains/blkcipher.info chown -R nginx:nginx /var/log/nginx ``` #### Logrotate configuration ```bash cp /etc/nginx/snippets/logrotate.d/nginx /etc/logrotate.d/ ``` #### Test your configuration ```bash nginx -t -c /etc/nginx/nginx.conf ``` ================================================ FILE: doc/HELPERS.md ================================================ # Helpers Go back to the **[Table of Contents](https://github.com/trimstray/nginx-admins-handbook#table-of-contents)** or **[What's next?](https://github.com/trimstray/nginx-admins-handbook#whats-next)** section. - **[≡ Helpers](#helpers)** * [Installing from prebuilt packages](#installing-from-prebuilt-packages) * [RHEL7 or CentOS 7](#rhel7-or-centos-7) * [Debian or Ubuntu](#debian-or-ubuntu) * [FreeBSD](#freebsd) * [Installing from source](#installing-from-source) * [Automatic installation on RHEL/Debian/BSD](#automatic-installation-on-rheldebianbsd) * [Nginx package](#nginx-package) * [Dependencies](#dependencies) * [Patches](#patches) * [3rd party modules](#3rd-party-modules) * [Configure options](#configure-options) * [Compiler and linker](#compiler-and-linker) * [Debugging Symbols](#debugging-symbols) * [SystemTap](#systemtap) * [stapxx](#stapxx) * [Installation Nginx on CentOS 7](#installation-nginx-on-centos-7) * [Pre installation tasks](#pre-installation-tasks) * [Dependencies](#dependencies) * [Get Nginx sources](#get-nginx-sources) * [Download 3rd party modules](#download-3rd-party-modules) * [Build Nginx](#build-nginx) * [Post installation tasks](#post-installation-tasks) * [Installation OpenResty on CentOS 7](#installation-openresty-on-centos-7) * [Installation Tengine on Ubuntu 18.04](#installation-tengine-on-ubuntu-1804) * [Installation Nginx on FreeBSD 11.3](#installation-nginx-on-freebsd-113) * [Installation Nginx on FreeBSD 12.1 (from ports)](#installation-nginx-on-freebsd-121-from-ports) * [Analyse configuration](#analyse-configuration) * [Monitoring](#monitoring) * [GoAccess](#goaccess) * [Build and install](#build-and-install) * [Analyse log file and enable all recorded statistics](#analyse-log-file-and-enable-all-recorded-statistics) * [Analyse compressed log file](#analyse-compressed-log-file) * [Analyse log file remotely](#analyse-log-file-remotely) * [Analyse log file and generate html report](#analyse-log-file-and-generate-html-report) * [Ngxtop](#ngxtop) * [Analyse log file](#analyse-log-file) * [Analyse log file and print requests with 4xx and 5xx](#analyse-log-file-and-print-requests-with-4xx-and-5xx) * [Analyse log file remotely](#analyse-log-file-remotely-1) * [Testing](#testing) * [Build OpenSSL 1.0.2-chacha version](HELPERS.md#build-openssl-102-chacha-version) * [Send request and show response headers](#send-request-and-show-response-headers) * [Send request with http method, user-agent, follow redirects and show response headers](#send-request-with-http-method-user-agent-follow-redirects-and-show-response-headers) * [Send multiple requests](#send-multiple-requests) * [Testing SSL connection](#testing-ssl-connection) * [Testing SSL connection (debug mode)](#testing-ssl-connection-debug-mode) * [Testing SSL connection with SNI support](#testing-ssl-connection-with-sni-support) * [Testing SSL connection with specific SSL version](#testing-ssl-connection-with-specific-ssl-version) * [Testing SSL connection with specific cipher](#testing-ssl-connection-with-specific-cipher) * [Testing OCSP Stapling](#testing-ocsp-stapling) * [Verify 0-RTT](#verify-0-rtt) * [Testing SCSV](#testing-scsv) * [Load testing with ApacheBench (ab)](#load-testing-with-apachebench-ab) * [Standard test](#standard-test) * [Test with Keep-Alive header](#test-with-keep-alive-header) * [Load testing with wrk2](#load-testing-with-wrk2) * [Standard scenarios](#standard-scenarios) * [POST call (with Lua)](#post-call-with-lua) * [Random paths (with Lua)](#random-paths-with-lua) * [Multiple paths (with Lua)](#multiple-paths-with-lua) * [Random server address to each thread (with Lua)](#random-server-address-to-each-thread-with-lua) * [Multiple json requests (with Lua)](#multiple-json-requests-with-lua) * [Debug mode (with Lua)](#debug-mode-with-lua) * [Analyse data pass to and from the threads](#analyse-data-pass-to-and-from-the-threads) * [Parsing wrk result and generate report](#parsing-wrk-result-and-generate-report) * [Load testing with locust](#load-testing-with-locust) * [Multiple paths](#multiple-paths) * [Multiple paths with different user sessions](#multiple-paths-with-different-user-sessions) * [TCP SYN flood Denial of Service attack](#tcp-syn-flood-denial-of-service-attack) * [HTTP Denial of Service attack](#tcp-syn-flood-denial-of-service-attack) * [Debugging](#debugging) * [Show information about processes](#show-information-about-processes) * [Check memory usage](#check-memory-usage) * [Show open files](#show-open-files) * [Check segmentation fault messages](#check-segmentation-fault-messages) * [Dump configuration](#dump-configuration) * [Get the list of configure arguments](#get-the-list-of-configure-arguments) * [Check if the module has been compiled](#check-if-the-module-has-been-compiled) * [Show the most accessed IP addresses](#show-the-most-accessed-ip-addresses) * [Show the most accessed IP addresses (ip and url)](#show-the-most-accessed-ip-addresses-ip-and-url) * [Show the most accessed IP addresses (method, code, ip, and url)](#show-the-most-accessed-ip-addresses-method-code-ip-and-url) * [Show the top 5 visitors (IP addresses)](#show-the-top-5-visitors-ip-addresses) * [Show the most requested urls](#show-the-most-requested-urls) * [Show the most requested urls containing 'string'](#show-the-most-requested-urls-containing-string) * [Show the most requested urls with http methods](#show-the-most-requested-urls-with-http-methods) * [Show the most accessed response codes](#show-the-most-accessed-response-codes) * [Analyse web server log and show only 2xx http codes](#analyse-web-server-log-and-show-only-2xx-http-codes) * [Analyse web server log and show only 5xx http codes](#analyse-web-server-log-and-show-only-5xx-http-codes) * [Show requests which result 502 and sort them by number per requests by url](#show-requests-which-result-502-and-sort-them-by-number-per-requests-by-url) * [Show requests which result 404 for php files and sort them by number per requests by url](#show-requests-which-result-404-for-php-files-and-sort-them-by-number-per-requests-by-url) * [Calculating amount of http response codes](#calculating-amount-of-http-response-codes) * [Calculating requests per second](#calculating-requests-per-second) * [Calculating requests per second with IP addresses](#calculating-requests-per-second-with-ip-addresses) * [Calculating requests per second with IP addresses and urls](#calculating-requests-per-second-with-ip-addresses-and-urls) * [Get entries within last n hours](#get-entries-within-last-n-hours) * [Get entries between two timestamps (range of dates)](#get-entries-between-two-timestamps-range-of-dates) * [Get line rates from web server log](#get-line-rates-from-web-server-log) * [Trace network traffic for all processes](#trace-network-traffic-for-all-nginx-processes) * [List all files accessed by a NGINX](#list-all-files-accessed-by-a-nginx) * [Check that the gzip_static module is working](#check-that-the-gzip_static-module-is-working) * [Which worker processing current request](#which-worker-processing-current-request) * [Capture only http packets](#capture-only-http-packets) * [Extract User Agent from the http packets](#extract-user-agent-from-the-http-packets) * [Capture only http GET and POST packets](#capture-only-http-get-and-post-packets) * [Capture requests and filter by source ip and destination port](#capture-requests-and-filter-by-source-ip-and-destination-port) * [Capture HTTP requests/responses in real time, filter by GET, HEAD and save to a file](#capture-http-requests--responses-in-real-time-filter-by-get-head-and-save-to-a-file) * [Dump a process's memory](#dump-a-processs-memory) * [GNU Debugger (gdb)](#gnu-debugger-gdb) * [Dump configuration from a running process](#dump-configuration-from-a-running-process) * [Show debug log in memory](#show-debug-log-in-memory) * [Core dump backtrace](#core-dump-backtrace) * [Debugging socket leaks](#debugging-socket-leaks) * [Shell aliases](#shell-aliases) * [Configuration snippets](#configuration-snippets) * [Nginx server header removal](#nginx-server-header-removal) * [Custom log formats](#custom-log-formats) * [Log only 4xx/5xx](#log-only-4xx5xx) * [Restricting access with basic authentication](#restricting-access-with-basic-authentication) * [Restricting access with client certificate](#restricting-access-with-client-certificate) * [Restricting access by geographical location](#restricting-access-by-geographical-location) * [GeoIP 2 database](#geoip-2-database) * [Dynamic error pages with SSI](#dynamic-error-pages-with-ssi) * [Blocking/allowing IP addresses](#blockingallowing-ip-addresses) * [Blocking referrer spam](#blocking-referrer-spam) * [Limiting referrer spam](#limiting-referrer-spam) * [Blocking User-Agent](#blocking-user-agent) * [Limiting User-Agent](#limiting-user-agent) * [Limiting the rate of requests with burst mode](#limiting-the-rate-of-requests-with-burst-mode) * [Limiting the rate of requests with burst mode and nodelay](#limiting-the-rate-of-requests-with-burst-mode-and-nodelay) * [Limiting the rate of requests per IP with geo and map](#limiting-the-rate-of-requests-per-ip-with-geo-and-map) * [Limiting the number of connections](#limiting-the-number-of-connections) * [Using trailing slashes](#using-trailing-slashes) * [Properly redirect all HTTP requests to HTTPS](#properly-redirect-all-http-requests-to-https) * [Adding and removing the www prefix](#adding-and-removing-the-www-prefix) * [Proxy/rewrite and keep the original URL](#proxyrewrite-and-keep-the-original-url) * [Proxy/rewrite and keep the part of original URL](#proxyrewrite-and-keep-the-part-of-original-url) * [Proxy/rewrite without changing the original URL (in browser)](#proxyrewrite-without-changing-the-original-url-in-browser) * [Modify 301/302 response body](#modify-301302-response-body) * [Redirect POST request with payload to external endpoint](#redirect-post-request-with-payload-to-external-endpoint) * [Route to different backends based on HTTP method](#route-to-different-backends-based-on-HTTP-method) * [Allow multiple cross-domains using the CORS headers](#allow-multiple-cross-domains-using-the-cors-headers) * [Set correct scheme passed in X-Forwarded-Proto](#set-correct-scheme-passed-in-x-forwarded-proto) * [Other snippets](#other-snippets) * [Recreate base directory](#recreate-base-directory) * [Create a temporary static backend](#create-a-temporary-static-backend) * [Create a temporary static backend with SSL support](#create-a-temporary-static-backend-with-ssl-support) * [Generate password file with htpasswd command](#generate-password-file-with-htpasswd-command) * [Generate private key without passphrase](#generate-private-key-without-passphrase) * [Generate private key with passphrase](#generate-private-key-with-passphrase) * [Remove passphrase from private key](#remove-passphrase-from-private-key) * [Encrypt existing private key with a passphrase](#encrypt-existing-private-key-with-a-passphrase) * [Generate CSR](#generate-csr) * [Generate CSR (metadata from existing certificate)](#generate-csr-metadata-from-existing-certificate) * [Generate CSR with -config param](#generate-csr-with--config-param) * [Generate private key and CSR](#generate-private-key-and-csr) * [List available EC curves](#list-available-ec-curves) * [Print ECDSA private and public keys](#print-ecdsa-private-and-public-keys) * [Generate ECDSA private key](#generate-ecdsa-private-key) * [Generate private key and CSR (ECC)](#generate-private-key-with-csr-ecc) * [Generate self-signed certificate](#generate-self-signed-certificate) * [Generate self-signed certificate from existing private key](#generate-self-signed-certificate-from-existing-private-key) * [Generate self-signed certificate from existing private key and csr](#generate-self-signed-certificate-from-existing-private-key-and-csr) * [Generate multidomain certificate (Certbot)](#generate-multidomain-certificate-certbot) * [Generate wildcard certificate (Certbot)](#generate-wildcard-certificate-certbot) * [Generate certificate with 4096 bit private key (Certbot)](#generate-certificate-with-4096-bit-private-key-certbot) * [Generate DH public parameters](#generate-dh-public-parameters) * [Display DH public parameters](#display-dh-public-parameters) * [Extract private key from pfx](#extract-private-key-from-pfx) * [Extract private key and certs from pfx](#extract-private-key-and-certs-from-pfx) * [Extract certs from p7b](#extract-certs-from-p7b) * [Convert DER to PEM](#convert-der-to-pem) * [Convert PEM to DER](#convert-pem-to-der) * [Verification of the certificate's supported purposes](#verification-of-the-certificates-supported-purposes) * [Check private key](#check-private-key) * [Verification of the private key](#verification-of-the-private-key) * [Get public key from private key](#get-public-key-from-private-key) * [Verification of the public key](#verification-of-the-public-key) * [Verification of the certificate](#verification-of-the-certificate) * [Verification of the CSR](#verification-of-the-csr) * [Check the private key and the certificate are match](#check-the-private-key-and-the-certificate-are-match) * [Check the private key and the CSR are match](#check-the-private-key-and-the-csr-are-match) * [TLSv1.3 and CCM ciphers](#tlsv13-and-ccm-ciphers) #### Installing from prebuilt packages > **:bookmark: [Always keep NGINX up-to-date - Hardening - P1](RULES.md#beginner-always-keep-nginx-up-to-date)** ##### RHEL7 or CentOS 7 ###### From EPEL ```bash # Install epel repository: yum install epel-release # or alternative: # wget -c --no-check-certificate -c https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # yum install epel-release-latest-7.noarch.rpm # Install NGINX: yum install nginx ``` ###### From Software Collections ```bash # Install and enable scl: yum install centos-release-scl yum-config-manager --enable rhel-server-rhscl-7-rpms # Install NGINX (rh-nginx14, rh-nginx16, rh-nginx18): yum install rh-nginx16 # Enable NGINX from SCL: scl enable rh-nginx16 bash ``` ###### From Official Repository ```bash # Where: # - is: rhel or centos cat > /etc/yum.repos.d/nginx.repo << __EOF__ [nginx] name=nginx repo baseurl=http://nginx.org/packages//$releasever/$basearch/ gpgcheck=0 enabled=1 __EOF__ # Install NGINX: yum install nginx ``` ##### Debian or Ubuntu Check available flavours of NGINX before install. For more information please see [this](https://askubuntu.com/a/556382) great answer by [Thomas Ward](https://askubuntu.com/users/10616/thomas-ward). ###### From Debian/Ubuntu Repository ```bash # Install NGINX: apt-get install nginx ``` ###### From Official Repository ```bash # Where: # - is: debian or ubuntu # - is: xenial, bionic, jessie, stretch or other cat > /etc/apt/sources.list.d/nginx.list << __EOF__ deb http://nginx.org/packages// nginx deb-src http://nginx.org/packages// nginx __EOF__ # Update packages list: apt-get update # Download the public key (or from your GPG error): apt-key adv --keyserver keyserver.ubuntu.com --recv-keys # Install NGINX: apt-get update apt-get install nginx ``` ##### FreeBSD ###### From FreeBSD Repository ```bash # Install NGINX: pkg install nginx ``` > If you install NGINX on FreeBSD/OpenBSD please see [Tuning FreeBSD for the highload](http://nginx.org/en/docs/freebsd_tuning.html). #### Installing from source > **:bookmark: [Always keep NGINX up-to-date - Hardening - P1](RULES.md#beginner-always-keep-nginx-up-to-date)** The build is configured using the `configure` command. The configure shell script attempts to guess correct values for various system-dependent variables used during compilation. It uses those values to create a `Makefile`. Of course you can adjust certain environment variables to make configure able to find the packages like a `zlib` or `openssl`, and of many other options (paths, modules). Before the beginning installation process please read these important articles which describes exactly the entire installation process and the parameters using the `configure` command: - [Installation and Compile-Time Options](https://www.nginx.com/resources/wiki/start/topics/tutorials/installoptions/) - [Installing NGINX Open Source](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#configure) - [Building nginx from Sources](https://nginx.org/en/docs/configure.html) In this chapter I'll present several very similar methods of installation: - [Installation Nginx on CentOS 7](#installation-nginx-on-centos-7) - [Installation OpenResty on CentOS 7](#installation-openresty-on-centos-7) - [Installation Tengine on Ubuntu 18.04](#installation-tengine-on-ubuntu-1804) - [Installation Nginx on FreeBSD 11.3](#installation-nginx-on-freebsd-113) - [Installation Nginx on FreeBSD 12.1 (from ports)](#installation-nginx-on-freebsd-121-from-ports) Each of them is suited towards a high performance as well as high-concurrency applications. They work great as a high-end proxy servers too. Of course, if you want you can use the default installation (remember about [dependencies](#dependencies)): ```bash ./configure make && make install ``` Look also on this short note about the system locations. That can be useful too: - For booting the system, rescues and maintenance: `/` - `/bin` - user programs - `/sbin` - system programs - `/lib` - shared libraries - Full running environment: `/usr` - `/usr/bin` - user programs - `/usr/sbin` - system programs - `/usr/lib` - shared libraries - `/usr/share` - manual pages, data - Added packages: `/usr/local` - `/usr/local/bin` - user programs - `/usr/local/sbin` - system programs - `/usr/local/lib` - shared libraries - `/usr/local/share` - manual pages, data ##### Automatic installation on RHEL/Debian/BSD Installing from source consists of multiple steps. If you don't want to pass through all of them manually, you can run automated script. I created it to facilitate the whole installation process. > It supports Debian and RHEL like distributions, and FreeBSD system. This tool is located in `lib/ngx_installer.sh`. Configuration file is in `lib/ngx_installer.conf`, variables is in `lib/ngx_installer.vars`. By default, it show prompt to confirm steps but you can disable it if you want: ```bash cd lib/ export NGX_PROMPT=0 ; bash ngx_installer.sh ``` ##### Nginx package There are currently two versions of NGINX: - **stable** - is recommended, doesn’t include all of the latest features, but has critical bug fixes from mainline release - **mainline** - is typically quite stable as well, includes the latest features and bug fixes and is always up to date You can download NGINX source code from an official read-only mirrors: > Detailed instructions about download and compile the NGINX sources can be found later in the handbook. - [NGINX source code](https://nginx.org/download/) - [NGINX GitHub repository](https://github.com/nginx/nginx) ##### Dependencies Mandatory requirements: > Download, compile and install or install prebuilt packages from repository of your distribution. - [OpenSSL](https://www.openssl.org/source/) library - [Zlib](https://zlib.net/) or [Cloudflare Zlib](https://github.com/cloudflare/zlib) library - [PCRE](https://ftp.pcre.org/pub/pcre/) library - [LuaJIT v2.1](https://github.com/LuaJIT/LuaJIT) or [OpenResty's LuaJIT2](https://github.com/openresty/luajit2) library - [jemalloc](https://github.com/jemalloc/jemalloc) library OpenResty's LuaJIT uses its own branch of LuaJIT with various important bug fixes and optimizations for OpenResty's use cases. I also use Cloudflare Zlib version due to performance. See below articles: - [A comparison of Zlib implementations](http://www.htslib.org/benchmarks/zlib.html) - [Improving Nginx Zlib Compression Performance](https://medium.com/@centminmod/improving-nginx-zlib-compression-performance-eb961f3ac0f4) If you download and compile above sources the good point is to install additional packages (dependent on the system version) before building NGINX: | Debian Like | RedHat Like | FreeBSD\*\* | Comment | | :--- | :--- | :--- | :--- | | `gcc`
`make`
`build-essential`
`linux-headers*`
`bison` | `gcc`
`gcc-c++`
`kernel-devel`
`bison` | `gcc`
`gmake`
`bison` | | | `perl`
`libperl-dev`
`libphp-embed` | `perl`
`perl-devel`
`perl-ExtUtils-Embed` | `perl5-devel` | | | `libssl-dev`* | `openssl-devel`* | | | | `zlib1g-dev`* | `zlib-devel`* | | | | `libpcre2-dev`* | `pcre-devel`* | `pcre`* | | | `lua5.1`
`libluajit-5.1-dev`* | `lua`
`luajit-devel`* | `lua51`
`luajit` | | | `libxslt-dev` | `libxslt libxslt-devel` | `libxslt` | | | `libgd-dev` | `gd gd-devel` | `libgd` | | | `libgeoip-dev` | `GeoIP-devel` | | | | `libxml2-dev` | `libxml2-devel` | `libxml2` | | | `libexpat-dev` | `expat-devel` | `expat` | | | `libgoogle-perftools-dev`
`libgoogle-perftools4` | `gperftools-devel` | | | | | `cpio` | | | | | `gettext-devel` | | | | `autoconf` | `autoconf` | `autoconf` | for `jemalloc` from sources | | `libjemalloc1`
`libjemalloc-dev`* | `jemalloc`
`jemalloc-devel`* | | for `jemalloc` | | `libpam0g-dev` | `pam-devel` | | for `ngx_http_auth_pam_module` | | `jq` | `jq` | `jq` | for [http error pages](https://github.com/trimstray/nginx-admins-handbook/tree/master/lib/nginx/snippets/http-error-pages) generator | | `git` | `git` | `git` | for `ngx_installer.sh` | | `wget` | `wget` | `wget` | for `ngx_installer.sh` | | | | `ncurses` | for `ngx_installer.sh` | * If you don't use from sources.
\*\* The package list for FreeBSD may be incomplete. Shell one-liners: ```bash # Ubuntu/Debian apt-get install gcc make build-essential bison perl libperl-dev lua5.1 libphp-embed libxslt-dev libgd-dev libgeoip-dev libxml2-dev libexpat-dev libgoogle-perftools-dev libgoogle-perftools4 autoconf apt-get install libssl-dev zlib1g-dev libpcre2-dev libluajit-5.1-dev apt-get install jq git wget logrotate # RedHat/CentOS yum install gcc gcc-c++ kernel-devel bison perl perl-devel perl-ExtUtils-Embed lua libxslt libxslt-devel gd gd-devel GeoIP-devel libxml2-devel expat-devel gperftools-devel cpio gettext-devel autoconf yum install openssl-devel zlib-devel pcre-devel luajit-devel yum install jq git wget logrotate # FreeBSD pkg install gcc gmake bison perl5-devel lua51 libxslt libgd libxml2 expat autoconf pkg install pcre luajit pkg install jq git wget ncurses texinfo gettext gettext-tools ``` ##### Patches - [nginx-remove-server-header.patch](https://gitlab.com/buik/nginx/blob/master/nginx-remove-server-header.patch) - to hide NGINX `Server` header (and more), see also this rule: [Hide Nginx server signature](RULES.md#beginner-hide-nginx-server-signature) - [TLSv1.3 and CCM ciphers](#tlsv13-and-ccm-ciphers) - to enable `TLS_AES_128_CCM_SHA256` and `TLS_AES_128_CCM_8_SHA256` cipher suites ##### 3rd party modules > Not all external modules can work properly with your currently NGINX version. You should read the documentation of each module before adding it to the modules list. You should also to check what version of module is compatible with your NGINX release. What's more, be careful before adding modules on production. Some of them can cause strange behaviors, increased memory and CPU usage, and also reduce the overall performance of NGINX. > Before installing external modules please read [Event-Driven architecture](NGINX_BASICS.md#event-driven-architecture) section to understand why poor quality 3rd party modules may reduce NGINX performance. > If you have running NGINX on your server, and if you want to add new modules, you'll need to compile them against the same version of NGINX that's currently installed (`nginx -v`) and to make new module compatible with the existing NGINX binary, you need to use the same compile flags (`nginx -V`). For more please see [How to Compile Dynamic NGINX Modules](https://gorails.com/blog/how-to-compile-dynamic-nginx-modules). > If you use, e.g. `--with-stream=dynamic`, then all those `stream_xxx` modules must also be built as NGINX dynamic modules. Otherwise you would definitely see those linker errors. Modules can be compiled as a shared object (`*.so` file) and then dynamically loaded into NGINX at runtime (`--add-dynamic-module`). On the other hand you can also built them into NGINX at compile time and linked to the NGINX binary statically (`--add-module`). I mixed both variants because some of the modules are built-in automatically even if I try them to be compiled as a dynamic modules (they are not support dynamic linking). You can download external modules from: - [NGINX 3rd Party Modules](https://www.nginx.com/resources/wiki/modules/) - [OpenResty Components](https://openresty.org/en/components.html) - [Tengine Modules](https://github.com/alibaba/tengine/tree/master/modules) A short description of the modules that I used in this step-by-step tutorial: - [`ngx_devel_kit`](https://github.com/simplresty/ngx_devel_kit)** - adds additional generic tools that module developers can use in their own modules - [`lua-nginx-module`](https://github.com/openresty/lua-nginx-module) - embed the Power of Lua into NGINX - [`set-misc-nginx-module`](https://github.com/openresty/set-misc-nginx-module) - various `set_xxx` directives added to NGINX rewrite module - [`echo-nginx-module`](https://github.com/openresty/echo-nginx-module) - module for bringing the power of `echo`, `sleep`, `time` and more to NGINX config file - [`headers-more-nginx-module`](https://github.com/openresty/headers-more-nginx-module) - set, add, and clear arbitrary output headers - [`replace-filter-nginx-module`](https://github.com/openresty/replace-filter-nginx-module) - streaming regular expression replacement in response bodies - [`array-var-nginx-module`](https://github.com/openresty/array-var-nginx-module) - add supports for array-typed variables to NGINX config files - [`encrypted-session-nginx-module`](https://github.com/openresty/encrypted-session-nginx-module) - encrypt and decrypt NGINX variable values - [`nginx-module-sysguard`](https://github.com/vozlt/nginx-module-sysguard) - module to protect servers when system load or memory use goes too high - [`nginx-access-plus`](https://github.com/nginx-clojure/nginx-access-plus) - allows limiting access to certain http request methods and client addresses - [`ngx_http_substitutions_filter_module`](https://github.com/yaoweibin/ngx_http_substitutions_filter_module) - can do both regular expression and fixed string substitutions - [`nginx-sticky-module-ng`](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/src) - module to add a sticky cookie to be always forwarded to the same - [`nginx-module-vts`](https://github.com/vozlt/nginx-module-vts) - Nginx virtual host traffic status module - [`ngx_brotli`](https://github.com/google/ngx_brotli) - module for Brotli compression - [`ngx_http_naxsi_module`](https://github.com/nbs-system/naxsi) - is an open-source, high performance, low rules maintenance WAF for NGINX - [`ngx_http_delay_module`](http://mdounin.ru/hg/ngx_http_delay_module) - allows to delay requests for a given time - [`nginx-backtrace`](https://github.com/alibaba/nginx-backtrace)* - module to dump backtrace when a worker process exits abnormally - [`ngx_debug_pool`](https://github.com/chobits/ngx_debug_pool)* - provides access to information of memory usage for NGINX memory pool - [`ngx_debug_timer`](https://github.com/hongxiaolong/ngx_debug_timer)* - provides access to information of timer usage for NGINX - [`nginx_upstream_check_module`](https://github.com/yaoweibin/nginx_upstream_check_module)* - health checks upstreams for NGINX - [`nginx-http-footer-filter`](https://github.com/alibaba/nginx-http-footer-filter)* - module that prints some text in the footer of a request upstream server - [`memc-nginx-module`](https://github.com/agentzh/memc-nginx-module) - extended version of the standard Memcached module - [`nginx-rtmp-module`](https://github.com/arut/nginx-rtmp-module) - NGINX-based Media Streaming Server - [`ngx-fancyindex`](https://github.com/aperezdc/ngx-fancyindex) - generates of file listings, like the built-in autoindex module does, but adding a touch of style - [`ngx_log_if`](https://github.com/cfsego/ngx_log_if) - allows you to control when not to write down access log - [`nginx-http-user-agent`](https://github.com/alibaba/nginx-http-user-agent) - module to match browsers and crawlers - [`ngx_http_auth_pam_module`](https://github.com/sto/ngx_http_auth_pam_module) - module to use PAM for simple http authentication - [`ngx_http_google_filter_module`](https://github.com/cuber/ngx_http_google_filter_module) - is a filter module which makes google mirror much easier to deploy - [`nginx-push-stream-module`](https://github.com/wandenberg/nginx-push-stream-module) - a pure stream http push technology for your Nginx setup - [`nginx_tcp_proxy_module`](https://github.com/yaoweibin/nginx_tcp_proxy_module) - add the feature of tcp proxy with nginx, with health check and status monitor - [`ngx_http_custom_counters_module`](https://github.com/lyokha/nginx-custom-counters-module) - customizable counters shared by all worker processes and virtual servers - [`ngx_chash_map`](https://github.com/Wine93/chash-map-nginx-module) - creates variables whose values are mapped to group by consistent hashing method - [`ngx_security_headers`](https://github.com/GetPageSpeed/ngx_security_headers) - adds security headers and removes insecure headers easily - [`ngx_http_ip2location_module`](https://github.com/ip2location/ip2location-nginx) - enables user to easily perform client's IP to geographical location lookup by using IP2Location database - [`ngx_http_ip2proxy`](https://github.com/ip2location/ip2location-nginx) - detects visitor IP addresses which are used as VPN anonymizer, open proxies, web proxies and Tor exits - [`nginx-length-hiding-filter-module`](https://github.com/nulab/nginx-length-hiding-filter-module) - provides functionality to append randomly generated HTML comment to the end of response body to hide correct response length and make it difficult for attackers to guess secure token * Available in Tengine Web Server (but these modules may have been updated/patched by Tengine Team).
** Is already being used in quite a few third party modules. ##### Configure options Out of the box you probably do not need to provide any flags yourself, the configure script should detect automatically some reasonable defaults. However, in order to optimize for speed and/or security, you should probably provide a few compiler flags. Red Hat published an article about the flag collections they consider good - see [Compiler and linker](#compiler-and-linker) chapter for more information. Another reasonable way to do it would be to copy the options used by your distribution provided packages. The maintainer probably knows what he was doing, and you at least know it works for your use case. There are some of the NGINX configuration options, for more information please see [Building nginx from Sources](http://nginx.org/en/docs/configure.html). ##### Compiler and linker Out of the box you probably do not need to provide any flags yourself, the configure script should detect automatically some reasonable defaults. However, in order to optimise for speed and/or security, you should probably provide a few compiler flags. See [this](https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/) recommendations by RedHat. You should also read [Compilation and Installation](https://wiki.openssl.org/index.php/Compilation_and_Installation) for OpenSSL. There are examples: ```bash # Example of use compiler options: # 1) --with-cc-opt="-I/usr/local/include -I${OPENSSL_INC} -I${LUAJIT_INC} -I${JEMALLOC_INC} -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC" # 2) --with-cc-opt="-I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -O3 -g -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -gsplit-dwarf" # 3) --with-cc-opt="-I/usr/local/include" # Example of use linker options: # 1) --with-ld-opt="-Wl,-E -L/usr/local/lib -ljemalloc -lpcre -Wl,-rpath,/usr/local/lib,-z,relro -Wl,-z,now -pie" # 2) --with-ld-opt="-L/usr/local/lib -ljemalloc -Wl,-lpcre -Wl,-z,relro -Wl,-rpath,/usr/local/lib" # 3) --with-ld-opt="-L/usr/local/lib" # For installation on FreeBSD: --with-cc-opt="-I/usr/local/include" --with-ld-opt="-L/usr/local/lib" ``` ###### Debugging Symbols Debugging symbols helps obtain additional information for debugging, such as functions, variables, data structures, source file and line number information. However, if you get the `No symbol table info available` error when you run a `(gdb) backtrace` you should to recompile NGINX with support of debugging symbols. For this it is essential to include debugging symbols with the `-g` flag and make the debugger output easier to understand by disabling compiler optimization with the `-O0` flag: > If you use `-O0` remember about disable `-D_FORTIFY_SOURCE=2`, if you don't do it you will get: `error: #warning _FORTIFY_SOURCE requires compiling with optimization (-O)`. ```bash ./configure --with-debug --with-cc-opt='-O0 -g' ... ``` Also if you get errors similar to one of them: ``` Missing separate debuginfo for /usr/lib64/libluajit-5.1.so.2 ... Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found) ... ``` You should also recompile libraries with `-g` compiler option and optional with `-O0`. For more information please read [3.9 Options for Debugging Your Program](https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html). ##### SystemTap SystemTap is a scripting language and tool for dynamically instrumenting running production Linux kernel-based operating systems. It's required for `openresty-systemtap-toolkit` for OpenResty. > It's good [all-in-one tutorial](https://gist.github.com/notsobad/b8f5ebb9b99f3a818f30) about install and configure SystemTap on CentOS 7/Ubuntu distributions. In case of problems please see this [SystemTap](https://github.com/shawfdong/hyades/wiki/SystemTap) document. > Hint: Do not specify `--with-debug` while profiling. It slows everything down significantly. ```bash cd /opt git clone --depth 1 https://github.com/openresty/openresty-systemtap-toolkit # RHEL/CentOS yum install yum-utils yum --enablerepo=base-debuginfo install kernel-devel-$(uname -r) kernel-headers-$(uname -r) kernel-debuginfo-$(uname -r) kernel-debuginfo-common-x86_64-$(uname -r) yum --enablerepo=base-debuginfo install systemtap systemtap-debuginfo reboot # Run this commands for testing SystemTap: stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}' stap -v -e 'probe begin { printf("Hello, World!\n"); exit() }' ``` For installation SystemTap on Ubuntu/Debian: - [Ubuntu Wiki - Systemtap](https://wiki.ubuntu.com/Kernel/Systemtap) - [Install SystemTap in Ubuntu 14.04](https://blog.jeffli.me/blog/2014/10/10/install-systemtap-in-ubuntu-14-dot-04/) ###### stapxx The author of OpenResty created great and simple macro language extensions to the SystemTap: [stapxx](https://github.com/openresty/stapxx). #### Installation Nginx on CentOS 7 ###### Pre installation tasks Set NGINX version (I use stable release): ```bash export ngx_version="1.16.0" ``` Set temporary variables: ```bash export ngx_src="/usr/local/src" export ngx_base="${ngx_src}/nginx-${ngx_version}" export ngx_master="${ngx_base}/master" export ngx_modules="${ngx_base}/modules" export NGX_PREFIX="/etc/nginx" export NGX_CONF="${NGX_PREFIX}/nginx.conf" ``` Create directories: ```bash for i in "${ngx_base}" "${ngx_master}" "${ngx_modules}" ; do mkdir "$i" done ``` Set user/group variables: ```bash export NGINX_USER="nginx" export NGINX_GROUP="nginx" export NGINX_UID="920" export NGINX_GID="920" ``` ###### Dependencies > In my configuration I used all prebuilt dependencies without `libssl-dev`, `zlib1g-dev`, `libluajit-5.1-dev`, and `libpcre2-dev` because I compiled them manually - for TLS 1.3 support and with OpenResty recommendation for LuaJIT. **Install prebuilt packages, export variables and set symbolic link:** ```bash # It's important and required, regardless of chosen sources: yum install gcc gcc-c++ kernel-devel bison perl perl-devel perl-ExtUtils-Embed lua libxslt libxslt-devel gd gd-devel GeoIP-devel libxml2-devel expat-devel gperftools-devel cpio gettext-devel autoconf jq git wget logrotate # In this example we use sources for all below packages so we do not install them: # yum install openssl-devel zlib-devel pcre-devel luajit-devel # For LuaJIT (luajit-devel): export LUAJIT_LIB="/usr/local/lib" # For original: # export LUAJIT_INC="/usr/local/include/luajit-2.0" # For OpenResty's: export LUAJIT_INC="/usr/local/include/luajit-2.1" for i in libluajit-5.1.so libluajit-5.1.so.2 liblua.so libluajit.so ; do # For original LuaJIT: # ln -sf /usr/local/lib/libluajit-5.1.so.2.0.5 ${LUAJIT_LIB}/${i} # For OpenResty's LuaJIT: ln -sf /usr/local/lib/libluajit-5.1.so.2.1.0 ${LUAJIT_LIB}/${i} done # ln -sf /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 ${LUAJIT_LIB}/liblua.so ``` > Remember to build [`sregex`](#sregex) also if you use above steps. **Or download and compile them:** PCRE: ```bash cd "${ngx_src}" export pcre_version="8.42" export PCRE_SRC="${ngx_src}/pcre-${pcre_version}" export PCRE_LIB="/usr/local/lib" export PCRE_INC="/usr/local/include" wget -c --no-check-certificate https://ftp.pcre.org/pub/pcre/pcre-${pcre_version}.tar.gz && tar xzvf pcre-${pcre_version}.tar.gz cd "$PCRE_SRC" # Add to compile with debugging symbols: # CFLAGS='-O0 -g' ./configure ./configure make -j2 && make test make install ``` Zlib: ```bash # I recommend to use Cloudflare Zlib version (cloudflare/zlib) instead an original Zlib (zlib.net), but both installation methods are similar: cd "${ngx_src}" export ZLIB_SRC="${ngx_src}/zlib" export ZLIB_LIB="/usr/local/lib" export ZLIB_INC="/usr/local/include" # For original Zlib: # export zlib_version="1.2.11" # wget -c --no-check-certificate http://www.zlib.net/zlib-${zlib_version}.tar.gz # mkdir -p zlib && tar xzvf zlib-${zlib_version}.tar.gz -C zlib # or: # git clone --depth 1 https://github.com/madler/zlib # For Cloudflare Zlib: git clone --depth 1 https://github.com/cloudflare/zlib cd "$ZLIB_SRC" ./configure make -j2 && make test make install ``` OpenSSL: ```bash cd "${ngx_src}" export openssl_version="1.1.1c" export OPENSSL_SRC="${ngx_src}/openssl-${openssl_version}" export OPENSSL_DIR="/usr/local/openssl-${openssl_version}" export OPENSSL_LIB="${OPENSSL_DIR}/lib" export OPENSSL_INC="${OPENSSL_DIR}/include" wget -c --no-check-certificate https://www.openssl.org/source/openssl-${openssl_version}.tar.gz && tar xzvf openssl-${openssl_version}.tar.gz cd "${ngx_src}/openssl-${openssl_version}" # Please run this and add as a compiler param: export __GCC_SSL=("__SIZEOF_INT128__:enable-ec_nistp_64_gcc_128") for _cc_opt in "${__GCC_SSL[@]}" ; do _cc_key=$(echo "$_cc_opt" | cut -d ":" -f1) _cc_value=$(echo "$_cc_opt" | cut -d ":" -f2) if [[ ! $(gcc -dM -E - /etc/profile.d/openssl.sh << __EOF__ #!/bin/sh export PATH=${OPENSSL_DIR}/bin:${PATH} export LD_LIBRARY_PATH=${OPENSSL_DIR}/lib:${LD_LIBRARY_PATH} __EOF__ chmod +x /etc/profile.d/openssl.sh && source /etc/profile.d/openssl.sh # To make the OpenSSL version visible globally first: if [[ -e "/usr/bin/openssl" ]] ; then _openssl_version=$(openssl version | awk '{print $2}') _openssl_date=$(date '+%Y%m%d%H%M%S') _openssl_str="openssl-${_openssl_version}-${_openssl_date}" mv /usr/bin/openssl /usr/bin/${_openssl_str} ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl else ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl fi cat > /etc/ld.so.conf.d/openssl.conf << __EOF__ ${OPENSSL_DIR}/lib __EOF__ ``` LuaJIT: ```bash # I recommend to use OpenResty's branch (openresty/luajit2) instead of LuaJIT (LuaJIT/LuaJIT), but both installation methods are similar: cd "${ngx_src}" export LUAJIT_SRC="${ngx_src}/luajit2" export LUAJIT_LIB="/usr/local/lib" # For original LuaJIT: # export LUAJIT_INC="/usr/local/include/luajit-2.0" # git clone http://luajit.org/git/luajit-2.0.git luajit2 # For OpenResty's LuaJIT: export LUAJIT_INC="/usr/local/include/luajit-2.1" git clone --depth 1 https://github.com/openresty/luajit2 cd "$LUAJIT_SRC" # Add to compile with debugging symbols: # CFLAGS='-g' make ... make && make install for i in libluajit-5.1.so libluajit-5.1.so.2 liblua.so libluajit.so ; do # For original LuaJIT: # ln -sf /usr/local/lib/libluajit-5.1.so.2.0.5 ${LUAJIT_LIB}/${i} # For OpenResty's LuaJIT: ln -sf /usr/local/lib/libluajit-5.1.so.2.1.0 ${LUAJIT_LIB}/${i} done # ln -sf /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 ${LUAJIT_LIB}/liblua.so ``` sregex: > Required for `replace-filter-nginx-module` module. ```bash cd "${ngx_src}" git clone --depth 1 https://github.com/openresty/sregex cd "${ngx_src}/sregex" make && make install ``` jemalloc: > To verify `jemalloc` in use: `lsof -n | grep jemalloc`. ```bash cd "${ngx_src}" export JEMALLOC_SRC="${ngx_src}/jemalloc" export JEMALLOC_INC="/usr/local/include/jemalloc" git clone --depth 1 https://github.com/jemalloc/jemalloc cd "$JEMALLOC_SRC" ./autogen.sh make && make install ``` Update links and cache to the shared libraries for both types of installation: ```bash ldconfig ``` ###### Get Nginx sources ```bash cd "${ngx_base}" wget -c --no-check-certificate https://nginx.org/download/nginx-${ngx_version}.tar.gz # or alternative: # git clone --depth 1 https://github.com/nginx/nginx master tar zxvf nginx-${ngx_version}.tar.gz -C "${ngx_master}" --strip 1 ``` ###### Download 3rd party modules ```bash cd "${ngx_modules}" for i in \ https://github.com/simplresty/ngx_devel_kit \ https://github.com/openresty/lua-nginx-module \ https://github.com/openresty/set-misc-nginx-module \ https://github.com/openresty/echo-nginx-module \ https://github.com/openresty/headers-more-nginx-module \ https://github.com/openresty/replace-filter-nginx-module \ https://github.com/openresty/array-var-nginx-module \ https://github.com/openresty/encrypted-session-nginx-module \ https://github.com/vozlt/nginx-module-sysguard \ https://github.com/nginx-clojure/nginx-access-plus \ https://github.com/yaoweibin/ngx_http_substitutions_filter_module \ https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng \ https://github.com/vozlt/nginx-module-vts \ https://github.com/google/ngx_brotli ; do git clone --depth 1 "$i" done wget -c --no-check-certificate http://mdounin.ru/hg/ngx_http_delay_module/archive/tip.tar.gz -O delay-module.tar.gz mkdir delay-module && tar xzvf delay-module.tar.gz -C delay-module --strip 1 ``` For `ngx_brotli`: ```bash cd "${ngx_modules}/ngx_brotli" git submodule update --init ``` I also use some modules from Tengine: - `ngx_backtrace_module` - `ngx_debug_pool` - `ngx_debug_timer` - `ngx_http_upstream_check_module` - `ngx_http_footer_filter_module` ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/alibaba/tengine ``` If you use NAXSI: ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/nbs-system/naxsi ``` ###### Build Nginx ```bash cd "${ngx_master}" # - you can also build NGINX without 3rd party modules # - remember about compiler and linker options # - don't set values for --with-openssl, --with-pcre, and --with-zlib if you select prebuilt packages for them # - add to compile with debugging symbols: -O0 -g # - and remove -D_FORTIFY_SOURCE=2 if you use above ./configure --prefix=$NGX_PREFIX \ --conf-path=$NGX_CONF \ --sbin-path=/usr/sbin/nginx \ --pid-path=/var/run/nginx.pid \ --lock-path=/var/run/nginx.lock \ --user=$NGINX_USER \ --group=$NGINX_GROUP \ --modules-path=${NGX_PREFIX}/modules \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --http-client-body-temp-path=/var/cache/nginx/client_temp \ --http-proxy-temp-path=/var/cache/nginx/proxy_temp \ --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \ --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \ --http-scgi-temp-path=/var/cache/nginx/scgi_temp \ --with-compat \ --with-debug \ --with-file-aio \ --with-threads \ --with-stream \ --with-stream_realip_module \ --with-stream_ssl_module \ --with-stream_ssl_preread_module \ --with-http_addition_module \ --with-http_auth_request_module \ --with-http_degradation_module \ --with-http_geoip_module \ --with-http_gunzip_module \ --with-http_gzip_static_module \ --with-http_image_filter_module \ --with-http_perl_module \ --with-http_random_index_module \ --with-http_realip_module \ --with-http_secure_link_module \ --with-http_ssl_module \ --with-http_stub_status_module \ --with-http_sub_module \ --with-http_v2_module \ --with-google_perftools_module \ --with-openssl=${OPENSSL_SRC} \ --with-openssl-opt="shared zlib no-ssl3 no-weak-ssl-ciphers -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong ${_openssl_gcc}" \ --with-pcre=${PCRE_SRC} \ --with-pcre-jit \ --with-zlib=${ZLIB_SRC} \ --without-http-cache \ --without-http_memcached_module \ --without-mail_pop3_module \ --without-mail_imap_module \ --without-mail_smtp_module \ --without-http_fastcgi_module \ --without-http_scgi_module \ --without-http_uwsgi_module \ --add-module=${ngx_modules}/ngx_devel_kit \ --add-module=${ngx_modules}/encrypted-session-nginx-module \ --add-module=${ngx_modules}/nginx-access-plus/src/c \ --add-module=${ngx_modules}/ngx_http_substitutions_filter_module \ --add-module=${ngx_modules}/nginx-sticky-module-ng \ --add-module=${ngx_modules}/nginx-module-vts \ --add-module=${ngx_modules}/ngx_brotli \ --add-module=${ngx_modules}/tengine/modules/ngx_backtrace_module \ --add-module=${ngx_modules}/tengine/modules/ngx_debug_pool \ --add-module=${ngx_modules}/tengine/modules/ngx_debug_timer \ --add-module=${ngx_modules}/tengine/modules/ngx_http_footer_filter_module \ --add-module=${ngx_modules}/tengine/modules/ngx_http_upstream_check_module \ --add-module=${ngx_modules}/tengine/modules/ngx_slab_stat \ --add-dynamic-module=${ngx_modules}/lua-nginx-module \ --add-dynamic-module=${ngx_modules}/set-misc-nginx-module \ --add-dynamic-module=${ngx_modules}/echo-nginx-module \ --add-dynamic-module=${ngx_modules}/headers-more-nginx-module \ --add-dynamic-module=${ngx_modules}/replace-filter-nginx-module \ --add-dynamic-module=${ngx_modules}/array-var-nginx-module \ --add-dynamic-module=${ngx_modules}/nginx-module-sysguard \ --add-dynamic-module=${ngx_modules}/delay-module \ --add-dynamic-module=${ngx_modules}/naxsi/naxsi_src \ --with-cc-opt="-I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -O2 -g -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -gsplit-dwarf" \ --with-ld-opt="-L/usr/local/lib -ljemalloc -Wl,-lpcre -Wl,-z,relro -Wl,-rpath,/usr/local/lib" make -j2 && make test make install ldconfig ``` Show NGINX version and parameters: ```bash nginx -V ``` And list all files in `/etc/nginx`: ```bash . ├── fastcgi.conf ├── fastcgi.conf.default ├── fastcgi_params ├── fastcgi_params.default ├── html │   ├── 50x.html │   └── index.html ├── koi-utf ├── koi-win ├── mime.types ├── mime.types.default ├── modules │   ├── ngx_http_array_var_module.so │   ├── ngx_http_delay_module.so │   ├── ngx_http_echo_module.so │   ├── ngx_http_headers_more_filter_module.so │   ├── ngx_http_lua_module.so │   ├── ngx_http_naxsi_module.so │   ├── ngx_http_replace_filter_module.so │   ├── ngx_http_set_misc_module.so │   └── ngx_http_sysguard_module.so ├── nginx.conf ├── nginx.conf.default ├── scgi_params ├── scgi_params.default ├── uwsgi_params ├── uwsgi_params.default └── win-utf 2 directories, 26 files ``` ###### Post installation tasks Create a system user/group: ```bash # Debian/Ubuntu groupadd -r -g $NGINX_GID $NGINX_GROUP adduser --system --home /non-existent --no-create-home --shell /usr/sbin/nologin --disabled-login --disabled-password --gecos \'nginx user\' --uid $NGINX_UID --group $NGINX_GROUP $NGINX_USER # RedHat/CentOS groupadd -r -g $NGINX_GID $NGINX_GROUP useradd --system --home-dir /non-existent --no-create-home --shell /usr/sbin/nologin --comment \'nginx user\' --uid $NGINX_UID --gid $NGINX_GROUP $NGINX_USER passwd -l $NGINX_USER ``` Create required directories: ```bash for i in \ /var/www \ /var/log/nginx \ /var/cache/nginx ; do mkdir -p "$i" && chown -R ${NGINX_USER}:${NGINX_GROUP} "$i" done ``` Include the necessary error pages: > You can also define them e.g. in `/etc/nginx/errors.conf` or other file and attach it as needed in server contexts. - default location: `/etc/nginx/html` ``` 50x.html index.html ``` Update modules list and include `modules.conf` to your configuration: ```bash _mod_dir="${NGX_PREFIX}/modules" :>"${_mod_dir}.conf" for _module in $(ls "${_mod_dir}/") ; do echo -en "load_module ${_mod_dir}/$_module;\n" >> "${_mod_dir}.conf" done ``` Create `logrotate` configuration: ```bash _logrotate_path="/etc/logrotate.d" cat > "${_logrotate_path}/nginx" << __EOF__ /var/log/nginx/*.log { daily missingok rotate 14 compress delaycompress notifempty create 0640 $NGINX_USER $NGINX_GROUP sharedscripts prerotate if [ -d ${_logrotate_path}/httpd-prerotate ]; then \ run-parts ${_logrotate_path}/httpd-prerotate; \ fi \ endscript postrotate invoke-rc.d nginx reload >/dev/null 2>&1 endscript } __EOF__ ``` Add systemd service: ```bash cat > /lib/systemd/system/nginx.service << __EOF__ # Stop dance for nginx # ======================= # # ExecStop sends SIGSTOP (graceful stop) to the nginx process. # If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control # and sends SIGTERM (fast shutdown) to the main process. # After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends # SIGKILL to all the remaining processes in the process group (KillMode=mixed). # # nginx signals reference doc: # http://nginx.org/en/docs/control.html # [Unit] Description=A high performance web server and a reverse proxy server Documentation=man:nginx(8) After=network.target [Service] Type=forking PIDFile=/run/nginx.pid ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;' ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;' ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid TimeoutStopSec=5 KillMode=mixed [Install] WantedBy=multi-user.target __EOF__ ``` Reload systemd manager configuration: ```bash systemctl daemon-reload ``` Enable NGINX service: ```bash systemctl enable nginx ``` Show NGINX version and parameters: ```bash nginx -V ``` Test NGINX configuration: ```bash nginx -t -c $NGX_CONF ``` #### Installation OpenResty on CentOS 7 > _OpenResty is a full-fledged web application server by bundling the standard nginx core, lots of 3rd-party nginx modules, as well as most of their external dependencies._ > > _This bundle is maintained by Yichun Zhang ([agentzh](https://github.com/agentzh))._ - Official github repository: [OpenResty](https://github.com/openresty/openresty) - Official website: [OpenResty](https://openresty.org/en/) - Official documentations: [OpenResty Getting Started](https://openresty.org/en/getting-started.html) and [OpenResty eBooks](https://openresty.org/en/ebooks.html) OpenResty is a more than web server. I would call it a superset of the NGINX web server. OpenResty comes with LuaJIT, a just-in-time compiler for the Lua scripting language and many Lua libraries, lots of high quality 3rd-party NGINX modules, and most of their external dependencies. OpenResty has good quality and performance. For me, the ability to run Lua scripts from within is also really great.
Show step-by-step OpenResty installation
* [Pre installation tasks](#pre-installation-tasks-1) * [Dependencies](#dependencies-1) * [Get OpenResty sources](#get-openresty-sources-1) * [Download 3rd party modules](#download-3rd-party-modules-1) * [Build OpenResty](#build-openresty) * [Post installation tasks](#post-installation-tasks-1) ###### Pre installation tasks Set the OpenResty version (I use newest and stable release): ```bash export ngx_version="1.15.8.1" ``` Set temporary variables: ```bash export ngx_src="/usr/local/src" export ngx_base="${ngx_src}/nginx-${ngx_version}" export ngx_master="${ngx_base}/master" export ngx_modules="${ngx_base}/modules" export NGX_PREFIX="/etc/nginx" export NGX_CONF="${NGX_PREFIX}/nginx.conf" ``` Create directories: ```bash for i in "${ngx_base}" "${ngx_master}" "${ngx_modules}" ; do mkdir "$i" done ``` Set user/group variables: ```bash export NGINX_USER="nginx" export NGINX_GROUP="nginx" export NGINX_UID="920" export NGINX_GID="920" ``` ###### Dependencies > In my configuration I used all prebuilt dependencies without `libssl-dev`, `zlib1g-dev`, and `libpcre2-dev` because I compiled them manually - for TLS 1.3 support. In addition, LuaJIT comes with OpenResty. **Install prebuilt packages, export variables and set symbolic link:** ```bash # It's important and required, regardless of chosen sources: yum install gcc gcc-c++ kernel-devel bison perl perl-devel perl-ExtUtils-Embed lua libxslt libxslt-devel gd gd-devel GeoIP-devel libxml2-devel expat-devel gperftools-devel cpio gettext-devel autoconf jq git wget logrotate # In this example we use sources for all below packages so we do not install them: # yum install openssl-devel zlib-devel pcre-devel ``` > Remember to build [`sregex`](#sregex) also if you use above steps. **Or download and compile them:** PCRE: ```bash cd "${ngx_src}" export pcre_version="8.42" export PCRE_SRC="${ngx_base}/pcre-${pcre_version}" export PCRE_LIB="/usr/local/lib" export PCRE_INC="/usr/local/include" wget -c --no-check-certificate https://ftp.pcre.org/pub/pcre/pcre-${pcre_version}.tar.gz && tar xzvf pcre-${pcre_version}.tar.gz cd "$PCRE_SRC" # Add to compile with debugging symbols: # CFLAGS='-O0 -g' ./configure ./configure make -j2 && make test make install ``` Zlib: ```bash # I recommend to use Cloudflare Zlib version (cloudflare/zlib) instead of an original Zlib (zlib.net), but both installation methods are similar: cd "${ngx_src}" export ZLIB_SRC="${ngx_src}/zlib" export ZLIB_LIB="/usr/local/lib" export ZLIB_INC="/usr/local/include" # For original Zlib: # export zlib_version="1.2.11" # wget -c --no-check-certificate http://www.zlib.net/zlib-${zlib_version}.tar.gz && tar xzvf zlib-${zlib_version}.tar.gz or git clone --depth 1 https://github.com/madler/zlib # cd "${ZLIB_SRC}-${zlib_version}" # For Cloudflare Zlib: git clone --depth 1 https://github.com/cloudflare/zlib cd "$ZLIB_SRC" ./configure make -j2 && make test make install ``` OpenSSL: ```bash cd "${ngx_src}" export openssl_version="1.1.1c" export OPENSSL_SRC="${ngx_src}/openssl-${openssl_version}" export OPENSSL_DIR="/usr/local/openssl-${openssl_version}" export OPENSSL_LIB="${OPENSSL_DIR}/lib" export OPENSSL_INC="${OPENSSL_DIR}/include" wget -c --no-check-certificate https://www.openssl.org/source/openssl-${openssl_version}.tar.gz && tar xzvf openssl-${openssl_version}.tar.gz cd "${ngx_src}/openssl-${openssl_version}" # Please run this and add as a compiler param: export __GCC_SSL=("__SIZEOF_INT128__:enable-ec_nistp_64_gcc_128") for _cc_opt in "${__GCC_SSL[@]}" ; do _cc_key=$(echo "$_cc_opt" | cut -d ":" -f1) _cc_value=$(echo "$_cc_opt" | cut -d ":" -f2) if [[ ! $(gcc -dM -E - /etc/profile.d/openssl.sh << __EOF__ #!/bin/sh export PATH=${OPENSSL_DIR}/bin:${PATH} export LD_LIBRARY_PATH=${OPENSSL_DIR}/lib:${LD_LIBRARY_PATH} __EOF__ chmod +x /etc/profile.d/openssl.sh && source /etc/profile.d/openssl.sh # To make the OpenSSL version visible globally first: if [[ -e "/usr/bin/openssl" ]] ; then _openssl_version=$(openssl version | awk '{print $2}') _openssl_date=$(date '+%Y%m%d%H%M%S') _openssl_str="openssl-${_openssl_version}-${_openssl_date}" mv /usr/bin/openssl /usr/bin/${_openssl_str} ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl else ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl fi cat > /etc/ld.so.conf.d/openssl.conf << __EOF__ ${OPENSSL_DIR}/lib __EOF__ ``` sregex: > Required for `replace-filter-nginx-module` module. ```bash cd "${ngx_src}" git clone --depth 1 https://github.com/openresty/sregex cd "${ngx_src}/sregex" make && make install ``` jemalloc: > To verify `jemalloc` in use: `lsof -n | grep jemalloc`. ```bash cd "${ngx_src}" export JEMALLOC_SRC="/usr/local/src/jemalloc" export JEMALLOC_INC="/usr/local/include/jemalloc" git clone --depth 1 https://github.com/jemalloc/jemalloc cd "$JEMALLOC_SRC" ./autogen.sh make && make install ``` Update links and cache to the shared libraries for both types of installation: ```bash ldconfig ``` ###### Get OpenResty sources ```bash cd "${ngx_base}" wget -c --no-check-certificate https://openresty.org/download/openresty-${ngx_version}.tar.gz tar zxvf openresty-${ngx_version}.tar.gz -C "${ngx_master}" --strip 1 ``` ###### Download 3rd party modules ```bash cd "${ngx_modules}" for i in \ https://github.com/openresty/replace-filter-nginx-module \ https://github.com/vozlt/nginx-module-sysguard \ https://github.com/nginx-clojure/nginx-access-plus \ https://github.com/yaoweibin/ngx_http_substitutions_filter_module \ https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng \ https://github.com/vozlt/nginx-module-vts \ https://github.com/google/ngx_brotli ; do git clone --depth 1 "$i" done wget -c --no-check-certificate http://mdounin.ru/hg/ngx_http_delay_module/archive/tip.tar.gz -O delay-module.tar.gz mkdir delay-module && tar xzvf delay-module.tar.gz -C delay-module --strip 1 ``` For `ngx_brotli`: ```bash cd "${ngx_modules}/ngx_brotli" git submodule update --init ``` I also use some modules from Tengine: - `ngx_backtrace_module` - `ngx_debug_pool` - `ngx_debug_timer` - `ngx_http_upstream_check_module` - `ngx_http_footer_filter_module` ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/alibaba/tengine ``` If you use NAXSI: ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/nbs-system/naxsi ``` ###### Build OpenResty ```bash cd "${ngx_master}" # - you can also build OpenResty without 3rd party modules # - remember about compiler and linker options # - don't set values for --with-openssl, --with-pcre, and --with-zlib if you select prebuilt packages for them # - add to compile with debugging symbols: -O0 -g # - and remove -D_FORTIFY_SOURCE=2 if you use above ./configure --prefix=$NGX_PREFIX \ --conf-path=$NGX_CONF \ --sbin-path=/usr/sbin/nginx \ --pid-path=/var/run/nginx.pid \ --lock-path=/var/run/nginx.lock \ --user=$NGINX_USER \ --group=$NGINX_GROUP \ --modules-path=${NGX_PREFIX}/modules \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --http-client-body-temp-path=/var/cache/nginx/client_temp \ --http-proxy-temp-path=/var/cache/nginx/proxy_temp \ --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \ --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \ --http-scgi-temp-path=/var/cache/nginx/scgi_temp \ --with-compat \ --with-debug \ --with-file-aio \ --with-threads \ --with-stream \ --with-stream_geoip_module \ --with-stream_realip_module \ --with-stream_ssl_module \ --with-stream_ssl_preread_module \ --with-http_addition_module \ --with-http_auth_request_module \ --with-http_degradation_module \ --with-http_geoip_module \ --with-http_gunzip_module \ --with-http_gzip_static_module \ --with-http_image_filter_module \ --with-http_perl_module \ --with-http_random_index_module \ --with-http_realip_module \ --with-http_secure_link_module \ --with-http_slice_module \ --with-http_ssl_module \ --with-http_stub_status_module \ --with-http_sub_module \ --with-http_v2_module \ --with-google_perftools_module \ --with-luajit \ --with-openssl=${OPENSSL_SRC} \ --with-openssl-opt="shared zlib no-ssl3 no-weak-ssl-ciphers -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong ${_openssl_gcc}" \ --with-pcre=${PCRE_SRC} \ --with-pcre-jit \ --with-zlib=${ZLIB_SRC} \ --without-http-cache \ --without-http_memcached_module \ --without-http_redis2_module \ --without-http_redis_module \ --without-http_rds_json_module \ --without-http_rds_csv_module \ --without-lua_redis_parser \ --without-lua_rds_parser \ --without-lua_resty_redis \ --without-lua_resty_memcached \ --without-lua_resty_mysql \ --without-lua_resty_websocket \ --without-mail_pop3_module \ --without-mail_imap_module \ --without-mail_smtp_module \ --without-http_fastcgi_module \ --without-http_scgi_module \ --without-http_uwsgi_module \ --add-module=${ngx_modules}/nginx-access-plus/src/c \ --add-module=${ngx_modules}/ngx_http_substitutions_filter_module \ --add-module=${ngx_modules}/nginx-module-vts \ --add-module=${ngx_modules}/ngx_brotli \ --add-module=${ngx_modules}/tengine/modules/ngx_backtrace_module \ --add-module=${ngx_modules}/tengine/modules/ngx_debug_pool \ --add-module=${ngx_modules}/tengine/modules/ngx_debug_timer \ --add-module=${ngx_modules}/tengine/modules/ngx_http_footer_filter_module \ --add-module=${ngx_modules}/tengine/modules/ngx_http_upstream_check_module \ --add-module=${ngx_modules}/tengine/modules/ngx_slab_stat \ --add-dynamic-module=${ngx_modules}/replace-filter-nginx-module \ --add-dynamic-module=${ngx_modules}/nginx-module-sysguard \ --add-dynamic-module=${ngx_modules}/delay-module \ --add-dynamic-module=${ngx_modules}/naxsi/naxsi_src \ --with-cc-opt="-I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -O2 -g -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -gsplit-dwarf" \ --with-ld-opt="-L/usr/local/lib -ljemalloc -Wl,-lpcre -Wl,-z,relro -Wl,-rpath,/usr/local/lib" make && make test make install ldconfig ``` Show OpenResty version and parameters: ```bash nginx -V ``` And list all files in `/etc/nginx`: ```bash . ├── bin │   ├── md2pod.pl │   ├── nginx-xml2pod │   ├── openresty -> /usr/sbin/nginx │   ├── opm │   ├── resty │   ├── restydoc │   └── restydoc-index ├── COPYRIGHT ├── fastcgi.conf ├── fastcgi.conf.default ├── fastcgi_params ├── fastcgi_params.default ├── koi-utf ├── koi-win ├── luajit │   ├── bin │   │   ├── luajit -> luajit-2.1.0-beta3 │   │   └── luajit-2.1.0-beta3 │   ├── include │   │   └── luajit-2.1 │   │   ├── lauxlib.h │   │   ├── luaconf.h │   │   ├── lua.h │   │   ├── lua.hpp │   │   ├── luajit.h │   │   └── lualib.h │   ├── lib │   │   ├── libluajit-5.1.a │   │   ├── libluajit-5.1.so -> libluajit-5.1.so.2.1.0 │   │   ├── libluajit-5.1.so.2 -> libluajit-5.1.so.2.1.0 │   │   ├── libluajit-5.1.so.2.1.0 │   │   ├── lua │   │   │   └── 5.1 │   │   └── pkgconfig │   │   └── luajit.pc │   └── share │   ├── lua │   │   └── 5.1 │   ├── luajit-2.1.0-beta3 │   │   └── jit │   │   ├── bc.lua │   │   ├── bcsave.lua │   │   ├── dis_arm64be.lua │   │   ├── dis_arm64.lua │   │   ├── dis_arm.lua │   │   ├── dis_mips64el.lua │   │   ├── dis_mips64.lua │   │   ├── dis_mipsel.lua │   │   ├── dis_mips.lua │   │   ├── dis_ppc.lua │   │   ├── dis_x64.lua │   │   ├── dis_x86.lua │   │   ├── dump.lua │   │   ├── p.lua │   │   ├── v.lua │   │   ├── vmdef.lua │   │   └── zone.lua │   └── man │   └── man1 │   └── luajit.1 ├── lualib │   ├── cjson.so │   ├── librestysignal.so │   ├── ngx │   │   ├── balancer.lua │   │   ├── base64.lua │   │   ├── errlog.lua │   │   ├── ocsp.lua │   │   ├── pipe.lua │   │   ├── process.lua │   │   ├── re.lua │   │   ├── resp.lua │   │   ├── semaphore.lua │   │   ├── ssl │   │   │   └── session.lua │   │   └── ssl.lua │   ├── resty │   │   ├── aes.lua │   │   ├── core │   │   │   ├── base64.lua │   │   │   ├── base.lua │   │   │   ├── ctx.lua │   │   │   ├── exit.lua │   │   │   ├── hash.lua │   │   │   ├── misc.lua │   │   │   ├── ndk.lua │   │   │   ├── phase.lua │   │   │   ├── regex.lua │   │   │   ├── request.lua │   │   │   ├── response.lua │   │   │   ├── shdict.lua │   │   │   ├── time.lua │   │   │   ├── uri.lua │   │   │   ├── utils.lua │   │   │   ├── var.lua │   │   │   └── worker.lua │   │   ├── core.lua │   │   ├── dns │   │   │   └── resolver.lua │   │   ├── limit │   │   │   ├── conn.lua │   │   │   ├── count.lua │   │   │   ├── req.lua │   │   │   └── traffic.lua │   │   ├── lock.lua │   │   ├── lrucache │   │   │   └── pureffi.lua │   │   ├── lrucache.lua │   │   ├── md5.lua │   │   ├── random.lua │   │   ├── sha1.lua │   │   ├── sha224.lua │   │   ├── sha256.lua │   │   ├── sha384.lua │   │   ├── sha512.lua │   │   ├── sha.lua │   │   ├── shell.lua │   │   ├── signal.lua │   │   ├── string.lua │   │   ├── upload.lua │   │   └── upstream │   │   └── healthcheck.lua │   └── tablepool.lua ├── mime.types ├── mime.types.default ├── modules │   ├── ngx_http_delay_module.so │   ├── ngx_http_naxsi_module.so │   ├── ngx_http_replace_filter_module.so │   └── ngx_http_sysguard_module.so ├── nginx │   └── html │   ├── 50x.html │   └── index.html ├── nginx.conf ├── nginx.conf.default ├── pod │   ├── array-var-nginx-module-0.05 │   │   └── array-var-nginx-module-0.05.pod │   ├── drizzle-nginx-module-0.1.11 │   │   └── drizzle-nginx-module-0.1.11.pod │   ├── echo-nginx-module-0.61 │   │   └── echo-nginx-module-0.61.pod │   ├── encrypted-session-nginx-module-0.08 │   │   └── encrypted-session-nginx-module-0.08.pod │   ├── form-input-nginx-module-0.12 │   │   └── form-input-nginx-module-0.12.pod │   ├── headers-more-nginx-module-0.33 │   │   └── headers-more-nginx-module-0.33.pod │   ├── iconv-nginx-module-0.14 │   │   └── iconv-nginx-module-0.14.pod │   ├── lua-5.1.5 │   │   └── lua-5.1.5.pod │   ├── lua-cjson-2.1.0.7 │   │   └── lua-cjson-2.1.0.7.pod │   ├── luajit-2.1 │   │   ├── changes.pod │   │   ├── contact.pod │   │   ├── ext_c_api.pod │   │   ├── extensions.pod │   │   ├── ext_ffi_api.pod │   │   ├── ext_ffi.pod │   │   ├── ext_ffi_semantics.pod │   │   ├── ext_ffi_tutorial.pod │   │   ├── ext_jit.pod │   │   ├── ext_profiler.pod │   │   ├── faq.pod │   │   ├── install.pod │   │   ├── luajit-2.1.pod │   │   ├── running.pod │   │   └── status.pod │   ├── luajit-2.1-20190507 │   │   └── luajit-2.1-20190507.pod │   ├── lua-rds-parser-0.06 │   ├── lua-redis-parser-0.13 │   │   └── lua-redis-parser-0.13.pod │   ├── lua-resty-core-0.1.17 │   │   ├── lua-resty-core-0.1.17.pod │   │   ├── ngx.balancer.pod │   │   ├── ngx.base64.pod │   │   ├── ngx.errlog.pod │   │   ├── ngx.ocsp.pod │   │   ├── ngx.pipe.pod │   │   ├── ngx.process.pod │   │   ├── ngx.re.pod │   │   ├── ngx.resp.pod │   │   ├── ngx.semaphore.pod │   │   ├── ngx.ssl.pod │   │   └── ngx.ssl.session.pod │   ├── lua-resty-dns-0.21 │   │   └── lua-resty-dns-0.21.pod │   ├── lua-resty-limit-traffic-0.06 │   │   ├── lua-resty-limit-traffic-0.06.pod │   │   ├── resty.limit.conn.pod │   │   ├── resty.limit.count.pod │   │   ├── resty.limit.req.pod │   │   └── resty.limit.traffic.pod │   ├── lua-resty-lock-0.08 │   │   └── lua-resty-lock-0.08.pod │   ├── lua-resty-lrucache-0.09 │   │   └── lua-resty-lrucache-0.09.pod │   ├── lua-resty-memcached-0.14 │   │   └── lua-resty-memcached-0.14.pod │   ├── lua-resty-mysql-0.21 │   │   └── lua-resty-mysql-0.21.pod │   ├── lua-resty-redis-0.27 │   │   └── lua-resty-redis-0.27.pod │   ├── lua-resty-shell-0.02 │   │   └── lua-resty-shell-0.02.pod │   ├── lua-resty-signal-0.02 │   │   └── lua-resty-signal-0.02.pod │   ├── lua-resty-string-0.11 │   │   └── lua-resty-string-0.11.pod │   ├── lua-resty-upload-0.10 │   │   └── lua-resty-upload-0.10.pod │   ├── lua-resty-upstream-healthcheck-0.06 │   │   └── lua-resty-upstream-healthcheck-0.06.pod │   ├── lua-resty-websocket-0.07 │   │   └── lua-resty-websocket-0.07.pod │   ├── lua-tablepool-0.01 │   │   └── lua-tablepool-0.01.pod │   ├── memc-nginx-module-0.19 │   │   └── memc-nginx-module-0.19.pod │   ├── nginx │   │   ├── accept_failed.pod │   │   ├── beginners_guide.pod │   │   ├── chunked_encoding_from_backend.pod │   │   ├── configure.pod │   │   ├── configuring_https_servers.pod │   │   ├── contributing_changes.pod │   │   ├── control.pod │   │   ├── converting_rewrite_rules.pod │   │   ├── daemon_master_process_off.pod │   │   ├── debugging_log.pod │   │   ├── development_guide.pod │   │   ├── events.pod │   │   ├── example.pod │   │   ├── faq.pod │   │   ├── freebsd_tuning.pod │   │   ├── hash.pod │   │   ├── howto_build_on_win32.pod │   │   ├── install.pod │   │   ├── license_copyright.pod │   │   ├── load_balancing.pod │   │   ├── nginx_dtrace_pid_provider.pod │   │   ├── nginx.pod │   │   ├── ngx_core_module.pod │   │   ├── ngx_google_perftools_module.pod │   │   ├── ngx_http_access_module.pod │   │   ├── ngx_http_addition_module.pod │   │   ├── ngx_http_api_module_head.pod │   │   ├── ngx_http_auth_basic_module.pod │   │   ├── ngx_http_auth_jwt_module.pod │   │   ├── ngx_http_auth_request_module.pod │   │   ├── ngx_http_autoindex_module.pod │   │   ├── ngx_http_browser_module.pod │   │   ├── ngx_http_charset_module.pod │   │   ├── ngx_http_core_module.pod │   │   ├── ngx_http_dav_module.pod │   │   ├── ngx_http_empty_gif_module.pod │   │   ├── ngx_http_f4f_module.pod │   │   ├── ngx_http_fastcgi_module.pod │   │   ├── ngx_http_flv_module.pod │   │   ├── ngx_http_geoip_module.pod │   │   ├── ngx_http_geo_module.pod │   │   ├── ngx_http_grpc_module.pod │   │   ├── ngx_http_gunzip_module.pod │   │   ├── ngx_http_gzip_module.pod │   │   ├── ngx_http_gzip_static_module.pod │   │   ├── ngx_http_headers_module.pod │   │   ├── ngx_http_hls_module.pod │   │   ├── ngx_http_image_filter_module.pod │   │   ├── ngx_http_index_module.pod │   │   ├── ngx_http_js_module.pod │   │   ├── ngx_http_keyval_module.pod │   │   ├── ngx_http_limit_conn_module.pod │   │   ├── ngx_http_limit_req_module.pod │   │   ├── ngx_http_log_module.pod │   │   ├── ngx_http_map_module.pod │   │   ├── ngx_http_memcached_module.pod │   │   ├── ngx_http_mirror_module.pod │   │   ├── ngx_http_mp4_module.pod │   │   ├── ngx_http_perl_module.pod │   │   ├── ngx_http_proxy_module.pod │   │   ├── ngx_http_random_index_module.pod │   │   ├── ngx_http_realip_module.pod │   │   ├── ngx_http_referer_module.pod │   │   ├── ngx_http_rewrite_module.pod │   │   ├── ngx_http_scgi_module.pod │   │   ├── ngx_http_secure_link_module.pod │   │   ├── ngx_http_session_log_module.pod │   │   ├── ngx_http_slice_module.pod │   │   ├── ngx_http_spdy_module.pod │   │   ├── ngx_http_split_clients_module.pod │   │   ├── ngx_http_ssi_module.pod │   │   ├── ngx_http_ssl_module.pod │   │   ├── ngx_http_status_module.pod │   │   ├── ngx_http_stub_status_module.pod │   │   ├── ngx_http_sub_module.pod │   │   ├── ngx_http_upstream_conf_module.pod │   │   ├── ngx_http_upstream_hc_module.pod │   │   ├── ngx_http_upstream_module.pod │   │   ├── ngx_http_userid_module.pod │   │   ├── ngx_http_uwsgi_module.pod │   │   ├── ngx_http_v2_module.pod │   │   ├── ngx_http_xslt_module.pod │   │   ├── ngx_mail_auth_http_module.pod │   │   ├── ngx_mail_core_module.pod │   │   ├── ngx_mail_imap_module.pod │   │   ├── ngx_mail_pop3_module.pod │   │   ├── ngx_mail_proxy_module.pod │   │   ├── ngx_mail_smtp_module.pod │   │   ├── ngx_mail_ssl_module.pod │   │   ├── ngx_stream_access_module.pod │   │   ├── ngx_stream_core_module.pod │   │   ├── ngx_stream_geoip_module.pod │   │   ├── ngx_stream_geo_module.pod │   │   ├── ngx_stream_js_module.pod │   │   ├── ngx_stream_keyval_module.pod │   │   ├── ngx_stream_limit_conn_module.pod │   │   ├── ngx_stream_log_module.pod │   │   ├── ngx_stream_map_module.pod │   │   ├── ngx_stream_proxy_module.pod │   │   ├── ngx_stream_realip_module.pod │   │   ├── ngx_stream_return_module.pod │   │   ├── ngx_stream_split_clients_module.pod │   │   ├── ngx_stream_ssl_module.pod │   │   ├── ngx_stream_ssl_preread_module.pod │   │   ├── ngx_stream_upstream_hc_module.pod │   │   ├── ngx_stream_upstream_module.pod │   │   ├── ngx_stream_zone_sync_module.pod │   │   ├── request_processing.pod │   │   ├── server_names.pod │   │   ├── stream_processing.pod │   │   ├── switches.pod │   │   ├── syntax.pod │   │   ├── sys_errlist.pod │   │   ├── syslog.pod │   │   ├── variables_in_config.pod │   │   ├── websocket.pod │   │   ├── welcome_nginx_facebook.pod │   │   └── windows.pod │   ├── ngx_coolkit-0.2 │   ├── ngx_devel_kit-0.3.1rc1 │   │   └── ngx_devel_kit-0.3.1rc1.pod │   ├── ngx_lua-0.10.15 │   │   └── ngx_lua-0.10.15.pod │   ├── ngx_lua_upstream-0.07 │   │   └── ngx_lua_upstream-0.07.pod │   ├── ngx_postgres-1.0 │   │   ├── ngx_postgres-1.0.pod │   │   └── todo.pod │   ├── ngx_stream_lua-0.0.7 │   │   ├── dev_notes.pod │   │   └── ngx_stream_lua-0.0.7.pod │   ├── opm-0.0.5 │   │   └── opm-0.0.5.pod │   ├── rds-csv-nginx-module-0.09 │   │   └── rds-csv-nginx-module-0.09.pod │   ├── rds-json-nginx-module-0.15 │   │   └── rds-json-nginx-module-0.15.pod │   ├── redis2-nginx-module-0.15 │   │   └── redis2-nginx-module-0.15.pod │   ├── redis-nginx-module-0.3.7 │   ├── resty-cli-0.24 │   │   └── resty-cli-0.24.pod │   ├── set-misc-nginx-module-0.32 │   │   └── set-misc-nginx-module-0.32.pod │   ├── srcache-nginx-module-0.31 │   │   └── srcache-nginx-module-0.31.pod │   └── xss-nginx-module-0.06 │   └── xss-nginx-module-0.06.pod ├── resty.index ├── scgi_params ├── scgi_params.default ├── site │   ├── lualib │   ├── manifest │   └── pod ├── uwsgi_params ├── uwsgi_params.default └── win-utf 78 directories, 305 files ``` ###### Post installation tasks > Check all post installation tasks from [Nginx on CentOS 7 - Post installation tasks](#post-installation-tasks) section.
#### Installation Tengine on Ubuntu 18.04 > _Tengine is a web server originated by Taobao, the largest e-commerce website in Asia. It is based on the NGINX HTTP server and has many advanced features. There’s a lot of features in Tengine that do not (yet) exist in NGINX._ - Official github repository: [Tengine](https://github.com/alibaba/tengine) - Official documentation: [Tengine Documentation](https://tengine.taobao.org/documentation.html) Generally, Tengine is a great solution, including many patches, improvements, additional modules, and most importantly it is very actively maintained. The build and installation process is very similar to [Installation Nginx on CentOS 7](#installation-nginx-on-centos-7). However, I will only specify the most important changes.
Show step-by-step Tengine installation
* [Pre installation tasks](#pre-installation-tasks-2) * [Dependencies](#dependencies-2) * [Get Tengine sources](#get-tengine-sources) * [Download 3rd party modules](#download-3rd-party-modules-2) * [Build Tengine](#build-tengine) * [Post installation tasks](#post-installation-tasks-2) ###### Pre installation tasks Set the Tengine version (I use newest and stable release): ```bash export ngx_version="2.3.0" ``` Set temporary variables: ```bash export ngx_src="/usr/local/src" export ngx_base="${ngx_src}/nginx-${ngx_version}" export ngx_master="${ngx_base}/master" export ngx_modules="${ngx_base}/modules" export NGX_PREFIX="/etc/nginx" export NGX_CONF="${NGX_PREFIX}/nginx.conf" ``` Create directories: ```bash for i in "${ngx_base}" "${ngx_master}" "${ngx_modules}" ; do mkdir "$i" done ``` Set user/group variables: ```bash export NGINX_USER="nginx" export NGINX_GROUP="nginx" export NGINX_UID="920" export NGINX_GID="920" ``` ###### Dependencies Install prebuilt packages, export variables and set symbolic link: ```bash apt-get install gcc make build-essential bison perl libperl-dev lua5.1 libphp-embed libxslt-dev libgd-dev libgeoip-dev libxml2-dev libexpat-dev libgoogle-perftools-dev libgoogle-perftools4 autoconf jq git wget logrotate # In this example we don't use zlib sources: apt-get install zlib1g-dev ``` PCRE: ```bash cd "${ngx_src}" export pcre_version="8.42" export PCRE_SRC="${ngx_base}/pcre-${pcre_version}" export PCRE_LIB="/usr/local/lib" export PCRE_INC="/usr/local/include" wget -c --no-check-certificate https://ftp.pcre.org/pub/pcre/pcre-${pcre_version}.tar.gz && tar xzvf pcre-${pcre_version}.tar.gz cd "$PCRE_SRC" # Add to compile with debugging symbols: # CFLAGS='-O0 -g' ./configure ./configure make -j2 && make test make install ``` OpenSSL: ```bash cd "${ngx_src}" export openssl_version="1.1.1c" export OPENSSL_SRC="${ngx_src}/openssl-${openssl_version}" export OPENSSL_DIR="/usr/local/openssl-${openssl_version}" export OPENSSL_LIB="${OPENSSL_DIR}/lib" export OPENSSL_INC="${OPENSSL_DIR}/include" wget -c --no-check-certificate https://www.openssl.org/source/openssl-${openssl_version}.tar.gz && tar xzvf openssl-${openssl_version}.tar.gz cd "${ngx_src}/openssl-${openssl_version}" # Please run this and add as a compiler param: export __GCC_SSL=("__SIZEOF_INT128__:enable-ec_nistp_64_gcc_128") for _cc_opt in "${__GCC_SSL[@]}" ; do _cc_key=$(echo "$_cc_opt" | cut -d ":" -f1) _cc_value=$(echo "$_cc_opt" | cut -d ":" -f2) if [[ ! $(gcc -dM -E - /etc/profile.d/openssl.sh << __EOF__ #!/bin/sh export PATH=${OPENSSL_DIR}/bin:${PATH} export LD_LIBRARY_PATH=${OPENSSL_DIR}/lib:${LD_LIBRARY_PATH} __EOF__ chmod +x /etc/profile.d/openssl.sh && source /etc/profile.d/openssl.sh # To make the OpenSSL version visible globally first: if [[ -e "/usr/bin/openssl" ]] ; then _openssl_version=$(openssl version | awk '{print $2}') _openssl_date=$(date '+%Y%m%d%H%M%S') _openssl_str="openssl-${_openssl_version}-${_openssl_date}" mv /usr/bin/openssl /usr/bin/${_openssl_str} ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl else ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl fi cat > /etc/ld.so.conf.d/openssl.conf << __EOF__ ${OPENSSL_DIR}/lib __EOF__ ``` LuaJIT: ```bash # I recommend to use OpenResty's branch (openresty/luajit2) instead of LuaJIT (LuaJIT/LuaJIT), but both installation methods are similar: cd "${ngx_src}" export LUAJIT_SRC="${ngx_src}/luajit2" export LUAJIT_LIB="/usr/local/lib" # For original LuaJIT: # export LUAJIT_INC="/usr/local/include/luajit-2.0" # git clone http://luajit.org/git/luajit-2.0.git luajit2 # For OpenResty's LuaJIT: export LUAJIT_INC="/usr/local/include/luajit-2.1" git clone --depth 1 https://github.com/openresty/luajit2 cd "$LUAJIT_SRC" # Add to compile with debugging symbols: # CFLAGS='-g' make ... make && make install for i in libluajit-5.1.so libluajit-5.1.so.2 liblua.so libluajit.so ; do # For original LuaJIT: # ln -sf /usr/local/lib/libluajit-5.1.so.2.0.5 ${LUAJIT_LIB}/${i} # For OpenResty's LuaJIT: ln -sf /usr/local/lib/libluajit-5.1.so.2.1.0 ${LUAJIT_LIB}/${i} done # ln -sf /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 ${LUAJIT_LIB}/liblua.so ``` sregex: > Required for `replace-filter-nginx-module` module. ```bash cd "${ngx_src}" git clone --depth 1 https://github.com/openresty/sregex cd "${ngx_src}/sregex" make && make install ``` jemalloc: > To verify `jemalloc` in use: `lsof -n | grep jemalloc`. ```bash cd "${ngx_src}" export JEMALLOC_SRC="/usr/local/src/jemalloc" export JEMALLOC_INC="/usr/local/include/jemalloc" git clone --depth 1 https://github.com/jemalloc/jemalloc cd "$JEMALLOC_SRC" ./autogen.sh make && make install ``` Update links and cache to the shared libraries for both types of installation: ```bash ldconfig ``` ###### Get Tengine sources ```bash cd "${ngx_base}" wget -c --no-check-certificate https://tengine.taobao.org/download/tengine-${ngx_version}.tar.gz # or alternative: # git clone --depth 1 https://github.com/alibaba/tengine master tar zxvf tengine-${ngx_version}.tar.gz -C "${ngx_master}" ``` ###### Download 3rd party modules > Not all modules from [this](#3rd-party-modules) section working properly with Tengine (e.g. `ndk_http_module` and other dependent on it). ```bash cd "${ngx_modules}" for i in \ https://github.com/openresty/echo-nginx-module \ https://github.com/openresty/headers-more-nginx-module \ https://github.com/openresty/replace-filter-nginx-module \ https://github.com/nginx-clojure/nginx-access-plus \ https://github.com/yaoweibin/ngx_http_substitutions_filter_module \ https://github.com/vozlt/nginx-module-vts \ https://github.com/google/ngx_brotli ; do git clone --depth 1 "$i" done wget -c --no-check-certificate http://mdounin.ru/hg/ngx_http_delay_module/archive/tip.tar.gz -O delay-module.tar.gz mkdir delay-module && tar xzvf delay-module.tar.gz -C delay-module --strip 1 ``` For `ngx_brotli`: ```bash cd "${ngx_modules}/ngx_brotli" git submodule update --init ``` If you use NAXSI: ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/nbs-system/naxsi ``` ###### Build Tengine ```bash cd "${ngx_master}" # - you can also build Tengine without 3rd party modules # - remember about compiler and linker options # - don't set values for --with-openssl, --with-pcre, and --with-zlib if you select prebuilt packages for them # - add to compile with debugging symbols: -O0 -g # - and remove -D_FORTIFY_SOURCE=2 if you use above ./configure --prefix=$NGX_PREFIX \ --conf-path=$NGX_CONF \ --sbin-path=/usr/sbin/nginx \ --pid-path=/var/run/nginx.pid \ --lock-path=/var/run/nginx.lock \ --user=$NGINX_USER \ --group=$NGINX_GROUP \ --modules-path=${NGX_PREFIX}/modules \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --http-client-body-temp-path=/var/cache/nginx/client_temp \ --http-proxy-temp-path=/var/cache/nginx/proxy_temp \ --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \ --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \ --http-scgi-temp-path=/var/cache/nginx/scgi_temp \ --with-compat \ --with-debug \ --with-file-aio \ --with-threads \ --with-stream \ --with-stream_geoip_module \ --with-stream_realip_module \ --with-stream_ssl_module \ --with-stream_ssl_preread_module \ --with-http_addition_module \ --with-http_auth_request_module \ --with-http_degradation_module \ --with-http_geoip_module \ --with-http_gunzip_module \ --with-http_gzip_static_module \ --with-http_image_filter_module \ --with-http_lua_module \ --with-http_perl_module \ --with-http_random_index_module \ --with-http_realip_module \ --with-http_secure_link_module \ --with-http_ssl_module \ --with-http_stub_status_module \ --with-http_sub_module \ --with-http_v2_module \ --with-google_perftools_module \ --with-openssl=${OPENSSL_SRC} \ --with-openssl-opt="shared zlib no-ssl3 no-weak-ssl-ciphers -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong ${_openssl_gcc}" \ --with-pcre=${PCRE_SRC} \ --with-pcre-jit \ --with-jemalloc=${JEMALLOC_SRC} \ --without-http-cache \ --without-http_memcached_module \ --without-mail_pop3_module \ --without-mail_imap_module \ --without-mail_smtp_module \ --without-http_fastcgi_module \ --without-http_scgi_module \ --without-http_uwsgi_module \ --without-http_upstream_keepalive_module \ --add-module=${ngx_master}/modules/ngx_backtrace_module \ --add-module=${ngx_master}/modules/ngx_debug_pool \ --add-module=${ngx_master}/modules/ngx_debug_timer \ --add-module=${ngx_master}/modules/ngx_http_footer_filter_module \ --add-module=${ngx_master}/modules/ngx_http_lua_module \ --add-module=${ngx_master}/modules/ngx_http_proxy_connect_module \ --add-module=${ngx_master}/modules/ngx_http_reqstat_module \ --add-module=${ngx_master}/modules/ngx_http_slice_module \ --add-module=${ngx_master}/modules/ngx_http_sysguard_module \ --add-module=${ngx_master}/modules/ngx_http_trim_filter_module \ --add-module=${ngx_master}/modules/ngx_http_upstream_check_module \ --add-module=${ngx_master}/modules/ngx_http_upstream_consistent_hash_module \ --add-module=${ngx_master}/modules/ngx_http_upstream_dynamic_module \ --add-module=${ngx_master}/modules/ngx_http_upstream_keepalive_module \ --add-module=${ngx_master}/modules/ngx_http_upstream_session_sticky_module \ --add-module=${ngx_master}/modules/ngx_http_user_agent_module \ --add-module=${ngx_master}/modules/ngx_slab_stat \ --add-module=${ngx_modules}/nginx-access-plus/src/c \ --add-module=${ngx_modules}/ngx_http_substitutions_filter_module \ --add-module=${ngx_modules}/nginx-module-vts \ --add-module=${ngx_modules}/ngx_brotli \ --add-dynamic-module=${ngx_modules}/echo-nginx-module \ --add-dynamic-module=${ngx_modules}/headers-more-nginx-module \ --add-dynamic-module=${ngx_modules}/replace-filter-nginx-module \ --add-dynamic-module=${ngx_modules}/delay-module \ --add-dynamic-module=${ngx_modules}/naxsi/naxsi_src \ --with-cc-opt="-I/usr/local/include -I${OPENSSL_INC} -I${LUAJIT_INC} -I${JEMALLOC_INC} -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC" \ --with-ld-opt="-Wl,-E -L/usr/local/lib -ljemalloc -lpcre -Wl,-rpath,/usr/local/lib/,-z,relro -Wl,-z,now -pie" make -j2 && make test make install ldconfig ``` Show Tengine version and parameters: ```bash nginx -V ``` And list all files in `/etc/nginx`: ```bash tree . ├── fastcgi.conf ├── fastcgi.conf.default ├── fastcgi_params ├── fastcgi_params.default ├── html │   ├── 50x.html │   └── index.html ├── koi-utf ├── koi-win ├── mime.types ├── mime.types.default ├── modules │   ├── ngx_http_delay_module.so │   ├── ngx_http_echo_module.so │   ├── ngx_http_headers_more_filter_module.so │   ├── ngx_http_naxsi_module.so │   └── ngx_http_replace_filter_module.so ├── nginx.conf ├── nginx.conf.default ├── scgi_params ├── scgi_params.default ├── uwsgi_params ├── uwsgi_params.default └── win-utf 2 directories, 22 files ``` ###### Post installation tasks > Check all post installation tasks from [Nginx on CentOS 7 - Post installation tasks](#post-installation-tasks) section.
#### Installation Nginx on FreeBSD 11.3 > The build and installation process is very similar to [Installation Nginx on CentOS 7](#installation-nginx-on-centos-7). However, I will only specify the most important changes. On FreeBSD you can also build NGINX from ports.
Show step-by-step NGINX installation
* [Pre installation tasks](#pre-installation-tasks-3) * [Dependencies](#dependencies-3) * [Get OpenResty sources](#get-openresty-sources-3) * [Download 3rd party modules](#download-3rd-party-modules-3) * [Build Nginx](#build-nginx-1) * [Post installation tasks](#post-installation-tasks-3) ###### Pre installation tasks Set NGINX version (I use stable release): ```bash export ngx_version="1.16.0" ``` Set temporary variables: ```bash export ngx_src="/usr/local/src" export ngx_base="${ngx_src}/nginx-${ngx_version}" export ngx_master="${ngx_base}/master" export ngx_modules="${ngx_base}/modules" export NGX_PREFIX="/etc/nginx" export NGX_CONF="${NGX_PREFIX}/nginx.conf" ``` Create directories: ```bash for i in "${ngx_base}" "${ngx_master}" "${ngx_modules}" ; do mkdir "$i" done ``` Set user/group variables: ```bash export NGINX_USER="nginx" export NGINX_GROUP="nginx" export NGINX_UID="920" export NGINX_GID="920" ``` ###### Dependencies > In my configuration I used all prebuilt dependencies without `openssl`, `zlib`, `luajit`, and `pcre` because I compiled them manually - for TLS 1.3 support and with OpenResty recommendation for LuaJIT. **Install prebuilt packages, export variables and set symbolic link:** ```bash # It's important and required, regardless of chosen sources: pkg install gcc gmake bison perl5-devel lua51 libxslt libgd libxml2 expat autoconf jq git wget ncurses texinfo gettext gettext-tools # In this example we use sources for all below packages so we do not install them: # pkg install pcre luajit # For LuaJIT (luajit): export LUAJIT_SRC="${ngx_src}/luajit2" export LUAJIT_LIB="/usr/local/lib" # For original LuaJIT: # export LUAJIT_INC="/usr/local/include/luajit-2.0" # git clone http://luajit.org/git/luajit-2.0.git luajit2 # For OpenResty's LuaJIT: export LUAJIT_INC="/usr/local/include/luajit-2.1" git clone --depth 1 https://github.com/openresty/luajit2 ``` > Remember to build [`sregex`](#sregex) also if you use above steps. **Or download and compile them:** PCRE: ```bash cd "${ngx_src}" export pcre_version="8.42" export PCRE_SRC="${ngx_src}/pcre-${pcre_version}" export PCRE_LIB="/usr/local/lib" export PCRE_INC="/usr/local/include" wget -c --no-check-certificate https://ftp.pcre.org/pub/pcre/pcre-${pcre_version}.tar.gz && tar xzvf pcre-${pcre_version}.tar.gz cd "$PCRE_SRC" # Add to compile with debugging symbols: # CFLAGS='-O0 -g' ./configure ./configure make -j2 && make test make install ``` Zlib: ```bash # I recommend to use Cloudflare Zlib version (cloudflare/zlib) instead an original Zlib (zlib.net), but both installation methods are similar: cd "${ngx_src}" export ZLIB_SRC="${ngx_src}/zlib" export ZLIB_LIB="/usr/local/lib" export ZLIB_INC="/usr/local/include" # For original Zlib: # export zlib_version="1.2.11" # wget -c --no-check-certificate http://www.zlib.net/zlib-${zlib_version}.tar.gz && tar xzvf zlib-${zlib_version}.tar.gz or git clone --depth 1 https://github.com/madler/zlib # cd "${ZLIB_SRC}-${zlib_version}" # For Cloudflare Zlib: git clone --depth 1 https://github.com/cloudflare/zlib cd "$ZLIB_SRC" ./configure gmake -j2 && gmake test gmake install ``` OpenSSL: ```bash cd "${ngx_src}" export openssl_version="1.1.1c" export OPENSSL_SRC="${ngx_src}/openssl-${openssl_version}" export OPENSSL_DIR="/usr/local/openssl-${openssl_version}" export OPENSSL_LIB="${OPENSSL_DIR}/lib" export OPENSSL_INC="${OPENSSL_DIR}/include" wget -c --no-check-certificate https://www.openssl.org/source/openssl-${openssl_version}.tar.gz && tar xzvf openssl-${openssl_version}.tar.gz cd "${ngx_src}/openssl-${openssl_version}" # Please run this and add as a compiler param: export __GCC_SSL=("") for _cc_opt in "${__GCC_SSL[@]}" ; do _cc_key=$(echo "$_cc_opt" | cut -d ":" -f1) _cc_value=$(echo "$_cc_opt" | cut -d ":" -f2) if [[ ! $(gcc -dM -E - > /etc/make.conf fi # After above, you will have to rebuild all required packages for build NGINX from sources. make -j2 && make test make install if [[ -e "/usr/bin/openssl" ]] ; then _openssl_version=$(openssl version | awk '{print $2}') _openssl_date=$(date '+%Y%m%d%H%M%S') _openssl_str="openssl-${_openssl_version}-${_openssl_date}" mv /usr/bin/openssl /usr/bin/${_openssl_str} ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl else ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl fi for i in libssl.so.1.1 libcrypto.so.1.1 ; do ln -sf ${ngx_src}/openssl-${openssl_version}/${i} /usr/lib/ done ``` Update links and cache to the shared libraries for both types of installation: ```bash ldconfig ``` LuaJIT: ```bash # I recommend to use OpenResty's branch (openresty/luajit2) instead of LuaJIT (LuaJIT/LuaJIT), but both installation methods are similar: cd "${ngx_src}" export LUAJIT_SRC="${ngx_src}/luajit2" export LUAJIT_LIB="/usr/local/lib" # For original LuaJIT: export LUAJIT_INC="/usr/local/include/luajit-2.0" git clone http://luajit.org/git/luajit-2.0.git luajit2 # For OpenResty's LuaJIT: # export LUAJIT_INC="/usr/local/include/luajit-2.1" # git clone --depth 1 https://github.com/openresty/luajit2 cd "$LUAJIT_SRC" # Add to compile with debugging symbols: # CFLAGS='-g' make ... gmake && gmake install # On FreeBSD you should set them manually or use the following instructions: for i in libluajit-5.1.so libluajit-5.1.so.2 liblua.so libluajit.so ; do # For original LuaJIT: ln -sf /usr/local/lib/libluajit-5.1.so.2.0.5 ${LUAJIT_LIB}/${i} # For OpenResty's LuaJIT: # ln -sf /usr/local/lib/libluajit-5.1.so.2.1.0 ${LUAJIT_LIB}/${i} done # ln -sf /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 ${LUAJIT_LIB}/liblua.so # Without this you get: /usr/bin/ld: /usr/local/lib/libluajit-5.1.a(lj_err.o): relocation R_X86_64_32S against `a local symbol' can not be used when making a shared object; recompile with -fPIC /usr/local/lib/libluajit-5.1.a: could not read symbols: Bad value cc: error: linker command failed with exit code 1 (use -v to see invocation) gmake[1]: *** [objs/Makefile:2165: objs/ngx_http_lua_module.so] Error 1 # Because: cd src && test -f libluajit.so && \ install -m 0755 libluajit.so /usr/local/lib/libluajit-5.1.so.2.1.0 && \ ldconfig -n /usr/local/lib && \ ln -sf libluajit-5.1.so.2.1.0 /usr/local/lib/libluajit-5.1.so && \ ln -sf libluajit-5.1.so.2.1.0 /usr/local/lib/libluajit-5.1.so.2 || : ldconfig: illegal option -- n usage: ldconfig [-32] [-aout | -elf] [-Rimrsv] [-f hints_file] [directory | file ...] ``` sregex: > Required for `replace-filter-nginx-module` module. ```bash cd "${ngx_src}" git clone --depth 1 https://github.com/openresty/sregex cd "${ngx_src}/sregex" gmake && gmake install ``` jemalloc: > To verify `jemalloc` in use: `lsof -n | grep jemalloc`. ```bash cd "${ngx_src}" export JEMALLOC_SRC="${ngx_src}/jemalloc" export JEMALLOC_INC="/usr/local/include/jemalloc" git clone --depth 1 https://github.com/jemalloc/jemalloc cd "$JEMALLOC_SRC" ./autogen.sh gmake && gmake install ``` Update links and cache to the shared libraries for both types of installation: ```bash ldconfig ``` ###### Get Nginx sources ```bash cd "${ngx_base}" wget -c --no-check-certificate https://nginx.org/download/nginx-${ngx_version}.tar.gz # or alternative: # git clone --depth 1 https://github.com/nginx/nginx master tar zxvf nginx-${ngx_version}.tar.gz -C "${ngx_master}" --strip 1 ``` ###### Download 3rd party modules ```bash cd "${ngx_modules}" for i in \ https://github.com/simplresty/ngx_devel_kit \ https://github.com/openresty/lua-nginx-module \ https://github.com/openresty/set-misc-nginx-module \ https://github.com/openresty/echo-nginx-module \ https://github.com/openresty/headers-more-nginx-module \ https://github.com/openresty/replace-filter-nginx-module \ https://github.com/openresty/array-var-nginx-module \ https://github.com/openresty/encrypted-session-nginx-module \ https://github.com/vozlt/nginx-module-sysguard \ https://github.com/nginx-clojure/nginx-access-plus \ https://github.com/yaoweibin/ngx_http_substitutions_filter_module \ https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng \ https://github.com/vozlt/nginx-module-vts \ https://github.com/google/ngx_brotli ; do git clone --depth 1 "$i" done wget -c --no-check-certificate http://mdounin.ru/hg/ngx_http_delay_module/archive/tip.tar.gz -O delay-module.tar.gz mkdir delay-module && tar xzvf delay-module.tar.gz -C delay-module --strip 1 ``` For `ngx_brotli`: ```bash cd "${ngx_modules}/ngx_brotli" git submodule update --init ``` I also use some modules from Tengine: - `ngx_backtrace_module` (build error) - `ngx_debug_pool` - `ngx_debug_timer` - `ngx_http_upstream_check_module` - `ngx_http_footer_filter_module` ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/alibaba/tengine ``` If you use NAXSI: ```bash cd "${ngx_modules}" git clone --depth 1 https://github.com/nbs-system/naxsi ``` ###### Build Nginx ```bash cd "${ngx_master}" # - you can also build NGINX without 3rd party modules # - remember about compiler and linker options # - don't set values for --with-openssl, --with-pcre, and --with-zlib if you select prebuilt packages for them # - add to compile with debugging symbols: -O0 -g # - and remove -D_FORTIFY_SOURCE=2 if you use above ./configure --prefix=$NGX_PREFIX \ --conf-path=$NGX_CONF \ --sbin-path=/usr/sbin/nginx \ --pid-path=/var/run/nginx.pid \ --lock-path=/var/run/nginx.lock \ --user=$NGINX_USER \ --group=$NGINX_GROUP \ --modules-path=${NGX_PREFIX}/modules \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --http-client-body-temp-path=/var/cache/nginx/client_temp \ --http-proxy-temp-path=/var/cache/nginx/proxy_temp \ --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \ --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \ --http-scgi-temp-path=/var/cache/nginx/scgi_temp \ --with-compat \ --with-debug \ --with-file-aio \ --with-threads \ --with-stream \ --with-stream_realip_module \ --with-stream_ssl_module \ --with-stream_ssl_preread_module \ --with-http_addition_module \ --with-http_auth_request_module \ --with-http_degradation_module \ --with-http_gunzip_module \ --with-http_gzip_static_module \ --with-http_image_filter_module \ --with-http_perl_module \ --with-http_random_index_module \ --with-http_realip_module \ --with-http_secure_link_module \ --with-http_ssl_module \ --with-http_stub_status_module \ --with-http_sub_module \ --with-http_v2_module \ --with-openssl=${OPENSSL_SRC} \ --with-openssl-opt="shared zlib no-ssl3 no-weak-ssl-ciphers -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong ${_openssl_gcc}" \ --with-pcre=${PCRE_SRC} \ --with-pcre-jit \ --with-zlib=${ZLIB_SRC} \ --without-http-cache \ --without-http_memcached_module \ --without-mail_pop3_module \ --without-mail_imap_module \ --without-mail_smtp_module \ --without-http_fastcgi_module \ --without-http_scgi_module \ --without-http_uwsgi_module \ --add-module=${ngx_modules}/ngx_devel_kit \ --add-module=${ngx_modules}/encrypted-session-nginx-module \ --add-module=${ngx_modules}/nginx-access-plus/src/c \ --add-module=${ngx_modules}/ngx_http_substitutions_filter_module \ --add-module=${ngx_modules}/nginx-sticky-module-ng \ --add-module=${ngx_modules}/nginx-module-vts \ --add-module=${ngx_modules}/ngx_brotli \ --add-module=${ngx_modules}/tengine/modules/ngx_debug_pool \ --add-module=${ngx_modules}/tengine/modules/ngx_debug_timer \ --add-module=${ngx_modules}/tengine/modules/ngx_http_footer_filter_module \ --add-module=${ngx_modules}/tengine/modules/ngx_http_upstream_check_module \ --add-module=${ngx_modules}/tengine/modules/ngx_slab_stat \ --add-dynamic-module=${ngx_modules}/lua-nginx-module \ --add-dynamic-module=${ngx_modules}/set-misc-nginx-module \ --add-dynamic-module=${ngx_modules}/echo-nginx-module \ --add-dynamic-module=${ngx_modules}/headers-more-nginx-module \ --add-dynamic-module=${ngx_modules}/replace-filter-nginx-module \ --add-dynamic-module=${ngx_modules}/array-var-nginx-module \ --add-dynamic-module=${ngx_modules}/nginx-module-sysguard \ --add-dynamic-module=${ngx_modules}/delay-module \ --add-dynamic-module=${ngx_modules}/naxsi/naxsi_src \ --with-cc-opt="-I/usr/local/include" \ --with-ld-opt="-L/usr/local/lib" # Unused modules (build errors): # --with-http_geoip_module \ # --with-google_perftools_module \ # --add-module=${ngx_modules}/tengine/modules/ngx_backtrace_module \ gmake -j2 && gmake test gmake install ldconfig ``` Show NGINX version and parameters: ```bash nginx -V ``` And list all files in `/etc/nginx`: ```bash . |-- fastcgi.conf |-- fastcgi.conf.default |-- fastcgi_params |-- fastcgi_params.default |-- html | |-- 50x.html | `-- index.html |-- koi-utf |-- koi-win |-- mime.types |-- mime.types.default |-- modules | |-- ngx_http_array_var_module.so | |-- ngx_http_delay_module.so | |-- ngx_http_echo_module.so | |-- ngx_http_headers_more_filter_module.so | |-- ngx_http_lua_module.so | |-- ngx_http_naxsi_module.so | |-- ngx_http_replace_filter_module.so | |-- ngx_http_set_misc_module.so | `-- ngx_http_sysguard_module.so |-- nginx.conf |-- nginx.conf.default |-- scgi_params |-- scgi_params.default |-- uwsgi_params |-- uwsgi_params.default `-- win-utf 2 directories, 26 files ``` ###### Post installation tasks Create a system user/group: ```bash pw group add -g $NGINX_GID $NGINX_GROUP pw user add -d /non-existent -n $NGINX_USER -g $NGINX_GROUP -s /usr/sbin/nologin -c \'nginx user\' -u $NGINX_UID -g $NGINX_GID -w no ``` Create required directories: ```bash for i in \ /var/www \ /var/log/nginx \ /var/cache/nginx ; do mkdir -p "$i" && chown -R ${NGINX_USER}:${NGINX_GROUP} "$i" done ``` Include the necessary error pages: > You can also define them e.g. in `${NGX_PREFIX}/errors.conf` or other file and attach it as needed in server contexts. - default location: `${NGX_PREFIX}/html` ``` 50x.html index.html ``` Update modules list and include `modules.conf` to your configuration: ```bash _mod_dir="${NGX_PREFIX}/modules" :>"${_mod_dir}.conf" for _module in $(ls "${_mod_dir}/") ; do echo -en "load_module ${_mod_dir}/$_module;\n" >> "${_mod_dir}.conf" done ``` Create `logrotate` configuration: ```bash _logrotate_path="/usr/local/etc/logrotate.d" cat > "${_logrotate_path}/nginx" << __EOF__ /var/log/nginx/*/*.log { daily rotate 90 missingok sharedscripts compress postrotate kill -HUP `cat /var/run/nginx.pid` endscript dateext } /var/log/nginx/*.log { daily rotate 90 missingok sharedscripts compress postrotate kill -HUP `cat /var/run/nginx.pid` endscript dateext } __EOF__ ``` Or `newsyslog` configuration: ```bash cat > "/etc/newsyslog.conf.d/nginx.conf" << __EOF__ /var/log/access.log 644 7 1024 * JC /var/run/nginx.pid /var/log/error.log 644 7 1024 * JC /var/run/nginx.pid __EOF__ ``` Turn on NGINX service: ```bash if ! grep -q nginx_enable=\"YES\" /etc/rc.conf ; then echo -en "nginx_enable=\"YES\"\\n" >> /etc/rc.conf fi # or: sysrc nginx_enable="YES" ``` Show NGINX version and parameters: ```bash nginx -V ``` Test NGINX configuration: ```bash nginx -t -c $NGX_CONF ```
#### Installation Nginx on FreeBSD 12.1 (from ports) > The installation process is different from the previous ones, in my opinion is much simpler, however, has some limitations. For more information please see: - [The FreeBSD ports system](https://www.lpthe.jussieu.fr/~talon/freebsdports.html) - [Exploring FreeBSD (3/3) – a tutorial from the Linux user’s perspective](https://eerielinux.wordpress.com/2015/10/25/exploring-freebsd-33-a-tutorial-from-the-linux-users-perspective/) - [Non-interactive customization of FreeBSD ports AND saving config to /var/db/ports/\*/options](https://stackoverflow.com/questions/50293239/non-interactive-customization-of-freebsd-ports-and-saving-config-to-var-db-port) - [Upgrading FreeBSD Ports](http://www.wonkity.com/~wblock/docs/html/portupgrade.html)
Show step-by-step NGINX installation
* [Pre installation tasks](#pre-installation-tasks-4) * [Update FreeBSD ports tree](#update-freebsd-ports-tree) * [Dependencies](#dependencies) * [Download 3rd party modules](#download-3rd-party-modules-4) * [Build Nginx](#build-nginx-2) * [Post installation tasks](#post-installation-tasks-4) ###### Pre installation tasks The following variables should be the same as the NGINX configuration (and compilation) options. > They do not affect the settings of the `configure` command. ```bash export ngx_src="/usr/local/src" # Default for NGINX port version: export NGX_PREFIX="/usr/local/etc/nginx" export NGX_CONF="${NGX_PREFIX}/nginx.conf" ``` Set user/group variables: ```bash # Default for NGINX port version: export NGINX_USER="www" export NGINX_GROUP="www" ``` ###### Update FreeBSD ports tree ```bash cd /usr/ports portsnap fetch portsnap extract portsnap update ``` ###### Dependencies **Install prebuilt packages, export variables and set symbolic link:** > Install the OpenSSL library only if the latest version is available. FreeBSD 12.1 has built-in OpenSSL 1.1.1d. If the latest available version is 1.1.1d you don't need to do anything more, go to the NGINX compilation and installation step. ```bash # It's important and required, regardless of chosen sources: pkg install gcc gmake bison perl5-devel pcre lua51 libxslt libgd libxml2 expat autoconf jq git wget ncurses texinfo gettext gettext-tools ``` OpenSSL (example 1): ```bash # Searches ports for NGINX: psearch openssl cd /usr/ports/security/openssl111 # Parameters (from: /var/db/ports/security_openssl111/options): OPTIONS_FILE_SET+=ASYNC OPTIONS_FILE_SET+=CT OPTIONS_FILE_SET+=MAN3 OPTIONS_FILE_UNSET+=RFC3779 OPTIONS_FILE_SET+=SHARED OPTIONS_FILE_SET+=ZLIB OPTIONS_FILE_UNSET+=ARIA OPTIONS_FILE_UNSET+=DES OPTIONS_FILE_UNSET+=GOST OPTIONS_FILE_UNSET+=IDEA OPTIONS_FILE_UNSET+=SM2 OPTIONS_FILE_UNSET+=SM3 OPTIONS_FILE_UNSET+=SM4 OPTIONS_FILE_UNSET+=RC2 OPTIONS_FILE_UNSET+=RC4 OPTIONS_FILE_UNSET+=RC5 OPTIONS_FILE_UNSET+=MD2 OPTIONS_FILE_UNSET+=MD4 OPTIONS_FILE_UNSET+=MDC2 OPTIONS_FILE_SET+=RMD160 OPTIONS_FILE_SET+=ASM OPTIONS_FILE_SET+=SSE2 OPTIONS_FILE_SET+=THREADS OPTIONS_FILE_SET+=EC OPTIONS_FILE_SET+=NEXTPROTONEG OPTIONS_FILE_SET+=SCTP OPTIONS_FILE_UNSET+=SSL3 OPTIONS_FILE_UNSET+=TLS1 OPTIONS_FILE_UNSET+=TLS1_1 OPTIONS_FILE_SET+=TLS1_2 make config-recursive make install # If you want to remove parameters from the options file: make rmconfig # If you want to recompile OpenSSL from ports: # - edit options file manually make clean make reinstall # make deinstall install # - remove options file (see above command) make config make clean make reinstall # make deinstall install # To use/link openssl* port from your system (world): if [[ ! $(grep -q "DEFAULT_VERSIONS+=ssl=openssl111" /etc/make.conf) ]] ; then echo -en "DEFAULT_VERSIONS+=ssl=openssl111\n" >> /etc/make.conf fi # After above, you will have to rebuild all required packages for build NGINX from sources. if [[ -e "/usr/bin/openssl" ]] ; then _openssl_version=$(openssl version | awk '{print $2}') _openssl_date=$(date '+%Y%m%d%H%M%S') _openssl_str="openssl-${_openssl_version}-${_openssl_date}" mv /usr/bin/openssl /usr/bin/${_openssl_str} ln -sf /usr/ports/security/openssl111/work/stage/usr/local/bin/openssl /usr/bin/openssl else ln -sf /usr/ports/security/openssl111/work/stage/usr/local/bin/openssl /usr/bin/openssl fi ``` OpenSSL (example 2): ```bash cd "${ngx_src}" export openssl_version="1.1.1c" export OPENSSL_SRC="${ngx_src}/openssl-${openssl_version}" export OPENSSL_DIR="/usr/local/openssl-${openssl_version}" export OPENSSL_LIB="${OPENSSL_DIR}/lib" export OPENSSL_INC="${OPENSSL_DIR}/include" wget -c --no-check-certificate https://www.openssl.org/source/openssl-${openssl_version}.tar.gz && tar xzvf openssl-${openssl_version}.tar.gz cd "${ngx_src}/openssl-${openssl_version}" # Please run this and add as a compiler param: export __GCC_SSL=("") for _cc_opt in "${__GCC_SSL[@]}" ; do _cc_key=$(echo "$_cc_opt" | cut -d ":" -f1) _cc_value=$(echo "$_cc_opt" | cut -d ":" -f2) if [[ ! $(gcc -dM -E - > /etc/make.conf fi # After above, you will have to rebuild all required packages for build NGINX from sources. make -j2 && make test make install if [[ -e "/usr/bin/openssl" ]] ; then _openssl_version=$(openssl version | awk '{print $2}') _openssl_date=$(date '+%Y%m%d%H%M%S') _openssl_str="openssl-${_openssl_version}-${_openssl_date}" mv /usr/bin/openssl /usr/bin/${_openssl_str} ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl else ln -sf ${OPENSSL_DIR}/bin/openssl /usr/bin/openssl fi for i in libssl.so.1.1 libcrypto.so.1.1 ; do ln -sf ${ngx_src}/openssl-${openssl_version}/${i} /usr/lib/ done ``` Update links and cache to the shared libraries for both types of installation: ```bash ldconfig ``` ###### Download 3rd party modules Work in progress. ###### Build Nginx Searches ports for NGINX: ```bash psearch nginx ``` Go to the main NGINX port directory: ```bash cd /usr/ports/www/nginx ``` Before these tasks create backup of your current NGINX config: ```bash tar czvfp /usr/backup/nginx.tgz /usr/local/etc/nginx ``` Parameters: ```bash # From /var/db/ports/www_nginx/options: OPTIONS_FILE_SET+=DEBUG OPTIONS_FILE_SET+=DEBUGLOG OPTIONS_FILE_SET+=DSO OPTIONS_FILE_SET+=FILE_AIO OPTIONS_FILE_UNSET+=IPV6 OPTIONS_FILE_SET+=THREADS OPTIONS_FILE_SET+=WWW OPTIONS_FILE_UNSET+=GSSAPI_BASE OPTIONS_FILE_UNSET+=GSSAPI_HEIMDAL OPTIONS_FILE_UNSET+=GSSAPI_MIT OPTIONS_FILE_UNSET+=MAIL OPTIONS_FILE_UNSET+=MAIL_IMAP OPTIONS_FILE_UNSET+=MAIL_POP3 OPTIONS_FILE_UNSET+=MAIL_SMTP OPTIONS_FILE_UNSET+=MAIL_SSL OPTIONS_FILE_UNSET+=GOOGLE_PERFTOOLS OPTIONS_FILE_SET+=HTTP OPTIONS_FILE_UNSET+=HTTP_ADDITION OPTIONS_FILE_SET+=HTTP_AUTH_REQ OPTIONS_FILE_SET+=HTTP_CACHE OPTIONS_FILE_UNSET+=HTTP_DAV OPTIONS_FILE_UNSET+=HTTP_FLV OPTIONS_FILE_SET+=HTTP_GUNZIP_FILTER OPTIONS_FILE_SET+=HTTP_GZIP_STATIC OPTIONS_FILE_UNSET+=HTTP_IMAGE_FILTER OPTIONS_FILE_UNSET+=HTTP_MP4 OPTIONS_FILE_UNSET+=HTTP_PERL OPTIONS_FILE_UNSET+=HTTP_RANDOM_INDEX OPTIONS_FILE_SET+=HTTP_REALIP OPTIONS_FILE_SET+=HTTP_REWRITE OPTIONS_FILE_UNSET+=HTTP_SECURE_LINK OPTIONS_FILE_UNSET+=HTTP_SLICE OPTIONS_FILE_UNSET+=HTTP_SLICE_AHEAD OPTIONS_FILE_SET+=HTTP_SSL OPTIONS_FILE_SET+=HTTP_STATUS OPTIONS_FILE_SET+=HTTP_SUB OPTIONS_FILE_UNSET+=HTTP_XSLT OPTIONS_FILE_SET+=HTTPV2 OPTIONS_FILE_SET+=STREAM OPTIONS_FILE_SET+=STREAM_SSL OPTIONS_FILE_SET+=STREAM_SSL_PREREAD OPTIONS_FILE_UNSET+=AJP OPTIONS_FILE_UNSET+=AWS_AUTH OPTIONS_FILE_UNSET+=BROTLI OPTIONS_FILE_UNSET+=CACHE_PURGE OPTIONS_FILE_UNSET+=CLOJURE OPTIONS_FILE_UNSET+=CT OPTIONS_FILE_UNSET+=DEVEL_KIT OPTIONS_FILE_UNSET+=ARRAYVAR OPTIONS_FILE_UNSET+=DRIZZLE OPTIONS_FILE_SET+=DYNAMIC_UPSTREAM OPTIONS_FILE_SET+=ECHO OPTIONS_FILE_UNSET+=ENCRYPTSESSION OPTIONS_FILE_UNSET+=FASTDFS OPTIONS_FILE_UNSET+=FORMINPUT OPTIONS_FILE_UNSET+=GRIDFS OPTIONS_FILE_SET+=HEADERS_MORE OPTIONS_FILE_UNSET+=HTTP_ACCEPT_LANGUAGE OPTIONS_FILE_UNSET+=HTTP_AUTH_DIGEST OPTIONS_FILE_UNSET+=HTTP_AUTH_KRB5 OPTIONS_FILE_UNSET+=HTTP_AUTH_LDAP OPTIONS_FILE_UNSET+=HTTP_AUTH_PAM OPTIONS_FILE_UNSET+=HTTP_DAV_EXT OPTIONS_FILE_UNSET+=HTTP_EVAL OPTIONS_FILE_UNSET+=HTTP_FANCYINDEX OPTIONS_FILE_SET+=HTTP_FOOTER OPTIONS_FILE_SET+=HTTP_GEOIP2 OPTIONS_FILE_SET+=HTTP_IP2LOCATION OPTIONS_FILE_SET+=HTTP_IP2PROXY OPTIONS_FILE_UNSET+=HTTP_JSON_STATUS OPTIONS_FILE_UNSET+=HTTP_MOGILEFS OPTIONS_FILE_UNSET+=HTTP_MP4_H264 OPTIONS_FILE_UNSET+=HTTP_NOTICE OPTIONS_FILE_UNSET+=HTTP_PUSH OPTIONS_FILE_UNSET+=HTTP_PUSH_STREAM OPTIONS_FILE_UNSET+=HTTP_REDIS OPTIONS_FILE_UNSET+=HTTP_RESPONSE OPTIONS_FILE_UNSET+=HTTP_SUBS_FILTER OPTIONS_FILE_UNSET+=HTTP_TARANTOOL OPTIONS_FILE_UNSET+=HTTP_UPLOAD OPTIONS_FILE_UNSET+=HTTP_UPLOAD_PROGRESS OPTIONS_FILE_SET+=HTTP_UPSTREAM_CHECK OPTIONS_FILE_SET+=HTTP_UPSTREAM_FAIR OPTIONS_FILE_SET+=HTTP_UPSTREAM_STICKY OPTIONS_FILE_UNSET+=HTTP_VIDEO_THUMBEXTRACTOR OPTIONS_FILE_UNSET+=HTTP_ZIP OPTIONS_FILE_UNSET+=ICONV OPTIONS_FILE_UNSET+=LET OPTIONS_FILE_UNSET+=LUA OPTIONS_FILE_UNSET+=MEMC OPTIONS_FILE_UNSET+=MODSECURITY OPTIONS_FILE_UNSET+=MODSECURITY3 OPTIONS_FILE_SET+=NAXSI OPTIONS_FILE_UNSET+=NJS OPTIONS_FILE_UNSET+=PASSENGER OPTIONS_FILE_UNSET+=POSTGRES OPTIONS_FILE_UNSET+=RDS_CSV OPTIONS_FILE_UNSET+=RDS_JSON OPTIONS_FILE_UNSET+=REDIS2 OPTIONS_FILE_UNSET+=RTMP OPTIONS_FILE_SET+=SET_MISC OPTIONS_FILE_UNSET+=SFLOW OPTIONS_FILE_UNSET+=SHIBBOLETH OPTIONS_FILE_UNSET+=SLOWFS_CACHE OPTIONS_FILE_UNSET+=SMALL_LIGHT OPTIONS_FILE_UNSET+=SRCACHE OPTIONS_FILE_UNSET+=VOD OPTIONS_FILE_SET+=VTS OPTIONS_FILE_UNSET+=XSS OPTIONS_FILE_UNSET+=WEBSOCKIFY ``` The simplest way to install: ```bash make install make clean ``` or with pre-install configuration: ```bash make config-recursive make install make clean ``` But if you want to run other tasks: - remove parameters from the options file: ```bash make rmconfig ``` - recompile NGINX from ports: ```bash # The following task are not necessery: # - edit options file manually # - regenerate options file with wizard: make config # - remove options file: make rmconfig # after this you might to run pre-install configuration: make config-recursive make clean make reinstall # make deinstall install ``` - to disable vulnerabilities (not recommend!): ```bash make DISABLE_VULNERABILITIES=yes reinstall ``` ###### Post installation tasks Include the necessary error pages: > You can also define them e.g. in `${NGX_PREFIX}/errors.conf` or other file and attach it as needed in server contexts. - default location: `${NGX_PREFIX}/html` ``` 50x.html index.html ``` Update modules list and include `modules.conf` to your configuration: ```bash NGX_PREFIX="/usr/local/etc/nginx" _mod_dir="/usr/local/libexec/nginx" :>"${NGX_PREFIX}/modules.conf" for _module in $(ls "${_mod_dir}/") ; do echo -en "load_module ${_mod_dir}/$_module;\n" >> "${NGX_PREFIX}/modules.conf" done ``` Create `logrotate` configuration: ```bash _logrotate_path="/usr/local/etc/logrotate.d" cat > "${_logrotate_path}/nginx" << __EOF__ /var/log/nginx/*/*.log { daily rotate 90 missingok sharedscripts compress postrotate kill -HUP `cat /var/run/nginx.pid` endscript dateext } /var/log/nginx/*.log { daily rotate 90 missingok sharedscripts compress postrotate kill -HUP `cat /var/run/nginx.pid` endscript dateext } __EOF__ ``` Or `newsyslog` configuration: ```bash cat > "/etc/newsyslog.conf.d/nginx.conf" << __EOF__ /var/log/access.log 644 7 1024 * JC /var/run/nginx.pid /var/log/error.log 644 7 1024 * JC /var/run/nginx.pid __EOF__ ``` Turn on NGINX service: ```bash if ! grep -q nginx_enable=\"YES\" /etc/rc.conf ; then echo -en "nginx_enable=\"YES\"\\n" >> /etc/rc.conf fi # or: sysrc nginx_enable="YES" ``` Show NGINX version and parameters: ```bash nginx -V ``` Test NGINX configuration: ```bash nginx -t -c $NGX_CONF ```
#### Analyse configuration It is an essential way for testing NGINX configuration: ```bash nginx -t -c /etc/nginx/nginx.conf ``` An external tool for analyse NGINX configuration is `gixy`. The main goal of this tool is to prevent security misconfiguration and automate flaw detection: ```bash gixy /etc/nginx/nginx.conf ``` #### Monitoring ##### GoAccess Standard paths to the configuration file: - `/etc/goaccess.conf` - `/etc/goaccess/goaccess.conf` - `/usr/local/etc/goaccess.conf` Prior to start GoAccess enable these parameters: ``` time-format %H:%M:%S date-format %d/%b/%Y log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" ``` ###### Build and install ```bash # Ubuntu/Debian apt-get install gcc make libncursesw5-dev libgeoip-dev libtokyocabinet-dev # RHEL/CentOS yum install gcc ncurses-devel geoip-devel tokyocabinet-devel cd /usr/local/src/ wget -c --no-check-certificate -c https://tar.goaccess.io/goaccess-1.3.tar.gz && \ tar xzvfp goaccess-1.3.tar.gz cd goaccess-1.3 ./configure --enable-utf8 --enable-geoip=legacy --with-openssl= --sysconfdir=/etc/ make -j2 && make install ln -sf /usr/local/bin/goaccess /usr/bin/goaccess ``` > You can always fetch default configuration from `/usr/local/src/goaccess-/config/goaccess.conf` source tree. ###### Analyse log file and enable all recorded statistics ```bash _fd="access.log" goaccess -f "$_fd" -a ``` ###### Analyse compressed log file ```bash _fd="access.log.1.gz" zcat "$_fd" | goaccess -a -p /etc/goaccess/goaccess.conf ``` ###### Analyse log file remotely ```bash _fd="access.log" ssh user@remote_host "$_fd" | goaccess -a ``` ###### Analyse log file and generate html report ```bash _fd="access.log" goaccess -p /etc/goaccess/goaccess.conf -f "$_fd" --log-format=COMBINED -o /var/www/index.html ``` ##### Ngxtop ###### Analyse log file ```bash _fd="access.log" ngxtop -l "$_fd" ``` ###### Analyse log file and print requests with 4xx and 5xx ```bash _fd="access.log" ngxtop -l "$_fd" -i 'status >= 400' print request status ``` ###### Analyse log file remotely ```bash _fd="access.log" ssh user@remote_host tail -f "$_fd" | ngxtop -f combined ``` #### Testing > You can change combinations and parameters of these commands. ###### Build OpenSSL 1.0.2-chacha version OpenSSL [1.0.2-chacha](https://github.com/PeterMosmans/openssl) fork is used as standard OpenSSL distribution for numerous SSL/TLS pentesting tools. It includes default support for ciphers that are deemed insecure, and has extensive starttls support. See also [Rebase OpenSSL 1.0.2-chacha to use TLS 1.3](https://www.onwebsecurity.com/announcements/rebase-openssl-1-0-2-chacha-to-use-tls-1-3.html). Set temporary variables: ```bash export OPENSSL_SRC=/usr/local/src/openssl-1.0.2-chacha export OPENSSL_DIR=/usr/local/openssl-1.0.2-chacha export OPENSSL_LIB="${OPENSSL_DIR}/lib" export OPENSSL_INC="${OPENSSL_DIR}/include" ``` Create directories: ```bash for i in "${OPENSSL_SRC}" "${OPENSSL_DIR}" ; do mkdir "$i" ; done ``` Clone repository: ```bash git clone https://github.com/PeterMosmans/openssl.git "${OPENSSL_SRC}" ``` Build and install: ```bash cd "${OPENSSL_SRC}" # Please run this and add as a compiler param: export __GCC_SSL=("__SIZEOF_INT128__:enable-ec_nistp_64_gcc_128") for _cc_opt in "${__GCC_SSL[@]}" ; do _cc_key=$(echo "$_cc_opt" | cut -d ":" -f1) _cc_value=$(echo "$_cc_opt" | cut -d ":" -f2) if [[ ! $(gcc -dM -E - ://: # 2) http -p Hh ://: # 3) htrace.sh -u ://: -h ``` ###### Send request with http method, user-agent, follow redirects and show response headers ```bash # 1) curl -Iks --location -X GET -A "x-agent" ://: # 2) http -p Hh GET ://: User-Agent:x-agent --follow # 3) htrace.sh -u ://: -M GET --user-agent "x-agent" -h ``` ###### Send multiple requests ```bash # URL sequence substitution with a dummy query string: curl -ks ://:?[1-20] # With shell 'for' loop: for i in {1..20} ; do curl -ks ://: ; done ``` ###### Testing SSL connection ```bash # 1) echo | openssl s_client -connect : # 2) gnutls-cli --disable-sni -p 443 ``` ###### Testing SSL connection (debug mode) ```bash echo | openssl s_client -connect : -showcerts -tlsextdebug -status ``` ###### Testing SSL connection with SNI support ```bash # 1) echo | openssl s_client -servername -connect : # 2) gnutls-cli -p 443 ``` ###### Testing SSL connection with specific SSL version ```bash openssl s_client -tls1_2 -connect : ``` ###### Testing SSL connection with specific cipher ```bash openssl s_client -cipher 'AES128-SHA' -connect : ``` ###### Testing OCSP Stapling ```bash openssl s_client -connect example.com:443 -servername example.com -tls1 -tlsextdebug -status echo | openssl s_client -connect example.com:443 -servername example.com -status 2> /dev/null | grep -A 17 'OCSP response:' ``` ###### Verify 0-RTT ```bash _host="example.com" cat > req.in << __EOF__ HEAD / HTTP/1.1 Host: $_host Connection: close __EOF__ openssl s_client -connect ${_host}:443 -tls1_3 -sess_out session.pem -ign_eof < req.in openssl s_client -connect ${_host}:443 -tls1_3 -sess_in session.pem -early_data req.in ``` ###### Testing SCSV ```bash _host="example.com" openssl s_client -connect ${_host}:443 -tlsextdebug -status -fallback_scsv -no_tls1_3 ``` ##### Load testing with ApacheBench (ab) > Project documentation: [Apache HTTP server benchmarking tool](https://httpd.apache.org/docs/2.4/programs/ab.html) Installation: ```bash # Debian like: apt-get install -y apache2-utils # RedHat like: yum -y install httpd-tools ``` This is a [great explanation](https://stackoverflow.com/a/12732410) about ApacheBench by [Mamsaac](https://stackoverflow.com/users/669111/mamsaac): > _The apache benchmark tool is very basic, and while it will give you a solid idea of some performance, it is a bad idea to only depend on it if you plan to have your site exposed to serious stress in production._ ###### Standard test ```bash ab -n 1000 -c 100 https://example.com/ ``` ###### Test with Keep-Alive header ```bash ab -n 5000 -c 100 -k -H "Accept-Encoding: gzip, deflate" https://example.com/index.php ``` ##### Load testing with wrk2 > Project documentation: [wrk2](https://github.com/giltene/wrk2) > See [this](https://github.com/giltene/wrk2/blob/master/SCRIPTING) chapter to use the Lua API for wrk2. Also take a look at [wrk2 scripts](https://github.com/giltene/wrk2/tree/master/scripts). Installation: ```bash # Debian like: apt-get install -y build-essential libssl-dev git zlib1g-dev git clone https://github.com/giltene/wrk2 && cd wrk2 make sudo cp wrk /usr/local/bin # RedHat like: yum -y groupinstall 'Development Tools' yum -y install openssl-devel git git clone https://github.com/giltene/wrk2 && cd wrk2 make sudo cp wrk /usr/local/bin ``` ###### Standard scenarios ```bash # 1) wrk -c 1 -t 1 -d 2s -R 5 -H "Host: example.com" https://example.com Running 2s test @ https://example.com 1 threads and 1 connections Thread Stats Avg Stdev Max +/- Stdev Latency 45.21ms 20.99ms 108.16ms 90.00% Req/Sec -nan -nan 0.00 0.00% 10 requests in 2.01s, 61.69KB read Requests/sec: 4.99 Transfer/sec: 30.76KB # RPS: 6 09/Jul/2019:08:00:25 5 09/Jul/2019:08:00:26 # 2) wrk -c 1 -t 1 -d 2s -R 25 -H "Host: example.com" https://example.com Running 2s test @ https://example.com 1 threads and 1 connections Thread Stats Avg Stdev Max +/- Stdev Latency 64.40ms 24.26ms 110.46ms 48.00% Req/Sec -nan -nan 0.00 0.00% 50 requests in 2.01s, 308.45KB read Requests/sec: 24.93 Transfer/sec: 153.77KB # RPS: 12 09/Jul/2019:08:02:09 26 09/Jul/2019:08:02:10 13 09/Jul/2019:08:02:11 # 3) wrk -c 5 -t 5 -d 2s -R 25 -H "Host: example.com" https://example.com Running 2s test @ https://example.com 5 threads and 5 connections Thread Stats Avg Stdev Max +/- Stdev Latency 47.97ms 25.79ms 136.45ms 90.00% Req/Sec -nan -nan 0.00 0.00% 50 requests in 2.01s, 308.45KB read Requests/sec: 24.92 Transfer/sec: 153.75KB # RPS: 25 09/Jul/2019:08:03:56 25 09/Jul/2019:08:03:57 5 09/Jul/2019:08:03:58 # 4) wrk -c 5 -t 5 -d 2s -R 50 -H "Host: example.com" https://example.com Running 2s test @ https://example.com 5 threads and 5 connections Thread Stats Avg Stdev Max +/- Stdev Latency 45.09ms 18.63ms 130.69ms 91.00% Req/Sec -nan -nan 0.00 0.00% 100 requests in 2.01s, 616.89KB read Requests/sec: 49.85 Transfer/sec: 307.50KB # RPS: 35 09/Jul/2019:08:05:00 50 09/Jul/2019:08:05:01 20 09/Jul/2019:08:05:02 # 5) wrk -c 24 -t 12 -d 30s -R 2500 -H "Host: example.com" https://example.com Running 30s test @ https://example.com 12 threads and 24 connections Thread calibration: mean lat.: 3866.673ms, rate sampling interval: 13615ms Thread calibration: mean lat.: 3880.487ms, rate sampling interval: 13606ms Thread calibration: mean lat.: 3890.279ms, rate sampling interval: 13615ms Thread calibration: mean lat.: 3872.985ms, rate sampling interval: 13606ms Thread calibration: mean lat.: 3876.076ms, rate sampling interval: 13615ms Thread calibration: mean lat.: 3883.463ms, rate sampling interval: 13606ms Thread calibration: mean lat.: 3870.145ms, rate sampling interval: 13623ms Thread calibration: mean lat.: 3873.675ms, rate sampling interval: 13623ms Thread calibration: mean lat.: 3898.842ms, rate sampling interval: 13672ms Thread calibration: mean lat.: 3890.278ms, rate sampling interval: 13615ms Thread calibration: mean lat.: 3882.429ms, rate sampling interval: 13631ms Thread calibration: mean lat.: 3896.333ms, rate sampling interval: 13639ms Thread Stats Avg Stdev Max +/- Stdev Latency 15.01s 4.32s 22.46s 57.62% Req/Sec 52.00 0.00 52.00 100.00% 18836 requests in 30.01s, 113.52MB read Requests/sec: 627.59 Transfer/sec: 3.78MB # RPS: 98 09/Jul/2019:08:06:13 627 09/Jul/2019:08:06:14 624 09/Jul/2019:08:06:15 640 09/Jul/2019:08:06:16 629 09/Jul/2019:08:06:17 627 09/Jul/2019:08:06:18 648 09/Jul/2019:08:06:19 624 09/Jul/2019:08:06:20 624 09/Jul/2019:08:06:21 631 09/Jul/2019:08:06:22 641 09/Jul/2019:08:06:23 627 09/Jul/2019:08:06:24 633 09/Jul/2019:08:06:25 636 09/Jul/2019:08:06:26 648 09/Jul/2019:08:06:27 626 09/Jul/2019:08:06:28 617 09/Jul/2019:08:06:29 636 09/Jul/2019:08:06:30 640 09/Jul/2019:08:06:31 627 09/Jul/2019:08:06:32 635 09/Jul/2019:08:06:33 639 09/Jul/2019:08:06:34 633 09/Jul/2019:08:06:35 598 09/Jul/2019:08:06:36 644 09/Jul/2019:08:06:37 632 09/Jul/2019:08:06:38 635 09/Jul/2019:08:06:39 624 09/Jul/2019:08:06:40 643 09/Jul/2019:08:06:41 635 09/Jul/2019:08:06:42 431 09/Jul/2019:08:06:43 # Other examples: wrk -c 24 -t 12 -d 30s -R 2500 --latency https://example.com/index.php ``` ###### POST call (with Lua) Based on: - [wrk2 scripts - post](https://github.com/giltene/wrk2/blob/master/scripts/post.lua) - [POST request with wrk?](https://stackoverflow.com/questions/15261612/post-request-with-wrk) Example 1: ```lua -- lua/post-call.lua request = function() wrk.method = "POST" wrk.body = "login=foo&password=bar" wrk.headers["Content-Type"] = "application/x-www-form-urlencoded" return wrk.format(wrk.method) end ``` Example 2: ```lua -- lua/post-call.lua request = function() path = "/forms/int/d/1FAI" wrk.method = "POST" wrk.body = "{\"hash\":\"ooJeiveenai6iequ\",\"timestamp\":\"1562585990\",\"data\":[[\"cache\",\"x-cache\",\"true\"]]}" wrk.headers["Content-Type"] = "application/json; charset=utf-8" wrk.headers["Accept"] = "application/json" return wrk.format(wrk.method, path) end ``` Example 3: ```lua -- lua/post-call.lua request = function() path = "/login" wrk.method = "POST" wrk.body = [[{ "hash": "ooJeiveenai6iequ", "timestamp": "1562585990", "data": { login: "foo", password: "bar" }, }]] wrk.headers["Content-Type"] = "application/json; charset=utf-8" return wrk.format(wrk.method, path) end ``` Command: ```bash # The first example: wrk -c 12 -t 12 -d 30s -R 12000 -s lua/post-call.lua https://example.com/login # Second and third example: wrk -c 12 -t 12 -d 30s -R 12000 -s lua/post-call.lua https://example.com ``` ###### Random paths (with Lua) Based on: - [Intelligent benchmark with wrk](https://medium.com/@felipedutratine/intelligent-benchmark-with-wrk-163986c1587f) - [Confusion about benchmarking Linkerd](https://discourse.linkerd.io/t/confusion-about-benchmarking-linkerd/280) Example 1: ```lua -- lua/random-paths.lua math.randomseed(os.time()) request = function() url_path = "/search?q=" .. math.random(0,100000) -- print(url_path) return wrk.format("GET", url_path) end ``` Example 2: ```lua -- lua/random-paths.lua math.randomseed(os.time()) local connected = false local host = "example.com" local path = "/search?q=" local url = "https://" .. host .. path wrk.headers["Host"] = host function ranValue(length) local res = "" for i = 1, length do res = res .. string.char(math.random(97, 122)) end return res end request = function() url_path = path .. ranValue(32) -- print(url_path) if not connected then connected = true return wrk.format("CONNECT", host) end return wrk.format("GET", url_path) end ``` Example 3: ```lua -- lua/random-paths.lua math.randomseed(os.time()) counter = 0 function ranValue(length) local res = "" for i = 1, length do res = res .. string.char(math.random(97, 122)) end return res end request = function() path = "/accounts/" .. counter rval = ranValue(32) wrk.method = "POST" wrk.body = [[{ "user": counter, "action": "insert", "amount": rval }]] wrk.headers["Content-Type"] = "application/json" wrk.headers["Accept"] = "application/json" io.write(string.format("id: %4d, path: %14s,\tvalue: %s\n", counter, path, rval)) counter = counter + 1 if counter>500 then counter = 1 end return wrk.format(wrk.method, path) end ``` Command: ```bash wrk -c 12 -t 12 -d 30s -R 12000 -s lua/random-paths.lua https://example.com/ ``` ###### Multiple paths (with Lua) Example 1: ```lua -- lua/multi-paths.lua math.randomseed(os.time()) math.random(); math.random(); math.random() function shuffle(paths) local j, k local n = #paths for i = 1, n do j, k = math.random(n), math.random(n) paths[j], paths[k] = paths[k], paths[j] end return paths end function load_url_paths_from_file(file) lines = {} local f=io.open(file,"r") if f~=nil then io.close(f) else return lines end for line in io.lines(file) do if not (line == '') then lines[#lines + 1] = line end end return shuffle(lines) end paths = load_url_paths_from_file("data/paths.list") if #paths <= 0 then print("No paths found. You have to create a file data/paths.list with one path per line.") os.exit() end counter = 0 request = function() url_path = paths[counter] if counter > #paths then counter = 0 end counter = counter + 1 return wrk.format(nil, url_path) end ``` - `data/paths.list`: ``` / - it's not recommend, requests are being duplicated if you add only '/' /foo/bar /articles/id=25 /3a06e672fad4bec2383748cfd82547ee.html ``` Command: ```bash wrk -c 12 -t 12 -d 60s -R 200 -s lua/multi-paths.lua https://example.com ``` ###### Random server address to each thread (with Lua) Based on: - [wrk2 scripts - addr](https://github.com/giltene/wrk2/blob/master/scripts/addr.lua) Example 1: ```lua -- lua/resolve-host.lua local addrs = nil function setup(thread) if not addrs then -- addrs = wrk.lookup(wrk.host, "443" or "http") addrs = wrk.lookup(wrk.host, wrk.port or "http") for i = #addrs, 1, -1 do if not wrk.connect(addrs[i]) then table.remove(addrs, i) end end end thread.addr = addrs[math.random(#addrs)] end function init(args) local msg = "thread remote socket: %s" print(msg:format(wrk.thread.addr)) end ``` Command: ```bash wrk -c 12 -t 12 -d 30s -R 600 -s lua/resolve-host.lua https://example.com ``` ###### Multiple json requests (with Lua) Based on: - [multi-request-json](https://github.com/jgsqware/wrk-report/blob/master/scripts/multi-request-json.lua) You should install `luarocks`, `lua`, `luajit` and `lua-cjson` before use `multi-req.lua`: ```bash # Debian like: apt-get install lua5.1 libluajit-5.1-dev luarocks # RedHat like: yum install lua luajit-devel luarocks # FreeBSD: pkg install lua51 luajit cd /usr/ports/devel/lua-luarocks && make install clean # cjson: luarocks install lua-cjson ``` ```lua -- lua/multi-req.lua local cjson = require "cjson" local cjson2 = cjson.new() local cjson_safe = require "cjson.safe" math.randomseed(os.time()) math.random(); math.random(); math.random() function shuffle(paths) local j, k local n = #paths for i = 1, n do j, k = math.random(n), math.random(n) paths[j], paths[k] = paths[k], paths[j] end return paths end function load_request_objects_from_file(file) local data = {} local content local f=io.open(file,"r") if f~=nil then content = f:read("*all") io.close(f) else return lines end data = cjson.decode(content) return shuffle(data) end requests = load_request_objects_from_file("data/requests.json") if #requests <= 0 then print("No requests found. You have to create a file data/requests.json.") os.exit() end print(" " .. #requests .. " requests") counter = 1 request = function() local request_object = requests[counter] counter = counter + 1 if counter > #requests then counter = 1 end return wrk.format(request_object.method, request_object.path, request_object.headers, request_object.body) end ``` - `data/requests.json`: ```json [ { "path": "/id/1", "body": "ceR1caesaed2nohJei", "method": "GET", "headers": { "X-Custom-Header-1": "foo", "X-Custom-Header-2": "bar" } }, { "path": "/id/2", "body": "{\"field\":\"value\"}", "method": "POST", "headers": { "Content-Type": "application/json", "X-Custom-Header-1": "foo", "X-Custom-Header-2": "bar" } } ] ``` Command: ```bash wrk -c 12 -t 12 -d 30s -R 200 -s lua/multi-req.lua https://example.com ``` ###### Debug mode (with Lua) Based on: - [wrk-debugging-environment](https://github.com/czerasz/wrk-debugging-environment/blob/master/environments/wrk/scripts/debug.lua) ```lua -- lua/debug.lua local file = io.open("data/debug.log", "w") file:write("\n----------------------------------------\n") file:write(os.date("%m/%d/%Y %I:%M %p")) file:write("\n----------------------------------------\n") file:close() local file = io.open("data/debug.log", "a") function typeof(var) local _type = type(var); if(_type ~= "table" and _type ~= "userdata") then return _type; end local _meta = getmetatable(var); if(_meta ~= nil and _meta._NAME ~= nil) then return _meta._NAME; else return _type; end end local function string(o) return '"' .. tostring(o) .. '"' end local function recurse(o, indent) if indent == nil then indent = '' end local indent2 = indent .. ' ' if type(o) == 'table' then local s = indent .. '{' .. '\n' local first = true for k,v in pairs(o) do if first == false then s = s .. ', \n' end if type(k) ~= 'number' then k = string(k) end s = s .. indent2 .. '[' .. k .. '] = ' .. recurse(v, indent2) first = false end return s .. '\n' .. indent .. '}' else return string(o) end end local function var_dump(...) local args = {...} if #args > 1 then var_dump(args) else print(recurse(args[1])) end end max_requests = 0 counter = 1 show_body = 0 function setup(thread) thread:set("id", counter) counter = counter + 1 end response = function (status, headers, body) file:write("\n----------------------------------------\n") file:write("Response " .. counter .. " with status: " .. status .. " on thread " .. id) file:write("\n----------------------------------------\n") file:write("[response] Headers:\n") for key, value in pairs(headers) do file:write("[response] - " .. key .. ": " .. value .. "\n") end if (show_body == 1) then file:write("[response] Body:\n") file:write(body .. "\n") end if (max_requests > 0) and (counter > max_requests) then wrk.thread:stop() end counter = counter + 1 end done = function () file:close() end ``` Command: ```bash wrk -c 12 -t 12 -d 15s -R 200 -s lua/debug.lua https://example.com ``` ###### Analyse data pass to and from the threads Based on: - [wrk2 - setup](https://github.com/giltene/wrk2/blob/master/scripts/setup.lua) ```lua -- lua/threads.lua local counter = 1 local threads = {} function setup(thread) thread:set("id", counter) table.insert(threads, thread) counter = counter + 1 end function init(args) requests = 0 responses = 0 -- local msg = "thread %d created" -- print(msg:format(id)) end function request() requests = requests + 1 return wrk.request() end function response(status, headers, body) responses = responses + 1 end function done(summary, latency, requests) io.write("\n----------------------------------------\n") io.write(" Summary") io.write("\n----------------------------------------\n") for index, thread in ipairs(threads) do local id = thread:get("id") local requests = thread:get("requests") local responses = thread:get("responses") local msg = "thread %d : %d req , %d res" print(msg:format(id, requests, responses)) end end ``` Command: ```bash wrk -c 12 -t 12 -d 5s -R 5000 -s lua/threads.lua https://example.com ``` ###### Parsing wrk result and generate report Installation: ```bash go get -u github.com/jgsqware/wrk-report ``` Command: ```bash wrk -c 12 -t 12 -d 15s -R 500 --latency https://example.com | wrk-report > report.html ```

wrk-report-01

##### Load testing with locust > Project documentation: [Locust Documentation](https://docs.locust.io/en/stable/) Installation: ```bash # Python 2.x python -m pip install locustio # Python 3.x python3 -m pip install locustio ``` About `locust`: - `Number of users to simulate` - the number of users testing your application. Each user opens a TCP connection to your application and tests it. - `Hatch rate (users spawned/second)` - for each second, how many users will be added to the current users until the total amount of users. Each hatch Locust calls the `on_start` function if you have. For example: - Number of users: 1000 - Hatch rate: 10 Each second 10 users added to current users starting from 0 so in 100 seconds you will have 1000 users. When it reaches to the number of users, the statistic will be reset. Locust tries to emulate user behavior, it will pause each individual 'User' between `min_wait` and `max_wait` ms, to simulate the time between normal user actions. > Each of tasks will be executed in a random order, with a delay of `min_wait` and `max_wait` between the beginning of each task. ###### Multiple paths ```python # python/multi-paths.py import urllib3 from locust import HttpLocust, TaskSet, task urllib3.disable_warnings() multiheaders = """{ "Host": "example.com", "User-Agent":"python-locust-test", } """ self.client.get("/", headers=h) def on_start(self): self.client.verify = False class UserBehavior(TaskSet): @task class NonLoggedUserBehavior(TaskSet): # Home page @task(1) def index(self): self.client.get("/", headers=multiheaders, verify=False) # Status @task(1) def status(self): self.client.get("/status", verify=False) # Article @task(1) def article(self): self.client.get("/article/1044162375/", headers=multiheaders, verify=False) # About # Twice as much of requests: @task(2) def about(self): with self.client.get("/about", catch_response=True) as response: if response.text.find("author@example.com") > 0: response.success() else: response.failure("author@example.com not found in response") class WebsiteUser(HttpLocust): task_set = UserBehavior min_wait = 1000 # ms, 1s max_wait = 5000 # ms, 5s ``` Command: ```bash # Without web interface: locust --host=https://example.com -f python/multi-paths.py -c 2000 -r 10 -t 1h 30m --no-web --print-stats --only-summary # With web interface locust --host=https://example.com -f python/multi-paths.py --print-stats --only-summary ``` ###### Multiple paths with different user sessions Look also: - [How to Run Locust with Different Users](https://www.blazemeter.com/blog/how-to-run-locust-with-different-users/) Create a file with user credentials: ```python # python/credentials.py USER_CREDENTIALS = [ ("user5", "ShaePhu8aen8"), ("user4", "Cei5ohcha3he"), ("user3", "iedie8booChu"), ("user2", "iCuo4es1ahzu"), ("user1", "eeSh0yi0woo8") # ... ] ``` ```python # python/diff-users.py import urllib3, logging, sys from locust import HttpLocust, TaskSet, task from credentials import USER_CREDENTIALS urllib3.disable_warnings() class UserBehavior(TaskSet): @task class LoggedUserBehavior(TaskSet): username = "default" password = "default" def on_start(self): if len(USER_CREDENTIALS) > 0: self.username, self.password = USER_CREDENTIALS.pop() self.client.post("/login", { 'username': self.username, 'password': self.password }) logging.info('username: %s, password: %s', self.username, self.password) def on_stop(self): self.client.post("/logout", verify=False) # Home page # 10x more often than other @task(10) def index(self): self.client.get("/", verify=False) # Enter specific url after client login @task(1) def random_gen(self): self.client.get("/random-generator", verify=False) # Client profile page @task(1) def profile(self): self.client.get("/profile", verify=False) # Contact page @task(1) def contact(self): self.client.post("/contact", { "email": "no-reply@example.com", "subject": "GNU/Linux and BSD", "message": "Free software, Yeah!" }) class WebsiteUser(HttpLocust): host = "https://api.example.com" task_set = UserBehavior min_wait = 2000 # ms, 2s max_wait = 15000 # ms, 15s ``` Command: ```bash # Without web interface (for 5 users, see credentials.py): locust -f python/diff-users.py -c 5 -r 5 -t 30m --no-web --print-stats --only-summary # With web interface (for 5 users, see credentials.py) locust -f python/diff-users.py --print-stats --only-summary ``` ###### TCP SYN flood Denial of Service attack ```bash hping3 -V -c 1000000 -d 120 -S -w 64 -p 80 --flood --rand-source ``` ###### HTTP Denial of Service attack ```bash # 1) slowhttptest -g -o http_dos.stats -H -c 1000 -i 15 -r 200 -t GET -x 24 -p 3 -u :///index.php slowhttptest -g -o http_dos.stats -B -c 5000 -i 5 -r 200 -t POST -l 180 -x 5 -u :///service/login # 2) pip3 install slowloris slowloris # 3) git clone https://github.com/jseidl/GoldenEye && cd GoldenEye ./goldeneye.py :// -w 150 -s 75 -m GET ``` #### Debugging > You can change combinations and parameters of these commands. When carrying out the analysis, remember about [debug log](RULES.md#beginner-use-debug-mode-for-debugging) and [log formats](RULES.md#beginner-use-custom-log-formats-for-debugging). ##### Show information about processes With `ps`: ```bash # For all processes (master + workers): ps axw -o pid,ppid,gid,user,etime,%cpu,%mem,vsz,rss,wchan,ni,command | egrep '([n]ginx|[P]ID)' ps aux | grep [n]ginx ps -lfC nginx # For master process: ps axw -o pid,ppid,gid,user,etime,%cpu,%mem,vsz,rss,wchan,ni,command | egrep '([n]ginx: master|[P]ID)' ps aux | grep "[n]ginx: master" # For worker/workers: ps axw -o pid,ppid,gid,user,etime,%cpu,%mem,vsz,rss,wchan,ni,command | egrep '([n]ginx: worker|[P]ID)' ps aux | grep "[n]ginx: worker" # Show only pid, user and group for all NGINX processes: ps -eo pid,comm,euser,supgrp | grep nginx ``` With `top`: ```bash # For all processes (master + workers): top -p $(pgrep -d , nginx) # For master process: top -p $(pgrep -f "nginx: master") top -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}') # For one worker: top -p $(pgrep -f "nginx: worker") top -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}') # For multiple workers: top -p $(pgrep -f "nginx: worker" | sed '$!s/$/,/' | tr -d '\n') top -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}' | sed '$!s/$/,/' | tr -d '\n') ``` ##### Check memory usage With `ps_mem`: ```bash # For all processes (master + workers): ps_mem -s -p $(pgrep -d , nginx) ps_mem -d -p $(pgrep -d , nginx) # For master process: ps_mem -s -p $(pgrep -f "nginx: master") ps_mem -s -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}') # For one worker: ps_mem -s -p $(pgrep -f "nginx: worker") ps_mem -s -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}') # For multiple workers: ps_mem -s -p $(pgrep -f "nginx: worker" | sed '$!s/$/,/' | tr -d '\n') ps_mem -s -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}' | sed '$!s/$/,/' | tr -d '\n') ``` With `pmap`: ```bash # For all processes (master + workers): pmap $(pgrep -d ' ' nginx) pmap $(pidof nginx) # For master process: pmap $(pgrep -f "nginx: master") pmap $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}') # For one and multiple workers: pmap $(pgrep -f "nginx: worker") pmap $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}') ``` ##### Show open files ```bash # For all processes (master + workers): lsof -n -p $(pgrep -d , nginx) # For master process: lsof -n -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}') # For one worker: lsof -n -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}') # For multiple workers: lsof -n -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}' | sed '$!s/$/,/' | tr -d '\n') ``` ##### Check segmentation fault messages ```bash dmesg | grep nginx | grep segfault # | wc -l ``` ##### Dump configuration From a configuration file and all attached files (from a disk, only what a new process would load): ```bash nginx -T nginx -T -c /etc/nginx/nginx.conf ``` From a running process: > For more information please see [GNU Debugger (gdb) - Dump configuration from a running process](#dump-configuration-from-a-running-process). ##### Get the list of configure arguments ```bash nginx -V 2>&1 | grep arguments ``` ##### Check if the module has been compiled ```bash nginx -V 2>&1 | grep -- 'http_geoip_module' ``` ##### Show the most accessed IP addresses ```bash # - add `head -n X` to the end to limit the result # - add `grep "string"` to the end to filter by specific string # - add this to the end for print header: # ... | xargs printf '%10s%20s\n%10s%20s\n' "AMOUNT" "IP_ADDRESS" _fd="access.log" awk '{print $1}' "$_fd" | sort | uniq -c | sort -nr ``` ##### Show the most accessed IP addresses (ip and url) ```bash # - add `head -n X` to the end to limit the result # - add `grep "string"` to the end to filter by specific string # - add this to the end for print header: # ... | xargs printf '%10s%20s\t%s\n%10s%20s\t%s\n' "AMOUNT" "IP" "URL" awk '{print $1 " " $7}' "$_fd" | sort | uniq -c | sort -nr ``` ##### Show the most accessed IP addresses (method, code, ip, and url) ```bash # - add `head -n X` to the end to limit the result # - add `grep "string"` to the end to filter by specific string # - add this to the end for print header: # ... | xargs printf '%10s%10s%10s%20s\t%s\n%10s%10s%10s%20s\t%s\n' "AMOUNT" "METHOD" "CODE" "IP" "URL" _fd="access.log" awk '{print $6 "\" " $9 " " $1 " " $7}' "$_fd" | sort | uniq -c | sort -nr ``` ##### Show the top 5 visitors (IP addresses) ```bash # - add this to the end for print header: # ... | xargs printf '%10s%10s%20s\n%10s%10s%20s\n' "NUM" "AMOUNT" "IP_ADDRESS" _fd="access.log" cut -d ' ' -f1 "$_fd" | sort | uniq -c | sort -nr | head -5 | nl ``` ##### Show the most requested urls ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL" _fd="access.log" awk -F\" '{print $2}' "$_fd" | awk '{print $2}' | sort | uniq -c | sort -nr ``` ##### Show the most requested urls containing 'string' ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL" _fd="access.log" awk -F\" '($2 ~ "/string") {print $2}' "$_fd" | awk '{print $2}' | sort | uniq -c | sort -nr ``` ##### Show the most requested urls with http methods ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s %8s\t%s\n%10s %8s\t%s\n' "AMOUNT" "METHOD" "URL" _fd="access.log" awk -F\" '{print $2}' "$_fd" | awk '{print $1 "\t" $2}' | sort | uniq -c | sort -nr ``` ##### Show the most accessed response codes ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "HTTP_CODE" _fd="access.log" awk '{print $9}' "$_fd" | sort | uniq -c | sort -nr ``` ##### Analyse web server log and show only 2xx http codes ```bash _fd="access.log" tail -n 100 -f "$_fd" | grep "HTTP/[1-2].[0-1]\" [2]" ``` ##### Analyse web server log and show only 5xx http codes ```bash _fd="access.log" tail -n 100 -f "$_fd" | grep "HTTP/[1-2].[0-1]\" [5]" ``` ##### Show requests which result 502 and sort them by number per requests by url ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL" _fd="access.log" awk '($9 ~ /502/)' "$_fd" | awk '{print $7}' | sort | uniq -c | sort -nr ``` ##### Show requests which result 404 for php files and sort them by number per requests by url ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL" _fd="access.log" awk '($9 ~ /401/)' "$_fd" | awk -F\" '($2 ~ "/*.php")' | awk '{print $7}' | sort | uniq -c | sort -nr ``` ##### Calculating amount of http response codes ```bash # Not less than 1 minute: _fd="access.log" tail -2000 "$_fd" | awk -v date=$(date -d '1 minutes ago' +"%d/%b/%Y:%H:%M") '$4 ~ date' | cut -d '"' -f3 | cut -d ' ' -f2 | sort | uniq -c | sort -nr # Last 2000 requests from log file: # - add this to the end for print header: # ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "HTTP_CODE" _fd="access.log" tail -2000 "$_fd" | cut -d '"' -f3 | cut -d ' ' -f2 | sort | uniq -c | sort -nr ``` ##### Calculating requests per second ```bash # In real time: _fd="access.log" tail -F "$_fd" | pv -lr >/dev/null # https://serverfault.com/a/641552 tail -F "$_fd" | pv --line-mode --rate --timer --average-rate -b >/dev/null # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s%24s%18s\n%10s%24s%18s\n' "AMOUNT" "DATE" "IP_ADDRESS" _fd="access.log" awk '{print $4}' "$_fd" | uniq -c | sort -nr | tr -d "[" ``` ##### Calculating requests per second with IP addresses ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s%24s%18s\n%10s%24s%18s\n' "AMOUNT" "DATE" "IP_ADDRESS" _fd="access.log" awk '{print $4 " " $1}' "$_fd" | uniq -c | sort -nr | tr -d "[" ``` ##### Calculating requests per second with IP addresses and urls ```bash # - add `head -n X` to the end to limit the result # - add this to the end for print header: # ... | xargs printf '%10s%24s%18s\t%s\n%10s%24s%18s\t%s\n' "AMOUNT" "DATE" "IP_ADDRESS" "URL" _fd="access.log" awk '{print $4 " " $1 " " $7}' "$_fd" | uniq -c | sort -nr | tr -d "[" ``` ##### Get entries within last n hours ```bash _fd="access.log" awk -v _date=`date -d 'now-6 hours' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > _date) print $0}' "$_fd" # date command shows output for specific locale, for prevent this you should set LANG variable: _fd="access.log" awk -v _date=$(LANG=en_us.utf-8 date -d 'now-6 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _date) print $0}' "$_fd" # or: _fd="access.log" export LANG=en_us.utf-8 awk -v _date=$(date -d 'now-6 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _date) print $0}' "$_fd" ``` ##### Get entries between two timestamps (range of dates) ```bash # 1) _fd="access.log" awk '$4>"[05/Feb/2019:02:10" && $4<"[15/Feb/2019:08:20"' "$_fd" # 2) # date command shows output for specific locale, for prevent this you should set LANG variable: _fd="access.log" awk -v _dateB=$(LANG=en_us.utf-8 date -d '10:20' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(LANG=en_us.utf-8 date -d '20:30' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' "$_fd" # or: _fd="access.log" export LANG=en_us.utf-8 awk -v _dateB=$(date -d '10:20' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(date -d '20:30' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' "$_fd" # 3) # date command shows output for specific locale, for prevent this you should set LANG variable: _fd="access.log" awk -v _dateB=$(LANG=en_us.utf-8 date -d 'now-12 hours' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(LANG=en_us.utf-8 date -d 'now-2 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' "$_fd" # or: _fd="access.log" export LANG=en_us.utf-8 awk -v _dateB=$(date -d 'now-12 hours' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(date -d 'now-2 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' "$_fd" ``` ##### Get line rates from web server log ```bash _fd="access.log" tail -F "$_fd" | pv -N RAW -lc 1>/dev/null ``` ##### Trace network traffic for all processes ```bash strace -q -e trace=network -p `pidof nginx | sed -e 's/ /,/g'` ``` ##### List all files accessed by a NGINX ```bash strace -q -ff -e trace=file nginx 2>&1 | perl -ne 's/^[^"]+"(([^\\"]|\\[\\"nt])*)".*/$1/ && print' ``` ##### Check that the `gzip_static` module is working ```bash strace -q -p `pidof nginx | sed -e 's/ /,/g'` 2>&1 | grep gz ``` ##### Which worker processing current request Example 1 (more elegant way): ```nginx log_format debug-req-trace '$pid - "$request_method $scheme://$host$request_uri" ' '$remote_addr:$remote_port $server_addr:$server_port ' '$request_id'; # Output example: 31863 - "GET https://example.com/" 35.228.233.xxx:63784 10.240.20.2:443 be90154db5beb0e9dd13c5d91c8ecd4c ``` Example 2: ```bash # Run strace in the background: nohup strace -q -s 256 -p `pidof nginx | sed -e 's/ /,/g'` 2>&1 -o /tmp/nginx-req.trace /dev/null 2>/dev/null & # Watch output file: watch -n 0.1 "awk '/Host:/ {print \"pid: \" \$1 \", \" \"host: \" \$6}' /tmp/nginx-req.trace | sed 's/\\\r\\\n.*//'" # Output example: Every 0.1s: awk '/Host:/ {print "pid: " $1 ", " "host: " $6}' /tmp/nginx-req.trace | sed 's/\\r\\n.*//' pid: 31863, host: example.com ``` ##### Capture only http packets ```bash ngrep -d eth0 -qt 'HTTP' 'tcp' ``` ##### Extract User Agent from the http packets ```bash tcpdump -ei eth0 -nn -A -s1500 -l | grep "User-Agent:" ``` ##### Capture only http GET and POST packets ```bash # 1) tcpdump -ei eth0 -s 0 -A -vv \ 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420' or 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354' # 2) tcpdump -ei eth0 -s 0 -v -n -l | egrep -i "POST /|GET /|Host:" ``` ##### Capture requests and filter by source ip and destination port ```bash ngrep -d eth0 "" src host 10.10.252.1 and dst port 80 ``` ##### Capture HTTP requests/responses in real time, filter by GET, HEAD and save to a file ```bash httpry -i eth0 -o output.dump -m get,head ``` * `-m` - monitor only specific HTTP methods * `-o` - output txt file, `-b` - output binary file (raw HTTP packets) ##### Check CLOSE_WAIT connections ```bash netstat -anp | grep CLOSE_WAIT | grep -c nginx ``` See also [this](https://github.com/openresty/openresty/issues/323#issuecomment-352516797) great answer by [agentzh](https://github.com/agentzh): > _If your NGINX worker processes' CPU usage is high when you see `CLOSE_WAIT` connections are growing, then you should sample a CPU flamegraph with `perf` or with `systemtap` (like `sample-bt`). If your NGINX worker proesses' CPU is low, then you should sample an off-CPU flamegraph to analyze (using `perf` or using a `systemtap` tool like `sample-bt-off-cpu`). In case of off-CPU blocking, use of tools like `strace` can be helpful as well._ ##### Dump a process's memory > For more information about analyse core dumps please see [GNU Debugger (gdb) - Core dump backtrace](#core-dump-backtrace). > Will make the debugger output easier to understand see [Debugging Symbols](#debugging-symbols). A core dump is a file containing a process's address space (memory) when the process terminates unexpectedly. In other words is an instantaneous picture of a failing process at the moment it attempts to do something very wrong. NGINX is unbelievably stable but sometimes it can happen that there is a unique termination of the running processes. I think the best practice for core dumps are a properly collected core files, and associated information, we can often solve, and otherwise extract valuable information about the failing process. To enable core dumps from NGINX configuration you should: ```bash # In the main NGINX configuration file: # - specify the maximum possible size of the core dump for worker processes # - specify the maximum number of open files for worker processes # - specify a working directory in which a core dump file will be saved # - enable global debugging (optional) worker_rlimit_core 500m; worker_rlimit_nofile 65535; working_directory /var/dump/nginx; error_log /var/log/nginx/error.log debug; # Make sure the /var/dump/nginx directory is writable: chown nginx:nginx /var/dump/nginx chmod 0770 /var/dump/nginx # Disable the limit for the maximum size of a core dump file: ulimit -c unlimited # or: sh -c "ulimit -c unlimited && exec su $LOGNAME" # Enable core dumps for the setuid and setgid processes: # %e.%p.%h.%t - ... echo "/var/dump/nginx/core.%e.%p.%h.%t" | tee /proc/sys/kernel/core_pattern sysctl -w fs.suid_dumpable=2 && sysctl -p ``` To generate a core dump of a running NGINX master process: ```bash _pid=$(pgrep -f "nginx: master") ; gcore -o core.master $_pid ``` To generate a core dump of a running NGINX worker processes: ```bash for _pid in $(pgrep -f "nginx: worker") ; do gcore -o core.worker $_pid ; done ``` Or other solution for above (to dump memory regions of running NGINX process): ```bash # Set pid of NGINX master process: _pid=$(pgrep -f "nginx: master") # Generate gdb commands from the process's memory mappings using awk: cat /proc/$_pid/maps | \ awk '$6 !~ "^/" {split ($1,addrs,"-"); print "dump memory mem_" addrs[1] " 0x" addrs[1] " 0x" addrs[2] ;}END{print "quit"}' > gdb.args # Use gdb with the -x option to dump these memory regions to mem_* files: gdb -p $_pid -x gdb.args # Look for some (any) nginx.conf text: grep -a worker_connections mem_* grep -a server_name mem_* # or: strings mem_* | grep worker_connections strings mem_* | grep server_name ``` ##### GNU Debugger (gdb) You can use GDB to extract very useful information about NGINX instances, e.g. the log from memory or configuration from running process. ###### Dump configuration from a running process > It's very useful when you need to verify which configuration has been loaded and restore a previous configuration if the version on disk has been accidentally removed or overwritten. > The `ngx_conf_t` is a type of a structure used for configuration parsing. It only exists during configuration parsing, and obviously you can't access it after configuration parsing is complete. For dump configuration from a running process you should use `ngx_conf_dump_t`. ```gdb # Save gdb arguments to a file, e.g. nginx.gdb: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf "Dumping %s to nginx.conf.running\n", $name append memory nginx.conf.running \ $elts[$nelts]->buffer.start $elts[$nelts]->buffer.end end # Run gdb in a batch mode: gdb -p $(pgrep -f "nginx: master") -batch -x nginx.gdb # And open NGINX config: less nginx.conf.running ``` Or other solution: ```gdb # Save gdb functions to a file, e.g. nginx.gdb: define dump_config set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf "Dumping %s to nginx.conf.running\n", $name append memory nginx.conf.running \ $elts[$nelts]->buffer.start $elts[$nelts]->buffer.end end end document dump_config Dump NGINX configuration. end # Run gdb in a batch mode: gdb -p $(pgrep -f "nginx: master") -iex "source nginx.gdb" -ex "dump_config" --batch # And open NGINX config: less nginx.conf.running ``` ###### Show debug log in memory First of all a buffer for debug logging should be enabled: ```nginx error_log memory:64m debug; ``` Next: ```gdb # Save gdb functions to a file, e.g. nginx.gdb: define dump_debug_log set $log = ngx_cycle->log while ($log != 0) && ($log->writer != ngx_log_memory_writer) set $log = $log->next end if ($log->wdata != 0) set $buf = (ngx_log_memory_buf_t *) $log->wdata dump memory debug_mem.log $buf->start $buf->end end end document dump_debug_log Dump in memory debug log. end # Run gdb in a batch mode: gdb -p $(pgrep -f "nginx: master") -iex "source nginx.gdb" -ex "dump_debug_log" --batch # truncate the file: sed -i 's/[[:space:]]*$//' debug_mem.log # And open NGINX debug log: less debug_mem.log ``` ###### Core dump backtrace > The above functions ([GNU Debugger (gdb)](#gnu-debugger-gdb)) under discussion can be used with core files. To backtrace core dumps which saved in `working_directory`: ```bash gdb /usr/sbin/nginx /var/dump/nginx/core.nginx.8125.x-9s-web01-prod.1561475764 (gdb) bt ``` You can use also this recipe: ```bash gdb --core /var/dump/nginx/core.nginx.8125.x-9s-web01-prod.1561475764 ``` #### Debugging socket leaks Typically a resource leak is defined as an erroneous condition of a program when it is allocating more resources than it actually needs. Debugging socket leaks may produce the following alerts in your error log: ``` 2015/12/10 01:36:39 [alert] 27263#27263: *241 open socket #71 left in connection 56 2015/12/10 01:36:39 [alert] 27263#27263: *242 open socket #73 left in connection 61 ``` > Disable third party modules and check your error log, it can be a good solution, added the warnings may not appear after that. The official documentation say: > _This directive is used for debugging. When internal error is detected, e.g. the leak of sockets on restart of working processes, enabling `debug_points` leads to a core file creation (abort) or to stopping of a process (stop) for further analysis using a system debugger. [...] This will result in `abort()` call once NGINX detects leak and core dump._ To debug this you should activates debug points (in the main context): ```nginx # Set 'abort' value to abort the debug point # and produce a core dump file whenever there is an internal error: debug_points abort; ``` That example comes from the official [Debugging - Debugging socket leaks](https://www.nginx.com/resources/wiki/start/topics/tutorials/debugging/#socket-leaks) tutorial: > Something like this in `gdb` should be usefull (assuming 456 is connection number from error message from the process which dumped core: `[...] left in connection 456`): ```gdb set $c = &ngx_cycle->connections[456] p $c->log->connection p *$c set $r = (ngx_http_request_t *) $c->data p *$r ``` > In particular, `p $c->log->connection` will print connection number as used in logs. It will be possible to grep debug log for relevant lines, e.g. ```bash fgrep ' *12345678 ' /var/log/nginx/error_log; ``` At the end, look also at these interesting explanations: [Socket leak](https://forum.nginx.org/read.php?29,239511,239511#msg-239511), [[nginx] Fixed socket leak with "return 444" in error_page (ticket #274)](https://forum.nginx.org/read.php?29,281339,281339#msg-281339) and [This is strictly a violation of the TCP specification](https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-specification/). #### Shell aliases ```bash alias ng.test='nginx -t -c /etc/nginx/nginx.conf' alias ng.stop='ng.test && systemctl stop nginx' alias ng.reload='ng.test && systemctl reload nginx' alias ng.reload='ng.test && kill -HUP $(cat /var/run/nginx.pid)' # ... kill -HUP $(ps auxw | grep [n]ginx | grep master | awk '{print $2}') alias ng.restart='ng.test && systemctl restart nginx' alias ng.restart='ng.test && kill -QUIT $(cat /var/run/nginx.pid) && /usr/sbin/nginx' # ... kill -QUIT $(ps auxw | grep [n]ginx | grep master | awk '{print $2}') ... ``` For more examples please see [Commands](NGINX_BASICS.md#commands) section. #### Configuration snippets ##### Nginx server header removal You could use a module like `ngx_headers_more` to disable or replace the server header. However, why compile, test and configure an extra module if it is also possible to change the upstream code with only a few simple lines? No module, not a multitude of code changes. Only one single patch. This [nginx-remove-server-header.patch](https://gitlab.com/buik/nginx/blob/master/nginx-remove-server-header.patch) will remove NGINX as server header. ##### Custom log formats ```nginx # Default main log format from nginx repository: log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; # Extended main log format: log_format main-level-0 '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time'; # Debug log formats: # - level 0 # - based on main-level-0 without "$http_referer" "$http_user_agent" log_format debug-level-0 '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '$request_id $pid $msec $request_time ' '$upstream_connect_time $upstream_header_time ' '$upstream_response_time "$request_filename" ' '$request_completion'; # - level 1 # - based on main-level-0 without "$http_referer" "$http_user_agent" log_format debug-level-1 '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '$request_id $pid $msec $request_time ' '$upstream_connect_time $upstream_header_time ' '$upstream_response_time "$request_filename" $request_length ' '$request_completion $connection $connection_requests'; # - level 2 # - based on main-level-0 without "$http_referer" "$http_user_agent" log_format debug-level-2 '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '$request_id $pid $msec $request_time ' '$upstream_connect_time $upstream_header_time ' '$upstream_response_time "$request_filename" $request_length ' '$request_completion $connection $connection_requests ' '$server_addr $server_port $remote_addr $remote_port'; # Debug log format for SSL: # - based on main-level-0 log_format debug-ssl-level-0 '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time ' '$ssl_protocol $ssl_cipher'; # Debug log format for GeoIP module (ngx_http_geoip_module): # - based on main-level-0 # - only if you enable ngx_http_geoip2_module and define geoip2 variables # log_format geoip-level-0 # '$remote_addr - $remote_user [$time_local] ' # '"$request_method $scheme://$host$request_uri ' # '$server_protocol" $status $body_bytes_sent ' # '"$http_referer" "$http_user_agent" ' # '$request_time ' # '"$geoip2_data_country_code $geoip2_data_country_name"'; # The following log format is very useful for debugging connection between proxy and upstream servers: # - based on main-level-0 log_format upstream_log '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time ' 'upstream_addr $upstream_addr ' 'upstream_bytes_received $upstream_bytes_received ' 'upstream_cache_status $upstream_cache_status ' 'upstream_connect_time $upstream_connect_time ' 'upstream_header_time $upstream_header_time ' 'upstream_response_length $upstream_response_length ' 'upstream_response_time $upstream_response_time upstream_status $upstream_status '; # Log only specific error codes: # Example: # - access_log /var/log/nginx/access.log main if=$error_codes; map $status $error_codes { default 1; ~^[23] 0; } map $status $error_codes_5xx { default 1; ~^[234] 0; } ``` ##### Log only 4xx/5xx ```nginx # 1) File: /etc/nginx/map/logs.conf # Map module: map $status $error_codes { default 1; ~^[23] 0; } # 2) Include this file in http context: include /etc/nginx/map/logs.conf; # 3) Turn on in a specific context (e.g. location): server { ... # Add if condition to access log: access_log /var/log/nginx/example.com-access.log combined if=$error_codes; } ``` ##### Restricting access with basic authentication ```bash # 1) Generate password file with htpasswd command: htpasswd -c htpasswd_example.com.conf # 2) Include this file in specific context: (e.g. server): server_name example.com; ... # These directives are optional, only if we need them: satisfy all; deny 10.255.10.0/24; allow 192.168.0.0/16; allow 127.0.0.1; deny all; # It's important: auth_basic "Restricted Area"; auth_basic_user_file /etc/nginx/acls/htpasswd_example.com.conf; location / { ... } location /public/ { auth_basic off; } ... ``` ##### Restricting access with client certificate If the client-side certificate failed to authenticate, NGINX show: `400 No required SSL certificate was sent`. ```nginx server { server_name example.com; ssl_client_certificate certs/client-X0.pem; ssl_verify_client on; ssl_verify_depth 3; proxy_set_header ClientDN $ssl_client_s_dn; # You can also show specific message to the client: location / { if ($ssl_client_verify != SUCCESS) { return 403; } } ... } ``` Read also this: [Nginx SSL certificate authentication signed by intermediate CA (chain)](https://stackoverflow.com/questions/8431528/nginx-ssl-certificate-authentication-signed-by-intermediate-ca-chain). ##### Restricting access by geographical location > The best explanation and technical reference is [Restricting Access by Geographical Location](https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-by-geoip/). Look also at [ngx_http_geoip_module](http://nginx.org/en/docs/http/ngx_http_geoip_module.html). I also recommend read the following resources: - [GeoIP discontinuation; Upgrade to GeoIP2 with nginx on CentOS](https://medium.com/@karljohnson/geoip-discontinuation-upgrade-to-geoip2-with-nginxon-centos-c2a3dbcf8fd) - [Blocking Country and Continent with nginx GeoIP on Ubuntu 18.04](https://guides.wp-bullet.com/blocking-country-and-continent-with-nginx-geoip-on-ubuntu-18-04/) - [Using NGINX With GeoIP MaxMind Database to Fetch Geolocation Data](https://dzone.com/articles/nginx-with-geoip-maxmind-database-to-fetch-user-ge) See also [ngx_http_geoip_module](NGINX_BASICS.md#ngx_http_geoip_module) chapter from this handbook. The NGINX must be compiled with the `ngx_http_geoip_module` or `ngx_http_geoip2_module` to use the GeoIP database. With this module you can blocking/allowing for example: - region - city - country ```nginx # 1) This allows all countries, except the three countries set to no. # Load geoip database to determine the country depending on the client IP address # (in a http context): geoip_country /usr/share/GeoIP/GeoIP.dat; # Define a map: map $geoip_country_code $allowed_country { default yes; AM no; BH no; GR no; } # In your location block: ... location / { if ($allowed_country = no) { return 444; } ... } # 2) This blocks all countries, except the three countries set to yes. # Load geoip database to determine the country depending on the client IP address # (in a http context): geoip_country /usr/share/GeoIP/GeoIP.dat; # Define a map: map $geoip_country_code $allowed_country { default no; AM yes; BH yes; GR yes; } # In your location block: ... location / { if ($allowed_country = no) { return 444; } ... } ``` For display GeoIP data in NGINX access log see [Custom log formats](HELPERS.md#custom-log-formats) chapter. ###### GeoIP 2 database Why should you use GeoIP2 instead of GeoIP Legacy? See [What’s New in GeoIP2](https://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/). GeoLite Legacy databases are discontinued as of January 2, 2019, they are not updated nor any longer available for download. Every user should move to GeoLite2 databases, a more contemporary versions of the GeoLite Legacy geolocation databases which are still available in a free version updated every month. For support GeoIP2 we have [ngx_http_geoip2_module](https://github.com/leev/ngx_http_geoip2_module). It creates variables based on the client IP address, using the precompiled MaxMind GeoIP2 databases, which provide localized name information not present in the original GeoIP databases. ```nginx # 1) This allows all countries, except the three countries set to no. # Tell NGINX about GeoIP2 databases (in http context): geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb { auto_reload 5m; $geoip2_metadata_country_build metadata build_epoch; $geoip2_data_country_code default=US country iso_code; $geoip2_data_country_name country names en; } geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb { $geoip2_data_city_name default=London city names en; } # Define a map: map $geoip2_data_country_code $allowed_country { default no; AM yes; BH yes; GR yes; } # 2) This allows all countries, except the three countries set to no and get source IP address # from X-Forwarded-For header. # First of all, you should extract the user IP address: map $http_x_forwarded_for $realip { ~^(\d+\.\d+\.\d+\.\d+) $1; default $remote_addr; } # You can also set source for the IP address: geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb { auto_reload 5m; $geoip2_metadata_country_build metadata build_epoch; $geoip2_data_country_code default=US source=$realip country iso_code; $geoip2_data_country_name source=$realip country names en; } geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb { $geoip2_data_city_name source=$realip city names en; $geoip2_data_time_zone source=$realip location time_zone; } # For both examples: # Add IP-Country header to confirm that NGINX is fetching all GeoIP information # (in a server context): more_set_headers "IP-Country: $geoip2_data_country_name"; # or: add_header IP-Country $geoip2_data_country_name; # In your location block: ... location / { if ($allowed_country = no) { return 403; } ... } ``` ##### Dynamic error pages with SSI Example 1: 1. Create error page template in `/var/www/error_pages/errors.html`: ```html Error

We are updating our website

This is only for a few seconds, you will be redirected.

``` or ```html <!--# echo var="status" default="" --> <!--# echo var="status_text" default="Something goes wrong..." -->

``` 2. Define error codes map in the http context or include it from a file: ```nginx map $status $status_text { default 'Something is wrong'; 400 'Bad Request'; 401 'Unauthorized'; 402 'Payment Required'; 403 'Forbidden'; 404 'Not Found'; 405 'Method Not Allowed'; 406 'Not Acceptable'; 407 'Proxy Authentication Required'; 408 'Request Timeout'; 409 'Conflict'; 410 'Gone'; 411 'Length Required'; 412 'Precondition Failed'; 413 'Payload Too Large'; 414 'URI Too Long'; 415 'Unsupported Media Type'; 416 'Range Not Satisfiable'; 417 'Expectation Failed'; 418 'I\'m a teapot'; 421 'Misdirected Request'; 422 'Unprocessable Entity'; 423 'Locked'; 424 'Failed Dependency'; 426 'Upgrade Required'; 428 'Precondition Required'; 429 'Too Many Requests'; 431 'Request Header Fields Too Large'; 451 'Unavailable For Legal Reasons'; 500 'Internal Server Error'; 501 'Not Implemented'; 502 'Bad Gateway'; 503 'Service Unavailable'; 504 'Gateway Timeout'; 505 'HTTP Version Not Supported'; 506 'Variant Also Negotiates'; 507 'Insufficient Storage'; 508 'Loop Detected'; 510 'Not Extended'; 511 'Network Authentication Required'; } ``` 3. Create an `error_page` in your context (e.g. server): ```nginx server { ... error_page 400 401 403 404 405 500 501 502 503 /errors.html; location = /errors.html { ssi on; internal; root /var/www/error_pages; } } ``` 4. Turn on the specific error page: ```nginx location = /404.html { return 404; } ``` Read also this: [Static error pages generator](https://github.com/trimstray/nginx-admins-handbook#static-error-pages-generator). ##### Blocking/allowing IP addresses Example 1: ```nginx # 1) File: /etc/nginx/acls/allow.map.conf # Map module: map $remote_addr $globals_internal_map_acl { # Status code: # - 0 = false # - 1 = true default 0; ### INTERNAL ### 10.255.10.0/24 1; 10.255.20.0/24 1; 10.255.30.0/24 1; 192.168.0.0/16 1; } # 2) Include this file in http context: include /etc/nginx/acls/allow.map.conf; # 3) Turn on in a specific context (e.g. location): server_name example.com; ... location / { proxy_pass http://localhost:80; client_max_body_size 10m; } location ~ ^/(backend|api|admin) { if ($globals_internal_map_acl) { set $pass 1; } if ($pass = 1) { proxy_pass http://localhost:80; client_max_body_size 10m; } if ($pass != 1) { rewrite ^(.*) https://example.com; } ... ``` Example 2: ```nginx # 1) File: /etc/nginx/acls/allow.geo.conf # Geo module: geo $globals_internal_geo_acl { # Status code: # - 0 = false # - 1 = true default 0; ### INTERNAL ### 10.255.10.0/24 1; 10.255.20.0/24 1; 10.255.30.0/24 1; 192.168.0.0/16 1; } # 2) Include this file in http context: include /etc/nginx/acls/allow.geo.conf; # 3) Turn on in a specific context (e.g. location): server_name example.com; ... location / { proxy_pass http://localhost:80; client_max_body_size 10m; } location ~ ^/(backend|api|admin) { if ($globals_internal_geo_acl = 0) { return 403; } proxy_pass http://localhost:80; client_max_body_size 10m; ... ``` Example 3: ```nginx # 1) File: /etc/nginx/acls/allow.conf ### INTERNAL ### allow 10.255.10.0/24; allow 10.255.20.0/24; allow 10.255.30.0/24; allow 192.168.0.0/16; ### EXTERNAL ### allow 35.228.233.xxx; # 2) Include this file in http context: include /etc/nginx/acls/allow.conf; # 3) Turn on in a specific context (e.g. server): server_name example.com; include /etc/nginx/acls/allow.conf; allow 35.228.233.xxx; deny all; ... ``` ##### Blocking referrer spam Example 1: ```nginx # 1) File: /etc/nginx/limits.conf map $http_referer $invalid_referer { hostnames; default 0; # Invalid referrers: "invalid.com" 1; "~*spamdomain4.com" 1; "~*.invalid\.org" 1; } # 2) Include this file in http context: include /etc/nginx/limits.conf; # 3) Turn on in a specific context (e.g. server): server_name example.com; if ($invalid_referer) { return 403; } ... ``` Example 2: ```nginx # 1) Turn on in a specific context (e.g. location): location /check_status { if ($http_referer ~ "spam1\.com|spam2\.com|spam3\.com") { return 444; } ... ``` How to test? ```bash siege -b -r 2 -c 40 -v https://example.com/storage/img/header.jpg -H "Referer: https://spamdomain4.com/" ** SIEGE 4.0.4 ** Preparing 5 concurrent users for battle. The server is now under siege... HTTP/1.1 403 0.11 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.12 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.18 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.18 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.19 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.10 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.11 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.11 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.12 secs: 124 bytes ==> GET /storage/img/header.jpg HTTP/1.1 403 0.12 secs: 124 bytes ==> GET /storage/img/header.jpg ... ``` ##### Limiting referrer spam Example 1: ```nginx # 1) File: /etc/nginx/limits.conf map $http_referer $limit_ip_key_by_referer { hostnames; # It's important because if you set numeric value, e.g. 0 rate limiting rule will be catch all referers: default ""; # Invalid referrers (we restrict them): "invalid.com" $binary_remote_addr; "~referer-xyz.com" $binary_remote_addr; "~*spamdomain4.com" $binary_remote_addr; "~*.invalid\.org" $binary_remote_addr; } limit_req_zone $limit_ip_key_by_referer zone=req_for_remote_addr_by_referer:1m rate=5r/s; # 2) Include this file in http context: include /etc/nginx/limits.conf; # 3) Turn on in a specific context (e.g. server): server_name example.com; limit_req zone=req_for_remote_addr_by_referer burst=2; ... ``` How to test? ```bash siege -b -r 2 -c 40 -v https://example.com/storage/img/header.jpg -H "Referer: https://spamdomain4.com/" ** SIEGE 4.0.4 ** Preparing 5 concurrent users for battle. The server is now under siege... HTTP/1.1 200 0.13 secs: 3174 bytes ==> GET /storage/img/header.jpg HTTP/1.1 503 0.14 secs: 206 bytes ==> GET /storage/img/header.jpg HTTP/1.1 503 0.15 secs: 206 bytes ==> GET /storage/img/header.jpg HTTP/1.1 503 0.10 secs: 206 bytes ==> GET /storage/img/header.jpg HTTP/1.1 503 0.10 secs: 206 bytes ==> GET /storage/img/header.jpg HTTP/1.1 503 0.10 secs: 206 bytes ==> GET /storage/img/header.jpg HTTP/1.1 200 0.63 secs: 3174 bytes ==> GET /storage/img/header.jpg HTTP/1.1 200 1.13 secs: 3174 bytes ==> GET /storage/img/header.jpg HTTP/1.1 200 1.00 secs: 3174 bytes ==> GET /storage/img/header.jpg HTTP/1.1 200 1.04 secs: 3174 bytes ==> GET /storage/img/header.jpg ... ``` ##### Blocking User-Agent Example 1: ```nginx # 1) File: /etc/nginx/limits.conf map $http_user_agent $invalid_ua { default 0; ~*scrapyproject 1; ~*netcrawler 1; ~*nmap 1; ~*sqlmap 1; ~*slowhttptest 1; ~*nikto 1; ~*python-requests 1; } # 2) Include this file in http context: include /etc/nginx/limits.conf; # 3) Turn on in a specific context (e.g. server): server_name example.com; if ($invalid_ua) { return 444; } ... ``` ##### Limiting User-Agent Example 1: ```nginx # 1) File: /etc/nginx/limits.conf map $http_user_agent $limit_ip_key_by_ua { default ""; ~*scrapyproject binary_remote_addr; ~*netcrawler binary_remote_addr; ~*nmap binary_remote_addr; ~*sqlmap binary_remote_addr; ~*slowhttptest binary_remote_addr; ~*nikto binary_remote_addr; ~*python-requests binary_remote_addr; } limit_req_zone $limit_ip_key_by_ua zone=req_for_remote_addr_by_ua:32k rate=10r/m; # 2) Include this file in http context: include /etc/nginx/limits.conf; # 3) Turn on in a specific context (e.g. server): server_name example.com; limit_req zone=req_for_remote_addr_by_ua burst=2; ... ``` ##### Limiting the rate of requests with burst mode ```nginx limit_req_zone $binary_remote_addr zone=req_for_remote_addr:64k rate=10r/m; ``` - key/zone type: `limit_req_zone` - the unique key for limiter: `$binary_remote_addr` - zone name: `req_for_remote_addr` - zone size: `64k` (1024 IP addresses) - rate is `0,16` request each second or `10` requests per minute (`1` request every `6` second) Example of use: ```nginx location ~ /stats { limit_req zone=req_for_remote_addr burst=5; ... ``` - set maximum requests as `rate` * `burst` in `burst` seconds - with bursts not exceeding `5` requests: + `0,16r/s` * `5` = `0.80` requests per `5` seconds + `10r/m` * `5` = `50` requests per `5` minutes Testing queue: ```bash # siege -b -r 1 -c 12 -v https://x409.info/stats/ ** SIEGE 4.0.4 ** Preparing 12 concurrent users for battle. The server is now under siege... HTTP/1.1 200 * 0.20 secs: 2 bytes ==> GET /stats/ HTTP/1.1 503 0.20 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.20 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.21 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.22 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.22 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.23 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 200 * 6.22 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 12.24 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 18.27 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 24.30 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 30.32 secs: 2 bytes ==> GET /stats/ | - burst=5 - 0,16r/s, 10r/m - 1r every 6 seconds Transactions: 6 hits Availability: 50.00 % Elapsed time: 30.32 secs Data transferred: 0.01 MB Response time: 15.47 secs Transaction rate: 0.20 trans/sec Throughput: 0.00 MB/sec Concurrency: 3.06 Successful transactions: 6 Failed transactions: 6 Longest transaction: 30.32 Shortest transaction: 0.20 ``` ##### Limiting the rate of requests with burst mode and nodelay ```nginx limit_req_zone $binary_remote_addr zone=req_for_remote_addr:50m rate=2r/s; ``` - key/zone type: `limit_req_zone` - the unique key for limiter: `$binary_remote_addr` - zone name: `req_for_remote_addr` - zone size: `50m` (800,000 IP addresses) - rate is `2` request each second or `120` requests per minute (`2` requests every `1` second) Example of use: ```nginx location ~ /stats { limit_req zone=req_for_remote_addr burst=5 nodelay; ... ``` - set maximum requests as `rate` * `burst` in `burst` seconds - with bursts not exceeding `5` requests + `2r/s` * `5` = `10` requests per `5` seconds + `120r/m` * `5` = `600` requests per `5` minutes - allocates slots in the queue according to the `burst` parameter with `nodelay` Testing queue: ```bash # siege -b -r 1 -c 12 -v https://x409.info/stats/ ** SIEGE 4.0.4 ** Preparing 12 concurrent users for battle. The server is now under siege... HTTP/1.1 200 * 0.18 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 0.18 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 0.19 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 0.19 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 0.19 secs: 2 bytes ==> GET /stats/ HTTP/1.1 200 * 0.19 secs: 2 bytes ==> GET /stats/ HTTP/1.1 503 0.19 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.19 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.20 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.21 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.21 secs: 1501 bytes ==> GET /stats/ HTTP/1.1 503 0.22 secs: 1501 bytes ==> GET /stats/ | - burst=5 with nodelay - 2r/s, 120r/m - 1r every 0.5 second Transactions: 6 hits Availability: 50.00 % Elapsed time: 0.23 secs Data transferred: 0.01 MB Response time: 0.39 secs Transaction rate: 26.09 trans/sec Throughput: 0.04 MB/sec Concurrency: 10.17 Successful transactions: 6 Failed transactions: 6 Longest transaction: 0.22 Shortest transaction: 0.18 ``` ##### Limiting the rate of requests per IP with geo and map ```nginx geo $limit_per_ip { default 0; 10.10.10.135 1; } map $limit_per_ip $limit_key { 0 ""; 1 $binary_remote_addr; } limit_req_zone $limit_key zone=per_ip:10m rate=20r/m; ``` - key/zone type: `limit_req_zone` - the unique key for limiter: `$limit_key` (`$binary_remote_addr`) - `$limit_per_ip` from geo module - match `$limit_per_ip` to `$limit_key` from map module - zone name: `per_ip` - zone size: `10m` (160,000 IP addresses) - rate is `0.3` request each second or `20` requests per minute (`1` request every `3` second) Example of use: ```nginx location ~ /stats { limit_req zone=per_ip; ... ``` ##### Limiting the number of connections ```nginx limit_conn_zone $binary_remote_addr zone=conn_for_remote_addr:1m; ``` - key/zone type: `limit_conn_zone` - the unique key for limiter: `$binary_remote_addr` - limit requests per IP as following - zone name: `conn_for_remote_addr` - zone size: `1m` (16,000 IP addresses) Example of use: ```nginx location ~ /stats { limit_conn conn_for_remote_addr 1; ... ``` - limit a single IP address to make no more than `1` connection from IP at the same time Testing queue: ```bash # siege -b -r 1 -c 100 -t 10s --no-parser https://x409.info/stats/ defaulting to time-based testing: 10 seconds ** SIEGE 4.0.4 ** Preparing 100 concurrent users for battle. The server is now under siege... Lifting the server siege... Transactions: 364 hits Availability: 32.13 % Elapsed time: 9.00 secs Data transferred: 1.10 MB Response time: 2.37 secs Transaction rate: 40.44 trans/sec Throughput: 0.12 MB/sec Concurrency: 95.67 Successful transactions: 364 Failed transactions: 769 Longest transaction: 1.10 Shortest transaction: 0.38 ``` ##### Using trailing slashes If you have something like: ```nginx location /api/ { proxy_pass http://bck_testing_01; } ``` And go to `http://example.com/api`, NGINX will automatically redirect you to `http://example.com/api/`. Even if you don't use one of these directives above, you could always do the redirect manually: ```nginx location = /api { rewrite ^ /api/ permanent; } ``` Or, if you don't want redirect you could use: ```nginx location = /api { proxy_pass http://bck_testing_01; } ``` If you want to rewrite/redirect the URL with trailing slash at end you could: ```nginx # 1. At the http level: map $request_uri $no_trailing_slash { default 1; ~.*[^/]$ 0; } # 2. At the server level: server { listen 80; server_name example.com; location / { if ($no_trailing_slash) { return 302 $request_uri/; } proxy_pass http://192.168.10.50:80; proxy_redirect off; server_name_in_redirect off; } } ``` ##### Properly redirect all HTTP requests to HTTPS None of the standard answers are safe to use if at any point you had unsecure HTTP set up and expect user content, have forms, host an API, or have configured any website, tool, application, or utility to speak to your site. The problem occurs when a `POST` request is made to your server. If the server response with a plain 30x redirect the `POST` content will be lost. To prevent this situation remember about the correct redirect HTTP code for `POST` request ([Redirect POST request with payload to external endpoint](#redirect-post-request-with-payload-to-external-endpoint)). It is therefore recommended to use the 301 code only as a response for `GET` or `HEAD` methods and to use the 308 Permanent Redirect for `POST` methods instead, as the method change is explicitly prohibited with this status (see [Mozilla Web Docs - 301 Moved Permanently](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301)). ```nginx server { listen 192.168.200.10:80; server_name example.com; if ($request_method = POST) { return 307 https://$server_name$request_uri; } # return 301 https://example.com$request_uri; return 301 https://$server_name$request_uri; ... } server { listen 192.168.200.10:443 ssl; server_name example.com; # add Strict-Transport-Security to prevent man in the middle attacks: add_header Strict-Transport-Security "max-age=31536000" always; ... } ``` > Look also at [Enable HTTP Strict Transport Security (from this handbook)](RULES.md#beginner-enable-http-strict-transport-security). ##### Proxy/rewrite and keep the original URL If you just want to mount several locations from upstreams - you do not need rewrites, just use: ```nginx location /v1/app1/ { proxy_pass http://localhost:9001/; } ``` But apps should use relative links or account for their absolute location. For more complex url manipulation you can use `break`-rewrites: ```nginx location /v1/app1/ { rewrite ^/v1/app1/(.*) /$1 break; proxy_pass http://192.168.252.10:9001/; } ``` Next, a few control groups: ```nginx location /api/ { rewrite ^ $request_uri; rewrite ^/api/(.*) $1 break; # If the second rewrite won't match: return 400; proxy_pass http://192.168.252.10:82/$uri; } location /bar/ { rewrite ^/bar(/.*) $1 break; proxy_pass http://192.168.252.10:82; } ``` ##### Proxy/rewrite and keep the part of original URL ```nginx location ~ /some/path/(?
.+)/index.html { proxy_pass http://192.168.252.10/$section/index.html; proxy_set_header Host $host; } ``` Or: ```nginx location /some/path/ { proxy_pass http://192.168.252.10/; # Note this slash -------------^ proxy_set_header Host $host; } ``` ##### Proxy/rewrite without changing the original URL (in browser) > Generally, this is not recommend, because you're changing hostnames. Browser security is tied to it, as is webserver configuration. > You can rewrite URLs within same hostname, but changing hostnames requires redirect or using a frame. You can change hostname only if you have the same backends under control. If you want to get resources from `app1.domain` and the data should comes from `app2.domain/app`: ```nginx server_name app1.domain; location / { rewrite ^/(.*)$ /app/$1 break; proxy_pass http://192.168.252.10:80; # upstream for app2.domain proxy_set_header Host app2.domain; } ``` Other example: ```nginx location /site99 { rewrite /site99(.*)$ /site99/page$1 break; proxy_pass http://tomcat_b9:8000; } ``` ##### Adding and removing the `www` prefix > Note that these solutions gets tricky if you use HTTPS, as you must then have a single certificate to cover all of your domain names if you want this to work properly. The best practice of using a separate `server` would still stand. - `www` to `non-www`: ```nginx server { ... server_name www.example.com; # $scheme will get the http or https protocol: return 301 $scheme://example.com$request_uri; } ``` You can also do it for multiple `www` to `non-www` (e.g. for subdomains): ```nginx server { ... server_name "~^www\.(api.example.com)$" "~^www\.(public.example.com)$" "~^www\.(static.example.com)$"; return 301 $scheme://$1$request_uri; } ``` Or: ```nginx server { ... server_name ~^www\.(?(?:example\.org|example\.com|subdomain\.example\.net))$; return 301 $scheme://$domain$request_uri; } ``` This matches all domain names pointed to the server starting with `www.` and redirects to `non-www`: ```nginx server { ... server_name ~^www\.(?.+)$; return 301 $scheme://$domain$request_uri; } ``` These final solutions is generally not considered to be the best practice, however, it still works and does the job. Both removes `www` prefix before any domain.: ```nginx server { ... if ($host ~ ^www\.(?.+)$) { return 301 $scheme://$domain$request_uri; } } ``` It is not a recommended if you don't care for the most ultimate performance (uses the `if` condition and regular expression) but in some cases it may be useful: ```nginx server { ... if ($host ~* ^www\.(.*)$) { rewrite / $scheme://$1 permanent; } } ``` - `non-www` to `www`: ```nginx server { ... server_name example.com; # $scheme will get the http or https protocol: return 301 $scheme://www.example.com$request_uri; } ``` This matches all domain names pointed to the server starting with whatever but `www.` and redirects to `www.`: ```nginx server { ... server_name ~^(?!www\.)(?.+)$; return 301 $scheme://www.$domain$request_uri; } ``` The following is very similar to above but uses `if`: ```nginx server { ... server_name www.example.com www.example2.com; if ($host ~ ^(?!www\.)(?.+)$) { return 301 $scheme://www.$domain$request_uri; } } ``` ##### Modify 301/302 response body By default, NGINX sent small document body for 301 and 302 redirects. [RFC 2616 - 301 Moved Permanently](https://tools.ietf.org/html/rfc2616#section-10.3.2) [IETF] and [RFC 2616 - 302 Found](https://tools.ietf.org/html/rfc2616#section-10.3.3) [IETF] specifies that the entity bodies should be present. Here you have an excellent explanation of the problem by [Michael Hampton](https://serverfault.com/users/126632/michael-hampton): [NGINX 301 and 302 serving small nginx document body. Any way to remove this behaviour?](https://serverfault.com/a/423685). On the other hand, 301/302 bodies never actually contains any locations - all pages contain just an error number and message without any revealing information. So I understand the reasons for deleting the 301/302 body content. Performance can also be an argument, however it depends on what you're optimising for; the difference might be quite substancial for traffic counting, for example; it might also be the tipping point between needing to send an extra packet for the body or not. However, please note that this change is not RFC compliant: > _This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course._ Example 1: ```nginx server { ... error_page 301 302 @30x; location @30x { default_type ""; return 300; } location / { ... } ... } ``` Example 2: ```nginx server { ... error_page 301 /redirect; location = /redirect { internal; # We have to return 200 to modify content. If we would use 301 we would modify Location header. # Don't worry, HTTP status will be 301: return 200 "

301 redirect

"; # Or without body (200's need to be with a resource in the response): # return 204; } location / { ... # Redirect has to be enclosed in location: return 301 https://$host$request_uri; } ... } ``` For more information about `return` directive and `HTTP 200/204` response codes please see [`return` directive](doc/NGINX_BASICS.md#return-directive) from this handbook. Example 3: - modify the source code: `src/http/ngx_http_special_response.c` (but I not tested this solution) ##### Redirect POST request with payload to external endpoint **POST** data is passed in the body of the request, which gets dropped if you do a standard redirect. Look at this: | DESCRIPTION | PERMANENT | TEMPORARY | | :--- | :--- | :--- | | allows changing the request method from POST to GET | 301 | 302 | | does not allow changing the request method from POST to GET | 308 | 307 | You can try with the HTTP status code 307, a RFC compliant browser should repeat the post request. You just need to write a NGINX rewrite rule with HTTP status code 307 or 308: ```nginx location /api { # HTTP 307 only for POST requests: if ($request_method = POST) { return 307 https://api.example.com$request_uri; } # You can keep this for non-POST requests: rewrite ^ https://api.example.com$request_uri permanent; client_max_body_size 10m; ... } ``` ##### Route to different backends based on HTTP method This snippet is helpful if you want to route requests to different backends based on method. For example: - `POST /v1/orders/` - would go to one backend pool - `GET /v1/orders/` - would go to another backend pool If you don't want to mix the two methods in the same backend, for example the reason being that sometimes POST needs to be always fast, while GET involves DB queries and can be slow. Example 1: > In this example users can only see specific resource (`/v1/id`). The rest is hidden for them. ```nginx # 1) File: /etc/nginx/map_methods.conf map $request_method $method_dest { default '/_get/v1/id'; GET '/_get/v1/id'; POST '/_post/v1/id'; } # 2) Include this file in http context: include /etc/nginx/map_methods.conf; # 3) Use it in a specific context (e.g. location): ... server_name example.com; # It's only accessible to the clients: location /v1/id { set $original_uri $uri; rewrite ^ $method_dest last; } # It's not accessible to the clients: location /_get/v1/id { internal; rewrite ^ $original_uri break; proxy_pass http://default.example.com-get-80; } # It's not accessible to the clients: location /_post/v1/id { internal; rewrite ^ $original_uri break; proxy_pass http://default.example.com-post-80; } ... ``` ##### Allow multiple cross-domains using the CORS headers Example 1: ```nginx location ~* \.(?:ttf|ttc|otf|eot|woff|woff2)$ { if ( $http_origin ~* (https?://(.+\.)?(domain1|domain2|domain3)\.(?:me|co|com)$) ) { add_header "Access-Control-Allow-Origin" "$http_origin"; } } ``` Example 2 (more slightly configuration; for GETs and POSTs): ```nginx location / { if ($http_origin ~* (^https?://([^/]+\.)*(domainone|domaintwo)\.com$)) { set $cors "true"; } # Determine the HTTP request method used: if ($request_method = 'GET') { set $cors "${cors}get"; } if ($request_method = 'POST') { set $cors "${cors}post"; } if ($cors = "true") { # Catch all in case there's a request method we're not dealing with properly: add_header 'Access-Control-Allow-Origin' "$http_origin"; } if ($cors = "trueget") { add_header 'Access-Control-Allow-Origin' "$http_origin"; add_header 'Access-Control-Allow-Credentials' 'true'; add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'; } if ($cors = "truepost") { add_header 'Access-Control-Allow-Origin' "$http_origin"; add_header 'Access-Control-Allow-Credentials' 'true'; add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type'; } } ``` ##### Set correct scheme passed in X-Forwarded-Proto ```nginx # Sets a $real_scheme variable whose value is the scheme passed by the load # balancer in X-Forwarded-Proto (if any), defaulting to $scheme. # Similar to how the HttpRealIp module treats X-Forwarded-For. map $http_x_forwarded_proto $real_scheme { default $http_x_forwarded_proto; '' $scheme; } ``` #### Other snippets ###### Recreate base directory Debian like distributions: ```bash # Remove even configuration files and records: apt-get purge nginx nginx-common nginx-full # Reinstall: apt-get install nginx # You can also try using --force-confmiss option of dpkg: dpkg --force-confmiss -i /var/cache/apt/archives/nginx-common_*.deb ``` ###### Create a temporary static backend Busybox: ```bash busybox httpd -p $PORT -h $HOME [-c httpd.conf] ``` Python 3.x: ```bash python3 -m http.server 8000 --bind 127.0.0.1 ``` Python 2.x: ```bash python -m SimpleHTTPServer 8000 ``` ###### Create a temporary static backend with SSL support Python 3.x: ```python from http.server import HTTPServer, BaseHTTPRequestHandler import ssl httpd = HTTPServer(('localhost', 4443), BaseHTTPRequestHandler) httpd.socket = ssl.wrap_socket (httpd.socket, keyfile="path/to/key.pem", certfile='path/to/cert.pem', server_side=True) httpd.serve_forever() ``` Python 2.x: ```python import BaseHTTPServer, SimpleHTTPServer import ssl httpd = BaseHTTPServer.HTTPServer(('localhost', 4443), SimpleHTTPServer.SimpleHTTPRequestHandler) httpd.socket = ssl.wrap_socket (httpd.socket, keyfile="path/tp/key.pem", certfile='path/to/cert.pem', server_side=True) httpd.serve_forever() ``` ###### Generate password file with `htpasswd` command ```bash htpasswd -c htpasswd_example.com.conf ``` ###### Generate private key without passphrase ```bash # _len: 2048, 4096 ( _fd="private.key" ; _len="2048" ; \ openssl genrsa -out ${_fd} ${_len} ) ``` ###### Generate private key with passphrase ```bash # _ciph: des3, aes128, aes256 # _len: 2048, 4096 ( _ciph="aes128" ; _fd="private.key" ; _len="2048" ; \ openssl genrsa -${_ciph} -out ${_fd} ${_len} ) ``` ###### Remove passphrase from private key ```bash ( _fd="private.key" ; _fd_unp="private_unp.key" ; \ openssl rsa -in ${_fd} -out ${_fd_unp} ) ``` ###### Encrypt existing private key with a passphrase ```bash # _ciph: des3, aes128, aes256 ( _ciph="aes128" ; _fd="private.key" ; _fd_pass="private_pass.key" ; \ openssl rsa -${_ciph} -in ${_fd} -out ${_fd_pass} ``` ###### Generate private key and CSR ```bash ( _fd="private.key" ; _fd_csr="request.csr" ; _len="2048" ; \ openssl req -out ${_fd_csr} -new -newkey rsa:${_len} -nodes -keyout ${_fd} ) ``` ###### Generate CSR ```bash ( _fd="private.key" ; _fd_csr="request.csr" ; \ openssl req -out ${_fd_csr} -new -key ${_fd} ) ``` ###### Generate CSR (metadata from existing certificate) > Where `private.key` is the existing private key. As you can see you do not generate this CSR from your certificate (public key). Also you do not generate the "same" CSR, just a new one to request a new certificate. ```bash ( _fd="private.key" ; _fd_csr="request.csr" ; _fd_crt="cert.crt" ; \ openssl x509 -x509toreq -in ${_fd_crt} -out ${_fd_csr} -signkey ${_fd} ) ``` ###### Generate CSR with -config param ```bash ( _fd="private.key" ; _fd_csr="request.csr" ; \ openssl req -new -sha256 -key ${_fd} -out ${_fd_csr} \ -config <( cat << __EOF__ [req] default_bits = 2048 default_md = sha256 prompt = no distinguished_name = dn req_extensions = req_ext [ dn ] C = "" ST = "" L = "" O = "" OU = "
" CN = "" [ req_ext ] subjectAltName = @alt_names [ alt_names ] DNS.1 = DNS.2 = DNS.3 = __EOF__ )) ``` Other values in `[ dn ]`: ``` countryName = "DE" # C= stateOrProvinceName = "Hessen" # ST= localityName = "Keller" # L= postalCode = "424242" # L/postalcode= postalAddress = "Keller" # L/postaladdress= streetAddress = "Crater 1621" # L/street= organizationName = "apfelboymschule" # O= organizationalUnitName = "IT Department" # OU= commonName = "example.com" # CN= emailAddress = "webmaster@example.com" # CN/emailAddress= ``` Example of `oids` (you'll probably also have to make OpenSSL know about the new fields required for EV by adding the following under `[new_oids]`): ``` [req] ... oid_section = new_oids [ new_oids ] postalCode = 2.5.4.17 streetAddress = 2.5.4.9 ``` Full example: ```bash ( _fd="private.key" ; _fd_csr="request.csr" ; \ openssl req -new -sha256 -key ${_fd} -out ${_fd_csr} \ -config <( cat << __EOF__ [req] default_bits = 2048 default_md = sha256 prompt = no distinguished_name = dn req_extensions = req_ext oid_section = new_oids [ new_oids ] serialNumber = 2.5.4.5 streetAddress = 2.5.4.9 postalCode = 2.5.4.17 businessCategory = 2.5.4.15 [ dn ] serialNumber=00001111 businessCategory=Private Organization jurisdictionC=DE C=DE ST=Hessen L=Keller postalCode=424242 streetAddress=Crater 1621 O=AV Company OU=IT CN=example.com [ req_ext ] subjectAltName = @alt_names [ alt_names ] DNS.1 = example.com __EOF__ )) ``` For more information please look at these great explanations: - [RFC 5280](https://tools.ietf.org/html/rfc5280) - [How to create multidomain certificates using config files](https://apfelboymchen.net/gnu/notes/openssl%20multidomain%20with%20config%20files.html) - [Generate a multi domains certificate using config files](https://gist.github.com/romainnorberg/464758a6620228b977212a3cf20c3e08) - [Your OpenSSL CSR command is out of date](https://expeditedsecurity.com/blog/openssl-csr-command/) - [OpenSSL example configuration file](https://www.tbs-certificats.com/openssl-dem-server-cert.cnf) - [Object Identifiers (OIDs)](https://www.alvestrand.no/objectid/) - [openssl objects.txt](https://github.com/openssl/openssl/blob/master/crypto/objects/objects.txt) ###### List available EC curves ```bash openssl ecparam -list_curves ``` ###### Print ECDSA private and public keys ```bash ( _fd="private.key" ; \ openssl ec -in ${_fd} -noout -text ) # For x25519 only extracting public key ( _fd="private.key" ; _fd_pub="public.key" ; \ openssl pkey -in ${_fd} -pubout -out ${_fd_pub} ) ``` ###### Generate ECDSA private key ```bash # _curve: prime256v1, secp521r1, secp384r1 ( _fd="private.key" ; _curve="prime256v1" ; \ openssl ecparam -out ${_fd} -name ${_curve} -genkey ) # _curve: X25519 ( _fd="private.key" ; _curve="x25519" ; \ openssl genpkey -algorithm ${_curve} -out ${_fd} ) ``` ###### Generate private key and CSR (ECC) ```bash # _curve: prime256v1, secp521r1, secp384r1 ( _fd="example.com.key" ; _fd_csr="example.com.csr" ; _curve="prime256v1" ; \ openssl ecparam -out ${_fd} -name ${_curve} -genkey ; \ openssl req -new -key ${_fd} -out ${_fd_csr} -sha256 ) ``` ###### Generate self-signed certificate ```bash # _len: 2048, 4096 ( _fd="domain.key" ; _fd_out="domain.crt" ; _len="2048" ; _days="365" ; \ openssl req -newkey rsa:${_len} -nodes \ -keyout ${_fd} -x509 -days ${_days} -out ${_fd_out} ) ``` ###### Generate self-signed certificate from existing private key ```bash # _len: 2048, 4096 ( _fd="domain.key" ; _fd_out="domain.crt" ; _days="365" ; \ openssl req -key ${_fd} -nodes \ -x509 -days ${_days} -out ${_fd_out} ) ``` ###### Generate self-signed certificate from existing private key and csr ```bash # _len: 2048, 4096 ( _fd="domain.key" ; _fd_csr="domain.csr" ; _fd_out="domain.crt" ; _days="365" ; \ openssl x509 -signkey ${_fd} -nodes \ -in ${_fd_csr} -req -days ${_days} -out ${_fd_out} ) ``` ###### Generate multidomain certificate (Certbot) ```bash certbot certonly -d example.com -d www.example.com ``` ###### Generate wildcard certificate (Certbot) ```bash certbot certonly --manual --preferred-challenges=dns -d example.com -d *.example.com ``` ###### Generate certificate with 4096 bit private key (Certbot) ```bash certbot certonly -d example.com -d www.example.com --rsa-key-size 4096 ``` ###### Generate DH public parameters ```bash ( _dh_size="2048" ; \ openssl dhparam -out /etc/nginx/ssl/dhparam_${_dh_size}.pem "$_dh_size" ) ``` ###### Display DH public parameters ```bash openssl pkeyparam -in dhparam.pem -text ``` ###### Extract private key from pfx ```bash ( _fd_pfx="cert.pfx" ; _fd_key="key.pem" ; \ openssl pkcs12 -in ${_fd_pfx} -nocerts -nodes -out ${_fd_key} ) ``` ###### Extract private key and certs from pfx ```bash ( _fd_pfx="cert.pfx" ; _fd_pem="key_certs.pem" ; \ openssl pkcs12 -in ${_fd_pfx} -nodes -out ${_fd_pem} ) ``` ###### Extract certs from p7b ```bash # PKCS#7 file doesn't include private keys. ( _fd_p7b="cert.p7b" ; _fd_pem="cert.pem" ; \ openssl pkcs7 -inform DER -outform PEM -in ${_fd_p7b} -print_certs > ${_fd_pem}) # or: openssl pkcs7 -print_certs -in -in ${_fd_p7b} -out ${_fd_pem}) ``` ###### Convert DER to PEM ```bash ( _fd_der="cert.crt" ; _fd_pem="cert.pem" ; \ openssl x509 -in ${_fd_der} -inform der -outform pem -out ${_fd_pem} ) ``` ###### Convert PEM to DER ```bash ( _fd_der="cert.crt" ; _fd_pem="cert.pem" ; \ openssl x509 -in ${_fd_pem} -outform der -out ${_fd_der} ) ``` ###### Verification of the certificate's supported purposes ```bash ( _fd_pem="cert.pem" ; \ openssl x509 -purpose -noout -in ${_fd_pem} ) ``` ###### Check private key ```bash ( _fd="private.key" ; \ openssl rsa -check -in ${_fd} ) ``` ###### Verification of the private key ```bash ( _fd="private.key" ; \ openssl rsa -noout -text -in ${_fd} ) ``` ###### Get public key from private key ```bash ( _fd="private.key" ; _fd_pub="public.key" ; \ openssl rsa -pubout -in ${_fd} -out ${_fd_pub} ) ``` ###### Verification of the public key ```bash # 1) ( _fd="public.key" ; \ openssl pkey -noout -text -pubin -in ${_fd} ) # 2) ( _fd="private.key" ; \ openssl rsa -inform PEM -noout -in ${_fd} &> /dev/null ; \ if [ $? = 0 ] ; then echo -en "OK\n" ; fi ) ``` ###### Verification of the certificate ```bash ( _fd="certificate.crt" ; # format: pem, cer, crt \ openssl x509 -noout -text -in ${_fd} ) ``` ###### Verification of the CSR ```bash ( _fd_csr="request.csr" ; \ openssl req -text -noout -in ${_fd_csr} ) ``` ###### Check the private key and the certificate are match ```bash (openssl rsa -noout -modulus -in private.key | openssl md5 ; \ openssl x509 -noout -modulus -in certificate.crt | openssl md5) | uniq ``` ###### Check the private key and the CSR are match ```bash (openssl rsa -noout -modulus -in private.key | openssl md5 ; \ openssl req -noout -modulus -in request.csr | openssl md5) | uniq ``` ###### TLSv1.3 and CCM ciphers > **:bookmark: [Use only strong ciphers - Hardening - P1](RULES.md#beginner-use-only-strong-ciphers)** By default, TLS 1.3 don't use the two missing CCM-mode suites. If you want enable them, e.g. in case anything decides to support them in the future you should edit `openssl-1.1.1*/include/openssl/ssl.h` file. Look for these lines (starting at these in OpenSSL 1.1.1d): ```c # if !defined(OPENSSL_NO_CHACHA) && !defined(OPENSSL_NO_POLY1305) # define TLS_DEFAULT_CIPHERSUITES "TLS_AES_256_GCM_SHA384:" \ "TLS_CHACHA20_POLY1305_SHA256:" \ "TLS_AES_128_GCM_SHA256" # else # define TLS_DEFAULT_CIPHERSUITES "TLS_AES_256_GCM_SHA384:" \ "TLS_AES_128_GCM_SHA256" # endif # endif ``` Once you've found them, modify both `#define` instructions to look like this (add `TLS_AES_128_CCM_SHA256` and `TLS_AES_128_CCM_8_SHA256`), and be careful with the colons, quotes, and end-of-line escaping: ```c # if !defined(OPENSSL_NO_CHACHA) && !defined(OPENSSL_NO_POLY1305) # define TLS_DEFAULT_CIPHERSUITES "TLS_AES_128_GCM_SHA256:" \ "TLS_AES_128_CCM_SHA256:" \ "TLS_AES_128_CCM_8_SHA256:" \ "TLS_CHACHA20_POLY1305_SHA256:" \ "TLS_AES_256_GCM_SHA384" # else /* We're definitely building with ChaCha20-Poly1305, so the "else" won't have any effect. Still... */ # define TLS_DEFAULT_CIPHERSUITES "TLS_AES_128_GCM_SHA256:" \ "TLS_AES_256_GCM_SHA384" #endif ``` ================================================ FILE: doc/HTTP_BASICS.md ================================================ # HTTP Basics Go back to the **[Table of Contents](https://github.com/trimstray/nginx-admins-handbook#table-of-contents)** or **[What's next?](https://github.com/trimstray/nginx-admins-handbook#whats-next)** section. - **[≡ HTTP Basics](#http-basics)** * [Introduction](#introduction) * [Features and architecture](#features-and-architecture) * [HTTP/2](#http2) * [How to debug HTTP/2?](#how-to-debug-http2) * [HTTP/3](#http3) * [URI vs URL](#uri-vs-url) * [Connection vs request](#connection-vs-request) * [HTTP Headers](#http-headers) * [Header compression](#header-compression) * [HTTP Methods](#http-methods) * [Request](#request) * [Request line](#request-line) * [Methods](#methods) * [Request URI](#request-uri) * [HTTP version](#http-version) * [Request header fields](#request-header-fields) * [Message body](#message-body) * [Generate requests](#generate-requests) * [Response](#response) * [Status line](#status-line) * [HTTP version](#http-version-1) * [Status codes and reason phrase](#status-codes-and-reason-phrase) * [Response header fields](#response-header-fields) * [Message body](#message-body-1) * [HTTP client](#http-client) * [IP address shortcuts](#ip-address-shortcuts) * [Back-End web architecture](#back-end-web-architecture) * [Useful video resources](#useful-video-resources) #### Introduction Simply put, HTTP stands for hypertext transfer protocol and is used for transmitting data (e.g. web pages) over the Internet. Some important information about HTTP: - all requests originate at the client (e.g. browser) - the server responds to a request - the requests and responses are in readable text - the requests are independent of each other and the server doesn’t need to track the requests I will not describe the HTTP protocol meticulously so you have to look at this as an introduction. I will discuss only the most important things because we have some great documents which describe this protocol in a great deal of detail: - [RFC 2616 - HTTP/1.1](https://tools.ietf.org/html/rfc2616) [IETF] - [RFC 7230 - HTTP/1.1: Message Syntax and Routing](https://tools.ietf.org/html/rfc7230) [IETF] - [HTTP Made Really Easy](https://www.jmarshall.com/easy/http/) - [MDN web docs - An overview of HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview) - [LWP in Action - Chapter 2. Web Basics](http://lwp.interglacial.com/ch02_01.htm) - [HTTP and everything you need to know about it](https://medium.com/faun/http-and-everything-you-need-to-know-about-it-8273bc224491) I also recommend to read: - [Mini HTTP guide for developers](https://charemza.name/blog/posts/abstractions/http/http-guide-for-developers/) - [10 Great Web Performance Blogs](https://www.aaronpeters.nl/blog/10-great-web-performance-blogs/) We have some interesting books: - [HTTP: The Definitive Guide](https://www.amazon.com/HTTP-Definitive-Guide-Guides-ebook/dp/B0043D2EKO) - [High Performance Browser Networking](https://hpbn.co/) Look also at the [useful video resources](#useful-video-resources) section of this chapter. And finally look at [this](https://github.com/bigcompany/know-your-http) amazing series of A1-sized posters about the HTTP protocol. #### Features and architecture The HTTP (1.0/1.1 = h1) protocol is a request/response protocol based on the client/server based architecture where web browsers, robots and search engines, etc. act like HTTP clients, and the Web server acts as a server. This is HTTP's message-based model. Every HTTP interaction includes a request and response. By its nature, HTTP is stateless. Stateless means that all requests are separate from each other. So each request from your browser must contain enough information on its own for the server to fulfill the request. Here is a brief explanation: - most often the HTTP communication uses the TCP protocol - the default port is TCP 80, but other ports can be used - HTTP allow multiple requests and responses to be carried over a single (persistent) connection ([RFC 7230 - Persistence](https://tools.ietf.org/html/rfc7230#section-6.3) [IETF]) - HTTP protocol is stateless (all requests are separate from each other) - each transaction of the message based model of HTTP is processed separately from the others - the HTTP client, i.e., a browser initiates an HTTP request and after a request is made, the client waits for the response - the HTTP server handles and processing requests from clients (and continues to listen, and to accept other requests), after that it sends a response to the client - any type of data can be sent by HTTP as long as both the client and the server know how to handle the data content - the server and client are aware of each other only during a current request. Afterwards, both of them forget about each other The HTTP protocol allows clients and servers to communicate. Clients send requests using an HTTP method request and servers listen for requests on a host and port. The following is a comparison: - **client** - the HTTP client sends a request to the server in the form of a request method, URI, and protocol version, followed by a MIME-like message containing request modifiers, client information, and possible body content over a TCP/IP connection - **server** - the HTTP server responds with a status line, including the message's protocol version and a success or error code, followed by a MIME-like message containing server information, entity meta information, and possible entity-body content

HTTP_steps

This infographic comes from [www.ntu.edu.sg - HTTP (HyperText Transfer Protocol)](https://www.ntu.edu.sg/home/ehchua/programming/webprogramming/HTTP_Basics.html). #### HTTP/2 > **:bookmark: [Use HTTP/2 - Performance - P1](RULES.md#beginner-use-http2)** HTTP/2 (h2) is a major revision of the HTTP network protocol, intended as a higher performance alternative to HTTP/1.1. It introduces several new features, while remaining semantically compatible. HTTP/2 enables more efficient communication between browsers and servers that support it. All recent versions of major browsers support HTTP/2, including Chrome, Edge, Firefox, Opera, and Safari. Look at the following comparison:

http_comparison

I will not describe HTTP/2 because there are brilliant studies: - [RFC 7540 - HTTP/2](https://tools.ietf.org/html/rfc7540) [IETF] - [HTTP/2 finalized - a quick overview](https://evertpot.com/http-2-finalized/) - [Introduction to HTTP/2](https://developers.google.com/web/fundamentals/performance/http2) - [HTTP2 Explained](https://daniel.haxx.se/http2/) - [HTTP/2 in Action](https://www.manning.com/books/http2-in-action) - [HTTP/2 Frequently Asked Questions](https://http2.github.io/faq/) - [How Does HTTP/2 Work?](https://sookocheff.com/post/networking/how-does-http-2-work/) - [HTTP/2: the difference between HTTP/1.1, benefits and how to use it](https://medium.com/@factoryhr/http-2-the-difference-between-http-1-1-benefits-and-how-to-use-it-38094fa0e95b) - [HTTP2 Vs. HTTP1 – Let’s Understand The Two Protocols](https://cheapsslsecurity.com/p/http2-vs-http1/) - [HTTP 2 protocol – it is faster, but is it also safer?](https://research.securitum.com/http-2-protocol-it-is-faster-but-is-it-also-safer/) However, you should know a client that does not support HTTP/2 will never ask the server for an HTTP/2 communication upgrade: the communication between them will be fully HTTP/1.1. A client that supports HTTP/2 never makes a HTTP/2 request by default and will ask the server (using HTTP/1.1 with `Upgrade: HTTP/2.0` header) for an HTTP/2 upgrade: - if the server is HTTP/2 ready, then the server will notice the client as such: the communication between them will be switched to HTTP/2 - if the server is not HTTP/2 ready, then the server will ignore the upgrade request answering with HTTP/1.1: the communication between them should stay plenty HTTP/1.1 See also [NGINX Updates Mitigate the August 2019 HTTP/2 Vulnerabilities](https://www.nginx.com/blog/nginx-updates-mitigate-august-2019-http-2-vulnerabilities/). ##### How to debug HTTP/2? There is a great explanation about [Tools for debugging, testing and using HTTP/2](https://blog.cloudflare.com/tools-for-debugging-testing-and-using-http-2/). Look also at [Useful tools for HTTP/2 debugging](https://community.akamai.com/customers/s/article/Useful-tools-for-HTTP-2-debugging?language=en_US). #### HTTP/3 - [A QUIC look at HTTP/3](https://lwn.net/SubscriberLink/814522/ab3bfaa8f75c60ce/) - [HTTP/3 explained](https://daniel.haxx.se/http3-explained/) - [HTTP/3: the past, the present, and the future](https://blog.cloudflare.com/http3-the-past-present-and-future/) - [What Is HTTP/3 – Lowdown on the Fast New UDP-Based Protocol](https://kinsta.com/blog/http3/) #### URI vs URL Uniform Resource Identifier (URI) is a string of characters used to identify a name or a resource on the Internet. A URI identifies a resource either by location, or a name, or both. A URI has two specializations known as URL and URN. I think the best explanation is here: [The Difference Between URLs, URIs, and URNs](https://danielmiessler.com/study/url-uri/) by [Daniel Miessler](https://danielmiessler.com/about/). For me, this short and clear comment is also interesting: > URIs **identify** and URLs **identify** and **locate**; however, **locators are also identifiers**, so every URL is also a URI, but there are URIs which are not URLs.

url_urn_uri

Look at the following examples to get your mind out of confusion and take it simple: | TYPE | DESCRIPTION | | :---: | :--- | | URL | `https://www.google.com/folder/page.html` | | URL | `http://example.com/resource?foo=bar#fragment` | | URL | `ftp://example.com/download.zip` | | URL | `mailto:user@example.com` | | URL | `file:///home/user/file.txt` | | URL | `/other/link.html` (a relative URL) | | URN | `www.pierobon.org/iis/review1.htm#one` | | URN | `urn:ietf:rfc:2648` | | URN | `urn:isbn:0451450523` | | URI | `http://www.pierobon.org/iis/review1.htm.html#one` | The graphic below explains the URL format:

url_format

If it is still unclear, I would advise you to look at the following articles: - [What is the difference between a URI, a URL and a URN?](https://stackoverflow.com/questions/176264/what-is-the-difference-between-a-uri-a-url-and-a-urn/1984225) - [The History of the URL: Path, Fragment, Query, and Auth](https://eager.io/blog/the-history-of-the-url-path-fragment-query-auth/) #### Connection vs request Basically connections are established to make requests using it. So you can have multiple requests per connection. Not at the same time, of course, but rendering a web page is a fairly lengthy process. A connection is a TCP-based reliable pipeline between two endpoints. Each connection requires keeping track of both endpoint addresses/ports, sequence numbers, and which packets are unacknowledged or presumed missing. A request is a HTTP request for a given resource with a particular method. There may be zero or more requests per session. (Yes, web browsers can and do open connections without sending a request.) Most modern browsers open several connections at once, and download different files (images, css, js) at the same time to speed up the page load. But, like I said, each connection can handle multiple requests (downloads), too. To summarize: - HTTP connections - client and server introduce themselves; making a connection with server involves TCP handshaking and it is basically creating a socket connection with the server - HTTP requests - client ask something from server; to make a HTTP request you should be already established a connection with the server. If you established connection with a server you can make multiple request using the same connection (HTTP/1.0 by default one request per connection, HTTP/1.1 by default it is keep alive) There is also great comparison: - opening 25 connections, one after another, and downloading 1 file through each connection (slowest) - opening 1 connection and downloading 25 files through it (slow) - opening 5 connections and downloading 5 files through each connection (fast) - opening 25 connections and downloading 1 file through each connection (waste of resources) So if you limit the number of connections or the number of requests too much, you will slow down your page load speeds. #### HTTP Headers When a client requests a resource from a server it uses HTTP. This request includes a set of key-value pairs giving information like the version of the browser or what file formats it understands. These key-value pairs are called request headers. The server answers with the requested resource but also sends response headers giving information on the resource or the server itself. See these articles about HTTP headers: - [HTTP headers](https://developer.mozilla.org/pl/docs/Web/HTTP/Headers) - [The HTTP Request Headers List](https://flaviocopes.com/http-request-headers/) - [The HTTP Response Headers List](https://flaviocopes.com/http-response-headers/) - [Exotic HTTP Headers](https://peteris.rocks/blog/exotic-http-headers/) - [Secure your web application with these HTTP headers](https://odino.org/secure-your-web-application-with-these-http-headers/) - [OWASP Secure Headers Project](https://owasp.org/www-project-secure-headers/) HTTP headers allow the client and server to pass additional information with the request or the response. An HTTP header consists of its case-insensitive name followed by a colon `:`, then by its value (without line breaks). ###### Header compression The role of header compression (HTTP/2): > _Header compression resulted in an ~88% reduction in the size of request headers and an ~85% reduction in the size of response headers. On the lower-bandwidth DSL link, in which the upload link is only 375 Kbps, request header compression in particular, led to significant page load time improvements for certain sites (i.e. those that issued large number of resource requests). We found a reduction of 45 - 1142 ms in page load time simply due to header compression._ - HTTP/2 supports a new dedicated header compression algorithm, called HPACK - HPACK is resilient to CRIME - HPACK uses three methods of compression: Static Dictionary, Dynamic Dictionary, Huffman Encoding Please see also: - [Designing Headers for HTTP Compression](https://www.mnot.net/blog/2018/11/27/header_compression) - [HPACK: Header Compression for HTTP/2](https://http2.github.io/http2-spec/compression.html) - [HPACK: the silent killer (feature) of HTTP/2](https://blog.cloudflare.com/hpack-the-silent-killer-feature-of-http-2/) #### HTTP Methods The HTTP protocol includes a set of methods that indicate which action to be done for a resource. The most common methods are `GET` and `POST`. But there are a few others, too: - `GET` - is used to retreive data from a server at the specified resource - `POST` - is used to create or append a resource to an existing resource - `PUT` - is used to create or update (replace) the existing resource - `HEAD` - is almost identical to `GET`, except without the response body - `TRACE` - is used for used for diagnostic/debugging purposes which echo's back input back to the user - `OPTIONS` - is used to describe the communication options (HTTP methods) that are available for the target resource - `PATCH` - is used to apply partial modifications to a resource - `DELETE` - is used to delete the specified resource #### Request A request consists of: `(1) a command or request + (2) optional headers + (4) optional body content`: ``` FIELDS OF HTTP REQUEST PART OF RFC 2616 --------------------------------------------------------------------- Request = (1) : Request-line Section 5.1 (2) : *(( general-header Section 4.5 | request-header Section 5.3 | entity-header ) CRLF) Section 7.1 (3) : CRLF (4) : [ message-body ] Section 4.3 ``` Example of form an HTTP request to fetch `/alerts/status` page from the web server running on `localhost:8000`:

http_request

Additional information about requests: - route to the endpoint - rewrite path/query - deny access (acls) - headers modification ##### Request line The Request-line begins with a method, followed by the Request-URI and the protocol version, and ending with CRLF (`\r\n` or `0d0a` in a hex). The elements are separated by space SP characters: ``` Request-Line = Method SP Request-URI SP HTTP-Version CRLF ``` ###### Methods | METHOD | DESCRIPTION | | :---: | :--- | | `GET` | is used to retreive data from a server at the specified resource ([RFC 2616 - GET](https://tools.ietf.org/html/rfc2616#section-9.3)) [IETF] | For example, say you have an API with a `/api/v2/users` endpoint. Making a `GET` request to that endpoint should return a list of all available users. > Requests with `GET` method does not change any data (the state of resource) and should never be used to submit new information to the server. This method is almost identical to `HEAD`. If `GET /users` returns a list of users, then `HEAD /users` will make the same request but won't get back the list of users. At a basic level, these things should be validated: - check that a valid `GET` request returns a `200` status code - ensure that a `GET` request to a specific resource returns the correct data When executing a `GET` request, you ask the server for one, or a set of entities. To allow the client to filter the result, it can use the so called "query string" (`?`) of the URL. | METHOD | DESCRIPTION | | :---: | :--- | | `POST` | is used to send data to the sever to modify and update a resource ([RFC 2616 - POST](https://tools.ietf.org/html/rfc2616#section-9.5)) [IETF] | Look at the definition of `POST`: > _The `POST` method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI [...] The posted entity is subordinate to that URI in the same way that a file is subordinate to a directory containing it, a news article is subordinate to a newsgroup to which it is posted, or a record is subordinate to a database._ When executing a `POST` request, the client is actually submitting a new document to the remote host but the server decides the location where it is stored under the user collection. > Requests with `POST` method change data (the state of resource) on the backend server by modifying or updating a resource. If you do not know the actual resource location, for instance, when you add a new article, but do not have any idea where to store it, you can `POST` it to an URL, and let the server decide the actual URL. Here are some tips for testing `POST` requests: - create a resource with a `POST` request (the server could respond with a `201 Created`) - next, make a `GET` request to ensure a `200 OK` status code is returned (the data was saved correctly) - add tests that ensure `POST` requests fail with incorrect or ill-formatted data Modify and update a resource: ``` POST /items/ HTTP/1.1 Host: example.com ``` The following is an error: ``` POST /items/ HTTP/1.1 Host: example.com ``` | METHOD | DESCRIPTION | | :---: | :--- | | `PUT` | is used to send data to the sever to create or overwrite a resource ([RFC 2616 - PUT](https://tools.ietf.org/html/rfc2616#section-9.6)) [IETF] | Look at the definition of `PUT`: > _The `PUT` method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource [...] the origin server can create the resource with that URI._ The `PUT` specification requires that you already know the URL of the resource you wish to create or update. On create, if the client chooses the identifier for a resource a `PUT` request will create the new resource at the specified URL. If you know that a resource already exists for a URL, you can make a `PUT` request to that URL to replace the state of that resource on the server. Check for these things when testing `PUT` requests: - repeatedly call a `PUT` request always returns the same result (idempotent) - if an existing resource is modified, either the `200 OK` or `204 No Content` response codes SHOULD be sent to indicate successful completion of the request - `PUT` requests should fail if invalid data is supplied in the request - nothing should be updated For a new resource: ``` PUT /items/ HTTP/1.1 Host: example.com ``` To overwrite an existing resource: ``` PUT /items/ HTTP/1.1 Host: example.com ``` For me, the difference between `POST` and `PUT` is sometimes unclear. I think both can be used for creating and the big difference is that a `POST` request is supposed to let the server decide how to (and if at all) create a new resource. In my opinion, there is one more good explanation: > The `POST` method is used to send user-generated data to the web server. For example, a `POST` method is used when a user comments on a forum or if they upload a profile picture. A `POST` method should also be used if you do not know the specific URL of where your newly created resource should reside. > The `PUT` method completely replaces whatever currently exists at the target URL with something else. With this method, you can create a new resource or overwrite an existing one given you know the exact Request-URI. Look also at [this](https://stackoverflow.com/a/630475) amazing answer by [Brian R. Bondy](https://stackoverflow.com/users/3153/brian-r-bondy). ###### Request URI The Request-URI is a Uniform Resource Identifier and identifies the resource upon which to apply the request. The exact resource identified by an Internet request is determined by examining both the Request-URI and the Host header field. The most common form of Request-URI is that used to identify a resource on an origin server or gateway. For example, a client wishing to retrieve a resource directly from the origin server would create a TCP connection to port 80 of the host `example.com` and send the following lines: ``` GET /pub/index.html HTTP/1.1 Host: example.com ``` The absoluteURI form is required when the request is being made to a proxy: ``` GET http://example.com/pub/index.html HTTP/1.1 ``` > Note that the absolute path cannot be empty; if none is present in the original URI, it MUST be given as `/` (the server root). The asterisk `*` is used when an HTTP request does not apply to a particular resource, but to the server itself, and is only allowed when the method used does not necessarily apply to a resource. ###### HTTP version The last part of the request indicating the client's supported HTTP version. HTTP has four versions — HTTP/0.9, HTTP/1.0, HTTP/1.1, and HTTP/2.0. Today the versions in common use are HTTP/1.1 and HTTP/2.0. Determining the appropriate version of the HTTP protocol is very important because it allows you to set specific HTTP method or required headers (e.g. `cache-control` for HTTP/1.1). There is a nice explanation [How does a HTTP 1.1 server respond to a HTTP 1.0 request?](https://stackoverflow.com/questions/35850518/how-does-a-http-1-1-server-respond-to-a-http-1-0-request). ##### Request header fields There are three types of HTTP message headers for requests: - **General-header** - applying to both requests and responses but with no relation to the data eventually transmitted in the body - **Request-header** - containing more information about the resource to be fetched or about the client itself - **Entity-header** - containing more information about the body of the entity, like its content length or its MIME-type The Request-header fields allow the client to pass additional information about the request, and about the client itself, to the server. What is an accepted maximum allowed size for HTTP headers? Read [this](https://stackoverflow.com/questions/686217/maximum-on-http-header-values) great answer. HTTP does not define any limit. However, the most web servers do limit size of headers they accept. Server will return `413 Entity Too Large` error if headers size exceeds that limit. ##### Message body Request (message) body is the part of the HTTP request where additional content can be sent to the server. It is optional. Most of the HTTP requests are `GET` requests without bodies. However, simulating requests with bodies is important to properly stress the proxy code and to test various hooks working with such requests. Most HTTP requests with bodies use `POST` or `PUT` request method. ##### Generate requests How to generate a request? - `curl` ```bash curl -Iks -v -X GET -H "Connection: keep-alive" -H "User-Agent: X-AGENT" https://example.com ``` - `httpie` ```bash http -p Hh GET https://example.com User-Agent:X-AGENT --follow ``` - `telnet` ```bash telnet example.com 80 GET /index.html HTTP/1.1 Host: example.com ``` - `openssl` ```bash openssl s_client -servername example.com -connect example.com:443 ... --- GET /index.html HTTP/1.1 Host: example.com ``` For more examples, see [Testing](HELPERS.md#testing) chapter. #### Response After receiving and interpreting a request message, a server responds with an HTTP response message: ``` FIELDS OF HTTP RESPONSE PART OF RFC 2616 --------------------------------------------------------------------- Response = (1) : Status-line Section 6.1 (2) : *(( general-header Section 4.5 | response-header Section 6.2 | entity-header ) CRLF) Section 7.1 (3) : CRLF (4) : [ message-body ] Section 7.2 ``` Example of form an HTTP response for a request to fetch the `/alerts/status` page from the web server running on `localhost:8000`:

http_request

Additional information about responses: - caching objects (browser/proxy) - headers modification - body modification ##### Status line The Status-line consisting of the protocol version followed by a numeric status code and its associated textual phrase. ``` Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF ``` ###### HTTP version > When an HTTP/1.1 message is sent to an HTTP/1.0 recipient or a recipient whose version is unknown, the HTTP/1.1 message is constructed such that it can be interpreted as a valid HTTP/1.0 message if all of the newer features are ignored. ###### Status codes and reason phrase - `1xx: Informational` - Request received, continuing process - `2xx: Success` - The action was successfully received, understood, and accepted - `3xx: Redirection` - Further action must be taken in order to complete the request - `4xx: Client Error` - The request contains bad syntax or cannot be fulfilled - `5xx: Server Error` - The server failed to fulfill an apparently valid request For more information please see: - [HTTP response status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) - [HTTP Status Codes](https://httpstatuses.com/) - [RFC 2616 - Status Code Definitions](https://tools.ietf.org/html/rfc2616#section-10) Be sure to take a look at this (it's genius work!):

http_decision_diagram

This diagram comes from [for-GET/http-decision-diagram](https://github.com/for-GET/http-decision-diagram) repository. The diagram follows the indications in [RFC7230](https://tools.ietf.org/html/rfc7230), [RFC7231](https://tools.ietf.org/html/rfc7231), [RFC7232](https://tools.ietf.org/html/rfc7232), [RFC7233](https://tools.ietf.org/html/rfc7233), [RFC7234](https://tools.ietf.org/html/rfc7234), [RFC7235](https://tools.ietf.org/html/rfc7235), and fills in the void where necessary. Under no circumstances does this diagram override the HTTP specifications. ##### Response header fields There are three types of HTTP message headers for responses: - **General-header** - applying to both requests and responses but with no relation to the data eventually transmitted in the body - **Response-header** - these header fields give information about the server and about further access to the resource identified by the Request-URI - **Entity-header** - containing more information about the body of the entity, like its content length or its MIME-type The response-header fields allow the server to pass additional information about the response. ##### Message body Contains the resource data that was requested by the client. #### HTTP client HTTP client is a client that is able to send a request to and get a response from the server in HTTP format. Clients also originates a connection, passes any necessary authentication tokens and delivers the request for a specific piece of data. > The clients are anything that send requests to the back-end. ##### IP address shortcuts IP addresses can be shortened by dropping the zeroes: ``` http://1.0.0.1 → http://1.1 http://127.0.0.1 → http://127.1 http://192.168.0.1 → http://192.168.1 http://0xC0A80001 or http://3232235521 → 192.168.0.1 http://192.168.257 → 192.168.1.1 http://192.168.516 → 192.168.2.4 ``` > This bypasses WAF filters for SSRF, open-redirect, etc where any IP as input gets blacklisted. For more information please see [How to Obscure Any URL](http://www.pc-help.org/obscure.htm) and [Magic IP Address Shortcuts](https://stuff-things.net/2014/09/25/magic-ip-address-shortcuts/). #### Back-End web architecture > I recommend to read these great repositories: > > - [Learn how to design large-scale systems](https://github.com/donnemartin/system-design-primer)
> - [Web Architecture 101](https://engineering.videoblocks.com/web-architecture-101-a3224e126947) The back-end, or the "server side", is all of the technology required to process the incoming request and generate and send the response to the client. This typically includes three major parts: - **server** - this is the computer that receives requests (backend/origin server) - **app** - this is the application running on the server that listens for requests, retrieves information from the database, and sends a response - **databases** - are used to organize and persist data #### Useful video resources - [HTTP Crash Course & Exploration](https://youtu.be/iYM2zFP3Zn0) - [CS50 2017 - Lecture 6 - HTTP](https://youtu.be/PUPDGbnpSjw) - [HTTP/2 101 (Chrome Dev Summit 2015)](https://youtu.be/r5oT_2ndjms) - [Headers for Hackers: Wrangling HTTP Like a Pro](https://youtu.be/TNlcoYLIGFk) - [Hacking with Andrew and Brad: an HTTP/2 client](https://youtu.be/yG-UaBJXZ80) - [Advanced HTTP Protocol Hacking](https://youtu.be/up8Vz5PTX3M) - [Hacking HTTP/2 - New Attacks on the Internet's Next Generation Foundation](https://youtu.be/kM8cc0kB21s) ================================================ FILE: doc/NGINX_BASICS.md ================================================ # NGINX Basics Go to the **[Table of Contents](https://github.com/trimstray/nginx-admins-handbook#table-of-contents)** or **[What's next?](https://github.com/trimstray/nginx-admins-handbook#whats-next)** section. - **[≡ NGINX Basics](#nginx-basics)** * [Directories and files](#directories-and-files) * [Commands](#commands) * [Processes](#processes) * [CPU pinning](#cpu-pinning) * [Shutdown of worker processes](#shutdown-of-worker-processes) * [Configuration syntax](#configuration-syntax) * [Comments](#comments) * [End of lines](#end-of-lines) * [Variables, Strings, and Quotes](#variables-strings-and-quotes) * [Directives, Blocks, and Contexts](#directives-blocks-and-contexts) * [External files](#external-files) * [Measurement units](#measurement-units) * [Regular expressions with PCRE](#regular-expressions-with-pcre) * [Enable syntax highlighting](#enable-syntax-highlighting) * [Connection processing](#connection-processing) * [Event-Driven architecture](#event-driven-architecture) * [Multiple processes](#multiple-processes) * [Simultaneous connections](#simultaneous-connections) * [HTTP Keep-Alive connections](#http-keep-alive-connections) * [sendfile, tcp_nodelay, and tcp_nopush](#sendfile-tcp_nodelay-and-tcp_nopush) * [Request processing stages](#request-processing-stages) * [Server blocks logic](#server-blocks-logic) * [Handle incoming connections](#handle-incoming-connections) * [Matching location](#matching-location) * [rewrite vs return](#rewrite-vs-return) * [URL redirections](#url-redirections) * [try_files directive](#try_files-directive) * [if, break, and set](#if-break-and-set) * [root vs alias](#root-vs-alias) * [internal directive](#internal-directive) * [External and internal redirects](#external-and-internal-redirects) * [allow and deny](#allow-and-deny) * [uri vs request_uri](#uri-vs-request_uri) * [Compression and decompression](#compression-and-decompression) * [What is the best NGINX compression gzip level?](#what-is-the-best-nginx-compression-gzip-level) * [Hash tables](#hash-tables) * [Server names hash table](#server-names-hash-table) * [Log files](#log-files) * [Conditional logging](#conditional-logging) * [Manually log rotation](#manually-log-rotation) * [Error log severity levels](#error-log-severity-levels) * [How to log the start time of a request?](#how-to-log-the-start-time-of-a-request) * [How to log the HTTP request body?](#how-to-log-the-http-request-body) * [NGINX upstream variables returns 2 values](#nginx-upstream-variables-returns-2-values) * [Reverse proxy](#reverse-proxy) * [Passing requests](#passing-requests) * [Trailing slashes](#trailing-slashes) * [Passing headers to the backend](#passing-headers-to-the-backend) * [Importance of the Host header](#importance-of-the-host-header) * [Redirects and X-Forwarded-Proto](#redirects-and-x-forwarded-proto) * [A warning about the X-Forwarded-For](#a-warning-about-the-x-forwarded-for) * [Improve extensibility with Forwarded](#improve-extensibility-with-forwarded) * [Response headers](#response-headers) * [Load balancing algorithms](#load-balancing-algorithms) * [Backend parameters](#backend-parameters) * [Upstream servers with SSL](#upstream-servers-with-ssl) * [Round Robin](#round-robin) * [Weighted Round Robin](#weighted-round-robin) * [Least Connections](#least-connections) * [Weighted Least Connections](#weighted-least-connections) * [IP Hash](#ip-hash) * [Generic Hash](#generic-hash) * [Other methods](#other-methods) * [Rate limiting](#rate-limiting) * [Variables](#variables) * [Directives, keys, and zones](#directives-keys-and-zones) * [Burst and nodelay parameters](#burst-and-nodelay-parameters) * [NAXSI Web Application Firewall](#naxsi-web-application-firewall) * [OWASP ModSecurity Core Rule Set (CRS)](#owasp-modsecurity-core-rule-set-crs) * [Core modules](#core-modules) * [ngx_http_geo_module](#ngx_http_geo_module) * [3rd party modules](#3rd-party-modules) * [ngx_set_misc](#ngx_set_misc) * [ngx_http_geoip_module](#ngx_http_geoip_module) #### Directories and files > If you compile NGINX with default parameters all files and directories are available from `/usr/local/nginx` location. For upstream NGINX packaging paths can be as follows (it depends on the type of system/distribution): - `/etc/nginx` - is the default configuration root for the NGINX service * other locations: `/usr/local/etc/nginx`, `/usr/local/nginx/conf` - `/etc/nginx/nginx.conf` - is the default configuration entry point used by the NGINX services, includes the top-level http block and all other configuration contexts and files * other locations: `/usr/local/etc/nginx/nginx.conf`, `/usr/local/nginx/conf/nginx.conf` - `/usr/share/nginx` - is the default root directory for requests, contains `html` directory and basic static files * other locations: `html/` in root directory - `/var/log/nginx` - is the default log (access and error log) location for NGINX * other locations: `logs/` in root directory - `/var/cache/nginx` - is the default temporary files location for NGINX * other locations: `/var/lib/nginx` - `/etc/nginx/conf` - contains custom/vhosts configuration files * other locations: `/etc/nginx/conf.d`, `/etc/nginx/sites-enabled` (I can't stand this debian/apache-like convention) - `/var/run/nginx` - contains information about NGINX process(es) * other locations: `/usr/local/nginx/logs`, `logs/` in root directory See also [Installation and Compile-Time Options - Files and Permissions](https://www.nginx.com/resources/wiki/start/topics/tutorials/installoptions/#files-and-permissions). #### Commands > **:bookmark: [Use reload option to change configurations on the fly - Base Rules - P2](RULES.md#beginner-use-reload-option-to-change-configurations-on-the-fly)** - `nginx -h` - shows the help - `nginx -v` - shows the NGINX version - `nginx -V` - shows the extended information about NGINX: version, build parameters, and configuration arguments - `nginx -t` - tests the NGINX configuration - `nginx -c ` - sets configuration file (default: `/etc/nginx/nginx.conf`) - `nginx -p ` - sets prefix path (default: `/etc/nginx/`) - `nginx -T` - tests the NGINX configuration and prints the validated configuration on the screen - `nginx -s ` - sends a signal to the NGINX master process: - `stop` - discontinues the NGINX process immediately - `quit` - stops the NGINX process after it finishes processing inflight requests - `reload` - reloads the configuration without stopping processes - `reopen` - instructs NGINX to reopen log files - `nginx -g ` - sets [global directives](https://nginx.org/en/docs/ngx_core_module.html) out of configuration file Some useful snippets for management of the NGINX daemon: - testing configuration: ```bash /usr/sbin/nginx -t -c /etc/nginx/nginx.conf /usr/sbin/nginx -t -q -g 'daemon on; master_process on;' # ; echo $? /usr/local/etc/rc.d/nginx status ``` - starting daemon: ```bash /usr/sbin/nginx -g 'daemon on; master_process on;' service nginx start systemctl start nginx /usr/local/etc/rc.d/nginx start # You can also start NGINX from start-stop-daemon script: /sbin/start-stop-daemon --quiet --start --exec /usr/sbin/nginx --background --retry QUIT/5 --pidfile /run/nginx.pid ``` - stopping daemon: ```bash # graceful shutdown (waiting for the worker processes to finish serving current requests) /usr/sbin/nginx -s quit # fast shutdown (kill connections immediately) /usr/sbin/nginx -s stop service nginx stop systemctl stop nginx /usr/local/etc/rc.d/nginx stop # You can also stop NGINX from start-stop-daemon script: /sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid ``` - reloading daemon: ```bash /usr/sbin/nginx -g 'daemon on; master_process on;' -s reload service nginx reload systemctl reload nginx /usr/local/etc/rc.d/nginx reload kill -HUP $(cat /var/run/nginx.pid) kill -HUP $(pgrep -f "nginx: master") ``` - restarting daemon: ```bash service nginx restart systemctl restart nginx /usr/local/etc/rc.d/nginx restart ``` Something about testing configuration: > You cannot test half-baked configurations. For example, you defined a server section for your domain in a separate file. Any attempt to test such a file will throw errors. The file has to be complete in all respects. #### Configuration syntax > **:bookmark: [Organising Nginx configuration - Base Rules - P2](RULES.md#beginner-organising-nginx-configuration)**
> **:bookmark: [Format, prettify and indent your Nginx code - Base Rules - P2](RULES.md#beginner-format-prettify-and-indent-your-nginx-code)** NGINX uses a micro programming language in the configuration files. This language's design is heavily influenced by Perl and Bourne Shell. Configuration syntax, formatting and definitions follow a so-called C-style convention. For me, NGINX configuration has a simple and very transparent structure. ##### Comments NGINX configuration files don't support comment blocks, they only accept `#` at the beginning of a line for a comment. ##### End of lines Lines containing directives must end with a semicolon (`;`), otherwise NGINX will fail to load the configuration and report an error. ##### Variables, Strings, and Quotes Variables start with `$` and that get set automaticaly for each request. The ability to set variables at runtime and control logic flow based on them is part of the rewrite module and not a general feature of NGINX. By default, we cannot modify built-in variables like `$host` or `$request_uri`. > There are some directives that do not support variables, e.g. `access_log` (is really the exception because can contain variables with restrictions) or `error_log`. Variables probably can't be (and shouldn't be because they are evaluated in the run-time during the processing of each request and rather costly compared to plain static configuration) declared anywhere, with very few exceptions: `root` directive can contains variables, `server_name` directive only allows strict `$hostname` built-in value as a variable-like notation (but it's more like a magic constant). If you use variables in `if` context, you can only set them in `if` conditions (and maybe rewrite directives). Don’t try to use them elsewhere. To assign value to the variable you should use a `set` directive: ```nginx set $var "value"; ``` > See [`if`, `break`, and `set`](#if-break-and-set) section to learn more about variables. Some interesting things about variables: > Make sure to read the [agentzh's Nginx Tutorials](https://openresty.org/download/agentzh-nginx-tutorials-en.html) - it's about NGINX tips & tricks. This guy is a NGINX Guru and creator of the OpenResty. In these tutorials he describes, amongst other things, variables in great detail. I also recommend [nginx built-in variables](http://siwei.me/blog/posts/nginx-built-in-variables) post. - the most variables in NGINX only exist at runtime, not during configuration time - the scope of variables spreads out all over configuration - variable assignment occurs when requests are actually being served - variable have exactly the same lifetime as the corresponding request - each request does have its own version of all those variables' containers (different containers values) - requests do not interfere with each other even if they are referencing a variable with the same name - the assignment operation is only performed in requests that access location Strings may be inputted without quotes unless they include blank spaces, semicolons or curly braces, then they need to be escaped with backslashes or enclosed in single/double quotes. Quotes are required for values which are containing space(s) and/or some other special characters, otherwise NGINX will not recognize them. You can either quote or `\`-escape some special characters like `" "` or `";"` in strings (characters that would make the meaning of a statement ambiguous). So the following instructions are the same: ```nginx # 1) add_header My-Header "nginx web server;"; # 2) add_header My-Header nginx\ web\ server\;; ``` Variables in quoted strings are expanded normally unless the `$` is escaped. ##### Directives, Blocks, and Contexts > Read this great article about [the NGINX configuration inheritance model](https://blog.martinfjordvald.com/understanding-the-nginx-configuration-inheritance-model/) by [Martin Fjordvald](https://blog.martinfjordvald.com/about/). Configuration options are called directives. We have four types of directives: - standard directive - one value per context: ```nginx worker_connections 512; ``` - array directive - multiple values per context: ```nginx error_log /var/log/nginx/localhost/localhost-error.log warn; ``` - action directive - something which does not just configure: ```nginx rewrite ^(.*)$ /msie/$1 break; ``` - `try_files` directive: ```nginx try_files $uri $uri/ /test/index.html; ``` Valid directives begin with a variable name and then state an argument or series of arguments separated by spaces. Directives are organised into groups known as **blocks** or **contexts**. Generally, context is a block directive that can have other directives inside braces. It appears to be organised in a tree-like structure, defined by sets of brackets - `{` and `}`. > The curly braces actually denote a new configuration context. As a general rule, if a directive is valid in multiple nested scopes, a declaration in a broader context will be passed on to any child contexts as default values. The children contexts can override these values at will. > Directives placed in the configuration file outside of any contexts are considered to be in the global/main context. > Special attention should be paid to some strange behavior associated with some directives. For more information please see [Set the HTTP headers with add_header and proxy_*_header directives properly](RULES.md#beginner-set-the-http-headers-with-add_header-and-proxy__header-directives-properly) rule. Directives can only be used in the contexts that they were designed for. NGINX will error out on reading a configuration file with directives that are declared in the wrong context. > If you want to review all directives see [alphabetical index of directives](https://nginx.org/en/docs/dirindex.html). Contexts can be layered within one another (a level of inheritance). Their structure looks like this: ``` Global/Main Context | | +-----» Events Context | | +-----» HTTP Context | | | | | +-----» Server Context | | | | | | | | +-----» Location Context | | | | | +-----» Upstream Context | | +-----» Mail Context ``` The most important contexts are shown in the following description. These will be the ones that you will be dealing with for the most part: - `global` - contains global configuration directives; is used to set the settings for NGINX globally and is the only context that is not surrounded by curly braces - `events` - configuration for the events module; is used to set global options for connection processing; contains directives that affect connection processing are specified - `http` - controls all the aspects of working with the HTTP module and holds directives for handling HTTP and HTTPS traffic; directives in this context can be grouped into: - HTTP client directives - HTTP file I/O directives - HTTP hash directives - HTTP socket directives - `server` - defines virtual host settings and describes a logical separation of a set of resources associated with a particular domain or IP address - `location` - define directives to handle client request and indicates a URI that comes either from the client or from an internal redirect - `upstream` - define a pool of back-end servers that NGINX can proxy the request; commonly used for defining either a web server cluster for load balancing NGINX also provides other contexts (e.g. used for mapping) such as: - `map` - is used to set the value of a variable depending on the value of another variable. It provides a mapping of one variable’s values to determine what the second variable should be set to - `geo` - is used to specify a mapping. However, this mapping is specifically used to categorize client IP addresses. It sets the value of a variable depending on the connecting IP address - `types` - is used to map MIME types to the file extensions that should be associated with them - `if` - provide conditional processing of directives defined within, execute the instructions contained if a given test returns `true` - `limit_except` - is used to restrict the use of certain HTTP methods within a location context Look also at the graphic below. It presents the most important contexts with reference to the configuration:

nginx-contexts

For HTTP, NGINX lookup starts from the http block, then through one or more server blocks, followed by the location block(s). ##### External files `include` directive may appear inside any contexts to perform conditional inclusion. It attaching another file, or files matching the specified mask: ```nginx include /etc/nginx/proxy.conf; # or: include /etc/nginx/conf/*.conf; ``` > You cannot use variables in NGINX config file includes. This is because includes are processed before any variables are evaluated. See also [this](http://nginx.org/en/docs/faq/variables_in_config.html): > _Variables should not be used as template macros. Variables are evaluated in the run-time during the processing of each request, so they are rather costly compared to plain static configuration. Using variables to store static strings is also a bad idea. Instead, a macro expansion and "include" directives should be used to generate configs more easily and it can be done with the external tools, e.g. sed + make or any other common template mechanism._ ##### Measurement units > It is recommended to always specify a suffix for the sake of clarity and consistency. Sizes can be specified in: - without a suffix: Bytes - `k` or `K`: Kilobytes - `m` or `M`: Megabytes - `g` or `G`: Gigabytes ```nginx client_max_body_size 2m; ``` Time intervals can be specified in: - without a suffix: Seconds - `ms`: Milliseconds - `s`: Seconds - `m`: Minutes - `h`: Hours - `d`: Days - `w`: Weeks - `M`: Months (30 days) - `y`: Years (365 days) ```nginx proxy_read_timeout 20; # =20s, default ``` Some of the time intervals can be specified only with a seconds resolution. You should also remember about this: > _Multiple units can be combined in a single value by specifying them in the order from the most to the least significant, and optionally separated by whitespace. For example, `1h 30m` specifies the same time as `90m` or `5400s`_. ##### Regular expressions with PCRE > **:bookmark: [Enable PCRE JIT to speed up processing of regular expressions - Performance - P2](RULES.md#beginner-enable-pcre-jit-to-speed-up-processing-of-regular-expressions)** Before start reading next chapters you should know what regular expressions are and how they works (they are not a black magic really). I recommend two great and short write-ups about regular expressions created by [Jonny Fox](https://medium.com/@jonny.fox): - [Regex tutorial — A quick cheatsheet by examples](https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285) - [Regex cookbook — Top 10 Most wanted regex](https://medium.com/factory-mind/regex-cookbook-most-wanted-regex-aa721558c3c1) Why? Regular expressions can be used in both the `server_name` and `location` (also in other) directives, and sometimes you must have a great skills of reading them. I think you should create the most readable regular expressions that do not become spaghetti code - impossible to debug and maintain. NGINX uses the [PCRE](https://www.pcre.org/) library to perform complex manipulations with your `location` blocks and use the powerful `rewrite` directive. To use a regular expression for string matching, it first needs to be compiled, which is usually done at the configuration phase. You can also enable `pcre_jit` to dynamic translation during execution (at run time) rather than prior to execution. This option can improve performance, however, in some cases `pcre_jit` may have a negative effect. So, before enabling it, I recommend you to read this great document: [PCRE Performance Project](https://zherczeg.github.io/sljit/pcre.html). Below is also something interesting about regular expressions and PCRE: - [Learn PCRE in Y minutes](https://learnxinyminutes.com/docs/pcre/) - [PCRE Regex Cheatsheet](https://www.debuggex.com/cheatsheet/regex/pcre) - [Regular Expression Cheat Sheet - PCRE](https://github.com/niklongstone/regular-expression-cheat-sheet) - [Regex cheatsheet](https://remram44.github.io/regex-cheatsheet/regex.html) - [Regular expressions in Perl](http://jkorpela.fi/perl/regexp.html) - [Regexp Security Cheatsheet](https://github.com/attackercan/regexp-security-cheatsheet) - [A regex cheatsheet for all those regex haters (and lovers)](https://dev.to/catherinecodes/a-regex-cheatsheet-for-all-those-regex-haters-and-lovers--2cj1) You can also use external tools for testing regular expressions. For more please see [online tools](https://github.com/trimstray/nginx-admins-handbook#online-tools) chapter. If you're good at it, check these very nice and brainstorming regex challenges: - [RegexGolf](https://alf.nu/RegexGolf) - [Regex Crossword](https://regexcrossword.com/) ##### Enable syntax highlighting ###### vi/vim ```bash # 1) Download vim plugin for NGINX: # Official NGINX vim plugin: mkdir -p ~/.vim/syntax/ wget "http://www.vim.org/scripts/download_script.php?src_id=19394" -O ~/.vim/syntax/nginx.vim # Improved NGINX vim plugin (incl. syntax highlighting) with Pathogen: mkdir -p ~/.vim/{autoload,bundle}/ curl -LSso ~/.vim/autoload/pathogen.vim https://tpo.pe/pathogen.vim echo -en "\nexecute pathogen#infect()\n" >> ~/.vimrc git clone https://github.com/chr4/nginx.vim ~/.vim/bundle/nginx.vim # 2) Set location of NGINX config files: cat > ~/.vim/filetype.vim << __EOF__ au BufRead,BufNewFile /etc/nginx/*,/etc/nginx/conf.d/*,/usr/local/nginx/conf/*,*/conf/nginx.conf if &ft == '' | setfiletype nginx | endif __EOF__ ``` > It may be interesting for you: [Highlight insecure SSL configuration in Vim](https://github.com/chr4/sslsecure.vim). ###### Sublime Text Install `cabal` - system for building and packaging Haskell libraries and programs (on Ubuntu): ```bash add-apt-repository -y ppa:hvr/ghc apt-get update apt-get install -y cabal-install-1.22 ghc-7.10.2 # Add this to the main configuration file of your shell: export PATH=$HOME/.cabal/bin:/opt/cabal/1.22/bin:/opt/ghc/7.10.2/bin:$PATH source $HOME/. cabal update ``` - `nginx-lint`: ```bash git clone https://github.com/temoto/nginx-lint cd nginx-lint && cabal install --global ``` - `sublime-nginx` + `SublimeLinter-contrib-nginx-lint`: Bring up the _Command Palette_ and type `install`. Among the commands you should see _Package Control: Install Package_. Type `nginx` to install [sublime-nginx](https://github.com/brandonwamboldt/sublime-nginx) and after that do the above again for install [SublimeLinter-contrib-nginx-lint](https://github.com/irvinlim/SublimeLinter-contrib-nginx-lint): type `SublimeLinter-contrib-nginx-lint`. #### Processes > **:bookmark: [Adjust worker processes - Performance - P3](RULES.md#beginner-adjust-worker-processes)**
> **:bookmark: [Improve debugging by disable daemon, master process, and all workers except one - Debugging - P4](RULES.md#beginner-improve-debugging-by-disable-daemon-master-process-and-all-workers-except-one)** NGINX has **one master process** and **one or more worker processes**. It has also cache loader and cache manager processes but only if you enable caching. The main purposes of the master process is to read and evaluate configuration files, as well as maintain the worker processes (respawn when a worker dies), handle signals, notify workers, opens log files, and, of course binding to ports. Master process should be started as root user, because this will allow NGINX to open sockets below 1024 (it needs to be able to listen on port 80 for HTTP and 443 for HTTPS). > To defines the number of worker processes you should set `worker_processes` directive. The worker processes do the actual processing of requests and get commands from master process. They runs in an event loop (registering events and responding when one occurs), handle network connections, read and write content to disk, and communicate with upstream servers. These are spawned by the master process, and the user and group will as specified (unprivileged). > The worker processes spend most of the time just sleeping and waiting for new events (they are in `S` state in `top`). The following signals can be sent to the NGINX master process: | SIGNAL | NUM | DESCRIPTION | | :--- | :---: | :--- | | `TERM`, `INT` | **15**, **2** | quick shutdown | | `QUIT` | **3** | graceful shutdown | | `KILL` | **9** | halts a stubborn process | | `HUP` | **1** | configuration reload, start new workers, gracefully shutdown the old worker processes | | `USR1` | **10** | reopen the log files | | `USR2` | **12** | upgrade executable on the fly | | `WINCH` | **28** | gracefully shutdown the worker processes | There’s no need to control the worker processes yourself. However, they support some signals too: | SIGNAL | NUM | DESCRIPTION | | :--- | :---: | :--- | | `TERM`, `INT` | **15**, **2** | quick shutdown | | `QUIT` | **3** | graceful shutdown | | `USR1` | **10** | reopen the log files | ###### CPU pinning Moreover, it is important to mention about `worker_cpu_affinity` directive (it's only supported on GNU/Linux). CPU affinity is used to control which CPUs NGINX utilizes for individual worker processes. By default, worker processes are not bound to any specific CPUs. What's more, system might schedule all worker processes to run on the same CPU which may not be efficient enough. CPU affinity is represented as a bitmask (given in hexadecimal), with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. [Here](https://www.kutukupret.com/2010/11/18/nginx-worker_cpu_affinity/) you will find an amazing explanation of this. There is a [worker_cpu_affinity configuration generator](https://github.com/cubicdaiya/nwcagen) for NGINX. After all, I would recommend to let the OS scheduler to do the work because there is no reason to ever set it up during normal operation. ###### Shutdown of worker processes This should come in useful if you want to tweak NGINX’s shutdown process, particularly if other servers or load balancers are relying upon predictable restart times or if it takes a long time to close worker processes. The `worker_shutdown_timeout` directive configures a timeout to be used when gracefully shutting down worker processes. When the timer expires, NGINX will try to close all the connections currently open to facilitate shutdown. NGINX’s [Maxim Dounin](https://twitter.com/mdounin) explains: > _The `worker_shutdown_timeout` directive is not expected to delay shutdown if there are no active connections. It was introduced to limit possible time spent in shutdown, that is, to ensure fast enough shutdown even if there are active connections._ When a worker process enters the "exiting" state, it does a few things: - mark itself as an exiting process - set a shutdown timer, if `worker_shutdown_timeout` is defined - close listening sockets - close idle connections Then, if the shutdown timer was set, after the `worker_shutdown_timeout` interval, all connections are closed. > By default, NGINX to wait for and process additional data from a client before fully closing a connection, but only if heuristics suggests that a client may be sending more data. Sometimes, you can see `nginx: worker process is shutting down` in your log file. The problem occurs when reloading the configuration - where NGINX usually exits the existing worker processes gracefully, but at times, it takes hours to close these processes. Every config reload may dropping a zombie workers, permanently eating up all of your system's memory. In this case, fast shutdown of worker processes might be a solution. In addition, setting `worker_shutdown_timeout` also solve the issue: ```nginx worker_shutdown_timeout 60s; ``` Test connection timeouts and how long your request is processed by a server, next adjust the `worker_shutdown_timeout` value to these values. 60 seconds is a value with a solid supply and nothing valid should last longer than that. In my experience, if you have multiple workers in a shutting down state, maybe you should first look at the loaded modules that may cause problems with hanging worker processes. #### Connection processing NGINX supports a variety of [connection processing methods](https://nginx.org/en/docs/events.html) which depends on the platform used. In general there are four types of event multiplexing: - `select` - is anachronism and not recommended but installed on all platforms as a fallback - `poll` - is anachronism and not recommended And the most efficient implementations of non-blocking I/O: - `epoll` - recommend if you're using GNU/Linux - `kqueue` - recommend if you're using BSD (it is technically superior to `epoll`) The `select` method can be enabled or disabled using the `--with-select_module` or `--without-select_module` configuration parameter. Similarly, the `poll` method can be enabled or disabled using the `--with-poll_module` or `--without-poll_module` configuration parameter. > `epoll` is an efficient method of processing connections available on Linux 2.6+. `kqueue` is an efficient method of processing connections available on FreeBSD 4.1+, OpenBSD 2.9+, and NetBSD 2.0+. There is normally no need to specify it explicitly, because NGINX will by default use the most efficient method. But if you want to set this: ```nginx use epoll; ``` There are also great resources (also makes comparisons) about them: - [Kqueue: A generic and scalable event notification facility](https://people.freebsd.org/~jlemon/papers/kqueue.pdf) - [poll vs select vs event-based](https://daniel.haxx.se/docs/poll-vs-select.html) - [select/poll/epoll: practical difference for system architects](http://www.ulduzsoft.com/2014/01/select-poll-epoll-practical-difference-for-system-architects/) - [Scalable Event Multiplexing: epoll vs. kqueue](https://people.eecs.berkeley.edu/~sangjin/2012/12/21/epoll-vs-kqueue.html) - [Async IO on Linux: select, poll, and epoll](https://jvns.ca/blog/2017/06/03/async-io-on-linux--select--poll--and-epoll/) - [A brief history of select(2)](https://idea.popcount.org/2016-11-01-a-brief-history-of-select2/) - [Select is fundamentally broken](https://idea.popcount.org/2017-01-06-select-is-fundamentally-broken/) - [Epoll is fundamentally broken](https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/) - [I/O Multiplexing using epoll and kqueue System Calls](https://austingwalters.com/io-multiplexing/) - [Benchmarking BSD and Linux](http://bulk.fefe.de/scalability/) - [The C10K problem](http://www.kegel.com/c10k.html) Look also at libevent benchmark (read about [libevent – an event notification library](http://libevent.org/)):

libevent-benchmark

This infographic comes from [daemonforums - An interesting benchmark (kqueue vs. epoll)](http://daemonforums.org/showthread.php?t=2124). You may also view why big players uses NGINX on FreeBSD instead of on GNU/Linux: - [FreeBSD NGINX Performance](https://devinteske.com/wp/freebsd-nginx-performance/) - [Why did Netflix use NGINX and FreeBSD to build their own CDN?](https://www.youtube.com/watch?v=KP_bKvXkoC4) NGINX means connections as follows (the following status information is provided by `ngx_http_stub_status_module`): - **Active connections** - the current number of active (open) client connections including waiting connections and connections to backends - **accepts** - the total number of accepted client connections - **handled** - the total number of handled connections. Generally, the parameter value is the same as `accepts` unless some resource limits have been reached (for example, the `worker_connections` limit) - **requests** - the total number of client requests - **Reading** - the current number of connections where NGINX is reading the request header - **Writing** - the current number of connections where NGINX is writing the response back to the client (reads request body, processes request, or writes response to a client) - **Waiting** - the current number of idle client connections waiting for a request, i.e. connection still opened waiting for either a new request, or the keepalive expiration (actually it is Active - (Reading + Writing)) > Waiting connections those are keepalive connections. They are usually not a problem but if you want to reduce them set the lower value of the `keepalive_timeout` directive. Be sure to recommend to read [this](https://trac.nginx.org/nginx/ticket/1610#comment:1): > Writing connections counter increasing might indicate one of the following: > > - crashed or killed worker processes. This is unlikely in your case though, as this would also result in other values growing as well, notably `Waiting` > - a real socket leak somewhere. These usually results in sockets in `CLOSE_WAIT` state (in a waiting state for the FIN packet terminating the connection), try looking at `netstat` output without `grep -v CLOSE_WAIT` filter. Leaked sockets are reported by NGINX during graceful shutdown of a worker process (for example, after a configuration reload) - if there are any leaked sockets, NGINX will write `open socket ... left in connection ...` alerts to the error log > > To further investigate things, please do the following: > - upgrade to the latest mainline versions, without any 3rd party modules, and check if you are able to reproduce the issue > - try disabling HTTP/2 to see if it fixes the issue > - check if you are seeing `open socket ... left in connection ...` (socket leaks) alerts on configuration reload See also [Debugging socket leaks (from this handbook)](HELPERS.md#debugging-socket-leaks). ##### Event-Driven architecture > [Thread Pools in NGINX Boost Performance 9x!](https://www.nginx.com/blog/thread-pools-boost-performance-9x/) - this official article is an amazing explanation about thread pools and generally about handling connections. I also recommend [Inside NGINX: How We Designed for Performance & Scale](https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale). Both are really great. NGINX uses Event-Driven architecture which heavily relies on Non-Blocking I/O. One advantage of non-blocking/asynchronous operations is that you can maximize the usage of a single CPU as well as memory because is that your thread can continue it's work in parallel. The end result is that even as load increases, memory and CPU usage remain manageable. > There is a perfectly good and brief [summary](https://stackoverflow.com/questions/8546273/is-non-blocking-i-o-really-faster-than-multi-threaded-blocking-i-o-how) about non-blocking I/O and multi-threaded blocking I/O by [Werner Henze](https://stackoverflow.com/users/1023911/werner-henze). I also recommend [asynchronous vs non-blocking](https://stackoverflow.com/a/2625565) by [Daniel Earwicker](https://stackoverflow.com/users/27423/daniel-earwicker). Take a look at this simple drawing:

blocking_non-blocking

This infographic comes from [Kansas State Polytechnic website](http://faculty.salina.k-state.edu/tim/ossg/Device/blocking.html). Blocking I/O system calls (a) do not return until the I/O is complete. Nonblocking I/O system calls return immediately. The process is later notified when the I/O is complete. There are forms of I/O and examples of POSIX functions: | Blocking | Non-blocking | Asynchronous | | :--- | :--- | :--- | | `write`, `read` | `write`, `read` + `poll/select` | `aio_write`, `aio_read` | Look also what the official documentation says about it: > _It’s well known that NGINX uses an asynchronous, event‑driven approach to handling connections. This means that instead of creating another dedicated process or thread for each request (like servers with a traditional architecture), it handles multiple connections and requests in one worker process. To achieve this, NGINX works with sockets in a non‑blocking mode and uses efficient methods such as epoll and kqueue._ > _Because the number of full‑weight processes is small (usually only one per CPU core) and constant, much less memory is consumed and CPU cycles aren’t wasted on task switching. The advantages of such an approach are well‑known through the example of NGINX itself. It successfully handles millions of simultaneous requests and scales very well._ I must not forget to mention here about Non-Blocking and 3rd party modules (also from official documentation): > _Unfortunately, many third‑party modules use blocking calls, and users (and sometimes even the developers of the modules) aren’t aware of the drawbacks. Blocking operations can ruin NGINX performance and must be avoided at all costs._ To handle concurrent requests with a single worker process NGINX uses the [reactor design pattern](https://stackoverflow.com/questions/5566653/simple-explanation-for-the-reactor-pattern-with-its-applications). Basically, it's a single-threaded but it can fork several processes to utilize multiple cores. However, NGINX is not a single threaded application. Each of worker processes is single-threaded and can handle thousands of concurrent connections. Workers are used to get request parallelism across multiple cores. When a request blocks, that worker will work on another request. NGINX does not create a new process/thread for each connection/requests but it starts several worker threads during start. It does this asynchronously with one thread, rather than using multi-threaded programming (it uses an event loop with asynchronous I/O). That way, the I/O and network operations are not a very big bottleneck (remember that your CPU would spend a lot of time waiting for your network interfaces, for example). This results from the fact that NGINX only use one thread to service all requests. When requests arrive at the server, they are serviced one at a time. However, when the code serviced needs other thing to do it sends the callback to the other queue and the main thread will continue running (it doesn't wait). Now you see why NGINX can handle a large amount of requests perfectly well (and without any problems). For more information take a look at following resources: - [Asynchronous, Non-Blocking I/O](https://medium.com/@entzik/on-asynchronous-non-blocking-i-o-4a2ac0af5c50) - [Asynchronous programming. Blocking I/O and non-blocking I/O](https://luminousmen.com/post/asynchronous-programming-blocking-and-non-blocking) - [Blocking I/O and non-blocking I/O](https://medium.com/coderscorner/tale-of-client-server-and-socket-a6ef54a74763) - [Non-blocking I/O](https://www.hellsoft.se/non-blocking-io/) - [About High Concurrency, NGINX architecture and internals](http://www.aosabook.org/en/nginx.html) - [A little holiday present: 10,000 reqs/sec with Nginx!](https://blog.webfaction.com/2008/12/a-little-holiday-present-10000-reqssec-with-nginx-2/) - [Nginx vs Apache: Is it fast, if yes, why?](http://planetunknown.blogspot.com/2011/02/why-nginx-is-faster-than-apache.html) - [How is Nginx handling its requests in terms of tasks or threading?](https://softwareengineering.stackexchange.com/questions/256510/how-is-nginx-handling-its-requests-in-terms-of-tasks-or-threading) - [Why nginx is faster than Apache, and why you needn’t necessarily care](https://djangodeployment.com/2016/11/15/why-nginx-is-faster-than-apache-and-why-you-neednt-necessarily-care/) - [How we scaled nginx and saved the world 54 years every day](https://blog.cloudflare.com/how-we-scaled-nginx-and-saved-the-world-54-years-every-day/) Finally, look at these great preview:

NGINX_non-blocking

Both infographic comes from [Inside NGINX: How We Designed for Performance & Scale](https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/). ##### Multiple processes NGINX uses only asynchronous I/O, which makes blocking a non-issue. The only reason NGINX uses multiple processes is to make full use of multi-core, multi-CPU, and hyper-threading systems. NGINX requires only enough worker processes to get the full benefit of symmetric multiprocessing (SMP). From official documentation: > _The NGINX configuration recommended in most cases - running one worker process per CPU core - makes the most efficient use of hardware resources._ NGINX uses a custom event loop which was designed specifically for NGINX - all connections are processed in a highly efficient run-loop in a limited number of single-threaded processes called workers. Worker processes accept new requests from a shared listen socket and execute a loop. There’s no specialized distribution of connections to the workers in NGINX; this work is done by the OS kernel mechanisms which notifies a workers. > _Upon startup, an initial set of listening sockets is created. workers then continuously accept, read from and write to the sockets while processing HTTP requests and responses._ - from [The Architecture of Open Source Applications - NGINX](http://aosabook.org/en/nginx.html). Multiplexing works by using a loop to increment through a program chunk by chunk operating on one piece of data/new connection/whatever per connection/object per loop iteration. It is all based on events multiplexing like `epoll()` or `kqueue()`. Within each worker NGINX can handle many thousands of concurrent connections and requests per second. > See [Nginx Internals](https://www.slideshare.net/joshzhu/nginx-internals) presentation as a lot of great stuff about the internals of the NGINX. NGINX does not fork a process or thread per connection (like Apache) so memory usage is very conservative and extremely efficient in the vast majority of cases. NGINX is a faster and consumes less memory than Apache and performs very well under load. It is also very friendly for CPU because there's no ongoing create-destroy pattern for processes or threads. Finally and in summary: - uses Non-Blocking "Event-Driven" architecture - uses the single-threaded reactor pattern to handle concurrent requests - uses highly efficient loop for connection processing - is not a single threaded application because it starts multiple worker processes (to handle multiple connections and requests) during start ##### Simultaneous connections Okay, so how many simultaneous connections can be processed by NGINX? ``` worker_processes * worker_connections = max connections ``` According to this: if you are running **4** worker processes with **4,096** worker connections per worker, you will be able to serve **16,384** connections. Of course, these are the NGINX settings limited by the kernel (number of connections, number of open files, or number of processes). > At this point, I would like to mention about [Understanding socket and port in TCP](https://medium.com/fantageek/understanding-socket-and-port-in-tcp-2213dc2e9b0c). It is a great and short explanation. I also recommend to read [Theoretical maximum number of open TCP connections that a modern Linux box can have](https://stackoverflow.com/questions/2332741/what-is-the-theoretical-maximum-number-of-open-tcp-connections-that-a-modern-lin). I've seen some admins does directly translate the sum of `worker_processes` and `worker_connections` into the number of clients that can be served simultaneously. In my opinion, it is a mistake because certain of clients (e.g. browsers which have different values for this) **opens a number of parallel connections** (see [this](https://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser) to confirm my words). Clients typically establish 4-8 TCP connections so that they can download resources in parallel (to download various components that compose a web page, for example, images, scripts, and so on). This increases the effective bandwidth and reduces latency. > That is a HTTP/1.1 limit (6-8) of concurrent HTTP calls. The best solution to improve performance (without upgrade the hardware and use cache at the middle (e.g. CDN, Varnish)) is using HTTP/2 ([RFC 7540](https://tools.ietf.org/html/rfc7540) [IETF]) instead of HTTP/1.1. > > HTTP/2 multiplex many HTTP requests on a single connection. When HTTP/1.1 has a limit of 6-8 roughly, HTTP/2 does not have a standard limit but say: "_It is recommended that this value (`SETTINGS_MAX_CONCURRENT_STREAMS`) be no smaller than 100_" (RFC 7540). That number is better than 6-8. Additionally, you must know that the `worker_connections` directive **includes all connections** per worker (e.g. connection structures are used for listen sockets, internal control sockets between NGINX processes, connections with proxied servers, and for upstream connections), not only incoming connections from clients. > Be aware that every worker connection (in the sleeping state) needs 256 bytes of memory, so you can increase it easily. The number of connections is especially limited by the maximum number of open files (`RLIMIT_NOFILE`) on your system (you can read about file descriptors and file handlers on [this](https://stackoverflow.com/questions/2423628/whats-the-difference-between-a-file-descriptor-and-file-pointer) great explanation). The reason is that the operating system needs memory to manage each open file, and memory is a limited resource. This limitation only affects the limits for the current process. The limits of the current process are bequeathed to children processes too, but each process has a separate count. To change the limit of the maximum file descriptors (that can be opened by a single worker process) you can also edit the `worker_rlimit_nofile` directive. With this, NGINX provides very powerful dynamic configuration capabilities with no service restarts. > The number of file descriptors is not the only one limitation of the number of connections - remember also about the kernel network (TCP/IP stack) parameters and the maximum number of processes. I don't like this piece of the NGINX documentation. Maybe I'm missing something but it says the `worker_rlimit_nofile` is a limit on the maximum number of open files for worker processes. I believe it is associated to a single worker process. If you set `RLIMIT_NOFILE` to 25,000 and `worker_rlimit_nofile` to 12,000, NGINX sets (only for workers) the maximum open files limit as a `worker_rlimit_nofile`. But the master process will have a set value of `RLIMIT_NOFILE`. Default value of `worker_rlimit_nofile` directive is `none` so by default NGINX sets the initial value of maximum open files from the system limits. ```bash # On GNU/Linux (or /usr/lib/systemd/system/nginx.service): grep "LimitNOFILE" /lib/systemd/system/nginx.service LimitNOFILE=5000 grep "worker_rlimit_nofile" /etc/nginx/nginx.conf worker_rlimit_nofile 256; PID SOFT HARD 24430 5000 5000 24431 256 256 24432 256 256 24433 256 256 24434 256 256 # To check fds on FreeBSD: sysctl kern.maxfiles kern.maxfilesperproc kern.openfiles kern.maxfiles: 64305 kern.maxfilesperproc: 57870 kern.openfiles: 143 ``` This is also controlled by the OS because the worker is not the only process running on the server. It would be very bad if your workers used up all of the file descriptors available to all processes, don't set your limits so that is possible. In my opinion, relying on the `RLIMIT_NOFILE` (and alternatives on other systems) than `worker_rlimit_nofile` value is more understandable and predictable. To be honest, it doesn't really matter which method is used to set, but you should keep a constant eye on the priority of the limits. > If you don't set the `worker_rlimit_nofile` directive manually, then the OS settings will determine how many file descriptors can be used by NGINX. I think that the chance of running out of file descriptors is minimal, but it might be a big problem on a high traffic websites. Ok, so how many fds are opens by NGINX? - one file handler for the client's active connection - one file handler for the proxied connection (that will open a socket handling these requests to remote or local host/process) - one file handler for opening file (e.g. static file) - other file handlers for internal connections, shared libraries, log files, and sockets Also important is: > NGINX can use up to two file descriptors per full-fledged connection. Look also at these diagrams: - 1 file handler for connection with client and 1 file handler for static file being served by NGINX: ``` # 1 connection, 2 file handlers +-----------------+ +----------+ | | | | 1 | | | CLIENT <---------------> NGINX | | | | ^ | +----------+ | | | | 2 | | | | | | | | | +------v------+ | | | STATIC FILE | | | +-------------+ | +-----------------+ ``` - 1 file handler for connection with client and 1 file handler for a open socket to the remote or local host/process: ``` # 2 connections, 2 file handlers +-----------------+ +----------+ | | +-----------+ | | 1 | | 2 | | | CLIENT <---------------> NGINX <---------------> BACKEND | | | | | | | +----------+ | | +-----------+ +-----------------+ ``` - 2 file handlers for two simultaneous connections from the same client (1, 4), 1 file handler for connection with other client (3), 2 file handlers for static files (2, 5), and 1 file handler for a open socket to the remote or local host/process (6), so in total it is 6 file descriptors: ``` # 4 connections, 6 file handlers 4 +-----------------------+ | +--------|--------+ +-----v----+ | | | | | 1 | v | 6 | CLIENT <-----+---------> NGINX <---------------+ | | | | ^ | +-----v-----+ +----------+ | | | | | | 3 | | 2 | 5 | | BACKEND | +----------+ | | | | | | | | | | | | +-----------+ | CLIENT <----+ | +------v------+ | | | | | STATIC FILE | | +----------+ | +-------------+ | +-----------------+ ``` In the first two examples: we can take that NGINX needs 2 file handlers for full-fledged connection (but still uses 2 worker connections). In the third example NGINX can take still 2 file handlers for every full-fledged connection (also if client uses parallel connections). So, to conclude, I think that the correct value of `worker_rlimit_nofile` per all connections of worker should be greater than `worker_connections`. In my opinion, the safe value of `worker_rlimit_nofile` (and system limits) is: ``` # 1 file handler for 1 connection: worker_connections + (shared libs, log files, event pool, etc.) = worker_rlimit_nofile # 2 file handlers for 1 connection: (worker_connections * 2) + (shared libs, log files, event pool, etc.) = worker_rlimit_nofile ``` That is probably how many files can be opened by each worker and should have a value greater than to the number of connections per worker (according to the above formula). In the most articles and tutorials we can see that this parameter has a value similar to the maximum number (or even more) of all open files by the NGINX. If we assume that this parameter applies to each worker separately these values are altogether excessive. However, after a deeper reflection they are rational because they allow one worker to use all the file descriptors so that they are not confined to other workers if something happens to them. Remember though that we are still limited by the connections per worker. May I remind you that any connection opens at least one file. So, moving on, the maximum number of open files by the NGINX should be: ``` (worker_processes * worker_connections * 2) + (shared libs, log files, event pool, etc.) = max open files ``` > To serve **16,384** connections by all workers (4,096 connections for each worker), and bearing in mind about the other handlers used by NGINX, a reasonably value of max files handlers in this case may be **35,000**. I think it's more than enough. Given the above to change/improve the limitations you should: 1. Edit the maximum, total, global number of file descriptors the kernel will allocate before choking (this step is optional, I think you should change this only for a very very high traffic): ```bash # Find out the system-wide maximum number of file handles: sysctl fs.file-max # Shows the current number of all file descriptors in kernel memory: # first value: # second value: # third value: # fs.file-max sysctl fs.file-nr # Set it manually and temporarily: sysctl -w fs.file-max=150000 # Set it permanently: echo "fs.file-max = 150000" > /etc/sysctl.d/99-fs.conf # And load new values of kernel parameters: sysctl -p # for /etc/sysctl.conf sysctl --system # for /etc/sysctl.conf and all of the system configuration files ``` 2. Edit the system-wide value of the maximum file descriptor number that can be opened by a single process: - for non-systemd systems: ```bash # Set the maximum number of file descriptors for the users logged in via PAM: # /etc/security/limits.conf nginx soft nofile 35000 nginx hard nofile 35000 ``` - for systemd systems: ```bash # Set the maximum number (hard limit) of file descriptors for the services started via systemd: # /etc/systemd/system.conf - global config (default values for all units) # /etc/systemd/user.conf - this specifies further per-user restrictions # /lib/systemd/system/nginx.service - default unit for the NGINX service # /etc/systemd/system/nginx.service - for your own instance of the NGINX service [Service] # ... LimitNOFILE=35000 # Reload a unit file and restart the NGINX service: systemctl daemon-reload && systemct restart nginx ``` 3. Adjusts the system limit on number of open files for the NGINX worker. The maximum value can not be greater than `LimitNOFILE` (in this example: 35,000). You can change it at any time: ```bash # Set the limit for file descriptors for a single worker process (change it as needed): # nginx.conf within the main context worker_rlimit_nofile 10000; # You need to reload the NGINX service: nginx -s reload ``` To show the current hard and soft limits applying to the NGINX processes (with `nofile`, `LimitNOFILE`, or `worker_rlimit_nofile`): ```bash for _pid in $(pgrep -f "nginx: [master,worker]") ; do echo -en "$_pid " grep "Max open files" /proc/${_pid}/limits | awk '{print $4" "$5}' done | xargs printf '%6s %10s\t%s\n%6s %10s\t%s\n' "PID" "SOFT" "HARD" ``` or use the following: ```bash # To determine the OS limits imposed on a process, read the file /proc/$pid/limits. # $pid corresponds to the PID of the process: for _pid in $(pgrep -f "nginx: [master,worker]") ; do echo -en ">>> $_pid\\n" cat /proc/$_pid/limits done ``` To list the current open file descriptors for each NGINX process: ```bash for _pid in $(pgrep -f "nginx: [master,worker]") ; do _fds=$(find /proc/${_pid}/fd/*) _fds_num=$(echo "$_fds" | wc -l) echo -en "\n\n##### PID: $_pid ($_fds_num fds) #####\n\n" # List all files from the proc/{pid}/fd directory: echo -en "$_fds\n\n" # List all open files (log files, memory mapped files, libs): lsof -as -p $_pid | awk '{if(NR>1)print}' done ``` You should also remember about the following rules: - `worker_rlimit_nofile` serves to dynamically change the maximum file descriptors the NGINX worker processes can handle, which is typically defined with the system's soft limit (`ulimit -Sn`) - `worker_rlimit_nofile` works only at the process level, it's limited to the system's hard limit (`ulimit -Hn`) - if you have SELinux enabled, you will need to run `setsebool -P httpd_setrlimit 1` so that NGINX has permissions to set its rlimit. To diagnose SELinux denials and attempts you can use `sealert -a /var/log/audit/audit.log`, or `audit2why` and `audit2allow` tools To sum up this example: - each of the NGINX processes (master + workers) have the ability to create up to 35,000 files - for all workers, the maximum number of file descriptors is 140,000 (`LimitNOFILE` per worker) - for each worker, the initial/current number of file descriptors is 10,000 (`worker_rlimit_nofile`) ``` nginx: master process = LimitNOFILE (35,000) \_ nginx: worker process = LimitNOFILE (35,000), worker_rlimit_nofile (10,000) \_ nginx: worker process = LimitNOFILE (35,000), worker_rlimit_nofile (10,000) \_ nginx: worker process = LimitNOFILE (35,000), worker_rlimit_nofile (10,000) \_ nginx: worker process = LimitNOFILE (35,000), worker_rlimit_nofile (10,000) = master (35,000), all workers: - 140,000 by LimitNOFILE - 40,000 by worker_rlimit_nofile ``` Look also at this great article about [Optimizing Nginx for High Traffic Loads](https://blog.martinfjordvald.com/optimizing-nginx-for-high-traffic-loads/). ##### HTTP Keep-Alive connections > **:bookmark: [Activate the cache for connections to upstream servers - Performance - P2](RULES.md#beginner-activate-the-cache-for-connections-to-upstream-servers)** Before starting this section I recommend to read the following articles: - [HTTP Keepalive Connections and Web Performance](https://www.nginx.com/blog/http-keepalives-and-web-performance/) - [Optimizing HTTP: Keep-alive and Pipelining](https://www.igvita.com/2011/10/04/optimizing-http-keep-alive-and-pipelining/) - [Evolution of HTTP — HTTP/0.9, HTTP/1.0, HTTP/1.1, Keep-Alive, Upgrade, and HTTPS](https://medium.com/platform-engineer/evolution-of-http-69cfe6531ba0) The original model of HTTP, and the default one in HTTP/1.0, is short-lived connections. Each HTTP request is completed on its own connection; this means a TCP handshake happens before each HTTP request, and these are serialized. The client creates a new TCP connection for each transaction (and the connection is torn down after the transaction completes). HTTP Keep-Alive connection or persistent connection is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses (Keep Alive's work between requests), as opposed to opening a new connection for every single request/response pair. When using keep alive the browser does not have to make multiple connections (keep in mind that establishing connections is expensive) but uses the already established connection and controls how long that stays active/open. So, the keep alive is a way to reduce the overhead of creating the connection, as, most of the time, a user will navigate through the site etc. (plus the multiple requests from a single page, to download css, javascript, images etc.). It takes a 3-way handshake to establish a TCP connection, so, when there is a perceivable latency between the client and the server, keepalive would greatly speed things up by reusing existing connections. This mechanism hold open the TCP connection between the client and the server after an HTTP transaction has completed. It's important because NGINX needs to close connections from time to time, even if you configure NGINX to allow infinite keep alive timeouts and a huge amount of acceptable requests per connection, to return results and as well errors and success messages.

closed_vs_keepalive

Persistent connection model keeps connections opened between successive requests, reducing the time needed to open new connections. The HTTP pipelining model goes one step further, by sending several successive requests without even waiting for an answer, reducing much of the latency in the network.

http_connections

This infographic comes from [Mozilla MDN - Connection management in HTTP/1.x](https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection_management_in_HTTP_1.x). However, at present, browsers are not using pipelined HTTP requests. For more information please see [Why is pipelining disabled in modern browsers?](https://stackoverflow.com/questions/30477476/why-is-pipelining-disabled-in-modern-browsers). Look also at this example that shows how a Keep-Alive header could be used: ``` Client Proxy Server | | | +- Keep-Alive: timeout=600 -->| | | Connection: Keep-Alive | | | +- Keep-Alive: timeout=1200 -->| | | Connection: Keep-Alive | | | | | |<-- Keep-Alive: timeout=300 --+ | | Connection: Keep-Alive | |<- Keep-Alive: timeout=5000 -+ | | Connection: Keep-Alive | | | | | ``` NGINX official documentation says: > _All connections are independently negotiated. The client indicates a timeout of 600 seconds (10 minutes), but the proxy is only prepared to retain the connection for at least 120 seconds (2 minutes). On the link between proxy and server, the proxy requests a timeout of 1200 seconds and the server reduces this to 300 seconds. As this example shows, the timeout policies maintained by the proxy are different for each connection. Each connection hop is independent._ Keepalive connections reduce overhead, especially when SSL/TLS is in use but they also have drawbacks; even when idling they consume server resources, and under heavy load, DoS attacks can be conducted. In such cases, using non-persistent connections, which are closed as soon as they are idle, can provide better performance. So, Keep-Alives will improve SSL/TLS performance by quite a big deal if clients are doing multiple requests but if you don't have the resources to handle them then they kill your servers. > NGINX closes keepalive connections when the `worker_connections` limit is reached (connections are kept in the cache till the origin server closes them). To better understand how Keep-Alive works, please see amazing [explanation](https://stackoverflow.com/a/38190172) by [Barry Pollard](https://stackoverflow.com/users/2144578/barry-pollard). NGINX provides the two layers to enable Keep-Alive: ###### Client layer - the maximum number of keepalive requests a client can make over a given connection, which means a client can make e.g. 256 successfull requests inside one keepalive connection: ```nginx # Default: 100 keepalive_requests 256; ``` - server will close connection after this time. A higher number may be required when there is a larger amount of traffic to ensure there is no frequent TCP connection re-initiated. If you set it lower, you are not utilizing keep-alives on most of your requests slowing down client: ```nginx # Default: 75s keepalive_timeout 10s; # Or tell the browser when it should close the connection by adding an optional second timeout # in the header sent to the browser (some browsers do not care about the header): keepalive_timeout 10s 25s; ``` > Increase this to allow the keepalive connection to stay open longer, resulting in faster subsequent requests. However, setting this too high will result in the waste of resources (mainly memory) as the connection will remain open even if there is no traffic, potentially: significantly affecting performance. I think this should be as close to your average response time as possible. You could also decrease little by little the timeout (75s -> 50s, then later 25s...) and see how the server behaves. ###### Upstream layer - the number of idle keepalive connections that remain open for each worker process. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process (when this number is exceeded, the least recently used connections are closed): ```nginx # Default: disable keepalive 32; ``` NGINX, by default, only talks on HTTP/1.0 to the upstream servers. To keep TCP connection alive both upstream section and origin server should be configured to not finalise the connection. > Please keep in mind that keepalive is a feature of HTTP 1.1, NGINX uses HTTP 1.0 per default for upstreams. Connection won't be reused by default because keepalive in the upstream section means no keepalive (each time you can see TCP stream number increases per every request to origin server). HTTP keepalive enabled in NGINX upstream servers reduces latency thus improves performance and it reduces the possibility that the NGINX runs out of ephemeral ports. > The connections parameter should be set to a number small enough to let upstream servers process new incoming connections as well. Update your upstream configuration to use keepalive: ```nginx upstream bk_x8080 { ... # Sets the maximum number of idle keepalive connections to upstream servers # that are preserved in the cache of each worker process. keepalive 16; } ``` And enable the HTTP/1.1 protocol in all upstream requests: ```nginx server { ... location / { # Default is HTTP/1, keepalive is only enabled in HTTP/1.1: proxy_http_version 1.1; # Remove the Connection header if the client sends it, # it could be "close" to close a keepalive connection: proxy_set_header Connection ""; proxy_pass http://bk_x8080; } } ... } ``` There are two basic cases when keeping connections alive is really beneficial: - fast backends, which produce responses is a very short time, comparable to a TCP handshake - distant backends, when a TCP handshake takes a long time, comparable to a backend response time Look at the test: - without keepalive for upstream: ```bash wrk -c 500 -t 6 -d 60s -R 15000 -H "Host: example.com" https://example.com/ Running 1m test @ https://example.com/ 6 threads and 500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 24.13s 10.68s 49.55s 59.06% Req/Sec 679.21 42.44 786.00 78.95% 228421 requests in 1.00m, 77.98MB read Socket errors: connect 0, read 0, write 0, timeout 1152 Non-2xx or 3xx responses: 4 Requests/sec: 3806.96 Transfer/sec: 1.30MB ``` - with keepalive for upstream: ```bash wrk -c 500 -t 6 -d 60s -R 15000 -H "Host: example.com" https://example.com/ Running 1m test @ https://example.com/ 6 threads and 500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 23.40s 9.53s 47.25s 60.67% Req/Sec 0.86k 50.19 0.94k 60.00% 294148 requests in 1.00m, 100.41MB read Socket errors: connect 0, read 0, write 0, timeout 380 Requests/sec: 4902.24 Transfer/sec: 1.67MB ``` ##### `sendfile`, `tcp_nodelay`, and `tcp_nopush` Before you start reading please review: - [Nginx optimization, understanding SENDFILE, TCP_NODELAY and TCP_NOPUSH](https://thoughts.t37.net/nginx-optimization-understanding-sendfile-tcp-nodelay-and-tcp-nopush-c55cdd276765) - [Nginx Tutorial #2: Performance](https://www.netguru.com/codestories/nginx-tutorial-performance) As you're making these changes, keep careful watch on your network traffic and see how each tweak impacts congestion. ###### `sendfile` > _By default, NGINX handles file transmission itself and copies the file into the buffer before sending it. Enabling the `sendfile` directive eliminates the step of copying the data into the buffer and enables direct copying data from one file descriptor to another._ Normally, when a file needs to be sent, the following steps are required: - `malloc` - allocate a local buffer for storing object data - `read` - retrieve and copy the object into the local buffer - `write` - copy the object from the local buffer into the socket buffer Look at this great explanation (from [Nginx Tutorial #2: Performance](https://www.netguru.com/codestories/nginx-tutorial-performance)): > _This involves two context switches (read, write) which make a second copy of the same object unnecessary. As you may see, it is not the optimal way. Thankfully, there is another system call that improves sending files, and it's called (surprise, surprise!): `sendfile(2)`. This call retrieves an object to the file cache, and passes the pointers (without copying the whole object) straight to the socket descriptor. Netflix states that using `sendfile(2)` increased the network throughput from [6Gbps to 30Gbps](https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf)._ When a file is transferred by a process, the kernel first buffers the data and then sends the data to the process buffers. The process, in turn, sends the data to the destination. NGINX employs a solution that uses the `sendfile` system call to perform a zero-copy data flow from disk to socket and saves context switching from userspace on read/write. `sendfile` tell how NGINX buffers or reads the file (trying to stuff the contents directly into the network slot, or buffer its contents first). This method is an improved method of data transfer, in which data is copied between file descriptors within the OS kernel space, that is, without transferring data to the application buffers. No additional buffers or data copies are required, and the data never leaves the kernel memory address space. In my opinion enabling this really won't make any difference unless NGINX is reading from something which can be mapped into the virtual memory space like a file (i.e. the data is in the cache). But please... do not let me influence you - you should in the first place be keeping an eye on this document: [Optimizing TLS for High–Bandwidth Applications in FreeBSD](https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf) [pdf]. By default NGINX disable the use of `sendfile`: ```nginx # http, server, location, if in location contexts # To turn on sendfile (my recommendation): sendfile on; # To turn off sendfile: sendfile off; # default ``` Look also at `sendfile_max_chunk` directive. NGINX documentation say: > _When set to a non-zero value, limits the amount of data that can be transferred in a single `sendfile()` call. Without the limit, one fast connection may seize the worker process entirely._ On fast local connection `sendfile()` in Linux may send tens of megabytes per one syscall blocking other connections. `sendfile_max_chunk` allows to limit the maximum size per one `sendfile()` operation. So, with this NGINX can reduce the maximum time spent in blocking `sendfile()` calls, since NGINX won’t try to send the whole file at once, but will do it in chunks. For example: ```nginx sendfile on; sendfile_max_chunk 512k; ``` ###### `tcp_nodelay` I recommend to read [The Caveats of TCP_NODELAY](https://eklitzke.org/the-caveats-of-tcp-nodelay) and [Rethinking the TCP Nagle Algorithm](http://ccr.sigcomm.org/archive/2001/jan01/ccr-200101-mogul.pdf) [pdf]. These great papers describes very interesting topics about `TCP_NODELAY` and `TCP_NOPUSH`. `tcp_nodelay` is used to manage Nagle's algorithm which is one mechanism for improving TCP efficiency by reducing the number of small packets sent over the network. If you set `tcp_nodelay on;`, NGINX adds the `TCP_NODELAY` options when opening a new socket. > The option only affects keep-alive connections. Otherwise there is 100ms delay when NGINX sends response tail in the last incomplete TCP packet. Additionally, it is enabled on SSL connections, for unbuffered proxying, and for WebSocket proxying. Maybe you should think about enabling Nagle's algorithm (`tcp_nodelay off;`) but it really depends on what is your specific workload and dominant traffic patterns on a service. `tcp_nodelay on;` is more reasonable for the modern web, the whole delay business of TCP was reasonable for terminals. Typically LANs have less issues with traffic congestion as compared to the WANs. The Nagle algorithm is most effective if TCP/IP traffic is generated sporadically by user input, not by applications using stream oriented protocols like a HTTP traffic. So, for me, the recipe is simple: - bulk sends or HTTP traffic - applications that require lower latency - non-interactive type of traffic There is no need for using Nagle's algorithm. You should also know [the Nagle’s algorithm author's interesting comment](https://news.ycombinator.com/item?id=9045125): > _If you're doing bulk file transfers, you never hit that problem. If you're sending enough data to fill up outgoing buffers, there's no delay. If you send all the data and close the TCP connection, there's no delay after the last packet. If you do send, reply, send, reply, there's no delay. If you do bulk sends, there's no delay. If you do send, send, reply, there's a delay._ > _The real problem is ACK delays. The 200ms "ACK delay" timer is a bad idea that someone at Berkeley stuck into BSD around 1985 because they didn't really understand the problem. A delayed ACK is a bet that there will be a reply from the application level within 200ms. TCP continues to use delayed ACKs even if it's losing that bet every time._ I think if you are dealing with non-interactive type of traffic or bulk transfers such as HTTP/web traffic then enabling `TCP_NODELAY` to disable Nagle's algorithm may be useful (is the default behavior of the NGINX). This is especially relevant if you're running applications or environments that only sometimes have highly interactive traffic and chatty protocols. By default NGINX enable the use of `TCP_NODELAY` option: ```nginx # http, server, location contexts # To turn on tcp_nodelay and at the same time to disable Nagle’s algorithm # (my recommendation, unless you turn tcp_nopush on): tcp_nodelay on; # default # To turn off tcp_nodelay and at the same time to enable Nagle’s algorithm: tcp_nodelay off; ``` ###### `tcp_nopush` This option is only available if you are using `sendfile` (NGINX uses `tcp_nopush` for requests served with `sendfile`). It causes NGINX to attempt to send its HTTP response head in one packet, instead of using partial frames. This is useful for prepending headers before calling `sendfile`, or for throughput optimization. > Normally, using `tcp_nopush` along with `sendfile` is very good. However, there are some cases where it can slow down things (specially from cache systems), so, run your own tests and find if it’s useful in that way. `tcp_nopush` enables `TCP_CORK` (more specifically, the `TCP_NOPUSH` socket option on FreeBSD or the `TCP_CORK` socket option on Linux) which aggressively accumulates data and which tells TCP to wait for the application to remove the cork before sending any packets. If `TCP_NOPUSH/TCP_CORK` (are not the same!) is enabled in a socket, it will not send data until the buffer fills to a fixed limit (allows application to control building of packet, e.g pack a packet with full HTTP response). To read more about it and get into the details of this option please read [TCP_CORK: More than you ever wanted to know](https://baus.net/on-tcp_cork/). Once, I read that `tcp_nopush` is opposite to `tcp_nodelay`. I don't agree with that because, as I understand it, the first one aggregates data based on buffer pressure instead whereas Nagle's algorithm aggregates data while waiting for a return ACK, which the latter option disables. It may appear that `tcp_nopush` and `tcp_nodelay` are mutually exclusive but if all directives are turned on, NGINX manages them very wisely: - ensure packages are full before sending them to the client - for the last packet, `tcp_nopush` will be removed, allowing TCP to send it immediately, without the 200ms delay And let's also remember (take a look at [Tony Finch notes](http://dotat.at/writing/nopush.html) - this guy developed a kernel patch for FreeBSD which makes `TCP_NOPUSH` work like `TCP_CORK`): - on Linux, `sendfile()` depends on the `TCP_CORK` socket option to avoid undesirable packet boundaries - FreeBSD has a similar option called `TCP_NOPUSH` - when `TCP_CORK` is turned off any buffered data is sent immediately, but this is not the case for `TCP_NOPUSH` By default NGINX disable the use of `TCP_NOPUSH` option: ```nginx # http, server, location contexts # To turn on tcp_nopush (my recommendation): tcp_nopush on; # To turn off tcp_nopush: tcp_nopush off; # default ``` ###### Mixing all together There are many opinions on this. My recommendation is to set all to `on`. However, I quote an interesting comment ([Mixing sendfile, tcp_nodelay and tcp_nopush illogical?](https://github.com/denji/nginx-tuning/issues/5)) that should dispel any doubts: > _When set indicates to always queue non-full frames. Later the user clears this option and we transmit any pending partial frames in the queue. This is meant to be used alongside `sendfile()` to get properly filled frames when the user (for example) must write out headers with a `write()` call first and then use `sendfile` to send out the data parts. `TCP_CORK` can be set together with `TCP_NODELAY` and it is stronger than `TCP_NODELAY`._ Summarizing: - `tcp_nodelay on;` is generaly at the odds with `tcp_nopush on;` as they are mutually exclusive - NGINX has special behavior that if you have `sendfile on;`, it uses `TCP_NOPUSH` for everything but the last package - and then turns `TCP_NOPUSH` off and enables `TCP_NODELAY` to avoid 200ms ACK delay So in fact, the most important changes are listed below: ```nginx sendfile on; tcp_nopush on; # with this, the tcp_nodelay does not really matter ``` #### Request processing stages > When building filtering rules (e.g. with `allow/deny`) you should always remember to test them and to know what happens at each of the phases (which modules are used). For additional information about the potential problems, look at [allow and deny](#allow-and-deny) section and [Take care about your ACL rules - Hardening - P1](RULES.md#beginner-take-care-about-your-acl-rules). There can be altogether 11 phases when NGINX handles (processes) a request: - `NGX_HTTP_POST_READ_PHASE` - first phase, read the request header - example modules: [ngx_http_realip_module](https://nginx.org/en/docs/http/ngx_http_realip_module.html) - `NGX_HTTP_SERVER_REWRITE_PHASE` - implementation of rewrite directives defined in a server block; to change request URI using PCRE regular expressions, return redirects, and conditionally select configurations - example modules: [ngx_http_rewrite_module](http://nginx.org/en/docs/http/ngx_http_rewrite_module.html) - `NGX_HTTP_FIND_CONFIG_PHASE` - replace the location according to URI (location lookup) - `NGX_HTTP_REWRITE_PHASE` - URI transformation on location level - example modules: [ngx_http_rewrite_module](http://nginx.org/en/docs/http/ngx_http_rewrite_module.html) - `NGX_HTTP_POST_REWRITE_PHASE` - URI transformation post-processing (the request is redirected to a new location) - example modules: [ngx_http_rewrite_module](http://nginx.org/en/docs/http/ngx_http_rewrite_module.html) - `NGX_HTTP_PREACCESS_PHASE` - authentication preprocessing request limit, connection limit (access restriction) - example modules: [ngx_http_limit_req_module](http://nginx.org/en/docs/http/ngx_http_limit_req_module.html), [ngx_http_limit_conn_module](http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html), [ngx_http_realip_module](https://nginx.org/en/docs/http/ngx_http_realip_module.html) - `NGX_HTTP_ACCESS_PHASE` - verification of the client (the authentication process, limiting access) - example modules: [ngx_http_access_module](https://nginx.org/en/docs/http/ngx_http_access_module.html), [ngx_http_auth_basic_module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) - `NGX_HTTP_POST_ACCESS_PHASE` - access restrictions check post-processing phase, the certification process, processing `satisfy any` directive - example modules: [ngx_http_access_module](https://nginx.org/en/docs/http/ngx_http_access_module.html), [ngx_http_auth_basic_module](https://nginx.org/en/docs/http/ngx_http_auth_basic_module.html) - `NGX_HTTP_PRECONTENT_PHASE` - generating content - example modules: [ngx_http_try_files_module](https://nginx.org/en/docs/http/ngx_http_core_module.html#try_files) - `NGX_HTTP_CONTENT_PHASE` - content processing - example modules: [ngx_http_index_module](https://nginx.org/en/docs/http/ngx_http_index_module.html), [ngx_http_autoindex_module](https://nginx.org/en/docs/http/ngx_http_autoindex_module.html), [ngx_http_gzip_module](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) - `NGX_HTTP_LOG_PHASE` - log processing - example modules: [ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html) You may feel lost now (me too...) so I let myself put this great and simple preview:

request-flow

This infographic comes from [Inside NGINX](https://www.nginx.com/resources/library/infographic-inside-nginx/) official library. On every phase you can register any number of your handlers. Each phase has a list of handlers associated with it. I recommend to read a great explanation about [HTTP request processing phases in Nginx](http://scm.zoomquiet.top/data/20120312173425/index.html) and, of course, official [Development guide](http://nginx.org/en/docs/dev/development_guide.html). I have also prepared a simple diagram that can help you understand what modules are used in each phase. It also contains short descriptions from official development guide:

nginx_phases

#### Server blocks logic > NGINX does have **server blocks** (like a virtual hosts in an Apache) that use `listen` directive to bind to TCP sockets and `server_name` directive to identify virtual hosts. It's a short example of two server block contexts with several regular expressions: ```nginx http { index index.html; root /var/www/example.com/default; server { listen 10.10.250.10:80; server_name www.example.com; access_log logs/example.access.log main; root /var/www/example.com/public; location ~ ^/(static|media)/ { ... } location ~* /[0-9][0-9](-.*)(\.html)$ { ... } location ~* \.(jpe?g|png|gif|ico)$ { ... } location ~* (?.*app)/(?.+\.php)$ { ... } ... } server { listen 10.10.250.11:80; server_name "~^(api.)?example\.com api.de.example.com"; access_log logs/example.access.log main; location ~ ^(/[^/]+)/api(.*)$ { ... } location ~ ^/backend/id/([a-z]\.[a-z]*) { ... } ... } } ``` ##### Handle incoming connections > **:bookmark: [Define the listen directives with address:port pair - Base Rules - P1](RULES.md#beginner-define-the-listen-directives-with-addressport-pair)**
> **:bookmark: [Prevent processing requests with undefined server names - Base Rules - P1](RULES.md#beginner-prevent-processing-requests-with-undefined-server-names)**
> **:bookmark: [Never use a hostname in a listen or upstream directives - Base Rules - P1](RULES.md#beginner-never-use-a-hostname-in-a-listen-or-upstream-directives)**
> **:bookmark: [Use exact names in a server_name directive if possible - Performance - P2](RULES.md#beginner-use-exact-names-in-a-server_name-directive-if-possible)**
> **:bookmark: [Separate listen directives for 80 and 443 ports - Base Rules - P3](RULES.md#beginner-separate-listen-directives-for-80-and-443-ports)**
> **:bookmark: [Use only one SSL config for the listen directive - Base Rules - P3](RULES.md#beginner-use-only-one-ssl-config-for-the-listen-directive)** NGINX uses the following logic to determining which virtual server (server block) should be used: 1. Match the `address:port` pair to the `listen` directive - that can be multiple server blocks with `listen` directives of the same specificity that can handle the request > NGINX use the `address:port` combination for handle incoming connections. This pair is assigned to the `listen` directive. The `listen` directive can be set to: - an IP address/port combination (`127.0.0.1:80;`) - a lone IP address, if only address is given, the port `80` is used (`127.0.0.1;`) - becomes `127.0.0.1:80;` - a lone port which will listen to every interface on that port (`80;` or `*:80;`) - becomes `0.0.0.0:80;` - the path to a UNIX domain socket (`unix:/var/run/nginx.sock;`) If the `listen` directive is not present then either `*:80` is used (runs with the superuser privileges), or `*:8000` otherwise. To play with `listen` directive NGINX must follow the following steps: - NGINX translates all incomplete `listen` directives by substituting missing values with their default values (see above) - NGINX attempts to collect a list of the server blocks that match the request most specifically based on the `address:port` - If any block that is functionally using `0.0.0.0`, will not be selected if there are matching blocks that list a specific IP - If there is only one most specific match, that server block will be used to serve the request - If there are multiple server blocks with the same level of matching, NGINX then begins to evaluate the `server_name` directive of each server block Look at this short example: ```nginx # From client side: GET / HTTP/1.0 Host: api.random.com # From server side: server { # This block will be processed: listen 192.168.252.10; # --> 192.168.252.10:80 ... } server { listen 80; # --> *:80 --> 0.0.0.0:80 server_name api.random.com; ... } ``` 2. Match the `Host` header field against the `server_name` directive as a string (the exact names hash table) 3. Match the `Host` header field against the `server_name` directive with a wildcard at the beginning of the string (the hash table with wildcard names starting with an asterisk) > If one is found, that block will be used to serve the request. If multiple matches are found, the longest match will be used to serve the request. 4. Match the `Host` header field against the `server_name` directive with a wildcard at the end of the string (the hash table with wildcard names ending with an asterisk) > If one is found, that block is used to serve the request. If multiple matches are found, the longest match will be used to serve the request. 5. Match the `Host` header field against the `server_name` directive as a regular expression > The first `server_name` with a regular expression that matches the `Host` header will be used to serve the request. 6. If all the `Host` headers doesn't match, then direct to the `listen` directive marked as `default_server` (makes the server block answer all the requests that doesn’t match any server block) 7. If all the `Host` headers doesn't match and there is no `default_server`, direct to the first server with a `listen` directive that satisfies first step 8. Finally, NGINX goes to the `location` context This list is based on [Mastering Nginx - The virtual server section](https://github.com/trimstray/nginx-admins-handbook#mastering-nginx). ##### Matching location > **:bookmark: [Make an exact location match to speed up the selection process - Performance - P3](RULES.md#beginner-make-an-exact-location-match-to-speed-up-the-selection-process)** > For each request, NGINX goes through a process to choose the best location block that will be used to serve that request. The location block enables you to handle several types of URIs/routes (Layer 7 routing based on URL), within a server block. Syntax looks like: ``` location optional_modifier location_match { ... } ``` `location_match` in the above defines what NGINX should check the request URI against. The `optional_modifier` below will cause the associated location block to be interpreted as follows (the order doesn't matter at this moment): - `(none)`: if no modifiers are present, the location is interpreted as a prefix match. To determine a match, the location will now be matched against the beginning of the URI - `=`: is an exact match, without any wildcards, prefix matching or regular expressions; forces a literal match between the request URI and the location parameter - `~`: if a tilde modifier is present, this location must be used for case sensitive matching (RE match) - `~*`: if a tilde and asterisk modifier is used, the location must be used for case insensitive matching (RE match) - `^~`: assuming this block is the best non-RE match, a carat followed by a tilde modifier means that RE matching will not take place And now, a short introduction to determines location priority: - the exact match is the best priority (processed first); ends search if match - the prefix match is the second priority; there are two types of prefixes: `^~` and `(none)`, if this match used the `^~` prefix, searching stops - the regular expression match has the lowest priority; there are two types of prefixes: `~` and `~*`; in the order they are defined in the configuration file - if regular expression searching yielded a match, that result is used, otherwise, the match from prefix searching is used So, look at this example, it comes from the [Nginx documentation - ngx_http_core_module](https://nginx.org/en/docs/http/ngx_http_core_module.html#location): ``` location = / { # Matches the query / only. [ configuration A ] } location / { # Matches any query, since all queries begin with /, but regular # expressions and any longer conventional blocks will be # matched first. [ configuration B ] } location /documents/ { # Matches any query beginning with /documents/ and continues searching, # so regular expressions will be checked. This will be matched only if # regular expressions don't find a match. [ configuration C ] } location ^~ /images/ { # Matches any query beginning with /images/ and halts searching, # so regular expressions will not be checked. [ configuration D ] } location ~* \.(gif|jpg|jpeg)$ { # Matches any request ending in gif, jpg, or jpeg. However, all # requests to the /images/ directory will be handled by # Configuration D. [ configuration E ] } ``` To help you understand how does location match works: - [Nginx location match tester](https://nginx.viraptor.info/) - [Nginx location match visible](https://detailyang.github.io/nginx-location-match-visible/) - [NGINX Regular Expression Tester](https://github.com/nginxinc/NGINX-Demos/tree/master/nginx-regex-tester) The process of choosing NGINX location block is as follows (a detailed explanation): 1. NGINX searches for an exact match. If a `=` modifier (e.g. `location = foo { ... }`) exactly matches the request URI, this specific location block is chosen right away - this block is processed - match-searching stops 2. Prefix-based NGINX location matches (no regular expression). Each location will be checked against the request URI. If no exact (meaning no `=` modifier) location block is found, NGINX will continue with non-exact prefixes. It starts with the longest matching prefix location for this URI, with the following approach: - In case the longest matching prefix location has the `^~` modifier (e.g. `location ^~ foo { ... }`), NGINX will stop its search right away and choose this location - the block of the longest (most explicit) of those matches is processed - match-searching stops - Assuming the longest matching prefix location doesn’t use the `^~` modifier, the match is temporarily stored and the process continues > I'm not sure about the order. In the official documentation it is not clearly indicated and external guides explain it differently. It seems logical to check the longest matching prefix location first. 3. As soon as the longest matching prefix location is chosen and stored, NGINX continues to evaluate the case-sensitive (e.g. `location ~ foo { ... }`) and insensitive regular expression (e.g. `location ~* foo { ... }`) locations. The first regular expression location that fits the URI is selected right away to process the request - the block of the first matching regex found (when parsing the config-file top-to-bottom) is processed - match-searching stops 4. If no regular expression locations are found that match the request URI, the previously stored prefix location (e.g. `location foo { ... }`) is selected to serve the request - `location /` kind of a catch all location - the block of the longest (most explicit) of those matches is processed - match-searching stops You should also know, that the non-regex match-types are fully declarative - order of definition in the config doesn't matter - but the winning regex-match (if processing even gets that far) is entirely based on its order of entry in the config file. In order, to better understand how this process work, please see this short cheatsheet that will allow you to design your location blocks in a predictable way:

nginx-location-cheatsheet

> I recommend to use external tools for testing regular expressions. For more please see [online tools](https://github.com/trimstray/nginx-admins-handbook#online-tools) chapter. Ok, so here's a more complicated configuration: ```nginx server { listen 80; server_name xyz.com www.xyz.com; location ~ ^/(media|static)/ { root /var/www/xyz.com/static; expires 10d; } location ~* ^/(media2|static2) { root /var/www/xyz.com/static2; expires 20d; } location /static3 { root /var/www/xyz.com/static3; } location ^~ /static4 { root /var/www/xyz.com/static4; } location = /api { proxy_pass http://127.0.0.1:8080; } location / { proxy_pass http://127.0.0.1:8080; } location /backend { proxy_pass http://127.0.0.1:8080; } location ~ logo.xcf$ { root /var/www/logo; expires 48h; } location ~* .(png|ico|gif|xcf)$ { root /var/www/img; expires 24h; } location ~ logo.ico$ { root /var/www/logo; expires 96h; } location ~ logo.jpg$ { root /var/www/logo; expires 48h; } } ``` And look the table with the results: | URL | LOCATIONS FOUND | FINAL MATCH | | :--- | :--- | :--- | | `/` | 1) prefix match for `/` | `/` | | `/css` | 1) prefix match for `/` | `/` | | `/api` | 1) exact match for `/api` | `/api` | | `/api/` | 1) prefix match for `/` | `/` | | `/backend` | 1) prefix match for `/`
2) prefix match for `/backend` | `/backend` | | `/static` | 1) prefix match for `/` | `/` | | `/static/header.png` | 1) prefix match for `/`
2) case sensitive regex match for `^/(media\|static)/` | `^/(media\|static)/` | | `/static/logo.jpg` | 1) prefix match for `/`
2) case sensitive regex match for `^/(media\|static)/` | `^/(media\|static)/` | | `/media2` | 1) prefix match for `/`
2) case insensitive regex match for `^/(media2\|static2)` | `^/(media2\|static2)` | | `/media2/` | 1) prefix match for `/`
2) case insensitive regex match for `^/(media2\|static2)` | `^/(media2\|static2)` | | `/static2/logo.jpg` | 1) prefix match for `/`
2) case insensitive regex match for `^/(media2\|static2)` | `^/(media2\|static2)` | | `/static2/logo.png` | 1) prefix match for `/`
2) case insensitive regex match for `^/(media2\|static2)` | `^/(media2\|static2)` | | `/static3/logo.jpg` | 1) prefix match for `/static3`
2) prefix match for `/`
3) case sensitive regex match for `logo.jpg$` | `logo.jpg$` | | `/static3/logo.png` | 1) prefix match for `/static3`
2) prefix match for `/`
3) case insensitive regex match for `.(png\|ico\|gif\|xcf)$` | `.(png\|ico\|gif\|xcf)$` | | `/static4/logo.jpg` | 1) priority prefix match for `/static4`
2) prefix match for `/` | `/static4` | | `/static4/logo.png` | 1) priority prefix match for `/static4`
2) prefix match for `/` | `/static4` | | `/static5/logo.jpg` | 1) prefix match for `/`
2) case sensitive regex match for `logo.jpg$` | `logo.jpg$` | | `/static5/logo.png` | 1) prefix match for `/`
2) case insensitive regex match for `.(png\|ico\|gif\|xcf)$` | `.(png\|ico\|gif\|xcf)$` | | `/static5/logo.xcf` | 1) prefix match for `/`
2) case sensitive regex match for `logo.xcf$` | `logo.xcf$` | | `/static5/logo.ico` | 1) prefix match for `/`
2) case insensitive regex match for `.(png\|ico\|gif\|xcf)$` | `.(png\|ico\|gif\|xcf)$` | ##### `rewrite` vs `return` Generally there are two ways of implementing redirects in NGINX: with `rewrite` and `return` directives. These directives (comes from the `ngx_http_rewrite_module`) are very useful but (from the NGINX documentation) the only 100% safe things which may be done inside if in a `location` context are: - `return ...;` - `rewrite ... last;` Anything else may possibly cause unpredictable behaviour, including potential `SIGSEGV`. ###### `rewrite` directive The `rewrite` directives are executed sequentially in order of their appearance in the configuration file. It's slower (but still extremely fast) than a `return` and returns HTTP 302 in all cases, irrespective of `permanent`. The `rewrite` directive just changes the request URI, not the response of request. Importantly only the part of the original url that matches the regex is rewritten. It can be used for temporary url changes. I sometimes used `rewrite` to capture elementes in the original URL, change or add elements in the path, and in general when I do something more complex: ```nginx location / { ... rewrite ^/users/(.*)$ /user.php?username=$1 last; # or: rewrite ^/users/(.*)/items$ /user.php?username=$1&page=items last; } ``` > You must know that rewrite returns only code 301 or 302. `rewrite` directive accept optional flags: - `break` - basically completes processing of rewrite directives, stops processing, and breakes location lookup cycle by not doing any location lookup and internal jump at all - if you use `break` flag inside `location` block: - no more parsing of rewrite conditions - internal engine continues to parse the current `location` block _Inside a location block, with `break`, NGINX only stops processing anymore rewrite conditions._ - if you use `break` flag outside `location` block: - no more parsing of rewrite conditions - internal engine goes to the next phase (searching for `location` match) _Outside a location block, with `break`, NGINX stops processing anymore rewrite conditions._ - `last` - basically completes processing of rewrite directives, stops processing, and starts a search for a new location matching the changed URI - if you use `last` flag inside `location` block: - no more parsing of rewrite conditions - internal engine **starts to look** for another location match based on the result of the rewrite result - no more parsing of rewrite conditions, even on the next location match _Inside a location block, with last, NGINX stops processing anymore rewrite conditions and then **starts to look** for a new matching of location block. NGINX also ignores any rewrites in the new location block._ - if you use `last` flag outside `location` block: - no more parsing of rewrite conditions - internal engine goes to the next phase (searching for `location` match) _Outside a location block, with `last`, NGINX stops processing anymore rewrite conditions._ - `redirect` - returns a temporary redirect with the 302 HTTP response code - `permanent` - returns a permanent redirect with the 301 HTTP response code Note: - that outside location blocks, `last` and `break` are effectively the same - processing of rewrite directives at server level may be stopped via `break`, but the location lookup will follow anyway This explanation is based on the awesome answer by [Pothi Kalimuthu](https://serverfault.com/users/102173/pothi-kalimuthu) to [nginx url rewriting: difference between break and last](https://serverfault.com/a/829148). Official documentation has a great tutorials about [Creating NGINX Rewrite Rules](https://www.nginx.com/blog/creating-nginx-rewrite-rules/) and [Converting rewrite rules](https://nginx.org/en/docs/http/converting_rewrite_rules.html). I also recommend [Clean Url Rewrites Using Nginx](https://www.codesmite.com/article/clean-url-rewrites-using-nginx). Finally, look at the difference between `last` and `break` flags in action: - `last` directive:

last

- `break` directive:

break

This infographic comes from [Internal rewrite - nginx](https://www.linkedin.com/pulse/internal-rewrite-nginx-ivan-dabi%C4%87) by [Ivan Dabic](https://www.linkedin.com/in/ivan-dabic). ###### `return` directive > **:bookmark: [Use return directive for URL redirection (301, 302) - Base Rules - P2](RULES.md#beginner-use-return-directive-for-url-redirection-301-302)**
> **:bookmark: [Use return directive instead of rewrite for redirects - Performance - P2](RULES.md#beginner-use-return-directive-instead-of-rewrite-for-redirects)** The other way is a `return` directive. It's faster than rewrite because there is no regexp that has to be evaluated. It's stops processing and returns HTTP 301 (by default) to a client (tells NGINX to respond directly to the request), and the entire url is rerouted to the url specified. I use `return` directive in the following cases: - force redirect from http to https: ```nginx server { ... return 301 https://example.com$request_uri; } ``` - redirect from www to non-www and vice versa: ```nginx server { ... # It's only example. You shouldn't use 'if' statement in the following case: if ($host = www.example.com) { return 301 https://example.com$request_uri; } } ``` - close the connection and log it internally: ```nginx server { ... return 444; } ``` - send 4xx HTTP response for a client without any other actions: ```nginx server { ... if ($request_method = POST) { return 405; } # or: if ($invalid_referer) { return 403; } # or: if ($request_uri ~ "^/app/(.+)$") { return 403; } # or: location ~ ^/(data|storage) { return 403; } } ``` - and sometimes for reply with HTTP code without serving a file or response body: ```nginx server { ... # NGINX will not allow a 200 with no response body (200's need to be with a resource in the response. # '204 No Content' is meant to say "I've completed the request, but there is no body to return"): return 204 "it's all okay"; # Or without body: return 204; # Because default Content-Type is application/octet-stream, browser will offer to "save the file". # If you want to see reply in browser you should add properly Content-Type: # add_header Content-Type text/plain; } ``` To the last example: be careful if you're using such a configuration to do a healthcheck. While a 204 HTTP code is semantically perfect for a healthcheck (success indication with no content), some services do not consider it a success. ###### URL redirections > **:bookmark: [Use return directive for URL redirection (301, 302) - Base Rules - P2](RULES.md#beginner-use-return-directive-for-url-redirection-301-302)**
> **:bookmark: [Use return directive instead of rewrite for redirects - Performance - P2](RULES.md#beginner-use-return-directive-instead-of-rewrite-for-redirects)** HTTP allows servers to redirect a client request to a different location. This is useful when moving content to a new URL, when deleting pages or when changing domain names or merging websites. URL redirection is done for various reasons: - for URL shortening - to prevent broken links when web pages are moved - to allow multiple domain names belonging to the same owner to refer to a single web site - to guide navigation into and out of a website - for privacy protection - for hostile purposes such as phishing attacks or malware distribution It comes from [Wikipedia - URL redirection](https://en.wikipedia.org/wiki/URL_redirection). I recommend to read: - [Redirections in HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections) - [301 101: How Redirects Work](https://www.digitalthirdcoast.com/blog/301-101-redirects-work) - [Modify 301/302 response body (from this handbook)](HELPERS.md#modify-301302-response-body) - [Redirect POST request with payload to external endpoint (from this handbook)](HELPERS.md#redirect-post-request-with-payload-to-external-endpoint) ##### `try_files` directive We have one more very interesting and important directive: `try_files` (from the `ngx_http_core_module`). This directive tells NGINX to check for the existence of a named set of files or directories (checks files conditionally breaking on success). I think the best explanation comes from the official documentation: > _`try_files` checks the existence of files in the specified order and uses the first found file for request processing; the processing is performed in the current context. The path to a file is constructed from the file parameter according to the root and alias directives. It is possible to check directory’s existence by specifying a slash at the end of a name, e.g. `$uri/`. If none of the files were found, an internal redirect to the uri specified in the last parameter is made._ Generally it may check files on disk, redirect to proxies or internal locations, and return error codes, all in one directive. Take a look at the following example: ```nginx server { ... root /var/www/example.com; location / { try_files $uri $uri/ /frontend/index.html; } location ^~ /images { root /var/www/static; try_files $uri $uri/ =404; } ... ``` - default root directory for all locations is `/var/www/example.com` - `location /` - matches all locations without more specific locations, e.g. exact names - `try_files $uri` - when you receive a URI that's matched by this block try `$uri` first > For example: `https://example.com/tools/en.js` - NGINX will try to check if there's a file inside `/tools` called `en.js`, if found it, serve it in the first place. - `try_files $uri $uri/` - if you didn't find the first condition try the URI as a directory > For example: `https://example.com/backend/` - NGINX will try first check if a file called `backend` exists, if can't find it then goes to second check `$uri/` and see if there's a directory called `backend` exists then it will try serving it. - `try_files $uri $uri/ /frontend/index.html` - if a file and directory not found, NGINX sends `/frontend/index.html` - `location ^~ /images` - handle any query beginning with `/images` and halts searching - default root directory for this location is `/var/www/static` - `try_files $uri` - when you receive a URI that's matched by this block try `$uri` first > For example: `https://example.com/images/01.gif` - NGINX will try to check if there's a file inside `/images` called `01.gif`, if found it, serve it in the first place. - `try_files $uri $uri/` - if you didn't find the first condition try the URI as a directory > For example: `https://example.com/images/` - NGINX will try first check if a file called `images` exists, if can't find it then goes to second check `$uri/` and see if there's a directory called `images` exists then it will try serving it. - `try_files $uri $uri/ =404` - if a file and directory not found, NGINX sends `HTTP 404` (Not Found) On the other hand, `try_files` is relatively primitive. When encountered, NGINX will look for any of the specified files physically in the directory matched by the location block. If they don’t exist, NGINX does an internal redirect to the last entry in the directive. Additionally, think about dont't check for the existence of directories: ```nginx # Use this to take out an extra filesystem stat(): try_files $uri @index; # Instead of this: try_files $uri $uri/ @index; ``` ##### `if`, `break` and `set` > **:bookmark: [Avoid checks server_name with if directive - Performance - P2](RULES.md#beginner-avoid-checks-server_name-with-if-directive)** The `ngx_http_rewrite_module` also provides additional directives: - `break` - stops processing, if is specified inside the `location`, further processing of the request continues in this location: ```nginx # It's useful for: if ($slow_resp) { limit_rate 50k; break; } ``` - `if` - you can use `if` inside a `server` but not the other way around, also notice that you shouldn't use `if` inside `location` as it may not work as desired. For example, `if` statements aren't a good way of setting custom headers because they may cause statements outside the if block to be ignored. The NGINX docs says: > _There are cases where you simply cannot avoid using an `if`, for example if you need to test a variable which has no equivalent directive._ You should also remember about this: > _The `if` context in NGINX is provided by the rewrite module and this is the primary intended use of this context. Since NGINX will test conditions of a request with many other purpose-made directives, `if` **should not** be used for most forms of conditional execution. This is such an important note that the NGINX community has created a page called [if is evil](https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/) (yes, it's really evil and in most cases not needed)._ A long time ago I found this: > _That’s actually not true and shows you don’t understand the problem with it. When the `if` statement ends with `return` directive, there is no problem and it’s safe to use._ On the other hand, official documentation say: > _Directive if has problems when used in location context, in some cases it doesn’t do what you expect but something completely different instead. In some cases it even segfaults. It’s generally a good idea to avoid it if possible._ - `set` - sets a value for the specified variable. The value can contain text, variables, and their combination Example of usage `if` and `set` directives: ```nginx # It comes from: https://gist.github.com/jrom/1760790: if ($request_uri = /) { set $test A; } if ($host ~* example.com) { set $test "${test}B"; } if ($http_cookie !~* "auth_token") { set $test "${test}C"; } if ($test = ABC) { proxy_pass http://cms.example.com; break; } ``` ##### `root` vs `alias` > Placing a `root` or `alias` directive in a location block overrides the `root` or `alias` directive that was applied at a higher scope. With `alias` you can map to another file name. With `root` forces you to name your file on the server. In the first case, NGINX replaces the string prefix e.g `/robots.txt` in the URL path with e.g. `/var/www/static/robots.01.txt` and then uses the result as a filesystem path. In the second, NGINX inserts the string e.g. `/var/www/static/` at the beginning of the URL path and then uses the result as a file system path. Look at this. There is a difference, when the `alias` is for a whole directory will work: ```nginx location ^~ /data/ { alias /home/www/static/data/; } ``` But the following code won't do: ```nginx location ^~ /data/ { root /home/www/static/data/; } ``` This would have to be: ```nginx location ^~ /data/ { root /home/www/static/; } ``` The `root` directive is typically placed in server and location blocks. Placing a `root` directive in the server block makes the `root` directive available to all location blocks within the same server block. This directive tells NGINX to take the request url and append it behind the specified directory. For example, with the following configuration block: ```nginx server { server_name example.com; listen 10.250.250.10:80; index index.html; root /var/www/example.com; location / { try_files $uri $uri/ =404; } location ^~ /images { root /var/www/static; try_files $uri $uri/ =404; } } ``` NGINX will map the request made to: - `http://example.com/images/logo.png` into the file path `/var/www/static/images/logo.png` - `http://example.com/contact.html` into the file path `/var/www/example.com/contact.html` - `http://example.com/about/us.html` into the file path `/var/www/example.com/about/us.html` Like you want to forward all requests which start `/static` and your data present in `/var/www/static` you should set: - first path: `/var/www` - last path: `/static` - full path: `/var/www/static` ```nginx location { root ; ... } ``` NGINX documentation on the `alias` directive suggests that it is better to use `root` over `alias` when the location matches the last part of the directive’s value. The `alias` directive can only be placed in a location block. The following is a set of configurations for illustrating how the `alias` directive is applied: ```nginx server { server_name example.com; listen 10.250.250.10:80; index index.html; root /var/www/example.com; location / { try_files $uri $uri/ =404; } location ^~ /images { alias /var/www/static; try_files $uri $uri/ =404; } } ``` NGINX will map the request made to: - `http://example.com/images/logo.png` into the file path `/var/www/static/logo.png` - `http://example.com/images/ext/img.png` into the file path `/var/www/static/ext/img.png` - `http://example.com/contact.html` into the file path `/var/www/example.com/contact.html` - `http://example.com/about/us.html` into the file path `/var/www/example.com/about/us.html` When location matches the last part of the directive's value it is better to use the root directive (it seems like an arbitrary style choice because authors don't justify that instruction at all). Look at this example from the official documentation: ```nginx location /images/ { alias /data/w3/images/; } # Better solution: location /images/ { root /data/w3; } ``` ##### `internal` directive This directive specifies that the location block is internal. In other words, the specified resource cannot be accessed by external requests. On the other hand, it specifies how external redirections, i.e. locations like `http://example.com/app.php/some-path` should be handled; while set, they should return 404, only allowing internal redirections. In brief, this tells NGINX it's not accessible from the outside (it doesn't redirect anything). Conditions handled as internal redirections are listed in the documentation for `internal` directive. Specifies that a given location can only be used for internal requests and are the following: - requests redirected by the `error_page`, `index`, `random_index`, and `try_files` directives - requests redirected by the `X-Accel-Redirect` response header field from an upstream server - subrequests formed by the `include virtual` command of the `ngx_http_ssi_module module`, by the `ngx_http_addition_module` module directives, and by `auth_request` and `mirror` directives - requests changed by the `rewrite` directive Example 1: ```nginx error_page 404 /404.html; location = /404.html { internal; } ``` Example 2: The files are served from the directory `/srv/hidden-files` by the path prefix `/hidden-files/`. Pretty straightforward. The internal declaration tells NGINX that this path is accessible only through rewrites in the NGINX config, or via the `X-Accel-Redirect `header in proxied responses. To use this, just return an empty response which contains that header. The content of the header should be the location you want to redirect to: ```nginx location /hidden-files/ { internal; alias /srv/hidden-files/; } ``` Example 3: Another use case for internal redirects in NGINX is to hide credentials. Often you need to make requests to 3rd party services. For example, you want to send text messages or access a paid maps server. It would be the most efficient to send these requests directly from your JavaScript front end. However, doing so means you would have to embed an access token in the front end. This means savvy users could extract this token and make requests on your account. An easy fix is to make an endpoint in your back end which initiates the actual request. We could make use of an HTTP client library inside the back end. However, this will again tie up workers, especially if you expect a barrage of requests and the 3rd party service is responding very slowly. ```nginx location /external-api/ { internal; set $redirect_uri "$upstream_http_redirect_uri"; set $authorization "$upstream_http_authorization"; # For performance: proxy_buffering off; # Pass on secret from backend: proxy_set_header Authorization $authorization; # Use URI determined by backend: proxy_pass $redirect_uri; } ``` Examples 2 and 3 (both are great!) comes from [How to use internal redirects in NGINX](https://clubhouse.io/developer-how-to/how-to-use-internal-redirects-in-nginx/). > There is a limit of 10 internal redirects per request to prevent request processing cycles that can occur in incorrect configurations. If this limit is reached, the error _HTTP 500 Internal Server Error_ is returned. In such cases, the `rewrite or internal redirection cycle` message can be seen in the error log. Look also at [Authentication Based on Subrequest Result](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/) from the official documentation. ##### External and internal redirects External redirects originate directly from the client. So, if the client fetched `https://example.com/directory` it would be directly fall into preceding `location` block. Internal redirect means that it doesn’t send a 302 response to the client, it simply performs an implicit rewrite of the url and attempts to process it as though the user typed the new url originally. The internal redirect is different from the external redirect defined by HTTP response code 302 and 301, client browser won't update its URI addresses. To begin rewriting internally, we should explain the difference between redirects and internal rewrite. When source points to a destination that is out of source domain that is what we call redirect as your request will go from source to outside domain/destination. With internal rewrite you would be, basically, doing the same only the destination is local path under same domain and not the outside location. There is also [great explanation](https://openresty.org/download/agentzh-nginx-tutorials-en.html#02-nginxdirectiveexecorder06) about internal redirects: > _The internal redirection (e.g. via the `echo_exec` or `rewrite` directive) is an operation that makes NGINX jump from one location to another while processing a request (are very similar to `goto` statement in the C language). This "jumping" happens completely within the server itself._ There are two different kinds of internal requests: - **internal redirects** - redirects the client requests internally. The URI is changed, and the request may therefore match another location block and become eligible for different settings. The most common case of internal redirects is when using the `rewrite` directive, which allows you to rewrite the request URI - **sub-requests** - additional requests that are triggered internally to generate (insert or append to the body of the original request) content that is complementary to the main request (`addition` or `ssi` modules) ##### `allow` and `deny` > **:bookmark: [Take care about your ACL rules - Hardening - P1](RULES.md#beginner-take-care-about-your-acl-rules)**
> **:bookmark: [Reject unsafe HTTP methods - Hardening - P1](RULES.md#beginner-reject-unsafe-http-methods)** Both comes from the `ngx_http_access_module` module and allows limiting access to certain client addresses. You can combining `allow/deny` rules. > `deny` will always return 403 error code. The easiest path would be to start out by denying all access, then only granting access to those locations you want. For example: ```nginx location / { # without 'satisfy any' both should be passed: satisfy any; allow 192.168.0/0/16; deny all; # sh -c "echo -n 'user:' >> /etc/nginx/.secret" # sh -c "openssl passwd -apr1 >> /etc/nginx/.secret" auth_basic "Restricted Area"; auth_basic_user_file /etc/nginx/.secret; root /usr/share/nginx/html; index index.html index.htm; } ``` Putting `satisfy any;` in your configuration tells NGINX to accept either http authentication, or IP restriction. By default, when you define both, it will expect both. See also [this](https://serverfault.com/a/748373) answer: > As you've found, it isn't advisable to but the auth settings at the server level because they will apply to all locations. While it is possible to turn basic auth off there doesn't appear to be a way to clear an existing IP whitelist. > > A better solution would be to add the authentication to the / location so that it isn't inherited by /hello. > > The problem comes if you have other locations that require the basic auth and IP whitelisting in which case it might be worth considering moving the auth components to an include file or nesting them under /. Both directives may work unexpectedly! Look at the following example: ```nginx server { server_name example.com; deny all; location = /test { return 200 "it's all okay"; more_set_headers 'Content-Type: text/plain'; } } ``` If you generate a reqeust: ```bash curl -i https://example.com/test HTTP/2 200 date: Wed, 11 Nov 2018 10:02:45 GMT content-length: 13 server: Unknown content-type: text/plain it's all okay ``` Why? Look at [Request processing stages](#request-processing-stages) chapter. That's because NGINX process request in phases, and `rewrite` phase (where `return` belongs) goes before `access` phase (where `deny` works). ##### `uri` vs `request_uri` > **:bookmark: [Use `$request_uri` to avoid using regular expressions - Performance - P2](RULES.md#beginner-use-request_uri-to-avoid-using-regular-expressions)** `$request_uri` is the original request (for example `/foo/bar.php?arg=baz` includes arguments and can't be modified) but `$uri` refers to the altered URI so `$uri` is not equivalent to `$request_uri`. See [this](https://stackoverflow.com/a/48709976) great and short explanation by [Richard Smith](https://stackoverflow.com/users/4862445/richard-smith): > The `$uri` variable is set to the URI that NGINX is currently processing - but it is also subject to normalisation, including: > > - removal of the `?` and query string > - consecutive `/` characters are replace by a single `/` > - URL encoded characters are decoded > > The value of `$request_uri` is always the original URI and is not subject to any of the above normalisations. > > Most of the time you would use `$uri`, because it is normalised. Using `$request_uri` in the wrong place can cause URL encoded characters to become doubly encoded. Both excludes the schema (`https://` and the port (implicit 443) in both examples above) as defined by [RFC 2616 - http URL](https://tools.ietf.org/html/rfc2616#section-3.2.2) [IETF] for the URL: ``` http_URL = "http(s):" "//" host [ ":" port ] [ abs_path [ "?" query ]] ``` Take a look at the following table: | URL | $request_uri | $uri | | :--- | :--- | :--- | | `https://example.com/foo` | `/foo` | `/foo` | | `https://example.com/foo/bar` | `/foo/bar` | `/foo/bar` | | `https://example.com/foo/bar/` | `/foo/bar/` | `/foo/bar/` | | `https://example.com/foo/bar?` | `/foo/bar?` | `/foo/bar` | | `https://example.com/foo/bar?do=test` | `/foo/bar?do=test` | `/foo/bar` | | `https://example.com/rfc2616-sec3.html#sec3.2` | `/rfc2616-sec3.html` | `/rfc2616-sec3.html` | Another way to repeat the location is to use the `proxy_pass` directive which is quite easy: ```nginx location /app/ { proxy_pass http://127.0.0.1:5000; # or: proxy_pass http://127.0.0.1:5000/api/app/; } ``` | LOCATION | proxy_pass | REQUEST | RECEIVED BY UPSTREAM | | :--- | :--- | :--- | :--- | | `/app/` | `http://localhost:5000/api$request_uri` | `/app/foo?bar=baz` | `/api/webapp/foo?bar=baz` | | `/app/` | `http://localhost:5000/api$uri` | `/app/foo?bar=baz` | `/api/webapp/foo` | #### Compression and decompression > **:bookmark: [Mitigation of CRIME/BREACH attacks - Hardening Rules - P2](RULES.md#beginner-mitigation-of-crimebreach-attacks)** By default, NGINX compresses responses only with MIME type text/html using the `gzip` method. So, if you send request with `Accept-Encoding: gzip` header you will not see the `Content-Encoding: gzip` in the response. To enable `gzip` compression: ```nginx gzip on; ``` To compress responses with other MIME types, include the `gzip_types` directive and list the additional types: ```nginx gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml; ``` > Remember: by default, NGINX doesn't compress image files using its per-request gzip module. I also highly recommend you read this (it's interesting observation about gzip and performance by [Barry Pollard](https://serverfault.com/users/268936/barry-pollard)): > _To be honest gzip is not very processor intensive these days and gzipping on the fly (and then unzipping in the browser) is often the norm. It’s something web browsers are very good at._ > > _So unless you are getting huge volumes of traffic you’ll probably not notice any performance or CPU load impact due to on the fly gzipping for most web files._ To test HTTP and Gzip compression I recommend two external tools: - [HTTP Compression Test](https://www.whatsmyip.org/http-compression-test/) - [HTTP Gzip Compression Test](http://www.visiospark.com/gzip-compression-test/) NGINX also compress large files and avoid the temptation to compress smaller files (such as images, executables, etc.), because very small files barely benefit from compression. You can tell NGINX not to compress files smaller than e.g. 128 bytes: ```nginx gzip_min_length 128; ``` For more information see [Finding the Nginx gzip_comp_level Sweet Spot](https://mjanja.ch/2015/03/finding-the-nginx-gzip_comp_level-sweet-spot/). Compressing resources on-the-fly adds CPU-load and latency (wait for the compression to be done) every time a resource is served. NGINX also provides static compression with static module. It is better, for 2 reasons: - you don't have to gzip for each request - you can use a higher gzip level For example: ```nginx # Enable static gzip compression: location ^~ /assets/ { gzip_static on; ... } ``` You should put the `gzip_static on;` inside the blocks that configure static files, but if you’re only running one site, it’s safe to just put it in the http block. > NGINX does not automatically compress the files for you. You will have to do this yourself. To compress files manually: ```bash cd assets/ while IFS='' read -r -d '' _fd; do gzip -N4c ${_fd} > ${_fd}.gz done < <(find . -maxdepth 1 -type f -regex ".*\.\(css\|js\|jpg\|gif\|png\|jpeg\)" -print0) ``` So, for example, to service a request for `/foo/bar/file`, NGINX tries to find and send the file `/foo/bar/file.gz` that directly, so no extra CPU-cost or latency is added to your requests, speeding up the serving of your app. ##### What is the best NGINX compression gzip level? The level of gzip compression simply determines how compressed the data is on a scale from 1-9, where 9 is the most compressed. The trade-off is that the most compressed data usually requires the most work to compress/decompress but look also at [this](https://stackoverflow.com/questions/28452429/does-gzip-compression-level-have-any-impact-on-decompression/37892065#37892065) great answer. Author explains that the level of gzip compression doesn't affect the difficulty to decompress. I think the ideal compression level seems to be between 4 and 6. The following directive set how much files will be compressed: ```nginx gzip_comp_level 6; ``` #### Hash tables > Before start reading this chapter I recommend [Hash tables explained](https://yourbasic.org/algorithms/hash-tables-explained/). To assist with the rapid processing of requests, NGINX uses hash tables. NGINX hash, though in principle is same as typical hash lists, but it has significant differences. They are not meant for applications that add and remove elements dynamicall but are specifically designed to hold set of init time elements arranged in hash list. All elements that are put in the hash list are known while creating the hash list itself. No dynamic addtion or deletion is possible here. This hash table is constructed and compiled during restart or reload and afterwards it's running very fast. Main purpose seems to be speeding up the lookup of one time added elements. Look at the [Setting up hashes](http://nginx.org/en/docs/hash.html) from official documentation: > _To quickly process static sets of data such as server names, map directive’s values, MIME types, names of request header strings, NGINX uses hash tables. During the start and each re-configuration NGINX selects the minimum possible sizes of hash tables such that the bucket size that stores keys with identical hash values does not exceed the configured parameter (hash bucket size). The size of a table is expressed in buckets. The adjustment is continued until the table size exceeds the hash max size parameter. Most hashes have the corresponding directives that allow changing these parameters._ I also recommend [Optimizations](https://www.nginx.com/resources/wiki/start/topics/tutorials/optimizations/) section and [nginx - Hashing scheme](http://netsecinfo.blogspot.com/2010/01/nginx-hashing-scheme.html) explanation. Some important information (based on [this](https://serverfault.com/questions/419847/nginx-setting-server-names-hash-max-size-and-server-names-hash-bucket-size/786726#786726) amazing research by [brablc](https://serverfault.com/users/94256/brablc)): - the general recommendation would be to keep both values as small as possible and as less collisions as possible (during startup and with each reconfiguration, NGINX selects the smallest possible size for the hash tables) - it depends on your setup, you can reduce the number of server from the table and `reload` the NGINX instead of `restart` - if NGINX gave out communication about the need for increasing `hash_max_size` or `hash_bucket_size`, then it is first necessary to increase the first parameter - bigger `hash_max_size` uses more memory, bigger `hash_bucket_size` uses more CPU cycles during lookup and more transfers from main memory to cache. If you have enough memory increase `hash_max_size` and try to keep `hash_bucket_size` as low as possible - each hash table entry consumes space in a bucket. The space required is the length of the key (with some overhead to store the domain’s actual length as well), e.g. domain name > Since `stage.api.example.com` is 21 characters, all entries consume at least 24 bytes in a bucket, and most consume 32 bytes or more. - as you increase the number of entries, you have to increase the size of the hash table and/or the number of hash buckets in the table > If NGINX complains increase `hash_max_size` first as long as it complains. If the number exceeds some big number (32769 for instance), increase `hash_bucket_size` to multiple of default value on your platform as long as it complains. If it does not complain anymore, decrease `hash_max_size` back as long as it does not complain. Now you have the best setup for your set of server names (each set of server names may need different setup). - with a hash bucket size of 64 or 128, a bucket is full after 4 or 5 entries hash to it - `hash_max_size` is not related to number of server names directly, if number of servers doubles, you may need to increase `hash_max_size` 10 times or even more to avoid collisions. If you cannot avoid them, you have to increase `hash_bucket_size` - if you have `hash_max_size` less than 10000 and small `hash_bucket_size`, you can expect long loading time because NGINX would try to find optimal hash size in a loop (see [src/core/ngx_hash.c](https://github.com/nginx/nginx/blob/c3aed0a23392a509f64b740064f5f6633e8c89d8/src/core/ngx_hash.c#L289)) - if you have `hash_max_size` bigger than 10000, there will be only 1000 loops performed before it would complain ##### Server names hash table The hash with the names of servers are controlled by the following directives (inside `http` context): - `server_names_hash_max_size` - sets the maximum size of the server names hash tables; default value: 512 - `server_names_hash_bucket_size` - sets the bucket size for the server names hash tables; default values: 32, 64, or 128 (the default value depends on the size of the processor’s cache line) > Parameter `server_names_hash_bucket_size` is always equalized to the size, multiple to the size of the line of processor cache. If server name is defined as `too.long.server.name.example.com` then NGINX will fail to start and display the error message like: ``` nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64 ``` To fix this, you should `reload` the NGINX or increase the `server_names_hash_bucket_size` directive value to the next power of two (in this case to 128). If a large number of server names are defined, and NGINX complained with the following error: ``` nginx: [emerg] could not build the server_names_hash, you should increase either server_names_hash_max_size: 512 or server_names_hash_bucket_size: 32 ``` Try to set the `server_names_hash_max_size` to a number close to the number of server names. Only if this does not help, or if NGINX's start time is unacceptably long, try to increase the `server_names_hash_bucket_size` parameter. #### Log files > **:bookmark: [Use custom log formats - Debugging - P4](RULES.md#beginner-use-custom-log-formats)** Log files are a critical part of the NGINX management. It writes information about client requests in the access log right after the request is processed (in the last phase: `NGX_HTTP_LOG_PHASE`). By default: - the access log is located in `logs/access.log`, but I suggest you take it to `/var/log/nginx` directory - data is written in the predefined `combined/main` format - `access.log` stores record of each request and log format is fully configurable - `error.log` contains important operational messages It is the equivalent to the following configuration: ```nginx # In nginx.conf (default log format): http { ... log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; # but I suggest you change: log_format main '$remote_addr - $remote_user [$time_local] ' '"$request_method $scheme://$host$request_uri ' '$server_protocol" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time'; } ``` For more information please see [Configuring Logging](https://docs.nginx.com/nginx/admin-guide/monitoring/logging/). > Set `access log off;` to completely turns off logging. > If you don't want 404 errors to show in your NGINX error logs, you should set `log_not_found off;`. > If you want to enable logging of subrequests into `access_log`, you should set `log_subrequest on;` and change the default logging format (you have to log `$uri` to see the difference). There is [great explanation](https://serverfault.com/questions/904396/how-to-identify-subrequests-in-nginx-log-files/922956#922956) about how to identify subrequests in NGINX log files. I also recommend to read: - [ngx_http_log_module](http://nginx.org/en/docs/http/ngx_http_log_module.html) - [ngx_http_upstream_module](http://nginx.org/en/docs/http/ngx_http_upstream_module.html) ##### Conditional logging Sometimes certain entries are there just to fill up the logs or are cluttering them. I sometimes exclude requests - by client IP or whatever else - when I want to debug log files more effective. So, in this example, if the `$error_codes` variable’s value is 0 - then log nothing (default action), but if 1 (e.g. `404` or `503` from backend) - to save this request to the log: ```nginx # Define map in the http context: http { ... map $status $error_codes { default 1; ~^[23] 0; } ... # Add if condition to the access log: access_log /var/log/nginx/example.com-access.log combined if=$error_codes; } ``` ##### Manually log rotation > **:bookmark: [Configure log rotation policy - Base Rules - P1](RULES.md#beginner-configure-log-rotation-policy)** NGINX will re-open its logs in response to the `USR1` signal: ```bash cd /var/log/nginx mv access.log access.log.0 kill -USR1 $(cat /var/run/nginx.pid) && sleep 1 # >= gzip-1.6: gzip -k access.log.0 # With any version: gzip < access.log.0 > access.log.0.gz # Test integrity and remove if test passed: gzip -t access.log.0 && rm -fr access.log.0 ``` ##### Error log severity levels > You can't specify your own format, but in NGINX build-in several level's of `error_log`-ing. The following is a list of all severity levels: | TYPE | DESCRIPTION | | :--- | :--- | | `debug` | information that can be useful to pinpoint where a problem is occurring | | `info` | informational messages that aren’t necessary to read but may be good to know | | `notice` | something normal happened that is worth noting | | `warn` | something unexpected happened, however is not a cause for concern | | `error` | something was unsuccessful, contains the action of limiting rules (default) | | `crit` | important problems that need to be addressed | | `alert` | severe situation where action is needed promptly | | `emerg` | the system is in an unusable state and requires immediate attention | For example: if you set `crit` error log level, messages of `crit`, `alert`, and `emerg` levels are logged. > For debug logging to work, NGINX needs to be built with `--with-debug`. Default values for the error level: - in the main section - `error` - in the HTTP section - `crit` - in the server section - `crit` ##### How to log the start time of a request? The most logging information requires the request to complete (status code, bytes sent, durations, etc). If you want to log the start time of a request in NGINX you should apply a [patch](https://gist.github.com/rkbodenner/318681) that exposes request start time as a variable. The `$time_local` variable contains the time when the log entry is written so when the HTTP request header is read, NGINX does a lookup of the associated virtual server configuration. If the virtual server is found, the request goes through six phases: - server rewrite phase - location phase - location rewrite phase (which can bring the request back to the previous phase) - access control phase - `try_files` phase - log phase Since the log phase is the last one, `$time_local` variable is much more close to the end of the request than it's start. ##### How to log the HTTP request body? Nginx doesn't parse the client request body unless it really needs to, so it usually does not fill the `$request_body` variable. The exceptions are when: - it sends the request to a proxy - or a fastcgi server So you really need to either add the `proxy_pass` or `fastcgi_pass` directives to your block. ```nginx # 1) Set log format: log_format req_body_logging '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" "$request_body"'; # 2) Limit the request body size: client_max_body_size 1k; client_body_buffer_size 1k; client_body_in_single_buffer on; # 3) Put the log format: server { ... location /api/v4 { access_log logs/access_req_body.log req_body_logging; proxy_pass http://127.0.0.1; ... } location = /post.php { access_log /var/log/nginx/postdata.log req_body_logging; fastcgi_pass php_cgi; ... } } ``` For this, you can also use [echo](https://github.com/openresty/echo-nginx-module) module. To log a request body, what we need is to use the `echo_read_request_body` directive and the `$request_body` variable (contains the request body of the echo module). > `echo_read_request_body` explicitly reads request body so that the `$request_body` variable will always have non-empty values (unless the body is so big that it has been saved by NGINX to a local temporary file). ```nginx http { log_format req_body_logging '$request_body'; access_log /var/log/nginx/access.log req_body_logging; ... server { location / { echo_read_request_body; ... } ... } } ``` ##### NGINX upstream variables returns 2 values For example: ``` upstream_addr 192.168.50.201:8080 : 192.168.50.201:8080 upstream_bytes_received 427 : 341 upstream_connect_time 0.001 : 0.000 upstream_header_time 0.003 : 0.001 upstream_response_length 0 : 0 upstream_response_time 0.003 : 0.001 upstream_status 401 : 200 ``` Below is a short description of each of them: - `$upstream_addr` - keeps the IP address and port, or the path to the UNIX-domain socket of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas, e.g. `192.168.1.1:80, 192.168.1.2:80, unix:/tmp/sock`. If an internal redirect from one server group to another happens, initiated by `X-Accel-Redirect` or `error_page`, then the server addresses from different groups are separated by colons, e.g. `192.168.1.1:80, 192.168.1.2:80, unix:/tmp/sock : 192.168.10.1:80, 192.168.10.2:80` - `$upstream_cache_status` - keeps the status of accessing a response cache (0.8.3). The status can be either `MISS`, `BYPASS`, `EXPIRED`, `STALE`, `UPDATING`, `REVALIDATED`, or `HIT` - `$upstream_connect_time` - time spent on establishing a connection with an upstream server - `$upstream_cookie_` - cookie with the specified name sent by the upstream server in the `Set-Cookie` response header field (1.7.1). Only the cookies from the response of the last server are saved - `$upstream_header_time` - time between establishing a connection and receiving the first byte of the response header from the upstream server - `$upstream_http_` - keep server response header fields. For example, the `Server` response header field is available through the `$upstream_http_server` variable. The rules of converting header field names to variable names are the same as for the variables that start with the `$http_` prefix. Only the header fields from the response of the last server are saved - `$upstream_response_length` - keeps the length of the response obtained from the upstream server (0.7.27); the length is kept in bytes. Lengths of several responses are separated by commas and colons like addresses in the `$upstream_addr` variable - `$upstream_response_time` - time between establishing a connection and receiving the last byte of the response body from the upstream server - `$upstream_status` - keeps status code of the response obtained from the upstream server. Status codes of several responses are separated by commas and colons like addresses in the `$upstream_addr` variable Official documentation say: > _[...] If several servers were contacted during request processing, their addresses are separated by commas. [...] If an internal redirect from one server group to another happens, initiated by “X-Accel-Redirect” or error_page, then the server addresses from different groups are separated by colons_ This means that it made multiple requests to a backend, most likely you either have a bare `proxy_pass` host that resolves to different IPs (frequently the case with something like Amazon ELB as an origin), are you have a configured upstream that has multiple servers. Unless disabled, the proxy module will make round robin attempts against all healthy backends. This can be configured from `proxy_next_upstream_*` directives. For example if this is not the desired behavior, you can just do (specifies in which cases a request should be passed to the next server): ```nginx # One should bear in mind that passing a request to the next server is only possible # if nothing has been sent to a client yet. That is, if an error or timeout occurs # in the middle of the transferring of a response, fixing this is impossible. proxy_next_upstream off; ``` For more information please see [ngx_http_upstream_module](http://nginx.org/en/docs/http/ngx_http_upstream_module.html) and [proxy_next_upstream](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream). #### Reverse proxy > After reading this chapter, please see: [Rules: Reverse Proxy](RULES.md#reverse-proxy). This is one of the greatest feature of the NGINX. In simplest terms, a reverse proxy is a server that comes in-between internal applications and external clients, forwarding client requests to the appropriate server. It takes a client request, passes it on to one or more servers, and subsequently delivers the server’s response back to the client. Official NGINX documentation says: > _Proxying is typically used to distribute the load among several servers, seamlessly show content from different websites, or pass requests for processing to application servers over protocols other than HTTP._ You can also read a very good explanation about [What's the difference between proxy server and reverse proxy server](https://stackoverflow.com/questions/224664/whats-the-difference-between-proxy-server-and-reverse-proxy-server/366212#366212). A reverse proxy can off load much of the infrastructure concerns of a high-volume distributed web application.

reverse-proxy_preview

This infographic comes from [Jenkins with NGINX - Reverse proxy with https](https://medium.com/@sportans300/nginx-reverse-proxy-with-https-466daa4da4fc). This allow you to have NGINX reverse proxy requests to unicorns, mongrels, webricks, thins, or whatever you really want to have run your servers. Reverse proxy gives you number of advanced features such as: - load balancing, failover, and transparent maintenance of the backend servers - increased security (e.g. SSL termination, hide upstream configuration) - increased performance (e.g. caching, load balancing) - simplifies the access control responsibilities (single point of access and maintenance) - centralised logging and auditing (single point of maintenance) - add/remove/modify HTTP headers In my opinion, the two most important things related to the reverse proxy are: - the way of requests forwarded to the backend - the type of headers forwarded to the backend If we talking about security of the proxy server look at this recommendations about [Guidelines on Securing Public Web Servers](https://www.nist.gov/publications/guidelines-securing-public-web-servers) [NIST]. This document is a good starting point. Is old but still has interesting solutions and suggestions. There is a [great explanation](https://serverfault.com/a/25095) about the benefits of improving security through the use of a reverse proxy server. > A reverse proxy gives you a couple things that may make your server more secure: > > - a place to monitor and log what is going on separate from the web server > - a place to filter separate from your web server if you know that some area of your system is vulnerable. Depending on the proxy you may be able to filter at the application level > - another place to implement ACLs and rules if you cannot be expressive enough for some reason on your web server > - a separate network stack that will not be vulnerable in the same ways as your web server. This is particularly true if your proxy is from a different vendor > - a reverse proxy with no filtering does not automatically protect you against everything, but if the system you need to protect is high-value then adding a reverse proxy may be worth the costs support and performance costs Another [great answer](https://security.stackexchange.com/questions/48347/documented-best-practices-for-reverse-proxy-implementation) about best practices for reverse proxy implementation: > In my experience some of the most important requirements and mitigations, in no particular order, are: > > - make sure that your proxy, back-end web (and DB) servers cannot establish direct outbound (internet) connections (including DNS and SMTP, and particularly HTTP). This means (forward) proxies/relays for required outbound access, if required > - make sure your logging is useful (§9.1 in the above), and coherent. You may have logs from multiple devices (router, firewall/IPS/WAF, proxy, web/app servers, DB servers). If you can't quickly, reliably and deterministically link records across each device together, you're doing it wrong. This means NTP, and logging any or all of: PIDs, TIDs, session-IDs, ports, headers, cookies, usernames, IP addresses and maybe more (and may mean some logs contain confidential information) > - understand the protocols, and make deliberate, informed decisions: including cipher/TLS version choice, HTTP header sizes, URL lengths, cookies. Limits should be implemented on the reverse-proxy. If you're migrating to a tiered architecture, make sure the dev team are in the loop so that problems are caught as early as possible > - run vulnerability scans from the outside, or get someone to do it for you. Make sure you know your footprint and that the reports highlight deltas, as well as the theoretical TLS SNAFU du-jour > - understand the modes of failure. Sending users a bare default "HTTP 500 - the wheels came off" when you have load or stability problems is sloppy > - monitoring, metrics and graphs: having normal and historic data is invaluable when investigating anomalies, and for capacity planning > - tuning: from TCP time-wait to listen backlog to SYN-cookies, again you need to make make deliberate, informed decisions > - follow basic OS hardening guidelines, consider the use of chroot/jails, host-based IDS, and other measures, where available ##### Passing requests > **:bookmark: [Use pass directive compatible with backend protocol - Reverse Proxy - P1](RULES.md#beginner-use-pass-directive-compatible-with-backend-protocol)** When NGINX proxies a request, it sends the request to a specified proxied server, fetches the response, and sends it back to the client. It is possible to proxy requests to: - an HTTP servers (e.g. NGINX, Apache, or other) with `proxy_pass` directive: ```nginx upstream bk_front { server 192.168.252.20:8080 weight=5; server 192.168.252.21:8080 } server { location / { proxy_pass http://bk_front; } location /api { proxy_pass http://192.168.21.20:8080; } location /info { proxy_pass http://localhost:3000; } location /ra-client { proxy_pass http://10.0.11.12:8080/guacamole/; } location /foo/bar/ { proxy_pass http://www.example.com/url/; } ... } ``` - a non-HTTP servers (e.g. PHP, Node.js, Python, Java, or other) with `proxy_pass` directive (as a fallback) or directives specially designed for this: - `fastcgi_pass` which passes a request to a FastCGI server ([PHP FastCGI Example](https://www.nginx.com/resources/wiki/start/topics/examples/phpfcgi/)): ```nginx server { ... location ~ ^/.+\.php(/|$) { fastcgi_pass 127.0.0.1:9000; include /etc/nginx/fcgi_params; } ... } ``` - `uwsgi_pass` which passes a request to a uWSGI server ([Nginx support uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html)): ```nginx server { location / { root html; uwsgi_pass django_cluster; uwsgi_param UWSGI_SCRIPT testapp; include /etc/nginx/uwsgi_params; } ... } ``` - `scgi_pass` which passes a request to an SCGI server: ```nginx server { location / { scgi_pass 127.0.0.1:4000; include /etc/nginx/scgi_params; } ... } ``` - `memcached_pass` which passes a request to a Memcached server: ```nginx server { location / { set $memcached_key "$uri?$args"; memcached_pass memc_instance:4004; error_page 404 502 504 = @memc_fallback; } location @memc_fallback { proxy_pass http://backend; } ... } ``` - `redis_pass` which passes a request to a Redis server ([HTTP Redis](https://www.nginx.com/resources/wiki/modules/redis/)): ```nginx server { location / { set $redis_key $uri; redis_pass redis_instance:6379; default_type text/html; error_page 404 = /fallback; } location @fallback { proxy_pass http://backend; } ... } ``` The `proxy_pass` and other `*_pass` directives specifies that all requests which match the location block should be forwarded to the specific socket, where the backend app is running. However, more complex apps may need additional directives: - `proxy_pass` - see [`ngx_http_proxy_module`](http://nginx.org/en/docs/http/ngx_http_proxy_module.html) directives explanation - `fastcgi_pass` - see [`ngx_http_fastcgi_module`](http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html) directives explanation - `uwsgi_pass` - see [`ngx_http_uwsgi_module`](http://nginx.org/en/docs/http/ngx_http_uwsgi_module.html) directives explanation - `scgi_pass` - see [`ngx_http_scgi_module`](http://nginx.org/en/docs/http/ngx_http_scgi_module.html) directives explanation - `memcached_pass` - see [`ngx_http_memcached_module`](http://nginx.org/en/docs/http/ngx_http_memcached_module.html) directives explanation - `redis_pass` - see [`ngx_http_redis_module`](https://www.nginx.com/resources/wiki/modules/redis/) directives explanation ##### Trailing slashes > **:bookmark: [Be careful with trailing slashes in proxy_pass directive - Reverse Proxy - P3](RULES.md#beginner-be-careful-with-trailing-slashes-in-proxy_pass-directive)** If you have something like: ```nginx location /public/ { proxy_pass http://bck_testing_01; } ``` And go to `http://example.com/public`, NGINX will automatically redirect you to `http://example.com/public/`. Look also at this example: ```nginx location /foo/bar/ { # proxy_pass http://example.com/url/; proxy_pass http://192.168.100.20/url/; } ``` If the URI is specified along with the address, it replaces the part of the request URI that matches the location parameter. For example, here the request with the `/foo/bar/page.html` URI will be proxied to `http://www.example.com/url/page.html`. If the address is specified without a URI, or it is not possible to determine the part of URI to be replaced, the full request URI is passed (possibly, modified). Here is an example with trailing slash in location, but no trailig slash in `proxy_pass`: ```nginx location /foo/ { proxy_pass http://127.0.0.1:8080/bar; } ``` See how `bar` and `path` concatenates. If one go to `http://yourserver.com/foo/path/id?param=1` NGINX will proxy request to `http://127.0.0.1/barpath/id?param=1`. As stated in NGINX documentation if `proxy_pass` used without URI (i.e. without path after `server:port`) NGINX will put URI from original request exactly as it was with all double slashes, `../` and so on. Look also at the configuration snippets: [Using trailing slashes](#using-trailing-slashes). Below are additional examples: | LOCATION | PROXY_PASS | REQUEST | RECEIVED BY UPSTREAM | | :--- | :--- | :--- | :--- | | `/app/` | `http://localhost:5000/api/` | `/app/foo?bar=baz` | `/api/foo?bar=baz` | | `/app/` | `http://localhost:5000/api` | `/app/foo?bar=baz` | `/apifoo?bar=baz` | | `/app` | `http://localhost:5000/api/` | `/app/foo?bar=baz` | `/api//foo?bar=baz` | | `/app` | `http://localhost:5000/api` | `/app/foo?bar=baz` | `/api/foo?bar=baz` | | `/app` | `http://localhost:5000/api` | `/appfoo?bar=baz` | `/apifoo?bar=baz` | In other words: > You usually always want a trailing slash, never want to mix with and without trailing slash, and only want without trailing slash when you want to concatenate a certain path component together (which I guess is quite rarely the case). Note how query parameters are preserved. ##### Passing headers to the backend > **:bookmark: [Set the HTTP headers with add_header and proxy_*_header directives properly - Base Rules - P1](RULES.md#beginner-set-the-http-headers-with-add_header-and-proxy__header-directives-properly)**
> **:bookmark: [Remove support for legacy and risky HTTP headers - Hardening - P1](RULES.md#beginner-remove-support-for-legacy-and-risky-http-headers)**
> **:bookmark: [Always pass Host, X-Real-IP, and X-Forwarded headers to the backend - Reverse Proxy - P2](RULES.md#beginner-always-pass-host-x-real-ip-and-x-forwarded-headers-to-the-backend)**
> **:bookmark: [Use custom headers without X- prefix - Reverse Proxy - P3](RULES.md#beginner-use-reload-option-to-change-configurations-on-the-fly)** By default, NGINX redefines two header fields in proxied requests: - the `Host` header is re-written to the value defined by the `$proxy_host` variable. This will be the IP address or name and port number of the upstream, directly as defined by the `proxy_pass` directive - the `Connection` header is changed to `close`. This header is used to signal information about the particular connection established between two parties. In this instance, NGINX sets this to `close` to indicate to the upstream server that this connection will be closed once the original request is responded to. The upstream should not expect this connection to be persistent When NGINX proxies a request, it automatically makes some adjustments to the request headers it receives from the client: - NGINX drop empty headers. There is no point of passing along empty values to another server; it would only serve to bloat the request - NGINX, by default, will consider any header that contains underscores as invalid. It will remove these from the proxied request. If you wish to have NGINX interpret these as valid, you can set the `underscores_in_headers` directive to `on`, otherwise your headers will never make it to the backend server. Underscores in header fields are allowed ([RFC 7230, sec. 3.2.](https://tools.ietf.org/html/rfc7230#section-3.2)), but indeed uncommon It is important to pass more than just the URI if you expect the upstream server handle the request properly. The request coming from NGINX on behalf of a client will look different than a request coming directly from a client. > Please read [Managing request headers](https://www.nginx.com/resources/wiki/start/topics/examples/headers_management/) from the official wiki. In NGINX does support arbitrary request header field. Last part of a variable name is the field name converted to lower case with dashes replaced by underscores: ``` $http_name_of_the_header_key ``` If you have `X-Real-IP = 127.0.0.1` in header, you can use `$http_x_real_ip` to get `127.0.0.1`. Use the `proxy_set_header` directive to sets headers that sends to the backend servers. > HTTP headers are used to transmit additional information between client and server. `add_header` sends headers to the client (browser) and will work on successful requests only, unless you set up `always` parameter. `proxy_set_header` sends headers to the backend server. If the value of a header field is an empty string then this field will not be passed to a proxied server. It's also important to distinguish between request headers and response headers. Request headers are for traffic inbound to the webserver or backend app. Response headers are going the other way (in the HTTP response you get back using client, e.g. curl or browser). Ok, so look at the following short explanation about proxy directives (for more information about valid header values please see [this](RULES.md#beginner-always-pass-host-x-real-ip-and-x-forwarded-stack-headers-to-the-backend) rule): - `proxy_http_version` - defines the HTTP protocol version for proxying, by default it it set to 1.0. For Websockets and keepalive connections you need to use the version 1.1: ```nginx proxy_http_version 1.1; ``` - `proxy_cache_bypass` - sets conditions under which the response will not be taken from a cache: ```nginx proxy_cache_bypass $http_upgrade; ``` - `proxy_intercept_errors` - means that any response with HTTP code 300 or greater is handled by the `error_page` directive and ensures that if the proxied backend returns an error status, NGINX will be the one showing the error page (as opposed to the error page on the backend side). If you want certain error pages still being delivered from the upstream server, then simply don't specify the `error_page ` on the reverse proxy (without this, NGINX will forward the error page coming from the upstream server to the client): ```nginx proxy_intercept_errors on; error_page 404 /404.html; # from proxy # To bypass error intercepting (if you have proxy_intercept_errors on): # 1 - don't specify the error_page 404 on the reverse proxy # 2 - go to the @debug location error_page 500 503 504 @debug; location @debug { proxy_intercept_errors off; proxy_pass http://backend; } ``` - `proxy_set_header` - allows redefining or appending fields to the request header passed to the proxied server - `Upgrade` and `Connection` - these header fields are required if your application is using Websockets: ```nginx proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; ``` - `Host` - the `$host` variable in the following order of precedence contains: host name from the request line, or host name from the Host request header field, or the server name matching a request: NGINX uses `Host` header for `server_name` matching. It does not use TLS SNI. This means that for an SSL server, NGINX must be able to accept SSL connection, which boils down to having certificate/key. The cert/key can be any, e.g. self-signed: ```nginx proxy_set_header Host $host; ``` - `X-Real-IP` - forwards the real visitor remote IP address to the proxied server: ```nginx proxy_set_header X-Real-IP $remote_addr; ``` - `X-Forwarded-For` - is the conventional way of identifying the originating IP address of the user connecting to the web server coming from either a HTTP proxy or load balancer: ```nginx proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ``` - `X-Forwarded-Proto` - identifies the protocol (HTTP or HTTPS) that a client used to connect to your proxy or load balancer: ```nginx proxy_set_header X-Forwarded-Proto $scheme; ``` - `X-Forwarded-Host` - defines the original host requested by the client: ```nginx proxy_set_header X-Forwarded-Host $host; ``` - `X-Forwarded-Port` - defines the original port requested by the client: ```nginx proxy_set_header X-Forwarded-Port $server_port; ``` If you want to read about custom headers, take a look at [Why we need to deprecate x prefix for HTTP headers?](https://tonyxu.io/posts/2018/http-deprecate-x-prefix/) and [this](https://stackoverflow.com/a/3561399) great answer by [BalusC](https://stackoverflow.com/users/157882/balusc). ###### Importance of the `Host` header > **:bookmark: [Set and pass Host header only with $host variable - Reverse Proxy - P2](RULES.md#beginner-set-and-pass-host-header-only-with-host-variable)** The `Host` header tells the webserver which virtual host to use (if set up). You can even have the same virtual host using several aliases (domains and wildcard-domains). This why the host header exists. The host header specifies which website or web application should process an incoming HTTP request. In NGINX, `$host` equals `$http_host`, lowercase and without the port number (if present), except when `HTTP_HOST` is absent or is an empty value. In that case, `$host` equals the value of the `server_name` directive of the server which processed the request. But look at this: > _An unchanged `Host` request header field can be passed with `$http_host`. However, if this field is not present in a client request header then nothing will be passed. In such a case it is better to use the `$host` variable - its value equals the server name in the `Host` request header field or the primary server name if this field is not present._ For example, if you set `Host: MASTER:8080`, `$host` will be "master" (while `$http_host` will be `MASTER:8080` as it just reflects the whole header). Look also at [$10k host header](https://www.ezequiel.tech/p/10k-host-header.html) and [What is a Host Header Attack?](https://www.acunetix.com/blog/articles/automated-detection-of-host-header-attacks/). ###### Redirects and `X-Forwarded-Proto` > **:bookmark: [Don't use X-Forwarded-Proto with $scheme behind reverse proxy - Reverse Proxy - P1](RULES.md#beginner-dont-use-x-forwarded-proto-with-scheme-behind-reverse-proxy)** This header is very important because it prevent a redirect loop. When used inside HTTPS server block each HTTP response from the proxied server will be rewritten to HTTPS. Look at the following example: 1. Client sends the HTTP request to the Proxy 2. Proxy sends the HTTP request to the Server 3. Server sees that the URL is `http://` 4. Server sends back 3xx redirect response telling the Client to connect to `https://` 5. Client sends an HTTPS request to the Proxy 6. Proxy decrypts the HTTPS traffic and sets the `X-Forwarded-Proto: https` 7. Proxy sends the HTTP request to the Server 8. Server sees that the URL is `http://` but also sees that `X-Forwarded-Proto` is https and trusts that the request is HTTPS 9. Server sends back the requested web page or data This explanation comes from [Purpose of the X-Forwarded-Proto HTTP Header](https://community.pivotal.io/s/article/Purpose-of-the-X-Forwarded-Proto-HTTP-Header). In step 6 above, the Proxy is setting the HTTP header `X-Forwarded-Proto: https` to specify that the traffic it received is HTTPS. In step 8, the Server then uses the `X-Forwarded-Proto` to determine if the request was HTTP or HTTPS. You can read about how to set it up correctly here: - [Set correct scheme passed in X-Forwarded-Proto](HELPERS.md#set-correct-scheme-passed-in-x-forwarded-proto) - [Don't use X-Forwarded-Proto with $scheme behind reverse proxy - Reverse Proxy - P1](RULES.md#beginner-dont-use-x-forwarded-proto-with-scheme-behind-reverse-proxy) ###### A warning about the `X-Forwarded-For` > **:bookmark: [Set properly values of the X-Forwarded-For header - Reverse Proxy - P1](RULES.md#beginner-set-properly-values-of-the-x-forwarded-for-header)** I think we should just maybe stop for a second. `X-Forwarded-For` is a one of the most important header that has the security implications. Where a connection passes through a chain of proxy servers, `X-Forwarded-For` can give a comma-separated list of IP addresses with the first being the furthest downstream (that is, the user). The HTTP `X-Forwarded-For` accepts two directives as mentioned above and described below: - `` - it is the IP address of the client - `` - it is the proxies that request has to go through. If there are multiple proxies then the IP addresses of each successive proxy is listed Syntax: ``` X-Forwarded-For: , , ``` `X-Forwarded-For` should not be used for any Access Control List (ACL) checks because it can be spoofed by attackers. Use the real IP address for this type of restrictions. HTTP request headers such as `X-Forwarded-For`, `True-Client-IP`, and `X-Real-IP` are not a robust foundation on which to build any security measures, such as access controls. [Set properly values of the X-Forwarded-For header (from this handbook)](RULES.md#beginner-set-properly-values-of-the-x-forwarded-for-header) - see this for more detailed information on how to set properly values of the `X-Forwarded-For` header. But that's not all. Behind a reverse proxy, the user IP we get is often the reverse proxy IP itself. If you use other HTTP server working between proxy and app server you should also set the correct mechanism for interpreting values of this header. I recommend to read [this](https://serverfault.com/questions/314574/nginx-real-ip-header-and-x-forwarded-for-seems-wrong/414166#414166) amazing explanation by [Nick M](https://serverfault.com/users/130923/nick-m). 1) Pass headers from proxy to the backend layer: - [Always pass Host, X-Real-IP, and X-Forwarded headers to the backend](RULES.md#beginner-always-pass-host-x-real-ip-and-x-forwarded-headers-to-the-backend) - [Set properly values of the X-Forwarded-For header (from this handbook)](RULES.md#beginner-set-properly-values-of-the-x-forwarded-for-header) 2) NGINX (backend) - modify the `set_real_ip_from` and `real_ip_header` directives: > For this, the `http_realip_module` must be installed (`--with-http_realip_module`). First of all, you should add the following lines to the configuration: ```nginx # Add these to the set_real_ip.conf, there are the real IPs where your traffic # is coming from (front proxy/lb): set_real_ip_from 192.168.20.10; # IP address of master set_real_ip_from 192.168.20.11; # IP address of slave # You can also add an entire subnet: set_real_ip_from 192.168.40.0/24; # Defines a request header field used to send the address for a replacement, # in this case we use X-Forwarded-For: real_ip_header X-Forwarded-For; # The real IP from your client address that matches one of the trusted addresses # is replaced by the last non-trusted address sent in the request header field: real_ip_recursive on; # Include it to the appropriate context: server { include /etc/nginx/set_real_ip.conf; ... } ``` 3) NGINX - add/modify and set log format: ```nginx log_format combined-1 '$remote_addr forwarded for $http_x_real_ip - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; # or: log_format combined-2 '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/example.com/access.log combined-1; ``` This way, e.g. the `$_SERVER['REMOTE_ADDR']` will be correctly filled up in PHP fastcgi. You can test it with the following script: ```php # tls_check.php '; print_r($_SERVER); echo ''; exit; ?> ``` And send request to it: ```bash curl -H Cache-Control: no-cache -ks https://example.com/tls-check.php?${RANDOM} | grep "HTTP_X_FORWARDED_FOR\|HTTP_X_REAL_IP\|SERVER_ADDR\|REMOTE_ADDR" [HTTP_X_FORWARDED_FOR] => 172.217.20.206 [HTTP_X_REAL_IP] => 172.217.20.206 [SERVER_ADDR] => 192.168.10.100 [REMOTE_ADDR] => 192.168.10.10 ``` ###### Improve extensibility with `Forwarded` Since 2014, the IETF has approved a standard header definition for proxy, called `Forwarded`, documented [here](https://tools.ietf.org/html/rfc7239) [IETF] and [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) that should be use instead of `X-Forwarded` headers. This is the one you should use reliably to get originating IP in case your request is handled by a proxy. Official NGINX documentation also gives you how to [Using the Forwarded header](https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/). In general, the proxy headers (`Forwarded` or `X-Forwarded-For`) are the right way to get your client IP only when you are sure they come to you via a proxy. If there is no proxy header or no usable value in, you should default to the `REMOTE_ADDR` server variable. ##### Response headers > **:bookmark: [Set the HTTP headers with add_header and proxy_*_header directives properly - Base Rules - P1](RULES.md#beginner-set-the-http-headers-with-add_header-and-proxy__header-directives-properly)** `add_header` directive allows you to define an arbitrary response header (mostly for informational/debugging purposes) and value to be included in all response codes which are equal to: - 2xx series: 200, 201, 204, 206 - 3xx series: 301, 302, 303, 304, 307, 308 For example: ```nginx add_header Custom-Header Value; ``` > To change (adding or removing) existing headers you should use a [headers-more-nginx-module](https://github.com/openresty/headers-more-nginx-module) module. There is one thing you must watch out for if you use `add_header` directive (also applies to `proxy_*_header` directives). See the following explanations: - [Nginx add_header configuration pitfall](https://blog.g3rt.nl/nginx-add_header-pitfall.html) - [Be very careful with your add_header in Nginx! You might make your site insecure](https://www.peterbe.com/plog/be-very-careful-with-your-add_header-in-nginx) This situation is described in the official documentation: > _There could be several `add_header` directives. These directives are inherited from the previous level if and only if there are no `add_header` directives defined on the current level._ However - and this is important - as you now have defined a header in your `server` context, all the remaining headers defined in the `http` context will no longer be inherited. Means, you’ve to define them in your `server` context again (or alternatively ignore them if they’re not important for your site). At the end, summary about directives to manipulate headers: - `proxy_set_header` is to sets or remove a request header (and pass it or not to the backend) - `add_header` is to add header to response - `proxy_hide_header` is to hide a response header We also have the ability to manipulate request and response headers using the [headers-more-nginx-module](https://github.com/openresty/headers-more-nginx-module) module: - `more_set_headers` - replaces (if any) or adds (if not any) the specified output headers - `more_clear_headers` - clears the specified output headers - `more_set_input_headers` - very much like `more_set_headers` except that it operates on input headers (or request headers) - `more_clear_input_headers` - very much like `more_clear_headers` except that it operates on input headers (or request headers) The following figure describes the modules and directives responsible for manipulating HTTP request and response headers:

headers_processing

#### Load balancing algorithms Load Balancing is in principle a wonderful thing really. You can find out about it when you serve tens of thousands (or maybe more) of requests every second. Of course, load balancing is not the only reason - think also about maintenance tasks without downtime. Generally load balancing is a technique used to distribute the workload across multiple computing resources and servers. I think you should always use this technique also if you have a simple app or whatever else what you're sharing with other. The configuration is very simple. NGINX includes a `ngx_http_upstream_module` to define backends (groups of servers or multiple server instances). More specifically, the `upstream` directive is responsible for this. > `upstream` defines the load balancing pool, only provide a list of servers, some kind of weight, and other parameters related to the backend layer. ##### Backend parameters > **:bookmark: [Tweak passive health checks - Load Balancing - P3](RULES.md#beginner-tweak-passive-health-checks)**
> **:bookmark: [Don't disable backends by comments, use down parameter - Load Balancing - P4](RULES.md#beginner-dont-disable-backends-by-comments-use-down-parameter)** Before we start talking about the load balancing techniques you should know something about `server` directive. It defines the address and other parameters of a backend servers. This directive accepts the following options: - `weight=` - sets the weight of the origin server, e.g. `weight=10` - `max_conns=` - limits the maximum number of simultaneous active connections from the NGINX proxy server to an upstream server (default value: `0` = no limit), e.g. `max_conns=8` - if you set `max_conns=4` the 5th will be rejected - if the server group does not reside in the shared memory (`zone` directive), the limitation works per each worker process - `max_fails=` - the number of unsuccessful attempts to communicate with the backend (default value: `1`, `0` disables the accounting of attempts), e.g. `max_fails=3;` - `fail_timeout=