^{Hi-diddle-diddle, he played on his
fiddle and danced with lady pigs.
Number three said, "Nicks on tricks!
I'll build my house with EN-jin-EKS!".
The Three Little Pigs: Who's Afraid of the Big Bad Wolf?}

• Introduction
  • General disclaimer
  • Contributing & Support
  • ToDo list
  • Reports: blkcipher.info
    • SSL Labs
    • Mozilla Observatory
  • Checklist to rule them all
  • Printable high-res hardening cheatsheets
• Books
  • Nginx Essentials
  • Nginx Cookbook
  • Nginx HTTP Server
  • Nginx High Performance
  • Mastering Nginx
  • ModSecurity 3.0 and NGINX: Quick Start Guide
  • Cisco ACE to NGINX: Migration Guide
• External Resources
  • Nginx official
  • Nginx distributions
  • Comparison reviews
  • Cheatsheets & References
  • Performance & Hardening
  • Presentations
  • Playgrounds
  • Config generators
  • Static analyzers
  • Log analyzers
  • Performance analyzers
  • Benchmarking tools
  • Debugging tools
  • Development
  • Online tools
  • Other stuff
• Helpers
  • Directories and files
  • Commands
  • Processes
  • Configuration syntax
    • Comments
    • Variables & Strings
    • Directives, Blocks, and Contexts
    • External files
    • Measurement units
    • Enable syntax highlighting
  • Connection processing
    • Event-Driven architecture
    • Multiple processes
    • Simultaneous connections
  • Request processing stages
  • Server blocks logic
    • Handle incoming connections
    • Matching location
    • rewrite vs return
    • try_files directive
    • if, break and set
  • Log files
    • Conditional logging
    • Manually log rotation
  • Error log severity levels
  • Load balancing algorithms
    • Backend parameters
    • Round Robin
    • Weighted Round Robin
    • Least Connections
    • Weighted Least Connections
    • IP Hash
    • Generic Hash
    • Other methods
  • Rate limiting
  • Analyse configuration
  • Monitoring
    • GoAccess
      • Build and install
      • Analyse log file and enable all recorded statistics
      • Analyse compressed log file
      • Analyse log file remotely
      • Analyse log file and generate html report
    • Ngxtop
      • Analyse log file
      • Analyse log file and print requests with 4xx and 5xx
      • Analyse log file remotely
  • Testing
    • Send request and show response headers
    • Send request with http method, user-agent, follow redirects and show response headers
    • Send multiple requests
    • Testing SSL connection
    • Testing SSL connection with SNI support
    • Testing SSL connection with specific SSL version
    • Testing SSL connection with specific cipher
    • Load testing with ApacheBench (ab)
      • Standard test
      • Test with KeepAlive header
    • Load testing with wrk2
      • Standard scenarios
      • POST call (with Lua)
      • Random paths (with Lua)
      • Multiple paths (with Lua)
      • Random server address to each thread (with Lua)
      • Multiple json requests (with Lua)
      • Debug mode (with Lua)
      • Analyse data pass to and from the threads
      • Parsing wrk result and generate report
    • Load testing with locust
      • Multiple paths
      • Multiple paths with different user sessions
    • TCP SYN flood Denial of Service attack
    • HTTP Denial of Service attack
  • Debugging
    • Show information about processes
    • Check memory usage
    • Show open files
    • Dump configuration
    • Get the list of configure arguments
    • Check if the module has been compiled
    • Show the most accessed IP addresses
    • Show the top 5 visitors (IP addresses)
    • Show the most requested urls
    • Show the most requested urls containing 'string'
    • Show the most requested urls with http methods
    • Show the most accessed response codes
    • Analyse web server log and show only 2xx http codes
    • Analyse web server log and show only 5xx http codes
    • Show requests which result 502 and sort them by number per requests by url
    • Show requests which result 404 for php files and sort them by number per requests by url
    • Calculating amount of http response codes
    • Calculating requests per second
    • Calculating requests per second with IP addresses
    • Calculating requests per second with IP addresses and urls
    • Get entries within last n hours
    • Get entries between two timestamps (range of dates)
    • Get line rates from web server log
    • Trace network traffic for all processes
    • List all files accessed by a NGINX
    • Check that the gzip_static module is working
    • Which worker processing current request
    • Capture only http packets
    • Extract User Agent from the http packets
    • Capture only http GET and POST packets
    • Capture requests and filter by source ip and destination port
    • Dump a process's memory
    • GNU Debugger (gdb)
      • Dump configuration from a running process
      • Show debug log in memory
      • Core dump backtrace
  • Shell aliases
  • Configuration snippets
    • Restricting access with basic authentication
    • Blocking/allowing IP addresses
    • Blocking referrer spam
    • Limiting referrer spam
    • Limiting the rate of requests with burst mode
    • Limiting the rate of requests with burst mode and nodelay
    • Limiting the number of connections
    • Adding and removing the www prefix
    • Redirect POST request with payload to external endpoint
    • Allow multiple cross-domains using the CORS headers
  • Other snippets
    • Create a temporary static backend
    • Create a temporary static backend with SSL support
    • Generate private key without passphrase
    • Generate CSR
    • Generate CSR (metadata from existing certificate)
    • Generate CSR with -config param
    • Generate private key and CSR
    • Generate ECDSA private key
    • Generate private key with CSR (ECC)
    • Generate self-signed certificate
    • Generate self-signed certificate from existing private key
    • Generate self-signed certificate from existing private key and csr
    • Generate multidomain certificate
    • Generate wildcard certificate
    • Generate certificate with 4096 bit private key
    • Generate DH Param key
    • Convert DER to PEM
    • Convert PEM to DER
    • Verification of the private key
    • Verification of the public key
    • Verification of the certificate
    • Verification of the CSR
    • Check whether the private key and the certificate match
  • Installation from prebuilt packages
    • RHEL7 or CentOS 7
    • Debian or Ubuntu
  • Installation from source
    • Automatic installation
    • Nginx package
    • Dependencies
    • 3rd party modules
    • Compiler and linker
      • Debugging Symbols
    • SystemTap
      • stapxx
    • Install Nginx on CentOS 7
      • Pre installation tasks
      • Install or build dependencies
      • Get Nginx sources
      • Download 3rd party modules
      • Build Nginx
      • Post installation tasks
    • Install OpenResty on CentOS 7
    • Install Tengine on Ubuntu 18.04
• Base Rules
  • Organising Nginx configuration
  • Format, prettify and indent your Nginx code
  • Use reload method to change configurations on the fly
  • Separate listen directives for 80 and 443
  • Define the listen directives explicitly with address:port pair
  • Prevent processing requests with undefined server names
  • Use only one SSL config for the listen directive
  • Use geo/map modules instead allow/deny
  • Map all the things...
  • Drop the same root inside location block
  • Configure log rotation policy
• Debugging
  • Use debug mode to track down unexpected behaviour
  • Use custom log formats
  • Memory analysis from core dumps
• Performance
  • Adjust worker processes
  • Use HTTP/2
  • Maintaining SSL sessions
  • Use exact names in a server_name directive where possible
  • Avoid checks server_name with if directive
  • Use try_files directive to ensure a file exists
  • Use return directive instead rewrite for redirects
  • Make an exact location match to speed up the selection process
  • Use limit_conn to improve limiting the download speed
• Hardening
  • Always keep NGINX up-to-date
  • Run as an unprivileged user
  • Disable unnecessary modules
  • Protect sensitive resources
  • Hide Nginx version number
  • Hide Nginx server signature
  • Hide upstream proxy headers
  • Force all connections over TLS
  • Use only the latest supported OpenSSL version
  • Use min. 2048-bit private keys
  • Keep only TLS 1.3 and TLS 1.2
  • Use only strong ciphers
  • Use more secure ECDH Curve
  • Use strong Key Exchange
  • Defend against the BEAST attack
  • Mitigation of CRIME/BREACH attacks)
  • HTTP Strict Transport Security
  • Reduce XSS risks (Content-Security-Policy)
  • Control the behaviour of the Referer header (Referrer-Policy)
  • Provide clickjacking protection (X-Frame-Options)
  • Prevent some categories of XSS attacks (X-XSS-Protection)
  • Prevent Sniff Mimetype middleware (X-Content-Type-Options)
  • Deny the use of browser features (Feature-Policy)
  • Reject unsafe HTTP methods
  • Prevent caching of sensitive data
  • Control Buffer Overflow attacks
  • Mitigating Slow HTTP DoS attacks (Closing Slow Connections)
• Reverse Proxy
• Load Balancing
  • Tweak passive health checks
  • Don't disable backends by comments, use down parameter
• Others
  • Enable DNS CAA Policy
  • Define security policies with security.txt
• Configuration Examples
  • Reverse Proxy
    • Installation
    • Configuration
    • Import configuration
    • Set bind IP address
    • Set your domain name
    • Regenerate private keys and certs
    • Update modules list
    • Generating the necessary error pages
    • Add new domain
    • Test your configuration

Before you start playing with NGINX please read an official Beginner’s Guide. It's a great introduction for everyone.

Nginx (/ˌɛndʒɪnˈɛks/ EN-jin-EKS, stylized as NGINX or nginx) is an open source HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server. It is originally written by Igor Sysoev. For a long time, it has been running on many heavily loaded Russian sites including Yandex, Mail.Ru, VK, and Rambler. In the April 2019 NGINX was the most commonly used HTTP server (see Netcraft survey).

NGINX is a fast, light-weight and powerful web server that can also be used as a:

• fast HTTP reverse proxy
• reliable load balancer
• high performance caching server
• full-fledged web platform

Generally it provides the core of complete web stacks and is designed to help build scalable web applications. When it comes to performance NGINX can easily handle a huge amount of traffic. The other main advantage of the NGINX is that allows you to do the same thing in different ways.

For me, it is a one of the best and most important service that I used in my SysAdmin career.

These essential documents should be the main source of knowledge for you:

• Getting Started
• NGINX Documentation

In addition, I would like to recommend two great docs focuses on the concept of the HTTP protocol:

• HTTP Made Really Easy
• Hypertext Transfer Protocol Specification

If you love security keep your eye on this one: Cryptology ePrint Archive. It provides access to recent research in cryptology and explores many subjects of security (e.g. Ciphers, Algorithms, SSL/TLS protocols).

When I was studying architecture of HTTP servers I became interested in NGINX. I found a lot of information about it but I've never found one guide that covers the most important things in suitable form. I was a little disappointed.

I was interested in everything: NGINX's internals, functions, security best practices, performance optimisations, tips & tricks, hacks and rules, but all documents treated the subject lightly.

Of course, I know that we also have great resources like Official Documentation, agentzh's Nginx Tutorials, Nginx Guts or Emiller’s Advanced Topics In Nginx Module Development. These are definitely the best assets for us and in the first place you should seek help there.

For me, however, there hasn't been a truly in-depth and reasonably simple cheatsheet which describe a variety of configurations and important cross-cutting topics for HTTP servers. That's why I created this repository.

This handbook is a collection of rules, helpers, notes and papers, best practices and recommendations collected and used by me (also in production environments). Many of these refer to external resources.

Throughout this handbook you will explore the many features and capabilities of the NGINX. You'll find out, for example, how to testing the performance or how to resolve debugging problems. You will learn configuration guidelines, security design patterns, ways to handle common issues and how to stay out of them.

In this handbook I added set of guidelines and examples has also been produced to help you administer of the NGINX server. They give us insight into NGINX's internals also.

If you do not have the time to read hundreds of articles (just like me) this multipurpose handbook may be useful. I created it in the hope that it will be useful especially for System Administrators and WebOps. I think it can also be a good complement to official documentations.

I did my best to make this handbook a single and consistent. Of course, I still have a lot to improve and to do. I hope you enjoy it, and fun with it.

Before you start remember about the two most important things:

Do not follow guides just to get 100% of something. Think about what you actually do at your server!

These guidelines provides (in some places) recommendations for very restrictive setup.

A real community, however, exists only when its members interact in a meaningful way that deepens their understanding of each other and leads to learning.

If you find something which doesn't make sense, or something doesn't seem right, please make a pull request and please add valid and well-reasoned explanations about your changes or comments.

Before adding a pull request, please see the contributing guidelines.

If this project is useful and important for you, you can bring positive energy by giving some good words or supporting this project. Thank you!

New chapters:

• Reverse Proxy
• Caching
• 3rd party modules
• Web Application Firewall
• ModSecurity
• Debugging

Existing chapters:

Other stuff:

• Add static error pages generator to NGINX snippets directory

Many of these recipes have been applied to the configuration of my private website.

An example configuration is in configuration examples chapter. It's also based on this version of printable high-res hardening cheatsheets.

Read about SSL Labs grading here (SSL Labs Grading 2018).

Short SSL Labs grades explanation:

A+ is clearly the desired grade, both A and B grades are acceptable and result in adequate commercial security. The B grade, in particular, may be applied to configurations designed to support very wide audiences (for old clients).

I finally got A+ grade and following scores:

• Certificate = 100%
• Protocol Support = 100%
• Key Exchange = 90%
• Cipher Strength = 90%

Read about Mozilla Observatory here.

I also got the highest note from Mozilla:

This checklist contains all rules (54) from this handbook.

Generally, I think that each of these principles is important and should be considered. I tried, however, to separate them into four levels of priority which I hope will help guide your decision.

PRIORITY	NAME	AMOUNT	DESCRIPTION
	critical	22	definitely use this rule, otherwise it will introduce high risks of your NGINX security, performance, and other
	major	18	it's also very important but not critical, and should still be addressed at the earliest possible opportunity
	normal	9	there is no need to implement but it is worth considering because it can improve the NGINX working and functions
	minor	5	as an option to implement or use (not required)

Remember, these are only guidelines. My point of view may be different from yours so if you feel these priority levels do not reflect your configurations commitment to security, performance or whatever else, you should adjust them as you see fit.

RULE	CHAPTER	PRIORITY
Define the listen directives explicitly with address:port pair Prevents soft mistakes which may be difficult to debug.	Base Rules
Prevent processing requests with undefined server names It protects against configuration errors e.g. don't pass traffic to incorrect backends.	Base Rules
Configure log rotation policy Save yourself trouble with your web server: configure appropriate logging policy.	Base Rules
Always keep NGINX up-to-date Use newest NGINX package to fix a vulnerabilities, bugs and to use new features.	Hardening
Run as an unprivileged user Use the principle of least privilege. This way only master process runs as root.	Hardening
Protect sensitive resources Hidden directories and files should never be web accessible.	Hardening
Hide upstream proxy headers Don't expose what version of software is running on the server.	Hardening
Force all connections over TLS Protects your website especially for handle sensitive communications.	Hardening
Use min. 2048-bit private keys 2048 bits private keys are sufficient for commercial use.	Hardening
Keep only TLS 1.3 and TLS 1.2 Use TLS with modern cryptographic algorithms and without protocol weaknesses.	Hardening
Use only strong ciphers Use only strong and not vulnerable cipher suites.	Hardening
Use more secure ECDH Curve Use ECDH Curves with according to NIST recommendations.	Hardening
Use strong Key Exchange Establishes a shared secret between two parties that can be used for secret communication.	Hardening
Defend against the BEAST attack The server ciphers should be preferred over the client ciphers.	Hardening
HTTP Strict Transport Security Tells browsers that it should only be accessed using HTTPS, instead of using HTTP.	Hardening
Reduce XSS risks (Content-Security-Policy) CSP is best used as defence-in-depth. It reduces the harm that a malicious injection can cause.	Hardening
Control the behaviour of the Referer header (Referrer-Policy) The default behaviour of referrer leaking puts websites at risk of privacy and security breaches.	Hardening
Provide clickjacking protection (X-Frame-Options) Defends against clickjacking attack.	Hardening
Prevent some categories of XSS attacks (X-XSS-Protection) Prevents to render pages if a potential XSS reflection attack is detected.	Hardening
Prevent Sniff Mimetype middleware (X-Content-Type-Options) Tells browsers not to sniff MIME types.	Hardening
Reject unsafe HTTP methods Only allow the HTTP methods for which you, in fact, provide services.	Hardening
Prevent caching of sensitive data It helps to prevent critical data (e.g. credit card details, or username) leaked.	Hardening
Organising Nginx configuration	Base Rules
Format, prettify and indent your Nginx code Formatted code is easier to maintain, debug, and can be read and understood in a short amount of time.	Base Rules
Use reload method to change configurations on the fly	Base Rules
Use HTTP/2 HTTP/2 will make our applications faster, simpler, and more robust.	Performance
Maintaining SSL sessions Improves performance from the clients’ perspective.	Performance
Use exact names in a server_name directive where possible	Performance
Avoid checks server_name with if directive Decreases NGINX processing requirements.	Performance
Use try_files directive to ensure a file exists Use it if you need to search for a file, it saving duplication of code also.	Performance
Use return directive instead rewrite for redirects Use return directive to more speedy response than rewrite.	Performance
Disable unnecessary modules Limits vulnerabilities, improve performance and memory efficiency.	Hardening
Hide Nginx version number Don't disclose sensitive information about NGINX.	Hardening
Hide Nginx server signature Don't disclose sensitive information about NGINX.	Hardening
Use only the latest supported OpenSSL version	Hardening
Mitigation of CRIME/BREACH attacks Disable HTTP compression or compress only zero sensitive content.	Hardening
Deny the use of browser features (Feature-Policy) A mechanism to allow and deny the use of browser features.	Hardening
Control Buffer Overflow attacks Prevents errors are characterised by the overwriting of memory fragments of the NGINX process.	Hardening
Mitigating Slow HTTP DoS attacks (Closing Slow Connections) Prevents attacks in which the attacker sends HTTP requests in pieces slowly.	Hardening
Enable DNS CAA Policy Allows domain name holders to indicate to CA whether they are authorized to issue digital certificates.	Others
Separate listen directives for 80 and 443	Base Rules
Use only one SSL config for the listen directive	Base Rules
Use geo/map modules instead allow/deny	Base Rules
Drop the same root inside location block	Base Rules
Adjust worker processes	Performance
Make an exact location match to speed up the selection process	Performance
Use limit_conn to improve limiting the download speed	Performance
Tweak passive health checks	Load Balancing
Define security policies with security.txt	Others
Map all the things...	Base Rules
Use debug mode to track down unexpected behaviour	Debugging
Use custom log formats	Debugging
Memory analysis from core dumps	Debugging
Don't disable backends by comments, use down parameter	Load Balancing

I created two versions of printable posters with hardening cheatsheets (High-Res 5000x8200) based on recipes from this handbook:

For *.xcf and *.pdf formats please see this directory.

• A+ with all 100%’s on @ssllabs and 120/100 on @mozilla observatory:

  It provides the highest scores of the SSL Labs test. Setup is very restrictive with 4096-bit private key, only TLS 1.2 and also modern strict TLS cipher suites (non 128-bits).

• A+ on @ssllabs and 120/100 on @mozilla observatory with TLS 1.3 support:

  It provides less restrictive setup with 2048-bit private key, TLS 1.3 and 1.2 and also modern strict TLS cipher suites (128/256-bits). The final grade is also in line with the industry standards. Recommend using this configuration.

Authors: Valery Kholodkov

Excel in Nginx quickly by learning to use its most essential features in real-life applications.

• Learn how to set up, configure, and operate an Nginx installation for day-to-day use
• Explore the vast features of Nginx to manage it like a pro, and use them successfully to run your website
• Example-based guide to get the best out of Nginx to reduce resource usage footprint

^{This short review comes from this book or the store.}

Authors: Derek DeJonghe

You’ll find recipes for:

• Traffic management and A/B testing
• Managing programmability and automation with dynamic templating and the NGINX Plus API
• Securing access through encrypted traffic, secure links, HTTP authentication subrequests, and more
• Deploying NGINX to AWS, Azure, and Google cloud-computing services
• Using Docker to deploy containers and microservices
• Debugging and troubleshooting, performance tuning, and practical ops tips

^{This short review comes from this book or the store.}

Authors: Martin Fjordvald, Clement Nedelcu

Harness the power of Nginx to make the most of your infrastructure and serve pages faster than ever.

• Discover possible interactions between Nginx and Apache to get the best of both worlds
• Learn to exploit the features offered by Nginx for your web applications
• Get your hands on the most updated version of Nginx (1.13.2) to support all your web administration requirements

^{This short review comes from this book or the store.}

Authors: Rahul Sharma

Optimize NGINX for high-performance, scalable web applications.

• Configure Nginx for best performance, with configuration examples and explanations
• Step–by-step tutorials for performance testing using open source software
• Tune the TCP stack to make the most of the available infrastructure

^{This short review comes from this book or the store.}

Authors: Dimitri Aivaliotis

Written for experienced systems administrators and engineers, this book teaches you from scratch how to configure Nginx for any situation. Step-by-step instructions and real-world code snippets clarify even the most complex areas.

^{This short review comes from this book or the store.}

Authors: Faisal Memon, Owen Garrett, Michael Pleshakov

Learn in this ebook how to get started with ModSecurity, the world’s most widely deployed web application firewall (WAF), now available for NGINX and NGINX Plus.

^{This short review comes from this book or the store.}

Authors: Faisal Memon

This ebook provides step-by-step instructions on replacing Cisco ACE with NGINX and off-the-shelf servers. NGINX helps you cut costs and modernize.

In this ebook you will learn:

• How to migrate Cisco ACE configuration to NGINX, with detailed examples
• Why you should go with a software load balancer, and not hardware

^{This short review comes from this book or the store.}

  :black_small_square: Nginx Project
  :black_small_square: Nginx Documentation
  :black_small_square: Nginx Wiki
  :black_small_square: Nginx Admin's Guide
  :black_small_square: Nginx Pitfalls and Common Mistakes
  :black_small_square: Nginx Forum
  :black_small_square: Nginx Mailing List
  :black_small_square: Nginx Read-only Mirror

  :black_small_square: OpenResty
  :black_small_square: The Tengine Web Server

  :black_small_square: NGINX vs. Apache (Pro/Con Review, Uses, & Hosting for Each)
  :black_small_square: Web cache server performance benchmark: nuster vs nginx vs varnish vs squid

  :black_small_square: agentzh's Nginx Tutorials
  :black_small_square: Nginx Guts
  :black_small_square: Nginx Cheatsheet
  :black_small_square: Nginx Tutorials, Linux Sysadmin Configuration & Optimizing Tips and Tricks
  :black_small_square: Nginx boilerplate configs
  :black_small_square: Awesome Nginx configuration template
  :black_small_square: Nginx Quick Reference
  :black_small_square: A collection of resources covering Nginx and more
  :black_small_square: A collection of useful Nginx configuration snippets

  :black_small_square: Nginx Tuning For Best Performance by Denji
  :black_small_square: TLS has exactly one performance problem: it is not used widely enough
  :black_small_square: SSL/TLS Deployment Best Practices
  :black_small_square: SSL Server Rating Guide
  :black_small_square: How to Build a Tough NGINX Server in 15 Steps
  :black_small_square: Top 25 Nginx Web Server Best Security Practices
  :black_small_square: Nginx Secure Web Server
  :black_small_square: Strong ciphers for Apache, Nginx, Lighttpd and more
  :black_small_square: Strong SSL Security on Nginx
  :black_small_square: Enable cross-origin resource sharing (CORS)
  :black_small_square: NAXSI - WAF for Nginx
  :black_small_square: ModSecurity for Nginx
  :black_small_square: Transport Layer Protection Cheat Sheet
  :black_small_square: Security/Server Side TLS

  :black_small_square: NGINX: Basics and Best Practices
  :black_small_square: NGINX Installation and Tuning
  :black_small_square: Nginx Internals (by Joshua Zhu)
  :black_small_square: Nginx internals (by Liqiang Xu)
  :black_small_square: How to secure your web applications with NGINX
  :black_small_square: Tuning TCP and NGINX on EC2
  :black_small_square: Extending functionality in nginx, with modules!
  :black_small_square: Nginx - Tips and Tricks.
  :black_small_square: Nginx Scripting - Extending Nginx Functionalities with Lua
  :black_small_square: How to handle over 1,200,000 HTTPS Reqs/Min
  :black_small_square: Using ngx_lua / lua-nginx-module in pixiv

  :black_small_square: NGINX Rate Limit, Burst and nodelay sandbox

  :black_small_square: Nginx config generator on steroids
  :black_small_square: Mozilla SSL Configuration Generator

  :black_small_square: gixy - is a tool to analyze Nginx configuration to prevent security misconfiguration and automate flaw detection.
  :black_small_square: nginx-config-formatter - Nginx config file formatter/beautifier written in Python.
  :black_small_square: nginxbeautifier - format and beautify Nginx config files.
  :black_small_square: nginx-minify-conf - creates a minified version of a Nginx configuration.

  :black_small_square: GoAccess - is a fast, terminal-based log analyzer (quickly analyze and view web server statistics in real time).
  :black_small_square: Graylog - is a leading centralized log management for capturing, storing, and enabling real-time analysis.
  :black_small_square: Logstash - is an open source, server-side data processing pipeline.

  :black_small_square: ngxtop - parses your Nginx access log and outputs useful, top-like, metrics of your Nginx server.

  :black_small_square: ab - is a single-threaded command line tool for measuring the performance of HTTP web servers.
  :black_small_square: siege - is an http load testing and benchmarking utility.
  :black_small_square: wrk - is a modern HTTP benchmarking tool capable of generating significant load.
  :black_small_square: wrk2 - is a constant throughput, correct latency recording variant of wrk.
  :black_small_square: bombardier - is a HTTP(S) benchmarking tool.
  :black_small_square: gobench - is a HTTP/HTTPS load testing and benchmarking tool.
  :black_small_square: hey - is a HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom.
  :black_small_square: boom - is a script you can use to quickly smoke-test your web app deployment.
  :black_small_square: httperf - the httperf HTTP load generator.
  :black_small_square: JMeter™ - is designed to load test functional behavior and measure performance.
  :black_small_square: Gatling - is a powerful open-source load and performance testing tool for web applications.
  :black_small_square: locust - is an easy-to-use, distributed, user load testing tool.
  :black_small_square: slowloris - low bandwidth DoS tool. Slowloris rewrite in Python.
  :black_small_square: slowhttptest - application layer DoS attack simulator.
  :black_small_square: GoldenEye - GoldenEye Layer 7 (KeepAlive+NoCache) DoS test tool.

  :black_small_square: strace - is a diagnostic, debugging and instructional userspace utility (linux syscall tracer) for Linux.
  :black_small_square: GDB - allows you to see what is going on `inside' another program while it executes.
  :black_small_square: SystemTap - provides infrastructure to simplify the gathering of information about the running Linux system.
  :black_small_square: stapxx - simple macro language extentions to SystemTap.
  :black_small_square: htrace.sh - is a simple Swiss Army knife for http/https troubleshooting and profiling.

  :black_small_square: Programming in Lua (first edition)
  :black_small_square: Emiller’s Guide To Nginx Module Development
  :black_small_square: An Introduction To OpenResty (nginx + lua) - Part 1
  :black_small_square: An Introduction To OpenResty - Part 2 - Concepts
  :black_small_square: An Introduction To OpenResty - Part 3

  :black_small_square: SSL Server Test by SSL Labs
  :black_small_square: SSL/TLS Capabilities of Your Browser
  :black_small_square: Test SSL/TLS (PCI DSS, HIPAA and NIST)
  :black_small_square: SSL analyzer and certificate checker
  :black_small_square: Test your TLS server configuration (e.g. ciphers)
  :black_small_square: Scan your website for non-secure content
  :black_small_square: Public Diffie-Hellman Parameter Service/Tool
  :black_small_square: Analyse the HTTP response headers by Security Headers
  :black_small_square: Analyze your website by Mozilla Observatory
  :black_small_square: CAA Record Helper
  :black_small_square: Linting tool that will help you with your site's accessibility, speed, security and more
  :black_small_square: Service to scan and analyse websites
  :black_small_square: Tool from above to either encode or decode a string of text
  :black_small_square: Online translator for search queries on log data
  :black_small_square: Online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript
  :black_small_square: Online tool to learn, build, & test Regular Expressions
  :black_small_square: Online Regex Tester & Debugger
  :black_small_square: A web app for encryption, encoding, compression and data analysis
  :black_small_square: Nginx location match tester
  :black_small_square: Nginx location match visible

  :black_small_square: OWASP Cheat Sheet Series
  :black_small_square: Mozilla Web Security
  :black_small_square: Application Security Wiki
  :black_small_square: OWASP ASVS 4.0
  :black_small_square: Transport Layer Security (TLS) Parameters
  :black_small_square: Security/Server Side TLS by Mozilla
  :black_small_square: TLS Security 6: Examples of TLS Vulnerabilities and Attacks
  :black_small_square: Guidelines for Setting Security Headers
  :black_small_square: Secure your web application with these HTTP headers
  :black_small_square: Security HTTP Headers
  :black_small_square: Analysis of various reverse proxies, cache proxies, load balancers, etc.
  :black_small_square: TLS Redirection (and Virtual Host Confusion)
  :black_small_square: HTTPS on Stack Overflow: The End of a Long Road
  :black_small_square: The Architecture of Open Source Applications - Nginx
  :black_small_square: BBC Digital Media Distribution: How we improved throughput by 4x
  :black_small_square: The C10K problem by Dan Kegel
  :black_small_square: The Secret To 10 Million Concurrent Connections
  :black_small_square: High Performance Browser Networking
  :black_small_square: jemalloc vs tcmalloc vs dlmalloc

If you compile NGINX server by default all files and directories are available from /usr/local/nginx location.

For upstream NGINX packaging paths can be as follows (it depends on the type of system):

• /etc/nginx - is the default configuration root for the NGINX service
  

  • other locations: /usr/local/etc/nginx, /usr/local/nginx/conf

• /etc/nginx/nginx.conf - is the default configuration entry point used by the NGINX services, includes the top-level http block and all other configuration contexts and files
  

  • other locations: /usr/local/etc/nginx/nginx.conf, /usr/local/nginx/conf/nginx.conf

• /usr/share/nginx - is the default root directory for requests, contains html directory and basic static files

• /var/log/nginx - is the default log (access and error log) location for NGINX

  • other locations: logs/ in root directory

• /var/cache/nginx - is the default temporary files location for NGINX
  

  • other locations: /var/lib/nginx

• /etc/nginx/conf - contains custom/vhosts configuration files
  

  • other locations: /etc/nginx/conf.d, /etc/nginx/sites-enabled (I can't stand this debian-like convention...)

• /var/run/nginx - contains information about NGINX process(es)
  

  • other locations: /usr/local/nginx/logs, logs/ in root directory

• nginx -h - shows the help
• nginx -v - shows the NGINX version
• nginx -V - shows the extended information about NGINX: version, build parameters and configuration arguments
• nginx -t - tests the NGINX configuration
• nginx -c - sets configuration file (default: /etc/nginx/nginx.conf)
• nginx -p - sets prefix path (default: /etc/nginx/)
• nginx -T - tests the NGINX configuration and prints the validated configuration on the screen
• nginx -s - sends a signal to the NGINX master process:
  • stop - discontinues the NGINX process immediately
  • quit - stops the NGINX process after it finishes processing inflight requests
  • reload - reloads the configuration without stopping processes
  • reopen - instructs NGINX to reopen log files
• nginx -g - sets global directives out of configuration file

Some useful snippets for process management:

• testing configuration:

/usr/sbin/nginx -t -c /etc/nginx/nginx.conf
/usr/sbin/nginx -t -q -g 'daemon on; master_process on;' # ; echo $?

• starting daemon:

/usr/sbin/nginx -g 'daemon on; master_process on;'

service nginx start
systemctl start nginx

# You can also start NGINX from start-stop-daemon script:
/sbin/start-stop-daemon --quiet --start --exec /usr/sbin/nginx --background --retry QUIT/5 --pidfile /run/nginx.pid

• stopping daemon:

/usr/sbin/nginx -s quit     # graceful shutdown (waiting for the worker processes to finish serving current requests)
/usr/sbin/nginx -s stop     # fast shutdown (kill connections immediately)

service nginx stopmicro
systemctl stop nginx

# You can also stop NGINX from start-stop-daemon script:
/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid

• reloading daemon:

/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload

service nginx reload
systemctl reload nginx

kill -HUP $(cat /var/run/nginx.pid)
kill -HUP $(pgrep -f "nginx: master")

NGINX uses a micro programming language in the configuration files. This language's design is heavily influenced by Perl and Bourne Shell. For me NGINX's configuration has a simple and very transparent structure.

NGINX's configuration files don't support comment blocks, they only accept # at the beginning of a line for a comment.

Lines containing directives must end with a ; or NGINX will fail to load the configuration and report an error.

Variables start with $. Some modules introduce variables can be used when setting directives.

There are some directives that do not support variables, e.g. access_log or error_log.

To assign value to the variable you should use a set directive:

set $var "value";

To learn more about variables see if, break and set section.

Some interesting things about variables in NGINX:

Make sure to read the agentzh's Nginx Tutorials - it's about NGINX tips & tricks. That guy is a Guru and creator of OpenResty. In these tutorials he describes, amongst other things, variables in great detail.

• the scope of variables spreads out all over configuration
• variable assignment occurs when requests are actually being served
• variable have exactly the same lifetime as the corresponding request
• each request does have its own version of all those variables' containers (different containers values)
• requests do not interfere with each other even if they are referencing a variable with the same name
• the assignment operation is only performed in requests that access location

Strings may be inputted without quotes unless they include blank spaces, semicolons or curly braces, then they need to be escaped with backslashes or enclosed in single/double quotes.

Variables in quoted strings are expanded normally unless the $ is escaped.

Read this great article about the NGINX configuration inheritance model by Martin Fjordvald.

Configuration options are called directives. We have four types of directives:

• standard directive - one value per context:

  worker_connections 512;

• array directive - multiple values per context:

  error_log /var/log/nginx/localhost/localhost-error.log warn;

• action directive - something which does not just configure:

  rewrite ^(.*)$ /msie/$1 break;

• try_files directive:

  try_files $uri $uri/ /test/index.html;

  If you want to review all directives see alphabetical index of directives.

Directives are organised into groups known as blocks or contexts. Generally context is a block directive can have other directives inside braces. It appears to be organised in a tree-like structure, defined by sets of brackets - { and }.

As a general rule, if a directive is valid in multiple nested scopes, a declaration in a broader context will be passed on to any child contexts as default values.

Directives placed in the configuration file outside of any contexts are considered to be in the global/main context.

Contexts can be layered within one another (a level of inheritance). Their structure looks like this:

Global/Main Context
        |
        |
        +-----» Events Context
        |
        |
        +-----» HTTP Context
        |          |
        |          |
        |          +-----» Server Context
        |          |          |
        |          |          |
        |          |          +-----» Location Context
        |          |
        |          |
        |          +-----» Upstream Context
        |
        |
        +-----» Mail Context

include directive may appear inside any contexts to perform conditional inclusion. It attaching another file, or files matching the specified mask:

include /etc/nginx/proxy.conf

Sizes can be specified in:

• k or K: Kilobytes
• m or M: Megabytes
• g or G: Gigabytes

client_max_body_size 2M;

Time intervals can be specified in:

• ms: Milliseconds
• s: Seconds (default, without a suffix)
• m: Minutes
• h: Hours
• d: Days
• w: Weeks
• M: Months (30 days)
• y: Years (365 days)

proxy_read_timeout 20s;

# 1) Download vim plugin for NGINX:

# Official NGINX vim plugin:
mkdir -p ~/.vim/syntax/

wget "http://www.vim.org/scripts/download_script.php?src_id=19394" -O ~/.vim/syntax/nginx.vim

# Improved NGINX vim plugin (incl. syntax highlighting) with Pathogen:
mkdir -p ~/.vim/{autoload,bundle}/

curl -LSso ~/.vim/autoload/pathogen.vim https://tpo.pe/pathogen.vim
echo -en "\nexecute pathogen#infect()\n" >> ~/.vimrc

git clone https://github.com/chr4/nginx.vim ~/.vim/bundle/nginx.vim

# 2) Set location of NGINX config files:
cat > ~/.vim/filetype.vim << __EOF__
au BufRead,BufNewFile /etc/nginx/*,/etc/nginx/conf.d/*,/usr/local/nginx/conf/*,*/conf/nginx.conf if &ft == '' | setfiletype nginx | endif
__EOF__

It may be interesting for you: Highlight insecure SSL configuration in Vim.

Install cabal - system for building and packaging Haskell libraries and programs (on Ubuntu):

add-apt-repository -y ppa:hvr/ghc
apt-get update

apt-get install -y cabal-install-1.22 ghc-7.10.2

# Add this to your shell main configuration file:
export PATH=$HOME/.cabal/bin:/opt/cabal/1.22/bin:/opt/ghc/7.10.2/bin:$PATH
source $HOME/.<shellrc>

cabal update

• nginx-lint:

  git clone https://github.com/temoto/nginx-lint
  
  cd nginx-lint && cabal install --global

• sublime-nginx + SublimeLinter-contrib-nginx-lint:

  Bring up the Command Palette and type install. Among the commands you should see Package Control: Install Package. Type nginx to install sublime-nginx and after that do the above again for install SublimeLinter-contrib-nginx-lint: type SublimeLinter-contrib-nginx-lint.

NGINX has one master process and one or more worker processes.

The main purposes of the master process is to read and evaluate configuration files, as well as maintain the worker processes (respawn when a worker dies), handle signals, notify workers, opens log files, and, of course binding to ports.

Master process should be started as root user, because this will allow NGINX to open sockets below 1024 (it needs to be able to listen on port 80 for HTTP and 443 for HTTPS).

The worker processes do the actual processing of requests and get commands from master process. They runs in an event loop (registering events and responding when one occurs), handle network connections, read and write content to disk, and communicate with upstream servers. These are spawned by the master process, and the user and group will as specified (unprivileged).

NGINX has also cache loader and cache manager processes but only if you enable caching.

The following signals can be sent to the NGINX master process:

There’s no need to control the worker processes yourself. However, they support some signals too:

NGINX supports a variety of connection processing methods which depends on the platform used.

In general there are four types of event multiplexing:

• select - is anachronism and not recommended but installed on all platforms as a fallback
• poll - is anachronism and not recommended

And the most efficient implementations of non-blocking I/O:

• epoll - recommend if you're using GNU/Linux
• kqueue - recommend if you're using BSD (is technically superior to epoll)

There are also great resources (also makes comparisons) about them:

• Kqueue: A generic and scalable event notification facility
• poll vs select vs event-based
• select/poll/epoll: practical difference for system architects
• Scalable Event Multiplexing: epoll vs. kqueue
• Async IO on Linux: select, poll, and epoll
• A brief history of select(2)
• Select is fundamentally broken
• Epoll is fundamentally broken
• I/O Multiplexing using epoll and kqueue System Calls
• Benchmarking BSD and Linux
• The C10K problem

Look also at libevent benchmark (read about libevent – an event notification library):

^{This infographic comes from daemonforums - An interesting benchmark (kqueue vs. epoll).}

You may also view why big players use NGINX on FreeBSD instead of on GNU/Linux:

• FreeBSD NGINX Performance
• Why did Netflix use NGINX and FreeBSD to build their own CDN?

Thread Pools in NGINX Boost Performance 9x! - this official article is an amazing explanation about thread pools and generally about handling connections. I also recommend Inside NGINX: How We Designed for Performance & Scale. Both are really great.

NGINX uses Event-Driven architecture which heavily relies on Non-Blocking I/O. One advantage of non-blocking/asynchronous operations is that you can maximize the usage of a single CPU as well as memory because is that your thread can continue it's work in parallel.

There is a perfectly good and brief summary about non-blocking I/O and multi-threaded blocking I/O by Werner Henze.

Look what the official documentation says about it:

It’s well known that NGINX uses an asynchronous, event‑driven approach to handling connections. This means that instead of creating another dedicated process or thread for each request (like servers with a traditional architecture), it handles multiple connections and requests in one worker process. To achieve this, NGINX works with sockets in a non‑blocking mode and uses efficient methods such as epoll and kqueue.

Because the number of full‑weight processes is small (usually only one per CPU core) and constant, much less memory is consumed and CPU cycles aren’t wasted on task switching. The advantages of such an approach are well‑known through the example of NGINX itself. It successfully handles millions of simultaneous requests and scales very well.

I must not forget to mention here about Non-Blocking and 3rd party modules (from official documentation):

Unfortunately, many third‑party modules use blocking calls, and users (and sometimes even the developers of the modules) aren’t aware of the drawbacks. Blocking operations can ruin NGINX performance and must be avoided at all costs.

To handle concurrent requests with a single worker process NGINX uses the reactor design pattern. Basically, it's a single-threaded but it can fork several processes to utilize multiple cores.

However, NGINX is not a single threaded application. Each of worker processes is single-threaded and can handle thousands of concurrent connections. NGINX does not create a new process/thread for each connection/requests but it starts several worker threads during start. It does this asynchronously with one thread, rather than using multi-threaded programming (it uses an event loop with asynchronous I/O).

That way, the I/O and network operations are not a very big bottleneck (remember that your CPU would spend a lot of time waiting for your network interfaces, for example). This results from the fact that NGINX only use one thread to service all requests. When requests arrive at the server, they are serviced one at a time. However, when the code serviced needs other thing to do it sends the callback to the other queue and the main thread will continue running (it doesn't wait).

Now you see why NGINX can handle a large amount of requests perfectly well (and without any problems).

For more information take a look at following resources:

• Asynchronous, Non-Blocking I/O
• About High Concurrency, NGINX architecture and internals
• A little holiday present: 10,000 reqs/sec with Nginx!
• Nginx vs Apache: Is it fast, if yes, why?
• How is Nginx handling its requests in terms of tasks or threading?
• Why nginx is faster than Apache, and why you needn’t necessarily care

NGINX uses only asynchronous I/O, which makes blocking a non-issue. The only reason NGINX uses multiple processes is to make full use of multi-core, multi-CPU and hyper-threading systems. NGINX requires only enough worker processes to get the full benefit of symmetric multiprocessing (SMP).

From NGINX documentation:

The NGINX configuration recommended in most cases – running one worker process per CPU core – makes the most efficient use of hardware resources.

NGINX uses a custom event loop which was designed specifically for NGINX - all connections are processed in a highly efficient run-loop in a limited number of single-threaded processes called workers.

Multiplexing works by using a loop to increment through a program chunk by chunk operating on one piece of data/new connection/whatever per connection/object per loop iteration. It is all based on events multiplexing like epoll(), kqueue() or select(). Within each worker NGINX can handle many thousands of concurrent connections and requests per second.

See Nginx Internals presentation as a lot of great stuff about the internals of NGINX.

NGINX does not fork a process or thread per connection (like Apache) so memory usage is very conservative and extremely efficient in the vast majority of cases. NGINX is a faster and consumes less memory than Apache. It is also very friendly for CPU because there's no ongoing create-destroy pattern for processes or threads.

Finally and in summary:

• uses Non-Blocking "Event-Driven" architecture
• uses the single-threaded reactor pattern to handle concurrent requests
• uses highly efficient loop for connection processing
• is not a single threaded application because it starts multiple worker processes (to handle multiple connections and requests) during start

Okay, so how many simultaneous connections can be processed by NGINX?

worker_processes * worker_connections = max connections

According to this: if you are running 4 worker processes with 4096 worker connections per worker, you will be able to serve 16384 connections.

I've seen some admins does directly translate the sum of worker_processes and worker_connections into the number of clients that can be served simultaneously. In my opinion it is mistake because each clients (e.g. browsers) opens a number of parallel connections to download various components that compose a web page, for example, images, scripts, and so on.

Additionally, you must know that worker_connections directive includes all connections (e.g. connections with proxied servers, among others), not only connections with clients.

Be aware that every worker connection (in the sleeping state) needs 256 bytes of memory, so you can increase it easily.

The number of connections is limited by the maximum number of open files (RLIMIT_NOFILE) on your system. To change the limit of the maximum file descriptors that can be opened by a single worker process (as oppose to the user running NGINX) you can edit the worker_rlimit_nofile directive (with this there's no need to restarting the main process).

A file descriptor is an opaque handle that is used in the interface between user and kernel space to identify file/socket resources.

I think that the chance of running out of file descriptors is minimal. However, you should know the following important rules:

• before increasing the number of worker_processes or worker_connections verify the open file limit, for this, following commands will be useful for you:

  # List all file descriptors in kernel memory:
  #   first value:  <allocated file handles>
  #  second value:  <unused-but-allocated file handles>
  #   third value:  <the system-wide max number of file handles>
  sysctl fs.file-nr
  
  # Find out the system-wide maximum number of file handles:
  sysctl fs.file-max
  
  # Current open file descriptors per NGINX worker process:
  for _pid in $(pgrep -f "nginx: worker") ; do
  
    echo -en "\n\n##### per worker pid: $_pid #####\n\n"
  
    # List files from proc directory:
    #   - ls -l /proc/${_pid}/fd
    ls /proc/${_pid}/fd | wc -l
  
    # List all open files (files, memory mapped files):
    lsof -as -p $_pid | awk '{if(NR>1)print}'
  
  done

• if you have SELinux enabled, you will need to run setsebool -P httpd_setrlimit 1 so that NGINX has permissions to set its rlimit. To diagnose SELinux denials and attempts you can use sealert -a /var/log/audit/audit.log.

• worker_rlimit_nofile serves to dynamically change the maximum file descriptors the NGINX worker process can handle, which is typically defined with the system's soft limit

• worker_rlimit_nofile works only at the process level, it's limited to the system's hard limit (ulimit -Hn)

So if you don't set this directive manually, then the OS settings will determine how many file descriptors can be used by NGINX.

Ok, so how many fds are opens by NGINX?

• one file handler for the client's active connection
• one file handler for opening file (e.g. static file)
• one file handler for the proxied connection (that will open a socket handling these requests to remote or local host/process)

Also important is:

NGINX can use up to two file descriptors per full-fledged connection.

Look also at these diagrams:

• 1 file handler for connection with client and 1 file handler for static file being served by NGINX:

                       +-----------------+
  +----------+         |                 |
  |          |    1    |                 |
  |  CLIENT <---------------> NGINX      |
  |          |         |        ^        |
  +----------+         |        |        |
                       |      2 |        |
                       |        |        |
                       |        |        |
                       | +------v------+ |
                       | | STATIC FILE | |
                       | +-------------+ |
                       +-----------------+
  

• 1 file handler for connection with client and 1 file handler for a open socket to the remote or local host/process:

                       +-----------------+
  +----------+         |                 |         +-----------+
  |          |    1    |                 |    2    |           |
  |  CLIENT <---------------> NGINX <---------------> BACKEND  |
  |          |         |                 |         |           |
  +----------+         |                 |         +-----------+
                       +-----------------+
  

• 2 file handlers for two simultaneous connections from the same client (1, 4), 1 file handler for connection with other client (3), 2 file handlers for static files (2, 5), and 1 file handler for a open socket to the remote or local host/process (6), so in total it is 6 file descriptors:

                    4
        +-----------------------+
        |              +--------|--------+
  +-----v----+         |        |        |
  |          |    1    |        v        |  6
  |  CLIENT <-----+---------> NGINX <---------------+
  |          |    |    |        ^        |    +-----v-----+
  +----------+    |    |        |        |    |           |
                3 |    |      2 | 5      |    |  BACKEND  |
  +----------+    |    |        |        |    |           |
  |          |    |    |        |        |    +-----------+
  |  CLIENT  <----+    | +------v------+ |
  |          |         | | STATIC FILE | |
  +----------+         | +-------------+ |
                       +-----------------+
  

I the first two examples: we can take that NGINX needs 2 file handlers for full-fledged connection (but still uses 2 worker connections). In the third example NGINX can take still 2 file handlers for every full-fledged connection (also if client uses parallel connections).

I think that the correct value of worker_rlimit_nofile per all connections of worker is:

worker_rlimit_nofile = worker_connections

So maximum number of open files by the NGINX should be:

worker_processes * worker_rlimit_nofile + (shared libs, log files, event pool etc.) = max open files

To serve 16384 connections by all workers, and bearing in mind about the other handlers used by NGINX, a reasonably value of max files handlers in this case may be 20000. I think it's more than enough.

To change/improve the limitations you should:

# Add to /etc/sysctl.d/99-fs.conf (system-wide value):
fs.file-max = 50000

# Add to /etc/security/limits.conf:
nginx       soft    nofile    10000
nginx       hard    nofile    20000

# Update worker_rlimit_nofile in nginx.conf within the main context:
worker_rlimit_nofile          20000;

You can test the hard and soft limits applying to the NGINX process with this: grep "Max open files" /proc/$(pgrep -f "nginx: master")/limits.

There is a great article about Optimizing Nginx for High Traffic Loads.

There can be altogether 11 phases when NGINX handles (processes) a request:

• NGX_HTTP_POST_READ_PHASE - first phase, read the request header

  • example modules: ngx_http_realip_module

• NGX_HTTP_SERVER_REWRITE_PHASE - implementation of rewrite directives defined in a server block; to change request URI using PCRE regular expressions, return redirects, and conditionally select configurations

  • example modules: ngx_http_rewrite_module

• NGX_HTTP_FIND_CONFIG_PHASE - replace the location according to URI (location lookup)

• NGX_HTTP_REWRITE_PHASE - URI transformation on location level

  • example modules: ngx_http_rewrite_module

• NGX_HTTP_POST_REWRITE_PHASE - URI transformation post-processing (the request is redirected to a new location)

  • example modules: ngx_http_rewrite_module

• NGX_HTTP_PREACCESS_PHASE - authentication preprocessing request limit, connection limit (access restriction)

  • example modules: ngx_http_limit_req_module, ngx_http_limit_conn_module, ngx_http_realip_module

• NGX_HTTP_ACCESS_PHASE - verification of the client (the authentication process, limiting access)

  • example modules: ngx_http_access_module, ngx_http_auth_basic_module

• NGX_HTTP_POST_ACCESS_PHASE - access restrictions check post-processing phase, the certification process, processing satisfy any directive

  • example modules: ngx_http_access_module, ngx_http_auth_basic_module

• NGX_HTTP_PRECONTENT_PHASE - generating content

  • example modules: ngx_http_try_files_module

• NGX_HTTP_CONTENT_PHASE - content processing

  • example modules: ngx_http_index_module, ngx_http_autoindex_module, ngx_http_gzip_module

• NGX_HTTP_LOG_PHASE - log processing

  • example modules: ngx_http_log_module

You may feel lost now (me too...) so I let myself put this great and simple preview:

^{This infographic comes from Inside NGINX official library.}

NGINX does have server blocks (like a virtual hosts in an Apache) that use listen and server_name directives to bind to tcp sockets.

Before start reading this chapter you should know what regular expressions are and how they works. I recommend two great and short write-ups about regular expressions created by Jonny Fox:

• Regex tutorial — A quick cheatsheet by examples
• Regex cookbook — Top 10 Most wanted regex

Why? Regular expressions can be used in both the server_name and location (also in other) directives, and sometimes you must have a great skill of reading them. I think you should create the most readable regular expressions that do not become spaghetti code - impossible to debug and maintain.

It's a short example of server block context (two server blocks):

http {

  index index.html;
  root /var/www/example.com/default;

  server {

    listen 10.10.250.10:80;
    server_name www.example.com;

    access_log logs/example.access.log main;

    root /var/www/example.com/public;

    ...

  }

  server {

    listen 10.10.250.11:80;
    server_name "~^(api.)?example\.com api.de.example.com";

    access_log logs/example.access.log main;

    proxy_pass http://localhost:8080;

    ...

  }

}

NGINX uses the following logic to determining which virtual server (server block) should be used:

1. Match the address:port pair to the listen directive - that can be multiple server blocks with listen directives of the same specificity that can handle the request

NGINX use the address:port combination for handle incoming connections. This pair is assigned to the listen directive.

The listen directive can be set to:

• an IP address/port combination (127.0.0.1:80;)

• a lone IP address, if only address is given, the port 80 is used (127.0.0.1;) - becomes 127.0.0.1:80;

• a lone port which will listen to every interface on that port (80; or *:80;) - becomes 0.0.0.0:80;

• the path to a UNIX-domain socket (unix:/var/run/nginx.sock;)

If the listen directive is not present then either *:80 is used (runs with the superuser privileges), or *:8000 otherwise.

The next steps are as follows:

- NGINX translates all incomplete `listen` directives by substituting missing values with their default values (see above)

- NGINX attempts to collect a list of the server blocks that match the request most specifically based on the `address:port`

- If any block that is functionally using `0.0.0.0`, will not be selected if there are matching blocks that list a specific IP

- If there is only one most specific match, that server block will be used to serve the request

- If there are multiple server blocks with the same level of matching, NGINX then begins to evaluate the `server_name` directive of each server block

Look at this short example:

# From client side:
GET / HTTP/1.0
Host: api.random.com

# From server side:
server {

  # This block will be processed:
  listen 192.168.252.10;  # --> 192.168.252.10:80

  ...

}

server {

  listen 80;  # --> *:80 --> 0.0.0.0:80
  server_name api.random.com;

  ...

}

2. Match the Host header field against the server_name directive as a string (the exact names hash table)

3. Match the Host header field against the server_name directive with a wildcard at the beginning of the string (the hash table with wildcard names starting with an asterisk)

If one is found, that block will be used to serve the request. If multiple matches are found, the longest match will be used to serve the request.

4. Match the Host header field against the server_name directive with a wildcard at the end of the string (the hash table with wildcard names ending with an asterisk)

If one is found, that block is used to serve the request. If multiple matches are found, the longest match will be used to serve the request.

5. Match the Host header field against the server_name directive as a regular expression

The first server_name with a regular expression that matches the Host header will be used to serve the request.

6. If all the Host headers doesn't match, then direct to the listen directive marked as default_server

7. If all the Host headers doesn't match and there is no default_server, direct to the first server with a listen directive that satisfies first step

8. Finally, NGINX goes to the location context

^{This list is based on Mastering Nginx - The virtual server section.}

For each request, NGINX goes through a process to choose the best location block that will be used to serve that request.

The location syntax looks like:

location optional_modifier location_match { ... }

location_match in the above defines what NGINX should check the request URI against. The optional_modifier below will cause the associated location block to be interpreted as follows:

• (none): if no modifiers are present, the location is interpreted as a prefix match. To determine a match, the location will now be matched against the beginning of the URI

• =: is an exact match, without any wildcards, prefix matching or regular expressions; forces a literal match between the request URI and the location parameter

• ~: if a tilde modifier is present, this location must be used for case sensitive matching (RE match)

• ~*: if a tilde and asterisk modifier is used, the location must be used for case insensitive matching (RE match)

• ^~: assuming this block is the best non-RE match, a carat followed by a tilde modifier means that RE matching will not take place

And now, a short introduction to determines location priority:

• the exact match is the best priority (processed first); ends search if match

• the prefix match is the second priority; there are two types of prefixes: ^~ and (none), if this match used the ^~ prefix, searching stops

• the regular expression match has the lowest priority; there are two types of prefixes: ~ and ~*; in the order they are defined in the configuration file

• if regular expression searching yielded a match, that result is used, otherwise, the match from prefix searching is used

So look at this example, it comes from the Nginx documentation - ngx_http_core_module:

location = / {
  # Matches the query / only.
  [ configuration A ]
}
location / {
  # Matches any query, since all queries begin with /, but regular
  # expressions and any longer conventional blocks will be
  # matched first.
  [ configuration B ]
}
location /documents/ {
  # Matches any query beginning with /documents/ and continues searching,
  # so regular expressions will be checked. This will be matched only if
  # regular expressions don't find a match.
  [ configuration C ]
}
location ^~ /images/ {
  # Matches any query beginning with /images/ and halts searching,
  # so regular expressions will not be checked.
  [ configuration D ]
}
location ~* \.(gif|jpg|jpeg)$ {
  # Matches any request ending in gif, jpg, or jpeg. However, all
  # requests to the /images/ directory will be handled by
  # Configuration D.
  [ configuration E ]
}

To help you understand how does location match works:

• Nginx location match tester
• Nginx location match visible

The process of choosing NGINX location block is as follows (a detailed explanation):

1. Prefix-based NGINX location matches (no regular expression). Each location will be checked against the request URI

2. NGINX searches for an exact match. If a = modifier exactly matches the request URI, this specific location block is chosen right away

3. If no exact (meaning no = modifier) location block is found, NGINX will continue with non-exact prefixes. It starts with the longest matching prefix location for this URI, with the following approach:

• In case the longest matching prefix location has the ^~ modifier, NGINX will stop its search right away and choose this location.

• Assuming the longest matching prefix location doesn’t use the ^~ modifier, the match is temporarily stored and the process continues.

4. As soon as the longest matching prefix location is chosen and stored, NGINX continues to evaluate the case-sensitive and insensitive regular expression locations. The first regular expression location that fits the URI is selected right away to process the request

5. If no regular expression locations are found that match the request URI, the previously stored prefix location is selected to serve the request

In order to better understand how this process work please see this short cheatsheet that will allow you to design your location blocks in a predictable way:

I recommend to use external tools for testing regular expressions. For more please see online tools chapter.

In conclusion, location picking order is as follows:

1. = - exactly, e.g. location = /path

2. ^~ - forward match, e.g. location ^~ /path

3. ~ - regular expression case sensitive, e.g. location ~ /path/

4. ~* - regular expression case insensitive, e.g. location ~* .(jpg|png|svg)

5. /, e.g. location /path

Ok, so here's a more complicated configuration:

server {

 listen           80;
 server_name      xyz.com www.xyz.com;

 location ~ ^/(media|static)/ {
  root            /var/www/xyz.com/static;
  expires         10d;
 }

 location ~* ^/(media2|static2) {
  root            /var/www/xyz.com/static2;
  expires         20d;
 }

 location /static3 {
  root            /var/www/xyz.com/static3;
 }

 location ^~ /static4 {
  root            /var/www/xyz.com/static4;
 }

 location = /api {
  proxy_pass      http://127.0.0.1:8080;
 }

 location / {
  proxy_pass      http://127.0.0.1:8080;
 }

 location /backend {
  proxy_pass      http://127.0.0.1:8080;
 }

 location ~ logo.xcf$ {
  root            /var/www/logo;
  expires         48h;
 }

 location ~* .(png|ico|gif|xcf)$ {
  root            /var/www/img;
  expires         24h;
 }

 location ~ logo.ico$ {
  root            /var/www/logo;
  expires         96h;
 }

 location ~ logo.jpg$ {
  root            /var/www/logo;
  expires         48h;
 }

}

And here's the table with the results:

Generally there are two ways of implementing redirects in NGINX: with rewrite and return.

These directives (they come from the ngx_http_rewrite_module) are very useful but (from the NGINX documentation) the only 100% safe things which may be done inside if in a location context are:

• return ...;
• rewrite ... last;

Anything else may possibly cause unpredictable behaviour, including potential SIGSEGV.

The rewrite directives are executed sequentially in order of their appearance in the configuration file. It's slower (but still extremely fast) than a return and returns HTTP 302 in all cases, irrespective of permanent.

Importantly only the part of the original url that matches the regex is rewritten. It can be used for temporary url changes.

I sometimes used rewrite to capture elementes in the original URL, change or add elements in the path, and in general when I do something more complex:

location / {

  ...

  rewrite   ^/users/(.*)$       /user.php?username=$1 last;

  # or:
  rewrite   ^/users/(.*)/items$ /user.php?username=$1&page=items last;

}

rewrite directive accept optional flags:

• break - basically completes processing of rewrite directives, stops processing, and breakes location lookup cycle by not doing any location lookup and internal jump at all

  • if you use break flag inside location block:

    • no more parsing of rewrite conditions
    • internal engine continues to parse the current location block

    Inside a location block, with break, NGINX only stops processing anymore rewrite conditions.

  • if you use break flag outside location block:

    • no more parsing of rewrite conditions
    • internal engine goes to the next phase (searching for location match)

    Outside a location block, with break, NGINX stops processing anymore rewrite conditions.

• last - basically completes processing of rewrite directives, stops processing, and starts a search for a new location matching the changed URI

  • if you use last flag inside location block:

    • no more parsing of rewrite conditions
    • internal engine starts to look for another location match based on the result of the rewrite result
    • no more parsing of rewrite conditions, even on the next location match

    Inside a location block, with last, NGINX stops processing anymore rewrite conditions and then starts to look for a new matching of location block! NGINX also ignores any rewrites in the new location block!

  • if you use last flag outside location block:

    • no more parsing of rewrite conditions
    • internal engine goes to the next phase (searching for location match)

    Outside a location block, with last, NGINX stops processing anymore rewrite conditions.

• redirect - returns a temporary redirect with the 302 HTTP response code

• permanent - returns a permanent redirect with the 301 HTTP response code

Note:

• that outside location blocks, last and break are effectively the same
• processing of rewrite directives at server level may be stopped via break, but the location lookup will follow anyway

^{This explanation is based on the awesome answer by Pothi Kalimuthu to nginx url rewriting: difference between break and last.}

Official documentation has a great tutorials about Creating NGINX Rewrite Rules and Converting rewrite rules.

The other way is a return directive. It's faster than rewrite because there is no regexp that has to be evaluated. It's stops processing and returns HTTP 301 (by default) to a client, and the entire url is rerouted to the url specified.

I use return directive to:

• force redirect from http to https:

  server {
  
    ...
  
    return  301 https://example.com$request_uri;
  
  }

• redirect from www to non-www and vice versa:

  server {
  
    ...
  
    if ($host = www.domain.com) {
  
      return  301 https://domain.com$request_uri;
  
    }
  
  }

• close the connection and log it internally:

  server {
  
    ...
  
    return 444;
  
  }

• send 4xx HTTP response for a client without any other actions:

  server {
  
    ...
  
    if ($request_method = POST) {
  
      return 405;
  
    }
  
    # or:
    if ($invalid_referer) {
  
      return 403;
  
    }
  
    # or:
    if ($request_uri ~ "^/app/(.+)$") {
  
      return 403;
  
    }
  
    # or:
    location ~ ^/(data|storage) {
  
      return 403;
  
    }
  
  }

• and sometimes for reply with HTTP code without serving a file:

  server {
  
    ...
  
    # NGINX will not allow a 200 with no response body (200's need to be with a resource in the response)
    return 204 "it's all okay";
    # Because default Content-Type is application/octet-stream, browser will offer to "save the file".
    # If you want to see reply in browser you should add properly Content-Type:
    # add_header Content-Type text/plain;
  
  }

We have one more very interesting and important directive: try_files (from ngx_http_core_module). This directive tells NGINX to check for the existence of a named set of files or directories (checks files conditionally breaking on success).

I think the best explanation come from official documentation:

try_files checks the existence of files in the specified order and uses the first found file for request processing; the processing is performed in the current context. The path to a file is constructed from the file parameter according to the root and alias directives. It is possible to check directory’s existence by specifying a slash at the end of a name, e.g. $uri/. If none of the files were found, an internal redirect to the uri specified in the last parameter is made.

Generally it may check files on disk, redirect to proxies, or internal locations, and return error codes, all in one directive.

Take a look at the following example:

server {

  ...

  root /var/www/example.com;

  location / {

    try_files $uri $uri/ /frontend/index.html;

  }

  location ^~ /images {

    root /var/www/static;
    try_files $uri $uri/ =404;

  }

  ...

• default root directory for all locations is /var/www/example.com

• location / - matches all locations without more specific locations, e.g. exact names

  • try_files $uri - when you receive a URI that's matched by this block try $uri first

    For example: https://example.com/tools/en.js - NGINX will try to check if there's a file inside /tools called en.js, if found it, serve it in the first place.

  • try_files $uri $uri/ - if you didn't find the first condition $uri try the URI as a directory

    For example: https://example.com/backend/ - NGINX will try first check if a file called backend exists, if can't find it then goes to second check $uri/ and see if there's a directory called backend exists then it will try serving it.

  • try_files $uri $uri/ /frontend/index.html - if a file and directory not found, NGINX sends /frontend/index.html

• location ^~ /images - handle any query beginning with /images and halts searching

  • default root directory for this location is /var/www/static

  • try_files $uri - when you receive a URI that's matched by this block try $uri first

    For example: https://example.com/images/01.gif - NGINX will try to check if there's a file inside /images called 01.gif, if found it, serve it in the first place.

  • try_files $uri $uri/ - if you didn't find the first condition $uri try the URI as a directory

    For example: https://example.com/images/ - NGINX will try first check if a file called images exists, if can't find it then goes to second check $uri/ and see if there's a directory called images exists then it will try serving it.

  • try_files $uri $uri/ =404 - if a file and directory not found, NGINX sends HTTP 404 (Not Found)

The ngx_http_rewrite_module also provides additional directives:

• break - stops processing, if is specified inside the location, further processing of the request continues in this location:

  # It's useful for:
  if ($slow_resp) {
  
    limit_rate 50k;
    break;
  
  }

• if - you can use if inside a server but not the other way around, also notice that you shouldn't use if inside location as it may not work as desired (see If Is Evil)

  The NGINX docs say also: There are cases where you simply cannot avoid using an if, for example if you need to test a variable which has no equivalent directive.

• set - sets a value for the specified variable. The value can contain text, variables, and their combination

Example of usage if and set directives:

# It comes from: https://gist.github.com/jrom/1760790:
if ($request_uri = /) {
  set $test  A;
}

if ($host ~* example.com) {
  set $test  "${test}B";
}

if ($http_cookie !~* "auth_token") {
  set $test  "${test}C";
}

if ($test = ABC) {
  proxy_pass http://cms.example.com;
  break;
}

Log files are a critical part of the NGINX management. It writes information about client requests in the access log right after the request is processed (in the last phase: NGX_HTTP_LOG_PHASE).

By default:

• the access log is located in logs/access.log, but I suggest you take it to /var/log/nginx directory
• data is written in the predefined combined format

Sometimes certain entries are there just to fill up the logs or are cluttering them. I sometimes exclude requests - by client IP or whatever else - when I want to debug log files more effective.

So in this example if the $error_codes variable’s value is 0 - then log nothing (default action), but if 1 (e.g. 404 or 503 from backend) - to save this request to the log:

# Define map in the http context:
http {

  ...

  map $status $error_codes {

    default   1;
    ~^[23]    0;

  }

  ...

  # Add if condition to access log:
  access_log /var/log/nginx/example.com-access.log combined if=$error_codes;

}

NGINX will re-open its logs in response to the USR1 signal:

cd /var/log/nginx

mv access.log access.log.0
kill -USR1 $(cat /var/run/nginx.pid) && sleep 1

# >= gzip-1.6:
gzip -k access.log.0
# With any version:
gzip < access.log.0 > access.log.0.gz

# Test integrity and remove if test passed:
gzip -t access.log.0 && rm -fr access.log.0

You can also read about how to configure log rotation policy.

The following is a list of all severity levels:

For example: if you set crit error log level, messages of crit, alert, and emerg levels are logged.

Load Balancing is in principle a wonderful thing really. You can find out about it when you serve tens of thousands (or maybe more) of requests every second. Of course load balancing is not the only reason - think also about maintenance tasks without downtime for example.

Generally load balancing is a technique used to distribute the workload across multiple computing resources and servers.

I think you should always use this technique also if you have a simple app or whatever else what you're sharing with other.

The configuration is very simple. NGINX includes a ngx_http_upstream_module to define backends (groups of servers or multiple server instances). More specifically, the upstream directive is responsible for this.

Before we start talking about the load balancing techniques you should know something about server directive. It defines the address and other parameters of a backend servers.

This directive accepts the following options:

• weight=<num> - sets the weight of the origin server, e.g. weight=10

• max_conns=<num> - limits the maximum number of simultaneous active connections from the NGINX proxy server to an upstream server (default value: 0 = no limit), e.g. max_conns=8

  • if you set max_conns=4 the 5th will be rejected
  • if the server group does not reside in the shared memory (zone directive), the limitation works per each worker process

• max_fails=<num> - the number of unsuccessful attempts to communicate with the backend (default value: 1, 0 disables the accounting of attempts), e.g. max_fails=3;

• fail_timeout=<time> - the time during which the specified number of unsuccessful attempts to communicate with the server should happen to consider the server unavailable (default value: 10 seconds), e.g. fail_timeout=30s;

• zone <name> <size> - defines shared memory zone that keeps the group’s configuration and run-time state that are shared between worker processes, e.g. zone backend 32k;

• backup - if server is marked as a backup server it does not receive requests unless both of the other servers are unavailable

• down - marks the server as permanently unavailable

It's the simpliest load balancing technique. Round Robin has the list of servers and forwards each request to each server from the list in order. Once it reaches the last server, the loop again jumps to the first server and start again.

upstream bck_testing_01 {

  # with default weight for all (weight=1)
  server 192.168.250.220:8080
  server 192.168.250.221:8080
  server 192.168.250.222:8080

}

In Weighted Round Robin load balancing algorithm, each server is allocated with a weight based on its configuration and ability to process the request.

This method is similar to the Round Robin in a sense that the manner by which requests are assigned to the nodes is still cyclical, albeit with a twist. The node with the higher specs will be apportioned a greater number of requests.

upstream bck_testing_01 {

  server 192.168.250.220:8080   weight=3
  server 192.168.250.221:8080              # default weight=1
  server 192.168.250.222:8080              # default weight=1

}

This method tells the load balancer to look at the connections going to each server and send the next connection to the server with the least amount of connections.

upstream bck_testing_01 {

  least_conn;

  # with default weight for all (weight=1)
  server 192.168.250.220:8080
  server 192.168.250.221:8080
  server 192.168.250.222:8080

}

For example: if clients D10, D11 and D12 attempts to connect after A4, C2 and C8 have already disconnected but A1, B3, B5, B6, C7 and A9 are still connected, the load balancer will assign client D10 to server 2 instead of server 1 and server 3. After that, client D11 will be assign to server 1 and client D12 will be assign to server 2.

This is, in general, a very fair distribution method, as it uses the ratio of the number of connections and the weight of a server. The server in the cluster with the lowest ratio automatically receives the next request.

upstream bck_testing_01 {

  least_conn;

  server 192.168.250.220:8080   weight=3
  server 192.168.250.221:8080              # default weight=1
  server 192.168.250.222:8080              # default weight=1

}

For example: if clients D10, D11 and D12 attempts to connect after A4, C2 and C8 have already disconnected but A1, B3, B5, B6, C7 and A9 are still connected, the load balancer will assign client D10 to server 2 or 3 (because they have a least active connections) instead of server 1. After that, client D11 and D12 will be assign to server 1 because it has the biggest weight parameter.

The IP Hash method uses the IP of the client to create a unique hash key and associates the hash with one of the servers. This ensures that a user is sent to the same server in future sessions (a basic kind of session persistence) except when this server is unavailable. If one of the servers needs to be temporarily removed, it should be marked with the down parameter in order to preserve the current hashing of client IP addresses.

This technique is especially helpful if actions between sessions has to be kept alive e.g. products put in the shopping cart or when the session state is of concern and not handled by shared memory of the application.

upstream bck_testing_01 {

  ip_hash;

  # with default weight for all (weight=1)
  server 192.168.250.220:8080
  server 192.168.250.221:8080
  server 192.168.250.222:8080

}

This technique is very similar to the IP Hash but for each request the load balancer calculates a hash that is based on the combination of a text string, variable, or a combination you specify, and associates the hash with one of the servers.

upstream bck_testing_01 {

  hash $request_uri;

  # with default weight for all (weight=1)
  server 192.168.250.220:8080
  server 192.168.250.221:8080
  server 192.168.250.222:8080

}

For example: load balancer calculate hash from the full original request URI (with arguments). Clients A4, C7, C8 and A9 sends requests to the /static location and will be assign to server 1. Similarly clients A1, C2, B6 which get /sitemap.xml resource they will be assign to server 2. Clients B3 and B5 sends requests to the /api/v4 and they will be assign to server 3.

It is similar to the Generic Hash method because you can also specify a unique hash identifier but the assignment to the appropriate server is under your control. I think it's a somewhat primitive method and I wouldn't say it is a full load balancing technique, but in some cases it is very useful.

Mainly this helps reducing the mess on the configuration made by a lot of location blocks with similar configurations.

First of all create a map:

map $request_uri $bck_testing_01 {

  default       "192.168.250.220:8080";

  /api/v4       "192.168.250.220:8080";
  /api/v3       "192.168.250.221:8080";
  /static       "192.168.250.222:8080";
  /sitemap.xml  "192.168.250.222:8080";

}

And add proxy_pass directive:

server {

  ...

  location / {

    proxy_pass    http://$bck_testing_01;

  }

  ...

}

All rate limiting rules (definitions) should be added to the NGINX http context.

Rate limiting rules are useful for:

• traffic shaping
• traffic optimising
• slow down the rate of incoming requests
• protect http requests flood
• protect against slow http attacks
• prevent consume a lot of bandwidth
• mitigating ddos attacks
• protect brute-force attacks

NGINX has following variables (unique keys) that can be used in a rate limiting rules:

^{Please see official doc for more information about variables.}

NGINX also provides following keys:

and directives:

Keys are used to store the state of each IP address and how often it has accessed a limited object. This information are stored in shared memory available from all NGINX worker processes.

Both keys also provides response status parameters indicating too many requests or connections with specific http code (default 503).

• limit_req_status <value>
• limit_conn_status <value>

For example, if you want to set the desired logging level for cases when the server limits the number of connections:

# Add this to http context:
limit_req_status 429;

# Set your own error page for 429 http code:
error_page 429 /rate_limit.html;
location = /rate_limit.html {

  root /usr/share/www/http-error-pages/sites/other;
  internal;

}

# And create this file:
cat > /usr/share/www/http-error-pages/sites/other/rate_limit.html << __EOF__
HTTP 429 Too Many Requests
__EOF__

Rate limiting rules also have zones that lets you define a shared space in which to count the incoming requests or connections.

All requests or connections coming into the same space will be counted in the same rate limit. This is what allows you to limit per URL, per IP, or anything else.

The zone has two required parts:

• <name> - is the zone identifier
• <size> - is the zone size

Example:

<key> <variable> zone=<name>:<size>;

State information for about 16,000 IP addresses takes 1 megabyte. So 1 kilobyte zone has 16 IP addresses.

The range of zones is as follows:

• http context

  http {
  
    ... zone=<name>;
  

• server context

  server {
  
    ... zone=<name>;
  

• location directive

  location /api {
  
    ... zone=<name>;
  

limit_req_zone key lets you set rate parameter (optional) - it defines the rate limited URL(s).

For enable queue you should use limit_req or limit_conn directives (see above). limit_req also provides optional parameters:

nodelay parameters are only useful when you also set a burst.

Without nodelay option NGINX would wait (no 503 response) and handle excessive requests with some delay.

It is an essential way for testing NGINX configuration:

nginx -t -c /etc/nginx/nginx.conf

An external tool for analyse NGINX configuration is gixy:

gixy /etc/nginx/nginx.conf

Standard paths to the configuration file:

• /etc/goaccess.conf
• /etc/goaccess/goaccess.conf
• /usr/local/etc/goaccess.conf

Prior to start GoAccess enable these parameters:

time-format %H:%M:%S
date-format %d/%b/%Y
log-format  %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

# Ubuntu/Debian
apt-get install gcc make libncursesw5-dev libgeoip-dev libtokyocabinet-dev

# RHEL/CentOS
yum install gcc ncurses-devel geoip-devel tokyocabinet-devel

cd /usr/local/src/

wget -c https://tar.goaccess.io/goaccess-1.3.tar.gz && \
tar xzvfp goaccess-1.3.tar.gz

cd goaccess-1.3

./configure --enable-utf8 --enable-geoip=legacy --with-openssl=<path_to_openssl_sources> --sysconfdir=/etc/

make -j2 && make install

ln -s /usr/local/bin/goaccess /usr/bin/goaccess

You can always fetch default configuration from /usr/local/src/goaccess-<version>/config/goaccess.conf source tree.

goaccess -f access.log -a

zcat access.log.1.gz | goaccess -a -p /etc/goaccess/goaccess.conf

ssh user@remote_host 'access.log' | goaccess -a

goaccess -p /etc/goaccess/goaccess.conf -f access.log --log-format=COMBINED -o /var/www/index.html

ngxtop -l access.log

ngxtop -l access.log -i 'status >= 400' print request status

ssh user@remote_host tail -f access.log | ngxtop -f combined

You can change combinations and parameters of these commands.

# 1)
curl -Iks <scheme>://<server_name>:<port>

# 2)
http -p Hh <scheme>://<server_name>:<port>

# 3)
htrace.sh -u <scheme>://<server_name>:<port> -h

# 1)
curl -Iks --location -X GET -A "x-agent" <scheme>://<server_name>:<port>

# 2)
http -p Hh GET <scheme>://<server_name>:<port> User-Agent:x-agent --follow

# 3)
htrace.sh -u <scheme>://<server_name>:<port> -M GET --user-agent "x-agent" -h

# URL sequence substitution with a dummy query string:
curl -ks <scheme>://<server_name>:<port>?[1-20]

# With shell 'for' loop:
for i in {1..20} ; do curl -ks <scheme>://<server_name>:<port> ; done

# 1)
echo | openssl s_client -connect <server_name>:<port>

# 2)
gnutls-cli --disable-sni -p 443 <server_name>

# 1)
echo | openssl s_client -servername <server_name> -connect <server_name>:<port>

# 2)
gnutls-cli -p 443 <server_name>

openssl s_client -tls1_2 -connect <server_name>:<port>

openssl s_client -cipher 'AES128-SHA' -connect <server_name>:<port>

Project documentation: Apache HTTP server benchmarking tool

Installation:

# Debian like:
apt-get install -y apache2-utils

# RedHat like:
yum -y install httpd-tools

This is a great explanation about ApacheBench by Mamsaac:

The apache benchmark tool is very basic, and while it will give you a solid idea of some performance, it is a bad idea to only depend on it if you plan to have your site exposed to serious stress in production.

ab -n 1000 -c 100 https://example.com/

ab -n 5000 -c 100 -k -H "Accept-Encoding: gzip, deflate" https://example.com/index.php

Project documentation: wrk2

See this chapter to use the Lua API for wrk2. Also take a look at wrk2 scripts.

Installation:

# Debian like:
apt-get install -y build-essential libssl-dev git zlib1g-dev
git clone https://github.com/giltene/wrk2 && cd wrk2
make
sudo cp wrk /usr/local/bin

# RedHat like:
yum -y groupinstall 'Development Tools'
yum -y install openssl-devel git
git clone https://github.com/giltene/wrk2 && cd wrk2
make
sudo cp wrk /usr/local/bin

# 1)
wrk -c 1 -t 1 -d 2s -R 5 -H "Host: example.com" https://example.com
Running 2s test @ https://example.com
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    45.21ms   20.99ms 108.16ms   90.00%
    Req/Sec       -nan      -nan   0.00      0.00%
  10 requests in 2.01s, 61.69KB read
Requests/sec:      4.99
Transfer/sec:     30.76KB

# RPS:
6 09/Jul/2019:08:00:25
5 09/Jul/2019:08:00:26

# 2)
wrk -c 1 -t 1 -d 2s -R 25 -H "Host: example.com" https://example.com
Running 2s test @ https://example.com
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    64.40ms   24.26ms 110.46ms   48.00%
    Req/Sec       -nan      -nan   0.00      0.00%
  50 requests in 2.01s, 308.45KB read
Requests/sec:     24.93
Transfer/sec:    153.77KB

# RPS:
12 09/Jul/2019:08:02:09
26 09/Jul/2019:08:02:10
13 09/Jul/2019:08:02:11

# 3)
wrk -c 5 -t 5 -d 2s -R 25 -H "Host: example.com" https://example.com
Running 2s test @ https://example.com
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    47.97ms   25.79ms 136.45ms   90.00%
    Req/Sec       -nan      -nan   0.00      0.00%
  50 requests in 2.01s, 308.45KB read
Requests/sec:     24.92
Transfer/sec:    153.75KB

# RPS:
25 09/Jul/2019:08:03:56
25 09/Jul/2019:08:03:57
 5 09/Jul/2019:08:03:58

# 4)
wrk -c 5 -t 5 -d 2s -R 50 -H "Host: example.com" https://example.com
Running 2s test @ https://example.com
  5 threads and 5 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    45.09ms   18.63ms 130.69ms   91.00%
    Req/Sec       -nan      -nan   0.00      0.00%
  100 requests in 2.01s, 616.89KB read
Requests/sec:     49.85
Transfer/sec:    307.50KB

# RPS:
35 09/Jul/2019:08:05:00
50 09/Jul/2019:08:05:01
20 09/Jul/2019:08:05:02

# 5)
wrk -c 24 -t 12 -d 30s -R 2500 -H "Host: example.com" https://example.com
Running 30s test @ https://example.com
  12 threads and 24 connections
  Thread calibration: mean lat.: 3866.673ms, rate sampling interval: 13615ms
  Thread calibration: mean lat.: 3880.487ms, rate sampling interval: 13606ms
  Thread calibration: mean lat.: 3890.279ms, rate sampling interval: 13615ms
  Thread calibration: mean lat.: 3872.985ms, rate sampling interval: 13606ms
  Thread calibration: mean lat.: 3876.076ms, rate sampling interval: 13615ms
  Thread calibration: mean lat.: 3883.463ms, rate sampling interval: 13606ms
  Thread calibration: mean lat.: 3870.145ms, rate sampling interval: 13623ms
  Thread calibration: mean lat.: 3873.675ms, rate sampling interval: 13623ms
  Thread calibration: mean lat.: 3898.842ms, rate sampling interval: 13672ms
  Thread calibration: mean lat.: 3890.278ms, rate sampling interval: 13615ms
  Thread calibration: mean lat.: 3882.429ms, rate sampling interval: 13631ms
  Thread calibration: mean lat.: 3896.333ms, rate sampling interval: 13639ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    15.01s     4.32s   22.46s    57.62%
    Req/Sec    52.00      0.00    52.00    100.00%
  18836 requests in 30.01s, 113.52MB read
Requests/sec:    627.59
Transfer/sec:      3.78MB

# RPS:
 98 09/Jul/2019:08:06:13
627 09/Jul/2019:08:06:14
624 09/Jul/2019:08:06:15
640 09/Jul/2019:08:06:16
629 09/Jul/2019:08:06:17
627 09/Jul/2019:08:06:18
648 09/Jul/2019:08:06:19
624 09/Jul/2019:08:06:20
624 09/Jul/2019:08:06:21
631 09/Jul/2019:08:06:22
641 09/Jul/2019:08:06:23
627 09/Jul/2019:08:06:24
633 09/Jul/2019:08:06:25
636 09/Jul/2019:08:06:26
648 09/Jul/2019:08:06:27
626 09/Jul/2019:08:06:28
617 09/Jul/2019:08:06:29
636 09/Jul/2019:08:06:30
640 09/Jul/2019:08:06:31
627 09/Jul/2019:08:06:32
635 09/Jul/2019:08:06:33
639 09/Jul/2019:08:06:34
633 09/Jul/2019:08:06:35
598 09/Jul/2019:08:06:36
644 09/Jul/2019:08:06:37
632 09/Jul/2019:08:06:38
635 09/Jul/2019:08:06:39
624 09/Jul/2019:08:06:40
643 09/Jul/2019:08:06:41
635 09/Jul/2019:08:06:42
431 09/Jul/2019:08:06:43

# Other examples:
wrk -c 24 -t 12 -d 30s -R 2500 --latency https://example.com/index.php

Based on:

• wrk2 scripts - post
• POST request with wrk?

Example 1:

-- lua/post-call.lua
request = function()

  wrk.method = "POST"
  wrk.body = "login=foo&password=bar"
  wrk.headers["Content-Type"] = "application/x-www-form-urlencoded"

  return wrk.format(wrk.method)

end

Example 2:

-- lua/post-call.lua

request = function()

  path = "/forms/int/d/1FAI"

  wrk.method = "POST"
  wrk.body = "{\"hash\":\"ooJeiveenai6iequ\",\"timestamp\":\"1562585990\",\"data\":[[\"cache\",\"x-cache\",\"true\"]]}"
  wrk.headers["Content-Type"] = "application/json; charset=utf-8"
  wrk.headers["Accept"] = "application/json"

  return wrk.format(wrk.method, path)

end

Example 3:

-- lua/post-call.lua

request = function()

  path = "/login"

  wrk.method = "POST"
  wrk.body = [[{
    "hash": "ooJeiveenai6iequ",
    "timestamp": "1562585990",
    "data":
    {
      login: "foo",
      password: "bar"
    },
  }]]
  wrk.headers["Content-Type"] = "application/json; charset=utf-8"

  return wrk.format(wrk.method, path)

end

Command:

# The first example:
wrk -c 12 -t 12 -d 30s -R 12000 -s lua/post-call.lua https://example.com/login

# Second and third example:
wrk -c 12 -t 12 -d 30s -R 12000 -s lua/post-call.lua https://example.com

Based on:

• Intelligent benchmark with wrk
• Confusion about benchmarking Linkerd

Example 1:

-- lua/random-paths.lua

math.randomseed(os.time())

request = function()

  url_path = "/search?q=" .. math.random(0,100000)

  -- print(url_path)

  return wrk.format("GET", url_path)

end

Example 2:

-- lua/random-paths.lua

math.randomseed(os.time())

local connected = false

local host = "example.com"
local path = "/search?q="
local url  = "https://" .. host .. path

wrk.headers["Host"] = host

function ranValue(length)

  local res = ""

  for i = 1, length do

    res = res .. string.char(math.random(97, 122))

  end

  return res

end

request = function()

  url_path = path .. ranValue(32)

  -- print(url_path)

   if not connected then

      connected = true
      return wrk.format("CONNECT", host)

   end

  return wrk.format("GET", url_path)

end

Example 3:

-- lua/random-paths.lua

math.randomseed(os.time())

counter = 0

function ranValue(length)

  local res = ""

  for i = 1, length do

    res = res .. string.char(math.random(97, 122))

  end

  return res

end

request = function()

  path = "/accounts/" .. counter

  rval = ranValue(32)

  wrk.method = "POST"
  wrk.body   = [[{
    "user": counter,
    "action": "insert",
    "amount": rval
  }]]
  wrk.headers["Content-Type"] = "application/json"
  wrk.headers["Accept"] = "application/json"

  io.write(string.format("id: %4d, path: %14s,\tvalue: %s\n", counter, path, rval))

  counter = counter + 1
  if counter>500 then

    counter = 1

  end

  return wrk.format(wrk.method, path)

end

Command:

wrk -c 12 -t 12 -d 30s -R 12000 -s lua/random-paths.lua https://example.com/

Example 1:

-- lua/multi-paths.lua

math.randomseed(os.time())
math.random(); math.random(); math.random()

function shuffle(paths)

  local j, k
  local n = #paths

  for i = 1, n do

    j, k = math.random(n), math.random(n)
    paths[j], paths[k] = paths[k], paths[j]

  end

  return paths

end

function load_url_paths_from_file(file)

  lines = {}

  local f=io.open(file,"r")
  if f~=nil then

    io.close(f)

  else

    return lines

  end

  for line in io.lines(file) do

    if not (line == '') then

      lines[#lines + 1] = line

    end

  end

  return shuffle(lines)

end

paths = load_url_paths_from_file("data/paths.list")

if #paths <= 0 then

  print("No paths found. You have to create a file data/paths.list with one path per line.")
  os.exit()

end

counter = 0

request = function()

  url_path = paths[counter]

  if counter > #paths then

    counter = 0

  end

  counter = counter + 1

  return wrk.format(nil, url_path)

end

• data/paths.list:

  / - it's not recommend, requests are being duplicated if you add only '/'
  /foo/bar
  /articles/id=25
  /3a06e672fad4bec2383748cfd82547ee.html
  

Command:

wrk -c 12 -t 12 -d 60s -R 200 -s lua/multi-paths.lua https://example.com

Based on:

• wrk2 scripts - addr

Example 1:

-- lua/resolve-host.lua

local addrs = nil

function setup(thread)

  if not addrs then

    -- addrs = wrk.lookup(wrk.host, "443" or "http")
    addrs = wrk.lookup(wrk.host, wrk.port or "http")

    for i = #addrs, 1, -1 do

      if not wrk.connect(addrs[i]) then

        table.remove(addrs, i)

      end

    end

  end

  thread.addr = addrs[math.random(#addrs)]

end

function init(args)

  local msg = "thread remote socket: %s"
  print(msg:format(wrk.thread.addr))

end

Command:

wrk -c 12 -t 12 -d 30s -R 600 -s lua/resolve-host.lua https://example.com

Based on:

• multi-request-json

You should install luarocks, lua, luajit and lua-cjson before use multi-req.lua:

# Debian like:
apt-get install lua5.1 libluajit-5.1-dev luarocks

# RedHat like:
yum install lua luajit-devel luarocks

# cjson:
luarocks install lua-cjson

-- lua/multi-req.lua

local cjson = require "cjson"
local cjson2 = cjson.new()
local cjson_safe = require "cjson.safe"

math.randomseed(os.time())
math.random(); math.random(); math.random()

function shuffle(paths)

  local j, k
  local n = #paths

  for i = 1, n do

    j, k = math.random(n), math.random(n)
    paths[j], paths[k] = paths[k], paths[j]

  end

  return paths

end

function load_request_objects_from_file(file)

  local data = {}
  local content

  local f=io.open(file,"r")
  if f~=nil then

    content = f:read("*all")
    io.close(f)

  else

    return lines

  end

  data = cjson.decode(content)

  return shuffle(data)

end

requests = load_request_objects_from_file("data/requests.json")

if #requests <= 0 then

  print("No requests found. You have to create a file data/requests.json.")
  os.exit()

end

print(" " .. #requests .. " requests")

counter = 1

request = function()

  local request_object = requests[counter]

  counter = counter + 1

  if counter > #requests then

    counter = 1

  end

  return wrk.format(request_object.method, request_object.path, request_object.headers, request_object.body)

end

• data/requests.json:

  [
    {
      "path": "/id/1",
      "body": "ceR1caesaed2nohJei",
      "method": "GET",
      "headers": {
        "X-Custom-Header-1": "foo",
        "X-Custom-Header-2": "bar"
      }
    },
    {
      "path": "/id/2",
      "body": "{\"field\":\"value\"}",
      "method": "POST",
      "headers": {
        "Content-Type": "application/json",
        "X-Custom-Header-1": "foo",
        "X-Custom-Header-2": "bar"
      }
    }
  ]

Command:

wrk -c 12 -t 12 -d 30s -R 200 -s lua/multi-req.lua https://example.com

Based on:

• wrk-debugging-environment

-- lua/debug.lua

local file = io.open("data/debug.log", "w")

file:write("\n----------------------------------------\n")
file:write(os.date("%m/%d/%Y %I:%M %p"))
file:write("\n----------------------------------------\n")
file:close()

local file = io.open("data/debug.log", "a")

function typeof(var)

  local _type = type(var);
  if(_type ~= "table" and _type ~= "userdata") then

    return _type;

  end

  local _meta = getmetatable(var);
  if(_meta ~= nil and _meta._NAME ~= nil) then

    return _meta._NAME;

  else

    return _type;

  end

end

local function string(o)

  return '"' .. tostring(o) .. '"'

end

local function recurse(o, indent)

  if indent == nil then indent = '' end

  local indent2 = indent .. '  '

  if type(o) == 'table' then

    local s = indent .. '{' .. '\n'
    local first = true

    for k,v in pairs(o) do

      if first == false then s = s .. ', \n' end
      if type(k) ~= 'number' then k = string(k) end
      s = s .. indent2 .. '[' .. k .. '] = ' .. recurse(v, indent2)
      first = false

    end

    return s .. '\n' .. indent .. '}'

  else

    return string(o)

  end

end

local function var_dump(...)

  local args = {...}
  if #args > 1 then

    var_dump(args)

  else

    print(recurse(args[1]))

  end

end

max_requests = 0
counter = 1
show_body = 0

function setup(thread)

  thread:set("id", counter)
  counter = counter + 1

end

response = function (status, headers, body)

  file:write("\n----------------------------------------\n")
  file:write("Response " .. counter .. " with status: " .. status .. " on thread " .. id)
  file:write("\n----------------------------------------\n")

  file:write("[response] Headers:\n")

  for key, value in pairs(headers) do

    file:write("[response]  - " .. key  .. ": " .. value .. "\n")

  end

  if (show_body == 1) then

    file:write("[response] Body:\n")
    file:write(body .. "\n")

  end

  if (max_requests > 0) and (counter > max_requests) then

    wrk.thread:stop()

  end

  counter = counter + 1

end

done = function ()

  file:close()

end

Command:

wrk -c 12 -t 12 -d 15s -R 200 -s lua/debug.lua https://example.com

Based on:

• wrk2 - setup

-- lua/threads.lua

local counter = 1
local threads = {}

function setup(thread)

  thread:set("id", counter)
  table.insert(threads, thread)

  counter = counter + 1

end

function init(args)

  requests  = 0
  responses = 0

  -- local msg = "thread %d created"
  -- print(msg:format(id))

end

function request()

  requests = requests + 1
  return wrk.request()

end

function response(status, headers, body)

  responses = responses + 1

end

function done(summary, latency, requests)

  io.write("\n----------------------------------------\n")
  io.write(" Summary")
  io.write("\n----------------------------------------\n")

  for index, thread in ipairs(threads) do

    local id        = thread:get("id")
    local requests  = thread:get("requests")
    local responses = thread:get("responses")

    local msg = "thread %d : %d req , %d res"

    print(msg:format(id, requests, responses))

  end

end

Command:

wrk -c 12 -t 12 -d 5s -R 5000 -s lua/threads.lua https://example.com

Installation:

go get -u github.com/jgsqware/wrk-report

Command:

wrk -c 12 -t 12 -d 15s -R 500 --latency https://example.com | wrk-report > report.html

Project documentation: Locust Documentation

Installation:

# Python 2.x
python -m pip install locustio

# Python 3.x
python3 -m pip install locustio

About locust:

• Number of users to simulate - the number of users testing your application. Each user opens a TCP connection to your application and tests it

• Hatch rate (users spawned/second) - for each second, how many users will be added to the current users until the total amount of users. Each hatch Locust calls the on_start function if you have

For example:

• Number of users: 1000
• Hatch rate: 10

Each second 10 users added to current users starting from 0 so in 100 seconds you will have 1000 users. When it reaches to the number of users, the statistic will be reset.

Locust tries to emulate user behavior, it will pause each individual 'User' between min_wait and max_wait ms, to simulate the time between normal user actions.

Each of tasks will be executed in a random order, with a delay of min_wait - max_wait between the beginning of each task.

# python/multi-paths.py

import urllib3

from locust import HttpLocust, TaskSet, task

urllib3.disable_warnings()

multiheaders = """{
"Host": "example.com",
"User-Agent":"python-locust-test",
}
"""

self.client.get("/", headers=h)

def on_start(self):
  self.client.verify = False

class UserBehavior(TaskSet):

  @task
  class NonLoggedUserBehavior(TaskSet):

    # Home page
    @task(1)
    def index(self):
      self.client.get("/", headers=multiheaders, verify=False)

    # Status
    @task(1)
    def status(self):
      self.client.get("/status", verify=False)

    # Article
    @task(1)
    def article(self):
      self.client.get("/article/1044162375/", headers=multiheaders, verify=False)

    # About
    # Twice as much of requests:
    @task(2)
    def about(self):
      with self.client.get("/about", catch_response=True) as response:
        if response.text.find("[email protected]") > 0:
          response.success()
        else:
          response.failure("[email protected] not found in response")

class WebsiteUser(HttpLocust):

  task_set = UserBehavior
  min_wait = 1000 # ms, 1s
  max_wait = 5000 # ms, 5s

Command:

# Without web interface:
locust --host=https://example.com -f python/multi-paths.py -c 2000 -r 10 -t 1h 30m --no-web --print-stats --only-summary

# With web interface
locust --host=https://example.com -f python/multi-paths.py --print-stats --only-summary

Look also:

• How to Run Locust with Different Users

Create a file with user credentials:

# python/credentials.py

USER_CREDENTIALS = [

  ("user5", "ShaePhu8aen8"),
  ("user4", "Cei5ohcha3he"),
  ("user3", "iedie8booChu"),
  ("user2", "iCuo4es1ahzu"),
  ("user1", "eeSh0yi0woo8")

  # ...

]

# python/diff-users.py

import urllib3, logging, sys

from locust import HttpLocust, TaskSet, task
from credentials import USER_CREDENTIALS

urllib3.disable_warnings()

class UserBehavior(TaskSet):

  @task
  class LoggedUserBehavior(TaskSet):

    username = "default"
    password = "default"

    def on_start(self):
      if len(USER_CREDENTIALS) > 0:
        self.username, self.password = USER_CREDENTIALS.pop()

      self.client.post("/login", {
        'username': self.username, 'password': self.password
      })
      logging.info('username: %s, password: %s', self.username, self.password)

    def on_stop(self):
      self.client.post("/logout", verify=False)

    # Home page
    # 10x more often than other
    @task(10)
    def index(self):
      self.client.get("/", verify=False)

    # Enter specific url after client login
    @task(1)
    def random_gen(self):
      self.client.get("/random-generator", verify=False)

    # Client profile page
    @task(1)
    def profile(self):
      self.client.get("/profile", verify=False)

    # Contact page
    @task(1)
    def contact(self):
      self.client.post("/contact", {
        "email": "[email protected]",
        "subject": "GNU/Linux and BSD",
        "message": "Free software, Yeah!"
      })

class WebsiteUser(HttpLocust):

  host = "https://api.example.com"
  task_set = UserBehavior
  min_wait = 2000   # ms, 2s
  max_wait = 15000  # ms, 15s

Command:

# Without web interface (for 5 users, see credentials.py):
locust -f python/diff-users.py -c 5 -r 5 -t 30m --no-web --print-stats --only-summary

# With web interface (for 5 users, see credentials.py)
locust -f python/diff-users.py --print-stats --only-summary

hping3 -V -c 1000000 -d 120 -S -w 64 -p 80 --flood --rand-source <remote_host>

# 1)
slowhttptest -g -o http_dos.stats -H -c 1000 -i 15 -r 200 -t GET -x 24 -p 3 -u <scheme>://<server_name>/index.php

slowhttptest -g -o http_dos.stats -B -c 5000 -i 5 -r 200 -t POST -l 180 -x 5 -u <scheme>://<server_name>/service/login

# 2)
pip3 install slowloris
slowloris <server_name>

# 3)
git clone https://github.com/jseidl/GoldenEye && cd GoldenEye
./goldeneye.py  <scheme>://<server_name> -w 150 -s 75 -m GET

You can change combinations and parameters of these commands. When carrying out the analysis, remember about debug log and log formats.

with ps:

# For all processes (master + workers):
ps axw -o pid,ppid,gid,user,etime,%cpu,%mem,vsz,rss,wchan,ni,command | egrep '([n]ginx|[P]ID)'

ps aux | grep [n]ginx
ps -lfC nginx

# For master process:
ps axw -o pid,ppid,gid,user,etime,%cpu,%mem,vsz,rss,wchan,ni,command | egrep '([n]ginx: master|[P]ID)'

ps aux | grep "[n]ginx: master"

# For worker/workers:
ps axw -o pid,ppid,gid,user,etime,%cpu,%mem,vsz,rss,wchan,ni,command | egrep '([n]ginx: worker|[P]ID)'

ps aux | grep "[n]ginx: worker"

# Show only pid, user and group for all NGINX processes:
ps -eo pid,comm,euser,supgrp | grep nginx

with top:

# For all processes (master + workers):
top -p $(pgrep -d , nginx)

# For master process:
top -p $(pgrep -f "nginx: master")
top -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}')

# For one worker:
top -p $(pgrep -f "nginx: worker")
top -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}')

# For multiple workers:
top -p $(pgrep -f "nginx: worker" | sed '$!s/$/,/' | tr -d '\n')
top -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}' | sed '$!s/$/,/' | tr -d '\n')

with ps_mem:

# For all processes (master + workers):
ps_mem -s -p $(pgrep -d , nginx)
ps_mem -d -p $(pgrep -d , nginx)

# For master process:
ps_mem -s -p $(pgrep -f "nginx: master")
ps_mem -s -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}')

# For one worker:
ps_mem -s -p $(pgrep -f "nginx: worker")
ps_mem -s -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}')

# For multiple workers:
ps_mem -s -p $(pgrep -f "nginx: worker" | sed '$!s/$/,/' | tr -d '\n')
ps_mem -s -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}' | sed '$!s/$/,/' | tr -d '\n')

with pmap:

# For all processes (master + workers):
pmap $(pgrep -d ' ' nginx)
pmap $(pidof nginx)

# For master process:
pmap $(pgrep -f "nginx: master")
pmap $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}')

# For one and multiple workers:
pmap $(pgrep -f "nginx: worker")
pmap $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}')

# For all processes (master + workers):
lsof -n -p $(pgrep -d , nginx)

# For master process:
lsof -n -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: master") { print $1}')

# For one worker:
lsof -n -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}')

# For multiple workers:
lsof -n -p $(ps axw -o pid,command | awk '($2 " " $3 ~ "nginx: worker") { print $1}' | sed '$!s/$/,/' | tr -d '\n')

From a configuration file and all attached files (from a disk, only what a new process would load):

nginx -T
nginx -T -c /etc/nginx/nginx.conf

From a running process:

For more information please see GNU Debugger (gdb) - Dump configuration from a running process.

nginx -V 2>&1 | grep arguments

nginx -V 2>&1 | grep -- 'http_geoip_module'

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s%20s\n%10s%20s\n' "AMOUNT" "IP_ADDRESS"
awk '{print $1}' access.log | sort | uniq -c | sort -nr

# - add this to the end for print header:
#   ... | xargs printf '%10s%10s%20s\n%10s%10s%20s\n' "NUM" "AMOUNT" "IP_ADDRESS"
cut -d ' ' -f1 access.log | sort | uniq -c | sort -nr | head -5 | nl

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL"
awk -F\" '{print $2}' access.log | awk '{print $2}' | sort | uniq -c | sort -nr

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL"
awk -F\" '($2 ~ "/string") { print $2}' access.log | awk '{print $2}' | sort | uniq -c | sort -nr

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s %8s\t%s\n%10s %8s\t%s\n' "AMOUNT" "METHOD" "URL"
awk -F\" '{print $2}' access.log | awk '{print $1 "\t" $2}' | sort | uniq -c | sort -nr

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "HTTP_CODE"
awk '{print $9}' access.log | sort | uniq -c | sort -nr

tail -n 100 -f access.log | grep "HTTP/[1-2].[0-1]\" [2]"

tail -n 100 -f access.log | grep "HTTP/[1-2].[0-1]\" [5]"

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL"
awk '($9 ~ /502/)' access.log | awk '{print $7}' | sort | uniq -c | sort -nr

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "URL"
awk '($9 ~ /401/)' access.log | awk -F\" '($2 ~ "/*.php")' | awk '{print $7}' | sort | uniq -c | sort -nr

# Not less than 1 minute:
tail -2000 access.log | awk -v date=$(date -d '1 minutes ago' +"%d/%b/%Y:%H:%M") '$4 ~ date' | cut -d '"' -f3 | cut -d ' ' -f2 | sort | uniq -c | sort -nr

# Last 2000 requests from log file:
# - add this to the end for print header:
#   ... | xargs printf '%10s\t%s\n%10s\t%s\n' "AMOUNT" "HTTP_CODE"
tail -2000 access.log | cut -d '"' -f3 | cut -d ' ' -f2 | sort | uniq -c | sort -nr

# In real time:
tail -F access.log | pv -lr >/dev/null

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s%24s%18s\n%10s%24s%18s\n' "AMOUNT" "DATE" "IP_ADDRESS"
awk '{print $4}' access.log | uniq -c | sort -nr | tr -d "["

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s%24s%18s\n%10s%24s%18s\n' "AMOUNT" "DATE" "IP_ADDRESS"
awk '{print $4 " " $1}' access.log | uniq -c | sort -nr | tr -d "["

# - add `head -n X` to the end to limit the result
# - add this to the end for print header:
#   ... | xargs printf '%10s%24s%18s\t%s\n%10s%24s%18s\t%s\n' "AMOUNT" "DATE" "IP_ADDRESS" "URL"
awk '{print $4 " " $1 " " $7}' access.log | uniq -c | sort -nr | tr -d "["

awk -v _date=`date -d 'now-6 hours' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > _date) print $0}' access.log

# date command shows output for specific locale, for prevent this you should set LANG variable:
awk -v _date=$(LANG=en_us.utf-8 date -d 'now-6 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _date) print $0}' access.log

# or:
export LANG=en_us.utf-8
awk -v _date=$(date -d 'now-6 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _date) print $0}' access.log

# 1)
awk '$4>"[05/Feb/2019:02:10" && $4<"[15/Feb/2019:08:20"' access.log

# 2)
# date command shows output for specific locale, for prevent this you should set LANG variable:
awk -v _dateB=$(LANG=en_us.utf-8 date -d '10:20' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(LANG=en_us.utf-8 date -d '20:30' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' access.log

# or:
export LANG=en_us.utf-8
awk -v _dateB=$(date -d '10:20' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(date -d '20:30' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' access.log

# 3)
# date command shows output for specific locale, for prevent this you should set LANG variable:
awk -v _dateB=$(LANG=en_us.utf-8 date -d 'now-12 hours' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(LANG=en_us.utf-8 date -d 'now-2 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' access.log

# or:
export LANG=en_us.utf-8
awk -v _dateB=$(date -d 'now-12 hours' +[%d/%b/%Y:%H:%M:%S) -v _dateE=$(date -d 'now-2 hours' +[%d/%b/%Y:%H:%M:%S) ' { if ($4 > _dateB && $4 < _dateE) print $0}' access.log