NGINX thundering herd problem

Solution 1: EPOLLEXCLUSIVE flag

If EPOLLEXCLUSIVE (Linux 4.5, glibc 2.24) is defined when compiling NGINX, NGINX could make use of it to reduce resource usage when volume of new connection is low.

NGINX discards EPOLLRDHUP if EPOLLEXCLUSIVE is enabled, keeping EPOLLIN and EPOLLOUT compatible with the CentOS 7 backport.

#if (NGX_HAVE_EPOLLEXCLUSIVE && NGX_HAVE_EPOLLRDHUP)
    if (flags & NGX_EXCLUSIVE_EVENT) {
        events &= ~EPOLLRDHUP;
    }
#endif

For every 16 requests handled, NGINX would re-add the socket in ngx_reorder_accept_events() to balance request across workers.

    if (c->requests++ % 16 != 0
        && ngx_accept_disabled <= 0)
    {
        return;
    }
 
    if (ngx_del_event(c->read, NGX_READ_EVENT, NGX_DISABLE_EVENT)
        == NGX_ERROR)
    {
        return;
    }
 
    if (ngx_add_event(c->read, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)
        == NGX_ERROR)
    {
        return;
    }

Solution 2: listen reuseport

SO_REUSEPORT could significantly increase the max latency in a degraded state. See https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/.

As a side effect, total length of pending connection queue (backlog) is multiplied by the number of workers because each worker has its own socket and therefore gets its own accept queue.

If you intend to set reuseport on listen, you should consider enabling the net.ipv4.tcp_migrate_req kernel parameter (Linux v5.14+ only) to reduce impact on in-flight request sockets during the handshake and established sockets in the accept queue during configuration reload.