NGINX graceful reload

Nginx is mostly graceful when reloading, except in the following situations:

1. Long-lived connections like WebSocket, SSE and requests that take a long time to complete.
2. A new request came in from an HTTP keep-alive connection right after reload is initiated.
3. A new TCP connection in the SYN backlog or accept queue gets dropped if reuseport is enabled.

The second case can be mitigated by setting appropriate worker_shutdown_timeout and keepalive_min_timeout values. keepalive_min_timeout allows a subsequent request on the connection to be processed, and returns Connection: close in the response HTTP header to properly terminate the connection without race conditions.

The third case can be resolved with net.ipv4.tcp_migrate_req, a new feature introduced in Linux 5.14.