Scaling K8s to a large cluster

For K3s:

1. Ensure master nodes have enough memory for etcd, Kubernetes API server, etc.
2. You should consider increasing the subnet size for the cluster CIDR so that you don’t run out of IPs for the pods. You can do that by passing the --cluster-cidr option to K3s server upon starting.
3. You should consider the same for the service CIDR. In K3s, the largest supported service-cidr mask is /12 for IPv4, and /112 for IPv6.
4. Add the flag --etcd-arg=quota-backend-bytes=5368709120 to all master nodes to increase the max size of the embedded etcd datastore to 5 GiB. 8 GiB is a suggested maximum size for normal environments.
5. Protocol buffers used by gRPC have a maximum size limit of < 2 GiB for any data in serialized form. This would only affect requests to Kubernetes API server with very large response body.
6. Kubernetes API server configuration --request-timeout defaults to a minute, and to prevent resource exhaustion you should keep it at 1m0s.

References

• https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs
• https://etcd.io/docs/v3.5/dev-guide/limit/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/