Acceleration Mechanisms on NVIDIA ConnectX-5 NICs

List of Mechanisms with Dedicated Counters

• TSO (TCP Segmentation Offload) - increasing outbound throughput and reducing CPU utilization by allowing the kernel to buffer multiple packets in a single large buffer. The NIC split the buffer into packet and transmits it
• LRO (Large Receive Offload) - increasing inbound throughput and reducing CPU utilization by aggregation of o multiple incoming packet of a single stream to a single buffer
  • There is also GRO (Generic Receive Offload), which is considered less broken according to a blog post (2009).
• CHECKSUM (Checksum) – calculation of TCP checksum (by the NIC). The following CSUM offload are available (refer to skbuff.h for detailed explanation)
  • CHECKSUM_UNNECESSARY
  • CHECKSUM_NONE – no CSUM acceleration was used
  • CHECKSUM_COMPLETE – Device provided CSUM on the entire packet
  • CHECKSUM_PARTIAL – Device provided CSUM
• CQE Compress – compression of Completion Queue Events (CQE) used for sparing bandwidth on PCIe and hence achieve better performance.

LRO

Requirements

• LRO is only enabled in Striding RQ (a.k.a. Multi-Packet Rx Queue, or MPRQ)
• Minimal hardware version is ConnectX-5

How to Enable Striding RQ

# ethtool --set-priv-flags eno0 rx_striding_rq on
# ethtool --show-priv-flags eno0
Private flags for eno0:
...
rx_striding_rq : on
... 

How to Enable LRO

LRO on a new kernel can be configured using ethtool commands:

1. To check LRO config, run:

# ethtool -k ens801f1 | grep large-receive-offload
large-receive-offload: off

2. To enable LRO, run:

# ethtool -K eth1 lro on

3. To disable LRO, run:

# ethtool -K eth1 lro off

4. To verify LRO configuration, run:

# ethtool -k ens801f1 | grep large-receive-offload
large-receive-offload: on

TSO

Requirements

• Minimal hardware version is ConnectX-4

How to Enable TSO

1. To check TSO config, run:

# ethtool -k ens801f1 | grep tcp-segmentation-offload
tcp-segmentation-offload: on

Following steps are similar to LRO.

ethtool configuration persistence

RHEL 9

1. To set persistent LRO configuration in the ens801f1 connection profile, run:

nmcli con modify ens801f1 ethtool.feature-rx on

1. To remove persistent LRO configuration, run:

nmcli con modify ens801f1 ethtool.feature-rx ""

RHEL 8

To persist LRO configuration on RHEL 8, add the following to /etc/sysconfig/network-scripts/ifcfg-* (replace * with interface name):

ETHTOOL_OPTS="-K ${DEVICE} lro on"

References

• https://github.com/torvalds/linux/commit/6c3a823e1e9c645f30c5b03fefe87fea8881060b
• https://doc.dpdk.org/guides/nics/mlx5.html
• https://www.ibm.com/docs/en/linux-on-systems?topic=practices-performance-tuning#d7439e143
• https://enterprise-support.nvidia.com/s/article/how-to-enable-large-receive-offload—lro-x
• https://enterprise-support.nvidia.com/s/article/understanding-mlx5-ethtool-counters
• https://www.redhat.com/en/blog/rhel-9-networking-say-goodbye-ifcfg-files-and-hello-keyfiles
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-ip-networking-with-ifcfg-files_configuring-and-managing-networking
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/configuring-ethtool-settings-in-networkmanager-connection-profiles_configuring-and-managing-networking