🪴 Zero's Garden

Changing logical sector size of NVMe SSDs

3 min read

Introduction

The default fdisk -l output of two example datacenter SSDs are respectively

Disk /dev/nvme2n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: INTEL SSDPF2KX019T1O
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/nvme2n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: Micron_7450_MTFDKCC1T9TFR
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

and after reformatting, you get

Disk /dev/nvme2n1: 1.75 TiB, 1920383410176 bytes, 468843606 sectors
Disk model: INTEL SSDPF2KX019T1O
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/nvme2n1: 1.75 TiB, 1920383410176 bytes, 468843606 sectors
Disk model: Micron_7450_MTFDKCC1T9TFR
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

You can check available LBA formats with nvme-cli:

apt-get install nvme-cli
for i in {0..3}; do
  echo "/dev/nvme$i:"
  nvme id-ctrl /dev/nvme${i} -H | grep "LBA Format"
  echo "---"
  nvme id-ns /dev/nvme${i}n1 -H | grep "LBA Format"
  echo
done
/dev/nvme2:
  [15:15] : 0   Extended LBA Formats Not Supported
---
  [6:5] : 0     Most significant 2 bits of Current LBA Format Selected
  [3:0] : 0     Least significant 4 bits of Current LBA Format Selected
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

You should use the LBA format with best relative performance, and in this case, LBA format 1.

NOTE

https://github.com/letsencrypt/openzfs-nvme-databases suggests using sst show -display SectorSize -ssd to check the sector size of Intel SSD drives, but for i in {0..3}; do sst show -display SectorSize,SectorDataSize -ssd $i; done results show that SectorSize and SectorDataSize have the same value and both of them correspond to the LBA format used by the namespace block device. Hence, you should use the more generic nvme-cli instead of sst.

Formatting

Formatting an NVMe namespace erases all data, so if you have already constructed a ZFS mirror on top of the disks, you must detach it first.

zpool detach rpool /dev/disk/by-id/nvme-Micron_...

Then, reformat the namespace with the desired LBA Format. Usually it should be “Metadata Size: 0 bytes - Data Size: 4096 bytes” and have the best Relative Performance level.

nvme format /dev/nvme2n1 --lbaf=1

If the disk was part of a ZFS mirror, attach it back to mirror the other device.

# zpool attach [-fsw] [-o property=value] pool device new_device
zpool attach -o ashift=12 rpool /dev/disk/by-id/nvme-Micron_... /dev/disk/by-id/nvme-Micron_...

and check fdisk -l results again.

Graph View