PCI3/NVMe performnace dramatically lower on 7.1.0-rc1 vs. 7.0.1

[by]

Describe the bug

When comparing benchmarks of an M.2 NVMe SSD via PCIE3 on an original Raspberry Pi M.2 HAT+, they vary significantly:

- 7.0.1 Benchmark Results:
DD Write: 624 MB/s
HDParm (Disk): 835.78 MB/s
HDParm (Cache): 836.21 MB/s
FIO 4KRandW: 127,204 IOPS
FIO 4KRandR: 217,872 IOPS
FIO 4KRandW: 508,819 KB/s
FIO 4KRandR: 871,489 KB/s
IOZone 4KR: 208,865 KB/s
IOZone 4KW: 268,703 KB/s
OZone 4KRandR: 83,905 KB/s
OZone 4KRandW: 352,386 KB/s

- 7.1.0-rc1 Benchmark Results:
DD Write: 381 MB/s
HDParm (Disk): 437.69 MB/s
HDParm (Cache): 434.26 MB/s
FIO 4KRandW: 106,113 IOPS
FIO 4KRandR: 110,702 IOPS
FIO 4KRandW: 424,455 KB/s
FIO 4KRandR: 442,810 KB/s
IOZone 4KR: 174,619 KB/s
IOZone 4KW: 209,005 KB/s
OZone 4KRandR: 73,331 KB/s
OZone 4KRandW: 223,401 KB/s

(both with dtparam=pciex1; dtparam=pciex1_gen=3)

Steps to reproduce the behaviour

Run benchmarking from https://pibenchmarks.com/ via sudo curl https://raw.githubusercontent.com/TheRemote/PiBenchmarks/master/Storage.sh | sudo bash

Device (s)

Raspberry Pi 1 Mod. B+, Raspberry Pi 5

System

vcgencmd version
2026/04/27 01:06:58
Copyright (c) 2012 Broadcom
version c2a7b9d3 (release) (embedded)

Logs

No response

Additional context

No response

[popcornmix]

I do see a new upstream commit that may affect may pcie link speed:
03f9209

(but it could well be something else).

[pelwell]

I think that specific patch is essentially just input validation (a range check before the array lookup), but there has clearly been work in that area.

[popcornmix]

A diff of lspci output shows

22c22
< 		LnkCap:	Port #0, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <2us
---
> 		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <2us
26c26
< 		LnkSta:	Speed 8GT/s, Width x1
---
> 		LnkSta:	Speed 5GT/s, Width x1
40,41c40,41
< 		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS+
< 		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
---
> 		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS+
> 		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
44,45c44,45
< 		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
< 			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
---
> 		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
> 			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
100c100
< 		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
---
> 		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
105c105
< 		LnkSta:	Speed 8GT/s, Width x1 (downgraded)
---
> 		LnkSta:	Speed 5GT/s (downgraded), Width x1 (downgraded)
120,121c120,121
< 		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
< 			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
---
> 		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
> 			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-

Looks like 7.0 kernel gets gen3 and 7.1 gets gen2 speeds.

[popcornmix]

It is the commit I referenced.
Reverting the change of:

        ret = of_pci_get_max_link_speed(np);
        pcie->gen = (ret < 0) ? 0 : ret;

to

        ret = of_pci_get_max_link_speed(np);
        if (pcie_get_link_speed(ret) == PCI_SPEED_UNKNOWN)
                pcie->gen = 0;

gets the speed back. I've pushed a commit.

[pelwell]

I stand corrected - it seems to be missing an assignment to pcie->gen in the non-error case.

Any suggestions, @ffainelli?

[ffainelli]

Would not this take care of what we need:

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index 714bcab97b60..ff3fe861429f 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -2072,7 +2072,8 @@ static int brcm_pcie_probe(struct platform_device *pdev)
                return PTR_ERR(pcie->clk);

        ret = of_pci_get_max_link_speed(np);
-       if (pcie_get_link_speed(ret) == PCI_SPEED_UNKNOWN)
+       pcie->gen = pci_get_link_speed(ret);
+       if (pcie->gen == PCI_SPEED_UNKNOWN)
                pcie->gen = 0;

        pcie->ssc = of_property_read_bool(np, "brcm,enable-ssc");

And if we don't want to be defensive because PCI_SPEED_UNKNOWN defaults to 0, we could just do this:

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index 714bcab97b60..db14acfa318d 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -2072,8 +2072,7 @@ static int brcm_pcie_probe(struct platform_device *pdev)
                return PTR_ERR(pcie->clk);

        ret = of_pci_get_max_link_speed(np);
-       if (pcie_get_link_speed(ret) == PCI_SPEED_UNKNOWN)
-               pcie->gen = 0;
+       pcie->gen = pci_get_link_speed(ret);

        pcie->ssc = of_property_read_bool(np, "brcm,enable-ssc");

[ffainelli]

A revert works just as well, not sure what the point of going through pci_get_link_speed() is, shouldn't have applied a Reviewed-by tag here.