From: Keith Busch kbusch@kernel.org
[ Upstream commit 5b2c214a95942f7997d1916a4c44017becbc3cac ]
NVMe devices from multiple vendors appear to get stuck in a reset state that we can't get out of with an NVMe level Controller Reset. The kernel would report these with messages that look like:
Device not ready; aborting reset, CSTS=0x1
These have historically required a power cycle to make them usable again, but in many cases, a PCIe FLR is sufficient to restart operation without a power cycle. Try it if the initial controller reset fails during any nvme reset attempt.
Signed-off-by: Keith Busch kbusch@kernel.org Reviewed-by: Chaitanya Kulkarni kch@nvidia.com Reviewed-by: Nitesh Shetty nj.shetty@samsung.com Signed-off-by: Christoph Hellwig hch@lst.de Signed-off-by: Sasha Levin sashal@kernel.org ---
LLM Generated explanations, may be completely bogus:
**Backport Status: YES**
This commit should be backported to stable kernel trees based on the following analysis:
## 1. Fixes a Real User-Impacting Bug
The commit addresses a significant hardware issue where NVMe devices from multiple vendors get stuck in a reset state showing "Device not ready; aborting reset, CSTS=0x1". This is a real-world problem that: - Affects multiple vendors' devices (not vendor-specific) - Historically required a full power cycle to recover - Renders the storage device completely unusable
## 2. Minimal and Contained Fix
The code change is very focused and minimal: - Only adds ~20 lines of code in a single location - The fix is contained within the `nvme_pci_configure_admin_queue()` function - Only executes when the initial controller reset fails (error path) - No architectural changes or new features
## 3. Clear Recovery Mechanism
The fix implements a well-understood recovery mechanism: ```c if (result < 0) { struct pci_dev *pdev = to_pci_dev(dev->dev); result = pcie_reset_flr(pdev, false); if (result < 0) return result;
pci_restore_state(pdev); result = nvme_disable_ctrl(&dev->ctrl, false); ```
It uses PCIe Function Level Reset (FLR) as a "bigger hammer" when the NVMe-level controller reset fails, which is a standard PCIe recovery mechanism.
## 4. Low Risk of Regression
- Only triggers in error conditions (when `nvme_disable_ctrl` fails) - Falls back gracefully if FLR also fails - Uses existing, well-tested PCIe APIs (`pcie_reset_flr`, `pci_restore_state`) - Adds informative logging when recovery succeeds
## 5. Follows Stable Tree Rules
- Fixes a bug that prevents device initialization - No new features or functionality - Small, self-contained change - Clear benefit (avoiding power cycles) with minimal risk
## 6. Similar Issues Fixed in Stable
The git history shows multiple commits addressing "stuck reset" issues in the NVMe driver (e.g., `3f674e7b670b`, `ebef7368571d`), indicating this is an ongoing class of problems that stable kernels need to handle.
The commit message explicitly states this affects devices from "multiple vendors" and provides a solution that avoids requiring power cycles, making it a clear candidate for stable backporting to improve system reliability.
drivers/nvme/host/pci.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 37fd1a8ace12..2bddc9f60fec 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1888,8 +1888,28 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev) * might be pointing at! */ result = nvme_disable_ctrl(&dev->ctrl, false); - if (result < 0) - return result; + if (result < 0) { + struct pci_dev *pdev = to_pci_dev(dev->dev); + + /* + * The NVMe Controller Reset method did not get an expected + * CSTS.RDY transition, so something with the device appears to + * be stuck. Use the lower level and bigger hammer PCIe + * Function Level Reset to attempt restoring the device to its + * initial state, and try again. + */ + result = pcie_reset_flr(pdev, false); + if (result < 0) + return result; + + pci_restore_state(pdev); + result = nvme_disable_ctrl(&dev->ctrl, false); + if (result < 0) + return result; + + dev_info(dev->ctrl.device, + "controller reset completed after pcie flr\n"); + }
result = nvme_alloc_queue(dev, 0, NVME_AQ_DEPTH); if (result)