Hi,
After updating to 6.14.2, the ethernet adapter is almost unusable, I get over 30% packet loss. Bisect says it's this commit:
commit 85f6414167da39e0da30bf370f1ecda5a58c6f7b Author: Vitaly Lifshits vitaly.lifshits@intel.com Date: Thu Mar 13 16:05:56 2025 +0200
e1000e: change k1 configuration on MTP and later platforms
[ Upstream commit efaaf344bc2917cbfa5997633bc18a05d3aed27f ]
Starting from Meteor Lake, the Kumeran interface between the integrated MAC and the I219 PHY works at a different frequency. This causes sporadic MDI errors when accessing the PHY, and in rare circumstances could lead to packet corruption.
To overcome this, introduce minor changes to the Kumeran idle state (K1) parameters during device initialization. Hardware reset reverts this configuration, therefore it needs to be applied in a few places.
Fixes: cc23f4f0b6b9 ("e1000e: Add support for Meteor Lake") Signed-off-by: Vitaly Lifshits vitaly.lifshits@intel.com Tested-by: Avigail Dahan avigailx.dahan@intel.com Signed-off-by: Tony Nguyen anthony.l.nguyen@intel.com Signed-off-by: Sasha Levin sashal@kernel.org
drivers/net/ethernet/intel/e1000e/defines.h | 3 ++ drivers/net/ethernet/intel/e1000e/ich8lan.c | 80 +++++++++++++++++++++++++++-- drivers/net/ethernet/intel/e1000e/ich8lan.h | 4 ++ 3 files changed, 82 insertions(+), 5 deletions(-)
My system is Novacustom V540TU laptop with Intel Core Ultra 5 125H. And the e1000e driver is running in a Xen HVM (with PCI passthrough). Interestingly, I have also another one with Intel Core Ultra 7 155H where the issue does not happen. I don't see what is different about network adapter there, they look identical on lspci (but there are differences about other devices)...
I see the commit above was already backported to other stable branches too...
#regzbot introduced: 85f6414167da39e0da30bf370f1ecda5a58c6f7b
On 4/14/2025 3:18 PM, Marek Marczykowski-Górecki wrote:
Hi,
After updating to 6.14.2, the ethernet adapter is almost unusable, I get over 30% packet loss. Bisect says it's this commit:
commit 85f6414167da39e0da30bf370f1ecda5a58c6f7b Author: Vitaly Lifshits <vitaly.lifshits@intel.com> Date: Thu Mar 13 16:05:56 2025 +0200 e1000e: change k1 configuration on MTP and later platformsMy system is Novacustom V540TU laptop with Intel Core Ultra 5 125H. And the e1000e driver is running in a Xen HVM (with PCI passthrough). Interestingly, I have also another one with Intel Core Ultra 7 155H where the issue does not happen. I don't see what is different about network adapter there, they look identical on lspci (but there are differences about other devices)...
I see the commit above was already backported to other stable branches too...
#regzbot introduced: 85f6414167da39e0da30bf370f1ecda5a58c6f7b
Thank you for this report.
Do you see the high packet loss without the virtualization? Can you please share the lspci output? Does your switch/link partner support flow control? if it is configurable can you try to enable it? Do you see any errors in dmesg related to the e1000e driver?
On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
Do you see the high packet loss without the virtualization?
I can't check that easily right now, will try later.
Can you please share the lspci output?
Sure:
00:07.0 Ethernet controller [0200]: Intel Corporation Device [8086:550a] (rev 20) Subsystem: CLEVO/KAPOK Computer Device [1558:a743] Physical Slot: 7 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin D routed to IRQ 69 Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=128K] Capabilities: [c8] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Kernel driver in use: e1000e Kernel modules: e1000e
Does your switch/link partner support flow control? if it is configurable can you try to enable it?
It does support it. Enabling it makes things much worse...
Do you see any errors in dmesg related to the e1000e driver?
Not really. dmesg | grep 'e1000e|ens7':
[ 3.088489] e1000e: Intel(R) PRO/1000 Network Driver [ 3.088512] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 3.093256] e1000e 0000:00:07.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 3.343378] e1000e 0000:00:07.0 0000:00:07.0 (uninitialized): registered PHC clock [ 3.718946] e1000e 0000:00:07.0 eth0: (PCI Express:2.5GT/s:Width x1) d4:93:90:3e:0d:bb [ 3.718966] e1000e 0000:00:07.0 eth0: Intel(R) PRO/1000 Network Connection [ 3.719101] e1000e 0000:00:07.0 eth0: MAC: 16, PHY: 12, PBA No: FFFFFF-0FF [ 3.759444] e1000e 0000:00:07.0 ens7: renamed from eth0 [ 8.632317] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 239.458205] e1000e 0000:00:07.0 ens7: NIC Link is Down [ 242.485869] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
(you can also see a test with flow control above)
And also ethtool output if useful: Settings for ens7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes
On 4/14/2025 3:58 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
Do you see the high packet loss without the virtualization?
I can't check that easily right now, will try later.
Can you please share the lspci output?
Sure:
00:07.0 Ethernet controller [0200]: Intel Corporation Device [8086:550a] (rev 20) Subsystem: CLEVO/KAPOK Computer Device [1558:a743] Physical Slot: 7 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin D routed to IRQ 69 Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=128K] Capabilities: [c8] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Kernel driver in use: e1000e Kernel modules: e1000e
Do you have mei modules running? Can you try if disabling them make things better?
Does your switch/link partner support flow control? if it is configurable can you try to enable it?
It does support it. Enabling it makes things much worse...
Do you see any errors in dmesg related to the e1000e driver?
Not really. dmesg | grep 'e1000e|ens7':
[ 3.088489] e1000e: Intel(R) PRO/1000 Network Driver [ 3.088512] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 3.093256] e1000e 0000:00:07.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 3.343378] e1000e 0000:00:07.0 0000:00:07.0 (uninitialized): registered PHC clock [ 3.718946] e1000e 0000:00:07.0 eth0: (PCI Express:2.5GT/s:Width x1) d4:93:90:3e:0d:bb [ 3.718966] e1000e 0000:00:07.0 eth0: Intel(R) PRO/1000 Network Connection [ 3.719101] e1000e 0000:00:07.0 eth0: MAC: 16, PHY: 12, PBA No: FFFFFF-0FF [ 3.759444] e1000e 0000:00:07.0 ens7: renamed from eth0 [ 8.632317] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 239.458205] e1000e 0000:00:07.0 ens7: NIC Link is Down [ 242.485869] e1000e 0000:00:07.0 ens7: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
(you can also see a test with flow control above)
And also ethtool output if useful: Settings for ens7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes
And also ethtool output if useful: Settings for ens7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
Can you please also share the output of ethtool -i? I would like to know the NVM version that you have on your device.
driver: e1000e version: 6.14.1+ firmware-version: 1.1-4 expansion-rom-version: bus-info: 0000:00:1f.6 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes
Your firmware version is not the latest, can you check with the board manufacturer if there is a BIOS update to your system?
I can check, but still, it's a regression in the Linux driver - old kernel did work perfectly well on this hw. Maybe new driver tries to use some feature that is missing (or broken) in the old firmware?
A little bit of context: I'm maintaining the kernel package for a Qubes OS distribution. While I can try to update firmware on my test system, I have no influence on what hardware users will use this kernel, and which firmware version they will use (and whether all the vendors provide newer firmware at all). I cannot ship a kernel that is known to break network on some devices.
Also, you mentioned that on another system this issue doesn't reproduce, do they have the same firmware version?
The other one has also 1.1-4 firmware. And I re-checked, e1000e from 6.14.2 works fine there.
Dear Marek,
Thank you for your detailed feedback and for providing the requested information.
We have conducted extensive testing of this patch across multiple systems and have not observed any packet loss issues. Upon comparing the mentioned setups, we noted that while the LAN controller is similar, the CPU differs. We believe that the issue may be related to transitions in the CPU's low power states.
Consequently, we kindly request that you disable the CPU low power state transitions in the S0 system state and verify if the issue persists. You can disable this in the kernel parameters on the command line with idle=poll. Please note that this command is intended for debugging purposes only, as it may result in higher power consumption.
Please inform us if disabling the CPU low power states resolves the issue or if further investigation is required. As previously mentioned, this patch is critical for the operation of Meteor Lake LAN devices, and therefore, we are unable to revert it.
Thank you for your cooperation.
On Mon, Apr 14, 2025 at 02:58:09PM +0200, Marek Marczykowski-Górecki wrote:
On Mon, Apr 14, 2025 at 03:38:39PM +0300, Lifshits, Vitaly wrote:
Do you see the high packet loss without the virtualization?
I can't check that easily right now, will try later.
Tried now, the same issue is without any virtualization too.
linux-stable-mirror@lists.linaro.org
-
Lifshits, Vitaly -
Marek Marczykowski-Górecki