Intel I226-V unable to change power state from D3cold to D0

SunnyWong · ‎08-17-2023

I got one of the TopTon devices with a Intel N200 cpu and 5x I226-V Network interfaces. I'm running Proxmox 8 on it and trying to make use of the PCIe pass-trough feature.

I enabled the virtualization features in the bios and added the required vfio modules. However, it seems there is a problem with accessing the devices. Furthermore, it seems that I'm not the only one having this issue, see intel-i226-v-pci-passthrough-failure.130632 .

I attached an excerpt from my journal log since the boot.

I think the major part sticking out is:

Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3: AER: Multiple Corrected error received: 0000:00:1c.3
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3: AER: can't find device of ID00e3
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3: AER: Multiple Corrected error received: 0000:00:1c.3
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3: AER: can't find device of ID00e3
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3: AER: Corrected error received: 0000:00:1c.3
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3:   device [8086:54bb] error status/mask=00000001/00002000
Aug 17 20:25:14 pve kernel: pcieport 0000:00:1c.3:    [ 0] RxErr                  (First)
Aug 17 20:25:15 pve kernel: igc 0000:04:00.0 enp4s0: PCIe link lost, device now detached
Aug 17 20:25:15 pve kernel: ------------[ cut here ]------------
Aug 17 20:25:15 pve kernel: igc: Failed to read reg 0x5b50!
Aug 17 20:25:15 pve kernel: WARNING: CPU: 2 PID: 54 at drivers/net/ethernet/intel/igc/igc_main.c:6412 igc_rd32+0xa4/0xc0 [igc]
Aug 17 20:25:15 pve kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_>
Aug 17 20:25:15 pve kernel:  cmdlinepart snd_timer i2c_algo_bit input_leds mei_me joydev syscopyarea snd sysfillrect spi_nor soundcore mtd mei sysimgblt acpi>
Aug 17 20:25:15 pve kernel: CPU: 2 PID: 54 Comm: kworker/2:1 Tainted: P           O       6.2.16-8-pve #1
Aug 17 20:25:15 pve kernel: Hardware name: Default string Default string/Default string, BIOS 5.27 05/15/2023
Aug 17 20:25:15 pve kernel: Workqueue: events igc_watchdog_task [igc]
Aug 17 20:25:15 pve kernel: RIP: 0010:igc_rd32+0xa4/0xc0 [igc]
Aug 17 20:25:15 pve kernel: Code: c7 c6 20 85 2b c0 e8 6b b2 aa c4 48 8b bb 28 ff ff ff e8 7f 76 60 c4 84 c0 74 b4 44 89 e6 48 c7 c7 48 85 2b c0 e8 0c 2e e4 >
Aug 17 20:25:15 pve kernel: RSP: 0018:ffffb420c035bc90 EFLAGS: 00010246
Aug 17 20:25:15 pve kernel: RAX: 0000000000000000 RBX: ffffa04dd31cac60 RCX: 0000000000000000
Aug 17 20:25:15 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Aug 17 20:25:15 pve kernel: RBP: ffffb420c035bca8 R08: 0000000000000000 R09: 0000000000000000
Aug 17 20:25:15 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000005b50
Aug 17 20:25:15 pve kernel: R13: ffffa04dd31ca000 R14: 0000000000020002 R15: 0000000000000002
Aug 17 20:25:15 pve kernel: FS:  0000000000000000(0000) GS:ffffa0512fb00000(0000) knlGS:0000000000000000
Aug 17 20:25:15 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 17 20:25:15 pve kernel: CR2: 000055ce7560fff4 CR3: 0000000201610000 CR4: 0000000000750ee0
Aug 17 20:25:15 pve kernel: PKRU: 55555554
Aug 17 20:25:15 pve kernel: Call Trace:
Aug 17 20:25:15 pve kernel:  <TASK>
Aug 17 20:25:15 pve kernel:  igc_get_hw_semaphore_i225+0x4e/0x190 [igc]
Aug 17 20:25:15 pve kernel:  igc_acquire_swfw_sync_i225+0x5e/0xd0 [igc]
Aug 17 20:25:15 pve kernel:  igc_acquire_phy_base+0x14/0x30 [igc]
Aug 17 20:25:15 pve kernel:  igc_read_phy_reg_gpy+0x38/0x1e0 [igc]
Aug 17 20:25:15 pve kernel:  igc_phy_has_link+0x94/0x170 [igc]
Aug 17 20:25:15 pve kernel:  igc_check_for_copper_link+0x98/0x160 [igc]
Aug 17 20:25:15 pve kernel:  igc_has_link+0x40/0xb0 [igc]
Aug 17 20:25:15 pve kernel:  igc_watchdog_task+0x44/0x570 [igc]
Aug 17 20:25:15 pve kernel:  ? psi_avgs_work+0x67/0xe0
ug 17 20:25:15 pve kernel:  process_one_work+0x222/0x430
Aug 17 20:25:15 pve kernel:  worker_thread+0x50/0x3e0
Aug 17 20:25:15 pve kernel:  ? __pfx_worker_thread+0x10/0x10
Aug 17 20:25:15 pve kernel:  kthread+0xe6/0x110
Aug 17 20:25:15 pve kernel:  ? __pfx_kthread+0x10/0x10
Aug 17 20:25:15 pve kernel:  ret_from_fork+0x29/0x50
Aug 17 20:25:15 pve kernel:  </TASK>
Aug 17 20:25:15 pve kernel: ---[ end trace 0000000000000000 ]---
Aug 17 20:25:16 pve kernel: pcieport 0000:00:1c.6: Data Link Layer Link Active not set in 1000 msec
Aug 17 20:25:16 pve kernel: vfio-pci 0000:05:00.0: Unable to change power state from D3cold to D0, device inaccessible
Aug 17 20:25:16 pve kernel: vfio-pci 0000:05:00.0: Unable to change power state from D3cold to D0, device inaccessible
Aug 17 20:25:17 pve kernel: vfio-pci 0000:05:00.0: timed out waiting for pending transaction; performing function level reset anyway
[...]
Aug 17 20:25:18 pve kernel: vfio-pci 0000:05:00.0: not ready 1023ms after FLR; waiting
Aug 17 20:25:19 pve kernel: vfio-pci 0000:05:00.0: not ready 2047ms after FLR; waiting
Aug 17 20:25:21 pve kernel: vfio-pci 0000:05:00.0: not ready 4095ms after FLR; waiting
Aug 17 20:25:26 pve kernel: vfio-pci 0000:05:00.0: not ready 8191ms after FLR; waiting
Aug 17 20:25:34 pve kernel: vfio-pci 0000:05:00.0: not ready 16383ms after FLR; waiting
Aug 17 20:25:52 pve kernel: vfio-pci 0000:05:00.0: not ready 32767ms after FLR; waiting
Aug 17 20:26:27 pve kernel: vfio-pci 0000:05:00.0: not ready 65535ms after FLR; giving up
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.6: Data Link Layer Link Active not set in 1000 msec
Aug 17 20:26:29 pve kernel: vfio-pci 0000:05:00.0: Unable to change power state from D3cold to D0, device inaccessible
Aug 17 20:26:29 pve kernel: igc 0000:04:00.0 enp4s0: PHC removed
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3: AER: Multiple Corrected error received: 0000:00:1c.3
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3:   device [8086:54bb] error status/mask=00002001/00002000
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3:    [ 0] RxErr                  (First)
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3: AER: Multiple Corrected error received: 0000:00:1c.3
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3: AER: can't find device of ID00e3
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3: AER: Multiple Corrected error received: 0000:00:1c.3
Aug 17 20:26:29 pve kernel: pcieport 0000:00:1c.3: AER: can't find device of ID00e3
[...]
Aug 17 20:26:32 pve kernel: vfio-pci 0000:04:00.0: Unable to change power state from unknown to D0, device inaccessible
Aug 17 20:26:32 pve kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Aug 17 20:26:32 pve kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Aug 17 20:26:32 pve kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Aug 17 20:26:33 pve kernel: vfio-pci 0000:04:00.0: timed out waiting for pending transaction; performing function level reset anyway
Aug 17 20:26:34 pve kernel: vfio-pci 0000:04:00.0: not ready 1023ms after FLR; waiting

From the previous link, there seems to be a workaround by disabling the power management with

`pcie_port_pm=off`. However, I hope I can raise awareness of this issue with that message.

I'm also more than open to help delivering more information and for testing.

Fariz_Intel · ‎08-21-2023

Hi SunnyWong,

We deeply apologize for any inconvenience caused by this issue.

We sincerely appreciate your information and would like to thank you for taking the time to share your thoughts regarding the Intel I226-V issue. Your insights are invaluable to us as we strive to keep improving our support.

Indeed, there's a known issue with the Intel I226-V, and it's recommended to disable the “Energy Efficient Ethernet (EEE)” mode or power management for both Windows and Linux advanced driver settings. By doing so, it might mitigate the issue, which could also be related to your problem.

A fix is included in the latest driver (for Windows); however, the results could vary depending on the motherboard and operating system. Intel will continue to work towards finding/providing a robust solution while simultaneously improving the driver.

We greatly appreciate your dedication to helping deliver more information and assisting with testing. You may continue to do so by following and posting in the topic titled 'Intel Communication: Intel® Ethernet Controller I226 Series-Random Connection Drops at Random Times,' as per the link below.

https://community.intel.com/t5/Ethernet-Products/Intel-Communication-Intel-Ethernet-Controller-I226-Series-Random/td-p/1453177/page/3

We've also noticed a few similar issues posted in the community (link below), mostly in Windows environments. Some of them have been resolved through driver updates, while others might require going through RMA with the OEM or retailer. However, we believe it's worth looking into.

Intel(R) Ethernet Controller I226-V Download Speed drop MSI MAG Z790 TOMAHAWK WIFI DDR5

https://community.intel.com/t5/Ethernet-Products/Intel-R-Ethernet-Contoller-I226-V-Download-Speed-drop-MSI-MAG/m-p/1450780

Intel I226-V Connection Issues

https://community.intel.com/t5/Ethernet-Products/Intel-I226-V-Connection-Issues/td-p/1465810

Thank you once again for bringing these matters to our attention, and please let us know if you would like to keep this post open for any further assistance or questions.

Best regards,

Fariz_Intel

Fariz_Intel · ‎09-05-2023

Hi SunnyWong,

If you have any additional questions or need further clarifications, please feel free to let us know.

Best regards,

Fariz_Intel

SunnyWong · ‎09-10-2023

Hi Fariz,

thanks for your fast response.

I checked the mirrored Linux repo on Github, in particular intel/igc and saw that you guys are working on the driver.

Telling me, that you're aware of this issue and actively working on it, is literally all I could ask for. Thanks for your kind response and to the linux driver dev team working on this!

My highest regards

- SW

Fariz_Intel · ‎10-20-2023

Hi SunnyWong,

Yes, this issue is acknowledged, and Intel will keep working towards improving the driver. Thank you very much for sharing the insight; we greatly appreciate it. If you need any additional information, please submit a new question, as this thread will no longer be monitored.

Best regards,

Fariz_Intel