i40e AMD-Vi IO_PAGE_FAULT

st6f9n · ‎12-14-2023

Hello,

one of our servers crashes regularly, apparently during heavy network load. The log files are then full of this message:

kernel: [514257.305733] i40e 0000:02:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0020 address=0x79ea8113f60 flags=0x0000]

This is the driver:

i40e: Intel(R) Ethernet Connection XL710 Network Driver
i40e: Copyright (c) 2013 - 2019 Intel Corporation.
i40e 0000:02:00.0: fw 8.5.67516 api 1.15 nvm 8.50 0x8000be1e 1.3295.0 [8086:15ff] [15d9:1c76]
i40e 0000:02:00.0: MAC address: 7c:c2:55:9d:d2:78
i40e 0000:02:00.0: FW LLDP is enabled
i40e 0000:02:00.0 eth0: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
i40e 0000:02:00.0: PCI-Express: Speed 8.0GT/s Width x4
i40e 0000:02:00.0: PCI-Express bandwidth available for this device may be insufficient for optimal performance.
i40e 0000:02:00.0: Please move the device to a different PCI-e link with more lanes and/or higher transfer rate.
i40e 0000:02:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 119 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
i40e 0000:02:00.1: fw 8.5.67516 api 1.15 nvm 8.50 0x8000be1e 1.3295.0 [8086:15ff] [15d9:1c76]
i40e 0000:02:00.1: MAC address: 7c:c2:55:9d:d2:79
i40e 0000:02:00.1: FW LLDP is enabled
i40e 0000:02:00.1: PCI-Express: Speed 8.0GT/s Width x4
i40e 0000:02:00.1: PCI-Express bandwidth available for this device may be insufficient for optimal performance.
i40e 0000:02:00.1: Please move the device to a different PCI-e link with more lanes and/or higher transfer rate.
i40e 0000:02:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 119 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
i40e 0000:02:00.0 enp2s0f0: renamed from eth0
i40e 0000:02:00.1 enp2s0f1: renamed from eth1
i40e 0000:02:00.0: entering allmulti mode.

This message stands out there:

i40e 0000:02:00.1: PCI-Express bandwidth available for this device may be insufficient for optimal performance.
i40e 0000:02:00.1: Please move the device to a different PCI-e link with more lanes and/or higher transfer rate.

Does somebody has any idea ? GRUB_CMDLINE_LINUX_DEFAULT="iommu=soft" is sometimes recommended in similar cases with IO_PAGE_FAULT. Maybe I should lower the speed from 10 Gbit/s to 1 Gbit/s as a test?

Thanks

Stefan

IntelSupport · ‎12-15-2023

Hello st6f9n,

Thank you for posting in Intel Communities.

Can you please update with further information for below:

The server model:

Intel(R) Ethernet Connection XL710 Network Driver Version:

PCIe lane used for XL710:

Is the XL710 prebuilt in the server or bought separately?:

Kindly let us know if you have any questions.

Regards,

Hayat

Intel Customer Support

st6f9n · ‎12-15-2023

Hello Hayat,

here are the requested data.

The server model: SUPERMICRO – 4U GPU A+ Server AS-4125GS-TNRT

Intel(R) Ethernet Connection XL710 Network Driver Version ("modinfo i40e"):

filename: /lib/modules/5.15.0-87-generic/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
license: GPL v2
description: Intel(R) Ethernet Connection XL710 Network Driver
author: Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion: 263AAE07A824C74401C1729
alias: pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias: pci:v00008086d0000158Asv*sd*bc*sc*i*
alias: pci:v00008086d00000D58sv*sd*bc*sc*i*
alias: pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias: pci:v00008086d00001588sv*sd*bc*sc*i*
alias: pci:v00008086d00001587sv*sd*bc*sc*i*
alias: pci:v00008086d000037D3sv*sd*bc*sc*i*
alias: pci:v00008086d000037D2sv*sd*bc*sc*i*
alias: pci:v00008086d000037D1sv*sd*bc*sc*i*
alias: pci:v00008086d000037D0sv*sd*bc*sc*i*
alias: pci:v00008086d000037CFsv*sd*bc*sc*i*
alias: pci:v00008086d000037CEsv*sd*bc*sc*i*
alias: pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias: pci:v00008086d0000104Esv*sd*bc*sc*i*
alias: pci:v00008086d000015FFsv*sd*bc*sc*i*
alias: pci:v00008086d00001589sv*sd*bc*sc*i*
alias: pci:v00008086d00001586sv*sd*bc*sc*i*
alias: pci:v00008086d00001585sv*sd*bc*sc*i*
alias: pci:v00008086d00001584sv*sd*bc*sc*i*
alias: pci:v00008086d00001583sv*sd*bc*sc*i*
alias: pci:v00008086d00001581sv*sd*bc*sc*i*
alias: pci:v00008086d00001580sv*sd*bc*sc*i*
alias: pci:v00008086d00001574sv*sd*bc*sc*i*
alias: pci:v00008086d00001572sv*sd*bc*sc*i*
depends:
retpoline: Y
intree: Y
name: i40e
vermagic: 5.15.0-87-generic SMP mod_unload modversions
sig_id: PKCS#7
signer: Build time autogenerated kernel key
sig_key: 24:9A:1B:01:50:7E:73:BC:D1:48:A7:32:C1:4C:25:07:D8:F9:A9:75
sig_hashalgo: sha512
signature: 18:D1:9A:1E:E6:90:3D:62:7C:DB:C5:BA:CB:E1:FF:3C:48:D4:CC:E2:
09:62:8F:C3:57:F8:A7:7F:D1:EA:FE:67:05:D0:82:98:0E:0D:C8:08:
0C:EE:3C:14:C3:AE:13:2B:85:2F:3E:DB:5F:4C:7A:28:13:FF:5F:D5:
E7:52:85:49:B8:8F:6B:AD:57:57:9D:77:FE:E4:77:BD:A6:B6:53:C8:
C6:C3:20:4C:72:63:64:BE:C2:88:53:CC:BE:9D:38:81:61:C2:89:A3:
D6:FB:4C:24:2B:6B:EA:C0:3B:D1:0C:5E:69:DD:76:50:31:C0:B3:98:
E4:73:DC:62:AE:A7:67:85:B0:C5:40:71:A5:12:F2:03:E3:70:3E:A4:
EC:67:17:2C:9A:68:5D:F0:66:A0:40:7A:66:60:FC:52:A9:B9:48:97:
BF:17:80:50:AE:38:AF:B4:9E:3F:2D:06:69:20:57:C4:03:B3:64:89:
FB:FE:7A:97:17:03:F8:B2:E8:FB:F8:69:B7:6F:C3:04:7C:DD:DD:4E:
0F:FB:D7:AF:69:53:57:2F:64:E9:5C:C1:49:6D:AB:80:A6:CF:E4:BE:
8B:DC:0B:00:80:27:FF:F6:8C:9F:C7:2A:A5:3B:12:AB:0B:4E:50:EA:
87:CF:4C:57:7C:A2:13:61:91:FD:32:45:EA:C9:F9:F1:88:DC:37:87:
B6:05:3C:A1:FA:8F:3B:8E:97:5D:1C:69:80:70:05:15:2B:01:35:D4:
8D:D2:B1:34:47:EE:56:CB:A3:4F:01:0B:A6:6A:1B:A8:D1:B4:C5:05:
DB:8C:A3:69:0F:C7:D9:79:54:CE:62:BE:0B:ED:98:8B:7E:FF:5C:0E:
4D:F8:D6:DC:F9:B5:02:2C:5E:D1:1F:89:71:D0:12:F7:92:CB:C5:B7:
F3:A6:18:D0:AC:EB:C0:CC:9F:68:BB:EA:DF:34:4F:4B:1C:AA:79:D2:
50:A3:E5:3D:39:50:20:A2:A3:02:BA:F7:44:1B:33:6D:1E:D7:F9:5E:
45:5F:A3:DF:B6:8F:EC:FA:E9:25:9D:43:D1:74:7A:E0:89:4E:B9:66:
B0:DD:A3:31:6E:2F:8B:B5:EA:5A:72:EB:01:AF:CA:92:98:02:4F:16:
95:D4:B1:AE:C1:53:ED:D8:32:73:6A:D7:D6:00:98:C8:5C:17:C6:D8:
4E:42:66:29:9A:5C:83:D5:A4:64:F4:CC:FC:43:6B:33:B6:77:F0:6B:
C4:CB:36:E8:57:4D:E0:35:E6:10:E3:C5:9C:D7:79:03:96:5B:1D:13:
FD:71:E5:26:46:FE:49:70:96:36:7B:15:35:FA:5C:DB:B7:42:75:D8:
0F:C2:5F:43:4C:65:5D:0C:D3:8A:C2:87
parm: debug:Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXX) (uint)

PCIe lane used for XL710:

I don't know, this is the description: "2x RJ45 10GbE LAN ports (rear)" and there is something about "PCI-E 5.0" in the description

Is the XL710 prebuilt in the server or bought separately?: prebuilt

Best regards,

Stefan

IntelSupport · ‎12-15-2023

Hi st6f9n,

Thank you for your response.

According to the information, it is like an OEM related issues.

It is advisable for you to contact Supermicro(OEM) for further information.

You might need to update your ethernet driver and the best way is to contact the OEM as they have different configurations and versions.

Kindly let us know if you have any questions. Else if none, kindly confirm for the case closure.

Regards,

Hayat

Intel Customer Support

display_name1 · ‎12-21-2023

Have you heard back from Supermicro? We are having a similar issue with our SUPERMICRO – 4U GPU A+ Server AS-4125GS-TNRT.