DELL Poweredge R6615: unable to see SR-IOV capabilities for the Intel X710/X557-AT 10GBASE-T

[LucaAstori]

Hi have an issue enabling SR-IOV for a DELL Poweredge R6615 server equipped with Intel X710-T4 Network Interface Card (PCIe, NOT onboard network card) .

Server:DELL Poweredge R6615
CPU: AMD
Network card: Intel X710-T4 Network Interface Card (PCIe)
OS: RedHat Enterprise Linux 9.4
hypervisor: KVM

- I verified that the SR-IoV feature and IOMMU are enabled into the BIOS
- verified that the RHEL load the SR-IOV modukle with AMD-IOMMU=on and IOMMU=pt

[usereref-fefwhyp-Ol ~]$ dmesg | grep -e IOMMU -e AND-VI

[ 0.457968] pci 0000:c0:00.2: AND-Vi: ; T‘" performance counters supported
[ 0.459133] pci 0000:80:00.2: AND-Vi: 1‘1"" performance counters supported
[ 0.459816] pci 0000:00:00.2: AND-Vi: WIT"V performance counters supported
[ 0.460827] pci 0000:40:00.2: AND-Vi: 1‘3", performance counters supported
[ 0.463142] perf/amd_iommu: Detected AMD "1‘7‘ #0 (2 banks, 4 counters/bank).
[ 0.463147] perf/amd_iommu: Detected AMD "I'T #1 (2 banks, 4 counters/bank).
[ 0.463152] perf/amd_iommu: Detected AND 1‘J #2 (2 banks, 4 counters/bank).
[ 0.463156] perf/amd_iommu: Detected AMD 1‘; #3 (2 banks, 4 counters/bank).

[user@ref-fefwhyp-01 ~]$ lspcil grep -i Ethernet
c3:00.0 , 3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCH5720 Gigabit ' '- , PCIe
c3:00.1 , Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit r '7 , PCIe
c4:00.0 ; Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT IOGBASE—T (rev 02)
c4:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT loGBASE-T (rev 02)
c4:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT loGBASE-T (rev 02)
c4:00.3 Ethernet controller: Intel Corporation Ethernet Controller x710/X557-AT 1068ASE-T (rev 02)

[user@ref-fefwhyp-01 ~]$

[user@ref-fefwhyp-01 ~]$ dmesg | grep -e IOMMU -e AMD-VI
[ 0.457968] pci 0000:c0:00.2: AND-Vi: IOMMU performance counters supported
[ 0.459133] pci 0000:80:00.2: AND-Vi: IOMMU performance counters pported
[ 0.459816] pci 0000:00:00.2: AMD-Vi: IOMMU performance Counters jfipported
[ 0.460827] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.463142] perf/amd_iommu: Detected AMD IOMMU (2 banks, 4 counters/bank).
[ 0.463147] perf/amd_iommu: Detected AMD IOMMU (2 banks, 4 counters/bank).
[ 0.463152] perf/amd_iommu: Detected AMD IOMMU (2 banks, 4 counters/bank).
[ 0.463156] perf/amd_iommu: Detected AMD IOMMU (2 banks, 4 counters/bank).
[user@ref-fefwhyp-01 ~]$

[root@ref-fefwhyp—01 ~]# lspci -s c4:00.2 —v

c4:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT 10GBASE-T (rev 02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-T
Flags: bus master, fast devsel, latency 0, IRQ 80, NUMA node 0, IOMMU group 12
Memory at c8000000 (64-bit, prefetchable) [size=16M]
Memory at cb008000 (64-bit, prefetchable) [size=32K]
Expansion ROM at cb400000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
Capabilities: [a0] Express Endpoint, M51 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number f0—64-80-ff-ff—fe-fd-3c
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1b0] Access Control Services
Kernel driver in use: i40e
Kernel modules: i40e

the capability:Capabilities: [160] Single Root I/O Virtualization (SR-IOV)seems missing.

1) Are you using a PCIe X710 or onboard? : it's a Dell purchased Intel PCIe X710
2) What is the brand and model of your system or motherboard?: DELL Poweredge R6615 (see below hw detail)

[roothEf-fefwhyp-Ol ~]# dmidecode -t 2

# dmidecode 3.6
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
Handle 0x0200, DMI type 2, 8 bytes

Base Board Information
Manufacturer:
Product Name: 067N9T
Version: A07
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
Handle 0x0400, DMI type 4, 48 bytes
Processor Information
Socket Designation: CPUl
Type: Central Processor
Family: Zen
Manufacturer: AMD
ID: 11 0F A1 00 FF FB 88 17
Signature: Family 25, Model 17, Stepping 1

Flags:

FPU (Floating-point unit on-chip)
VME (Virtual mode extension)
DE (Debugging extension)
PSE (Page size extension)
TSC (Time stamp counter)
MSR (Model specific registers)
PAE (Physical address extension)
MCE (Machine check exception)
CX8 (CMPXCHGS instruction supported)
APIC (On-chip APIC hardware supported)
SEP (Fast system call)
MTRR (Memory type range registers)
PGE (Page global enable)
MCA (Machine check architecture)
CMOV (Conditional move instruction supported)
PAT (Page attribute table)
PSE-36 (36-bit page size extension)
CLFSH (CLFLUSH instruction supported)
MMX (MMX technology supported)
FXSR (FXSAVE and FXSTOR instructions supported)
SSE (Streaming SIMD extensions)
SSE2 (Streaming SIMD extensions 2)
HTT (Multi-threading)
Version: AMD EPYC 9124 16-Core Processor
Voltage: 1.8 V
External Clock: Unknown
Max Speed: 4400 MHz
Current Speed: 3000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: 0x0700
L2 Cache Handle: 0x0701
L3 Cache Handle: 0x0702
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 16
Core Enabled: 16
Thread Count: 32
Characteristics:
64-bit capable
Multi-Core
Hardware Thread
Execute Protection
Enhanced Virtualization

3) If you are using a PCIe card, can you send photos of the card on both sides for us to validate if the card is not an OEM unit? No now i can't open the server chassis

Any ideas?

[Fikri_Intel]

Hi LucaAstori,

Thank you for contacting Intel.

We appreciate you reaching out to us regarding the reported issue. To assist you more effectively and provide an accurate solution, we kindly request the following details:

1- Do the Intel X710 and X557 come with the system or purchased separately?

2- Have you followed the requirement of the below article:

https://www.intel.com/content/www/us/en/support/articles/000005722/ethernet-products.html

Once we have this information, we will be able to guide you better and provide the appropriate support.

Looking forward to your response.

Regards,

Fikri O.

[Fikri_Intel]

Hi LucaAstori,

Good day.

I just wanted to follow up with you. Please let us know if you have further concerns.

Looking forward to your response.

Regards,

Fikri O.

[Fikri_Intel]

Hi LucaAstori,

Hope you are doing well.

'

Since we have not seen an update, we will be closing this forum case from our end. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Regards,

Fikri O.

[LucaAstori]

Hi, sorry for late answer, but was urgent find a solution, and the exit strategy was to change the network card from a x710 (purchased separately from the server but always through Dell), With an 540 that was immediately recognized and correctly identify with SR-IOV functionality.

forgive me, I'm really sorry for the late answer, and thanks for your help

you can close this thread 

regards

Luca