X540 Windows 2016 SR-IOV and VLANs

[DGint1]

In my Windows 2012 VMs under KVM (Proxmox), I can set a VLAN on the virtual function adapter and it creates a second interface on that VLAN - easy!

In Win 2016, I have no such option.

Do I need to go back to 2012 or is there some handle to jiggle to make this work in 2016?

[idata]

Hello Pneumatic,

Thank you for posting in Intel Wired Ethernet Communities. Is this an Intel Retail X540, or an OEM? Which Linux version are you running on the host? Is the Virtual Function already listed in device manager of the guest 2016? Did you have to install the VF from a driver package on 2012? Please provide the lspci | grep X540 output to see if the VF is loaded onto the host. See that the VF is the connection for the PCI host device in VM properties. If you have any questions please let us know.Best regards,Daniel D

[DGint1]

The adapter is embedded on a Supermicro motherboard. The VF adapter works fine in the 2016 guest without a VLAN. It also works fine with or without VLAN in 2012R2 guests. Here's the uname and lspci:

Linux xxxxxxxxxx-host1 4.15.17-3-pve # 1 SMP PVE 4.15.17-13 (Mon, 18 Jun 2018 17:15:04 +0200) x86_64 GNU/Linux

00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 01)

00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)

00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)

00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)

00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)

00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)

00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)

00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)

00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)

00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)

00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)

00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)

00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)

00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01)

00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01)

00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01)

00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 01)

00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)

00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)

00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)

00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller # 1 (rev 05)

00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller # 2 (rev 05)

00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller # 2 (rev 05)

00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port # 1 (rev d5)

00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port # 5 (rev d5)

00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller # 1 (rev 05)

00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)

00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)

00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

01:10.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.3 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.5 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:10.7 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.3 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.5 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:11.7 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.3 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.5 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:12.7 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.3 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.5 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:13.7 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:14.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:14.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:14.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

01:14.3 Ethernet con...

[idata]

Hello Pneumatic,

Have you checked with Supermicro if there is a firmware update that might support this issue? Will investigate this issue and update you as soon as possible. Please let us know if you have any questions.Best regards,Daniel D

[idata]

Hello Pneumatic,

You might also check with KVM's community or support. They may have seen this issue before. Since VLAN's work in Windows Server 2012 it may be an issue relating to KVM. Please let us know if Supermicro or KVM are able to provide a solution. If you have any questions please do not hesitate to ask us.Best regards,Daniel D

[idata]

Hello Pneumatic,

If you still need assistance after contacting Supermicro and KVM please let us know.Best regards,Daniel D

[idata]

Hello Pneumatic,

Please let us know if you have any other questions.Best regards,Daniel D

[DGint1]

The problem persists: the virtual function adapters come up just fine from the linux (4.15.18) Proxmox host and pass through just fine into any and all guests.

The issue is simply that the driver in the Windows 2016 guests do not have any provision for setting a VLAN, which works fine in Windows 2012R2.

I will simply move back to 2012R2 until you guys can figure out the issue. Everything works fine in 2012R2.

[DGint1]

I should note that there is no way for me to install Advanced Networking Services in Windows 2016. It is not an option during the driver install and it is not available for download separately.

[idata]

Hello Pneumatic,

Thank you for the reply. If you have already moved back to 2012 R2 this information may not be readily available, but it will help us investigate the issue further. In the Windows Server 2016 guest were you getting any errors in Windows Event Viewer? Is the Windows revision the latest release? Are you using the latest https://downloadcenter.intel.com/download/26092/Intel-Network-Adapter-Driver-for-Windows-Server-2016-?product=83418 Advanced Network Services package for 2016, and was there a check box to install ANS? Any information you can provide might help. Please let us know if you have any other questions.Best regards,Daniel D

[DGint1]

There is no error - the only difference between the 2012R2 build and the 2016 build is that I am not prompted with "Advanced Networking Services" during the driver install in the latter (it does not appear in the list of things to install). These are server templates that I am building using the latest ISOs directly from Microsoft. I simply finish the install, apply all Windows updates and then move onto drivers.

This works fine in 2012R2 but not in 2016. I am using the latest driver that you have linked. There is no way to download ANS separately. If I could get this, perhaps I could install it manually instead of allowing the driver installer to make this decision for me.

[DGint1]

What does the ANS do specifically to the system in order to set the VLAN ID? I've tried making some changes in the registry to no avail. If I can just set the VLAN manually, then I'll be all set. The driver installs and functions just fine - I just need to put the adapter on a VLAN.

[idata]

Hello Pneumatic,

Thank you for your replies. We will further look into this issue. We will update you as soon as possible.Best regards,Daniel D

[DGint1]

Anything would be helpful if you have it. Otherwise, I need to move on.

[idata]

Hello Pneumatic,

To manually install ANS from the driver package please use the following process:

1. Extract the driver file

2. Locate DxSetup.exe from APPS\PROSETDX\Win64 folder3. Open command prompt with admin rights and navigate to the folder4. Type in - DxSetup.exe /qn ANS=1 /liew install.log

You may wish to try it, but we have only tested it on host systems. We have not tested it with Proxmox. Please double check with Supermicro support and Proxmox support if there is a work around on their end. If you have any other questions please let us know.

Best regards,

Daniel D

[DGint1]

Thanks - the log shows a failure to get the hardware id from the passthrough device. This is likely the culprit:

=== Logging started: 9/19/2018 8:24:44 ===

Action start 8:24:44: INSTALL.

Action start 8:24:44: FindRelatedProducts.

Action ended 8:24:44: FindRelatedProducts. Return value 0.

Action start 8:24:44: AppSearch.

Action ended 8:24:44: AppSearch. Return value 1.

Action start 8:24:44: CA_SET_OSVERSION.

Action ended 8:24:44: CA_SET_OSVERSION. Return value 1.

Action start 8:24:44: LaunchConditions.

Action ended 8:24:44: LaunchConditions. Return value 1.

Action start 8:24:44: ValidateProductID.

Action ended 8:24:44: ValidateProductID. Return value 1.

Action start 8:24:44: CA_RESET_BASE_DRIVERS.

Action ended 8:24:44: CA_RESET_BASE_DRIVERS. Return value 1.

Action start 8:24:44: CostInitialize.

Action ended 8:24:44: CostInitialize. Return value 1.

Action start 8:24:44: CA_RESET_LR_DMIX.

Action ended 8:24:44: CA_RESET_LR_DMIX. Return value 1.

Action start 8:24:44: CA_RESET_DCB.

Action ended 8:24:44: CA_RESET_DCB. Return value 1.

Action start 8:24:44: CA_RESET_FCOE.

Action ended 8:24:44: CA_RESET_FCOE. Return value 1.

Action start 8:24:44: CA_RESET_ADVANCED_INSTALL.

Action ended 8:24:44: CA_RESET_ADVANCED_INSTALL. Return value 1.

Action start 8:24:44: CA_RESET_ISCSI.

Action ended 8:24:44: CA_RESET_ISCSI. Return value 1.

Action start 8:24:44: CA_RESET_IWARP.

Action ended 8:24:44: CA_RESET_IWARP. Return value 1.

Action start 8:24:44: CA_RESET_BASIC_INSTALL.

Action ended 8:24:44: CA_RESET_BASIC_INSTALL. Return value 1.

Action start 8:24:44: CA_SETUP_FILES.

1: [08:24:44] 2: Entering SetupFiles

1: [08:24:44] 2: Local Language is:

1: [08:24:44] 2: ENU

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpCHS.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpCHT.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpDEU.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpENU.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpESN.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpFRA.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpKOR.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpITA.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpJPN.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\DMIXSetpPTB.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\StrProv.DLL

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\StrProvR.DLL

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\uninstall.reg

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\WdfCoInstaller.inf

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\WdfCx32F.DLL

1: [08:24:44] 2: MsiViewFetch failed

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\WdfCx64F.DLL

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\rules.dat

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\rules.map

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\rule.dll

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\SaveRestore.ps1

1: [08:24:44] 2: C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\IntED65.tmp\\UnbindANS.ps1

1: [08:24:44] 2: Exiting SetupFiles

Action ended 8:24:44: CA_SETUP_FILES. Return value 1.

Action start 8:24:44: CA_CHECK_PCI_DEVICE_RULES.

1: [08:24:44] 2: Path to Rule.dll 3: C:\Users\ADMINI~1\AppData\Local\Temp\IntED65.tmp\Rule.dll

1: [08:24:44] 2: Entering -> RefreshPCIBusData()

1: [08:24:44] 2: Call to CM_Reenumerate_DevNode successful

1: [08:24:44] 2: Exiting -> RefreshPCIBusData()

1: [08:24:44] 2: Evaluating device ID : 3: PCI\VEN_1AF4&DEV_1000&SUBSYS_00011AF4&REV_00

1: [08:24:44] 2: Evaluating device ID : 3: PCI\VEN_8086&DEV_1515&SUBSYS_152815D9&REV_01

1: [08:24:44] 2: Evaluating device ID : 3: PCI\VEN_8086&DEV_7000&SUBSYS_11001AF4&REV_00

1: [08:24:44] 2: Evaluating device ID : 3: STORAGE\Volume

1: [08:24:44] 2: Evaluating device ID : 3: ACPI\VEN_PNP&DEV_0A06

1: [08:24:44] 2: Evaluating device ID : 3: SCSI\DiskRed_Hat___________VirtIO0001

1: [08:24:44] 2: Evaluating device ID : 3: PRINTENUM\LocalPrintQueue

1: [08:24:44] 2: Evaluating device ID : 3: ROOT\VOLMGR

1: [08:24:44] 2: Evaluating device ID : 3: PCI\VEN_1B36&DEV_0001&SUBSYS_00000000&REV_00

1: [08:24:44] 2: Evaluating device ID : 3: PCI\VEN_1B36&DEV_0001&SUBSYS_00000000&REV_00

1: [08:24:44] 2: Evaluating device ID : 3: TS_INPT\TS_KBD

1: [08:24:44] 2: Evaluating device ID : 3: ROOT\BasicDisplay

1: [08:24:44] 2: Evaluating device ID : 3: (null)

1: [08:24:44] 2: Evaluating device ID : 3: PCI\VEN_8086&DEV_7020&SUBSYS_11001AF4&REV_01

1: [08:24:44] 2: Evaluating device ID :...

[DGint1]

Here's the relevant businfo from the host (where the VF adapters have an "illegal vendor id":

lshw -c network -businfo

Bus info Device Class Description

=======================================================

pci@0000:01:00.0 eth0 network Ethernet Controller 10-Gigabit X540-AT2

pci@0000:01:00.1 eth1 network Ethernet Controller 10-Gigabit X540-AT2

pci@0000:01:10.0 network Illegal Vendor ID

pci@0000:01:10.2 enp1s16f2 network Illegal Vendor ID

pci@0000:01:10.4 enp1s16f4 network Illegal Vendor ID

pci@0000:01:10.6 enp1s16f6 network Illegal Vendor ID

pci@0000:01:11.0 enp1s17 network Illegal Vendor ID

pci@0000:01:11.2 enp1s17f2 network Illegal Vendor ID

pci@0000:01:11.4 enp1s17f4 network Illegal Vendor ID

pci@0000:01:11.6 enp1s17f6 network Illegal Vendor ID

pci@0000:01:12.0 enp1s18 network Illegal Vendor ID

pci@0000:01:12.2 enp1s18f2 network Illegal Vendor ID

pci@0000:01:12.4 enp1s18f4 network Illegal Vendor ID

pci@0000:01:12.6 enp1s18f6 network Illegal Vendor ID

pci@0000:01:13.0 enp1s19 network Illegal Vendor ID

pci@0000:01:13.2 enp1s19f2 network Illegal Vendor ID

pci@0000:01:13.4 enp1s19f4 network Illegal Vendor ID

pci@0000:01:13.6 enp1s19f6 network Illegal Vendor ID

vmbr0 network Ethernet interface

tap107i0 network Ethernet interface

tap103i0 network Ethernet interface

tap112i0 network Ethernet interface

vmbr0v20 network Ethernet interface

tap106i0 network Ethernet interface

tap105i0 network Ethernet interface

eth0.20 network Ethernet interface

tap101i0 network Ethernet interface

tap104i0 network Ethernet interface

tap102i0 network Ethernet interface

tap108i0 network Ethernet interface

tap109i0 network Ethernet interface

[idata]

Hello Pneumatic,

Thank you for the information. We are currently checking on the Illegal Vendor ID in lshw. Advanced network services is currently not supported on Windows 2016. A single VLAN can be assigned in the advanced tab of device properties. For multiple VLANs a team can be set in Server Manager then VLANs can be assigned to the team. For more information please see the following https://www.intel.com/content/www/us/en/support/articles/000022706/network-and-i-o/ethernet-products.html link. If you have any questions please do not hesitate to ask.

Best regards,

Daniel D

[idata]

Hello Pneumatic,

The Illegal Vendor ID in the bus info is expected in VF adapters. You can even see it in the following link under https://software.intel.com/en-us/articles/configure-sr-iov-network-virtual-functions-in-linux-kvm SR-IOV Virtual Function PCI Passthrough Device. If you have any other questions please do not hesitate to ask.Best regards,Daniel DIntel Customer Support

[idata]

Hello Pneumatic,

Please let us know if you were able to assign VLANs using the process recommended with Server Manager. If you have any questions or concerns please let us know. Best regards,Daniel DIntel Customer Support

[idata]

Hello Pneumatic,

If you have any other questions please let us know.Best regards,

Daniel D

Intel Customer Support