Hyper-V vSwitches do not work with 2.x drivers on a I225-V

[Elu]

I have a system with an I225-V running driver version 1.1.3.28, with a Hyper-V vSwitch configured against it and multiple management OS vNICs (with both VLAN and not). This works correctly.

When any driver from the 2.x branch is installed, e.g. 2.1.3.15, this fails. The vNICs appear to stop receiving any network traffic (packets received count stops incrementing).

This also occurs with a clean setup without VLANs: even with a basic vSwitch, the default management vNIC does not appear to receive any packets.

NIC: (3) I225-V

Working driver version: 1.1.3.28, any other 1.x

Failing driver versions: 2.1.3.15, any other 2.x (e.g. 2.1.1.7)

OS: Windows 11 22H2 Build 22621.2283

Motherboard: Asus Maximus Z690 Hero

BIOS: 2703 (latest)

Intel ME: 16.1.27.2176 (latest)

[B_Y]

Hello @Elu,

Thank you for posting in Intel Ethernet Communities. 

Kindly provide the following information for us to check on your query.

1.) System Support Utility(SSU) log of your system. This will allow us to check your Adapter details and configuration. This would also help us identify if you are using an OEM or retail version of Intel Ethernet Adapter. Kindly refer to the steps below.

a- Go to https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-?product=91600 and download the software.

b- Open SSU.exe

c- Mark the box "Everything" and then click "Scan".

d- When finished scanning, click "Next".

e- Click on "Save" and attach the file to a post.

2.) Since you are using OEM/asus motherboard, have you tried using the driver provided by your manufacturer? Please list down the driver you have tried and its link.

3.) How many systems are affected by this issue?

Looking forward to your response.

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[Elu]

Hi,

1. Attached logs. Logs were gathered while driver version 2.1.3.3 as supplied by Asus/OEM was installed and in a non-working state, with Hyper-V vNICs configured but unable to receive packets.

2. Asus provides driver version 2.1.3.3 from https://dlcdnets.asus.com/pub/ASUS/mb/04LAN/DRV_LAN_Intel_I225_I226_TSD_W11_64_V11438_20230321R.zip?model=ROG%20MAXIMUS%20Z690%20HERO. This does not work either.

Please note that I was initially forcibly updated to 2.1.3.15 via Windows Update, so even if the OEM version worked it would still not be a good long-term solution if I need to manually rollback drivers after every update.

Currently tested/failing driver versions:

2.1.3.15 via Windows Update

2.1.3.15 via Intel (28.2.1 driver package)

2.1.3.3 via Asus

2.1.1.7 via Intel (27.2 driver package)

Currently working version:

1.1.3.28, unknown source

3. I currently only have the one system with both Windows 11 and an Intel NIC present.

Edit: forgot attachment, logs should be there now

[Elu]

I've just run a test against a fresh Windows 11 22H2 install, to rule out any issues caused by operating system configuration. I've attached the SSU logs from that system.

Default inbox driver version 1.0.2.8 worked. Updated version 2.1.3.15 installed via Windows Update ("Intel - Net - 2.1.3.15" WU package) did not work.

Minimal steps to set up/repro against a clean Win11 install (more or less equivalent to those described at https://taeffner.net/2022/04/multiple-vlans-windows-10-11-onboard-tools-hyper-v/) follow:

1. Install Windows Features:
   • Hyper-V => Hyper-V Management Tools => Hyper-V Module for Windows PowerShell
   • Hyper-V => Hyper-V Platform => Hyper-V Services
2. Check current driver version is on 1.x branch. It should be if using a fresh Windows install before updates.
3. Configure a new vSwitch and add a VLAN-tagged vNIC to it via PowerShell (see below):
   • New-VMSwitch -name VLAN-vSwitch -NetAdapterName Ethernet -AllowManagementOS $true
   • Add-VMNetworkAdapter -ManagementOS -Name "VLAN1" -SwitchName "VLAN-vSwitch" -Passthru | Set-VMNetworkAdapterVlan -Access -VlanId 1
4. The newly created vNIC should work, and be on a tagged VLAN 1.
5. Install updated 2.x branch driver. Either via Windows Update or via Intel driver package.
6. The previously-created vNIC no longer receives any packets. Re-creating the vSwitch and vNIC from scratch after updated driver installation also does not work.

PowerShell to configure vNIC (because the post editor doesn't seem to like code blocks in the middle of a list):

New-VMSwitch -name VLAN-vSwitch -NetAdapterName Ethernet -AllowManagementOS $true
Add-VMNetworkAdapter -ManagementOS -Name "VLAN1" -SwitchName "VLAN-vSwitch" -Passthru | Set-VMNetworkAdapterVlan -Access -VlanId 1

[B_Y]

Hello @Elu,

Noted on the information. Please give us some time to look into it and answer appropriately. We will keep you posted.

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[B_Y]

Hello @Elu,

Thank you for updated information. Let us check the details and advise accordingly.

Best regards,

BY_Intel

Intel Customer Support 

[cailyera]

I use Hyper-V with a vSwitch connected to external network with a (3) I225-V. Since the update, if i set a vlan ID for the host or any VM, it's not working anymore.  Suddently stop working after the update. It only work on my native vlan. (no ID) I have a docker VM with Macvlan setup. Each docker container can use different vlan. They all stopped working...

When I restore the driver(1.0.2.14). Tagged network start working again for the host but macvlan docker interface receive no packet only TX. 

The problem seem related to this one.

NIC: (3) I225-V

Failing driver versions: 2.1.3.15

OS: Windows 11 22H2 Build 22621.2283

Motherboard: Asus ROG STRIX B550-I GAMING

BIOS: 3301 (latest)

[B_Y]

Hello @Elu,

Thank you for your patience. It appears that the default or original inbox driver version (1.0.2.8, 1.0.2.14) worked till there is a windows/new update. Inbox driver usually came specifically from Original Equipment Manufacturer (OEM) / Asus' customization to ensure it works accordingly at minimum state (which is expected).

Please be aware, we should only use the software application that came with the Asus system to perform the update, but in hyper V environment, we many need to investigate the minidump or the changes happened after post update with the later drivers.

Since the issue definitely occurring after an update, contact your motherboard manufacturer / Asus for further assistance as the I225-V controller is embedded to the system.

Just to add, you may consider reach out Asus to update the NVM file for the I225, because the log didn't display the version, such as "NVM Version:"Not Available"

We hope this helps to answer your concern.

Best regards,

BY_Intel

Intel Customer Support 

[Elu]

Hi,

> we should only use the software application that came with the Asus system to perform the update

I do not believe we are being given a choice in this update, as it has been pushed down by Windows Update automatically. As far as I can tell, this appears to be pushed down by WU under the branding of Intel, with no involvement from Asus. Notably, it is actually a different version from the one provided by Asus (which also does not work for this purpose).

Just to be clear: to me it looks like Intel is providing the drivers to Microsoft, who are then pushing them down via Windows Update and in this process breaking previously working vSwitch setups, all automatically. I did not initiate this update process. I would have been fine staying on the older working version, if I had a choice in the matter.

> but in hyper V environment, we many need to investigate the minidump or the changes happened after post update with the later drivers

Please note that although my primary environment is running under a Hyper-V hypervisor, the fresh test setup I was using was not. It was only using Hyper-V services to provide the vSwitch and vNICs, but the OS itself was running bare-metal (i.e. no hypervisor was involved). This issue has nothing to do with virtualisation itself, but is solely an interaction between the new drivers and the vSwitch/vNIC.

Since I can already see this becoming an endless back-and-forth (where you insist this is an OEM issue, and Asus will likely insist it is not), could you please tell me if there is any way I can convince you otherwise? If I purchase a non-Asus PCIe card with an I-225V on it and manage to reproduce the issue there, will you again tell me that it is an OEM issue?

[CGGX]

I can also confirm this issue with a I226-V PCI-e card. VLAN on Hyper-V Switch/Adapter doesn't work with any 2.x version driver; but works fine on the latest 1.x driver which is currently version 1.1.4.42 from the 28.2.1 driver package. Also Wake-on-LAN is not working with both versions.

[B_Y]

Hello @Elu,

Thank you for feedback. May I seek your help to share the vlan, event viewer log from the host machine for each working (default driver) and non working state (preferably with OEM/Asus' latest chipset/LAN drivers) with us so we can check internally.

Looking forward to hearing from you

Best regards,

BY_Intel

Intel Customer Support 

[Elu]

Are you asking for just the SSU logs again or did you want me to export/upload some other set of logs? I'm not certain which "vlan, event viewer log" is referring to.

Also are you looking for logs from me (Asus onboard NIC) or from @CGGX who is experiencing the same issue on a PCIe I226-V?

I will also reiterate that part of the issue here is we're being forced to update to an Intel-provided driver via Windows Update regardless of which version the OEM provides on their website.

Lastly, is there any chance you could attempt repro with the steps I provided above? It may provide more obvious evidence of the issue at hand than simply logs. I am, of course, still happy to provide logs once you've confirmed which specific logs you want from me.

[CGGX]

I think the problem has something to do with the new NetAdapter architecture of the 2.x driver (e2fn.sys) Network Adapter WDF Class Extension (NetAdapterCx) - Windows drivers | Microsoft Learn

The 1.x driver appears to be NDIS 6.8x (e2f.sys) and VLAN works correctly with Hyper-V. The problem is that now under Windows 11 and newer version of Windows 10, the 2.x driver are pushed to us over Windows Update. That was how I discovered the issue, because my virtual adapters using VLAN stopped working after 2.1.3.15 was automatically installed by Windows Update. I had to manually install 1.1.4.42 from the NDIS68 folder of the driver package zip file to have working virtual adapters again. Luckily the 2.x version is no longer automatically reinstalled by Windows Update, but I doubt this will only last until the next time when newer drivers are available on Windows Update again.

As for reproduction: you can easily reproduce this problem even with a clean Windows installation with the steps listed by @Elu in the comment above https://community.intel.com/t5/Ethernet-Products/Hyper-V-vSwitches-do-not-work-with-2-x-drivers-on-a-I225-V/m-p/1531012#M34821

My PC has an add-on PCI-e card with the I226-V chip. It's not the network adapter from the motherboard, which has a Realtek chip. Also this onboard Realtek NIC also has a new NetAdapter driver and VLAN + Hyper-V work fine.

[Elu]

Hi @B_Y ,

Could you please confirm what logs you actually want from us at this point? As I said in my previous reply, it is unclear if you are looking for the SSU logs again or some specific event logs. It's also unclear if you're looking for more logs from me (Asus onboard I225-V) or @CGGX (PCIe I226-V).

It would also be good to get a response regarding whether Intel has attempted or will attempt the steps to repro outlined above. Are you asking for more information/logs because those repro steps are not working? Have they been tried yet?

[B_Y]

Hello @Elu,

Thank you for patience. Based on the 2 logs, if I understand correctly that they were not at working state. We have limited resource and deep knowledge to test or reproduce the issue from an OEM system and Windows to understand current issue.

May I know if you have the log from the working state and inbox driver version 1.0.2.8?

We also need following file from non working state:

1. Memory dump:

https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/generate-a-kernel-or-complete-crash-dump

Follow the manually generate memory dump file 

After restart MEMORY.DMP will be found in C:\Windows\memory.dmp

2. Event viewer: In event viewer choose Windows Logs > System > Save All Event As and save the file

With these information, we will then check with engineers.

Thank you very much for your help.

Best regards,

BY_Intel

Intel Customer Support 

[Elu]

I have attached both event logs and SSU logs from both working and broken states.

Please tell me how you propose I send the memory dump. The kernel-only dump is 2 GB, and I didn't even bother taking a "full" memory dump as that would be 64 GB. Even with compression, the kernel-only dump is 210 MB, too big to upload here.

One thing of interest is that with another fresh Windows 11 22H2 install, Windows Update now only attempts to update to 1.1.3.28. It does not appear to automatically update to 2.x at the moment. Was the update withdrawn?

@B_Y, could you please pass along the above steps to repro to your engineers? The ones in this message: https://community.intel.com/t5/Ethernet-Products/Hyper-V-vSwitches-do-not-work-with-2-x-drivers-on-a-I225-V/m-p/1531012/highlight/true#M34821

Also, could you please comment on the fact that another user, CGGX, is experiencing this on a PCIe card which indicates this is not isolated to a specific OEM's onboard NICs? So far you seem to have ignored my repeated requests for clarification of whether you want logs from them and their system, or just me.

Thanks

[Elu]

I have uploaded a zipped copy of the memory dump to OneDrive: https://1drv.ms/u/s!AnsMJLywVyaw5xutLRSl-fxN5rs8?e=4iVKZw

[B_Y]

Hello @Elu,

Thank you for sharing all the provided information. Please give us some time to look into it and answer appropriately. We will keep you posted.

Thank you.

Best regards,

BY_Intel

Intel Customer Support 

[CZ_4]

I also had this issue. I am using I266-V, 2.x version of NIC driver, THE VMs HAVE NO ACCESS TO VLANS NETWORK ON HYPER V SWITCH.

This issue does not exist if i switch to use Realtek NIC.

I am not using ASUS computer, nor ASUS OEM motherboard. It seems to be a common issue regardless the OEMs.

I registered this account just to reply on this thread. I just felt sorry that this issue can be very easy to be reproduced and customer has provided the steps, but you Intel guys kept asking for logs from customers without even trying to repro it and collecting your own log.  Where is your QA team and what do they do? Are they having their lunch? 

[j3b0]

I also made an account just to share that the same thing is broken for me on Windows 11 23H2 22631. I found that Intel 2.x drivers (Windows auto-installs 2.1.3.15, dated 2023/06/29) DO NOT PASS VLAN TRAFFIC. The Untagged adapter works fine in all driver versions, but the VLAN-aware adapters do not work. Rolling back to the 1.1.4.38 driver (dated 2023/02/09) in Device Manager instantly allows VLANs to work. Ridiculous

My install steps are below:

- Install Hyper V but only the Services (under Platform) and the Powershell module, and reboot
- Open powershell in admin mode

- Create a vswitch bridged to the actual ethernet port

New-VMSwitch -name VLAN-vSwitch -NetAdapterName Ethernet -AllowManagementOS $true

- Configure the different VLANs

Add-VMNetworkAdapter -ManagementOS -Name "LAN100" -SwitchName "VLAN-vSwitch" -Passthru | Set-VMNetworkAdapterVlan -Access -VlanId 100

Add-VMNetworkAdapter -ManagementOS -Name "DMZ101" -SwitchName "VLAN-vSwitch" -Passthru | Set-VMNetworkAdapterVlan -Access -VlanId 101

Add-VMNetworkAdapter -ManagementOS -Name "WAN99" -SwitchName "VLAN-vSwitch" -Passthru | Set-VMNetworkAdapterVlan -Access -VlanId 99

- Configure the UNTAG VLAN (-Access and -VlanId must be passed together, and are not needed for this untagged adapter) https://learn.microsoft.com/en-us/powershell/module/hyper-v/set-vmnetworkadaptervlan?view=windowsserver2022-ps

Add-VMNetworkAdapter -ManagementOS -Name "UNTAG" -SwitchName "VLAN-vSwitch" -Passthru | Set-VMNetworkAdapterVlan -Untagged

- Remove any unused or duplicate adapters, such as VLAN-vSwitch as it does the same thing as UNTAG https://learn.microsoft.com/en-us/powershell/module/hyper-v/remove-vmnetworkadapter?view=windowsserver2022-ps

Remove-VMNetworkAdapter -VMNetworkAdapterName "VLAN-vSwitch" -SwitchName "VLAN-vSwitch" -ManagementOS

- See what we've set up (rename in the Settings if desired)

PS C:\Windows\system32> Get-NetAdapter

Name InterfaceDescription ifIndex Status MacAddress LinkSpeed
---- -------------------- ------- ------ ---------- ---------
UNTAG Hyper-V Virtual Ethernet Adapter #6 39 Disabled 00-15-5D-BF-AA-05 1 Gbps
DMZ101 Hyper-V Virtual Ethernet Adapter #4 42 Disabled 00-15-5D-BF-AA-01 1 Gbps
Ethernet Intel(R) Ethernet Controller (3) I225-V 12 Up 7C-10-C9-42-D3-11 1 Gbps
LAN100 Hyper-V Virtual Ethernet Adapter #3 38 Up 00-15-5D-BF-AA-00 1 Gbps
WAN99 Hyper-V Virtual Ethernet Adapter #5 46 Disabled 00-15-5D-BF-AA-02 1 Gbps

[Irwan_Intel]

Hello Elu,

Would you be able to confirm the replication steps when you measuring received packets?

Regards,

Irwan_Intel