AX200 and Cisco 9130AXI Access Point / IOS-XE 17.3.1- Win 10 x64 Frequent BSOD

[MondelezINT]

Good evening, we have just completed an installation of x35 Cisco 9130 AXI Access Points and 9800-CL Wireless Controllers.

Code on WLC us 17.3.1

This is a brownfield deployment replacing existing Cisco 2602i access points.

802.11n/ac/ax are enabled.

802.11r/k/v are also enabled on BSSID.

Clients are authenticating using 802.1x and EAP-TLS.

Client mix is HP EliteBook 840 G3, ProBook 640 G4, ProBook 840 G5, ProBook 840 G6 and Elitebook x360 G4.

All clients successfully connected to the network before replacement, no BSOD, drop outs etc

Clients associated to the network using the AX200 chipset now frequently BSOD with various error codes. (See Below)

Have tried disabling 802.11ax and 802.ac on both AP and client side but same issue persists.

Frequent Error Codes

*BAD_POOL_CALLER

*PAGE_FAULT_IN_NONPAGED_AREA

*DRIVER IRQL NOT LESS OR EQUAL - TCPIP.sys

*KERNEL MODE HEAP_CORRUPTION

We have tried various drivers.

• 21.120.2 – Regular BSOD
• 21.110.1.1 – Regular BSOD
• 21.80.2.1 – Seems more stable

Has anyone had any similar experiences please?

[AndrewG_Intel]

Hello MondelezINT

Thank you for posting on the Intel® communities.

Could you please provide the following information to understand better your environment?

1- We understand that you have serveral systems. Just to make sure, do all the devices listed here (HP EliteBook 840 G3, ProBook 640 G4, ProBook 840 G5, ProBook 840 G6 and Elitebook x360 G4) have only an Intel® Wi-Fi 6 AX200? Are all of them experiencing the issue? Please provide details

2- Are there other devices in this environment using different Intel® Wireless card models that are working fine? For instance, Intel® Wireless card 9000 Series? Is the issue happening only with Intel® Wi-Fi 6 AX200?

3- Is the Intel® Wi-Fi 6 AX200 the original wireless adapter that came pre-installed on the systems or did you install/integrate it?

4- Regarding the devices affected, please confirm Windows® full version and build number, and BIOS version:

5- Have you checked with CISCO* if the new Access Points and Wireless Controllers might required special settings to be configured or may be a firmware update?

Best regards,

Andrew G.

Intel Customer Support Technician

[MondelezINT]

Thank you for your reply Andrew, we have started escalation procedures with our global account team within HP, but for the benefit of the community sharing what we have found thus far.

1- We understand that you have serveral systems. Just to make sure, do all the devices listed here (HP EliteBook 840 G3, ProBook 640 G4, ProBook 840 G5, ProBook 840 G6 and Elitebook x360 G4) have only an Intel® Wi-Fi 6 AX200? Are all of them experiencing the issue? Please provide details

Only machines with the AX200 chipset are exhibiting this behaviour.

2- Are there other devices in this environment using different Intel® Wireless card models that are working fine? For instance, Intel® Wireless card 9000 Series? Is the issue happening only with Intel® Wi-Fi 6 AX200?

Yes, we have a HP ProBook 640 G4 with the AC8265 chipset which works absolutely fine 

3- Is the Intel® Wi-Fi 6 AX200 the original wireless adapter that came pre-installed on the systems or did you install/integrate it?

Yes, all these machines are OEM from the factory, no modiciations

4- Regarding the devices affected, please confirm Windows® full version and build number, and BIOS version:

Windows 10 x64 v1809 and 1909

5- Have you checked with CISCO* if the new Access Points and Wireless Controllers might required special settings to be configured or may be a firmware update?

The code on the wireless controller is 17.3.1, this is the latest release, downgrading to previous v16.14.4a resolves the issue.

There are a number of enhacements in 17.3.1 to enable 802.11ax specific features including Uplink MU-MIMO and OFDMA in Cisco Catalyst 9130 APs.

Laptops on Test

Model	WLC Version	AP Model	WLAN Chipset	Driver Versions Tested	MDLZ Image	PHY	Auth	Notes
HP ProBook 640 G4	17.3.1	C9130AXI-E	Intel AC8265 Chipset	20.70.18.2	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD Reported
HP ProBook 640 G4	16.12.4a	C9130AXI-E	Intel AC8265 Chipset	20.70.18.2	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD Reported
HP ProBook 640 G5	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.80.2.1	Yes	5Ghz 802.11ax	EAP-TLS	Seems stable, no BSOD
HP ProBook 640 G5	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	Yes	5Ghz 802.11ax	EAP-TLS	Regular BSOD
HP ProBook 640 G5	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.120.2	Yes	5Ghz 802.11ax	EAP-TLS	Regular BSOD
HP ProBook 640 G5	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD
HP ProBook 640 G5	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.120.2	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD
HP Elitebook 840 G6	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.80.2.1	Yes	5Ghz 802.11ax	EAP-TLS	Seems stable, no BSOD
HP Elitebook 840 G6	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	Yes	5Ghz 802.11ax	EAP-TLS	Regular BSOD
HP Elitebook 840 G6	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.120.2	Yes	5Ghz 802.11ax	EAP-TLS	Regular BSOD
HP Elitebook 840 G6	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD
HP Elitebook 840 G6	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.120.2	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD
HP Elitbook x360 G4	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.80.2.1	Yes	5Ghz 802.11ax	EAP-TLS	Seems stable, no BSOD
HP Elitbook x360 G4	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	Yes	5Ghz 802.11ax	EAP-TLS	Regular BSOD
HP Elitbook x360 G4	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.120.2	Yes	5Ghz 802.11ax	EAP-TLS	Regular BSOD
HP Elitbook x360 G4	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD
HP Elitbook x360 G4	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.120.2	Yes	5Ghz 802.11ax	EAP-TLS	No BSOD
HP Elitebook 840 G6	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.80.2.1	No	5Ghz 802.11ax	PEAP	Seems stable, no BSOD
HP Elitebook 840 G6	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	No	5Ghz 802.11ax	PEAP	Regular BSOD
HP Elitebook 840 G6	17.3.1	C9130AXI-E	Intel AX200 Chipset	21.120.2	No	5Ghz 802.11ax	PEAP	Regular BSOD
HP Elitebook 840 G6	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.110.1.1	No	5Ghz 802.11ax	PEAP	No BSOD
HP Elitebook 840 G6	16.12.4a	C9130AXI-E	Intel AX200 Chipset	21.120.2	No	5Ghz 802.11ax	PEAP	No BSOD

Frequent Error Codes

• *BAD_POOL_CALLER
• *PAGE_FAULT_IN_NONPAGED_AREA
• *DRIVER IRQL NOT LESS OR EQUAL - TCPIP.sys
• *KERNEL MODE HEAP_CORRUPTION

Driver Versions

• 21.120.2 – Latest Intel Driver
• 21.110.1.1 – Latest HP Approved Driver
• 21.80.2.1 – Older HP Approved Driver

[AndrewG_Intel]

Hello MondelezINT

Thank you for all the information provided.

Please allow us to look into this and we will be posting back in this thread as soon as possible either to provide more information or to request additional details from your environment.

Best regards,

Andrew G.

Intel Customer Support Technician

[Reuben_Farrelly]

Add me to this.  Same AP's, same controller code version, same AX200 cards, same drivers, and the same big mess of blue screens, crashes and instability on a Dell Inspiron 7373.   Basically, these network cards are unusable.

[AndrewG_Intel]

Hello all,

We investigated to see if this could be related to a driver regression; however, after checking this further and based on the facts and evidence that the tests are showing from your end (for instance, the different driver versions tested work fine with the previous controller firmware 16.x), this actually seems to be related to the latest revision of the firmware 17.x on the wireless controllers.

Having said that, our conclusion is that this seems to be a behavior triggered by the latest firmware. This conclusion is based on the previous facts and also considering that we have seen this behavior with the same firmware version on other wireless controllers, but when testing with other Access Points and networks the behavior is not present. For instance, this thread:

https://community.intel.com/t5/Wireless/Intel-AX201-get-bluescreen-on-Cisco-9120AX-WIFI-6/m-p/1215822#M31173

In this case, our suggestion is to stay (if possible) with firmware version 16.x on the wireless controllers and to report this to Cisco support explaining the behavior when upgrading from firmware 16.x to 17.x so they may investigate this further from their side.

Best regards,

Andrew G.

Intel Customer Support Technician

[Reuben_Farrelly]

@AndrewG_Intel I agree with you in so far as that it may be the later code for the Cisco APs and Controller that is the trigger here.  The question is then "what is the root cause?".

I have access to Cisco engineers in the Wireless Networking BU and I did ask about this very thing from my Cisco contact a week ago.  I was told that they were aware that "there are many unique driver issues reported by customer having Intel AX200 and that it looks like Intel is coming up with client side fixes."   I also have access to pre-release code from Cisco and the problem is still present with that code too.  I'm happy to follow this up again and get more detail if need, be if Intel is having issues working with them.

But regardless of this, and if there is something amiss with what the APs send to the client devices (and yes, that should be fixed as well), the theory that any remote networking device can cause a complete crash of the kernel on a Windows machine is by definition the theory that there is a serious security vulnerability at play here.  There is no way, ever, in any way, that a remote device should be able to cause a complete kernel crash even if it sends garbage to the client or anything else that the client device does not like.   For that reason, Intel is the only party who can investigate this in detail since the driver is the component which is crashing the Windows kernel.

If the cause is not anything to do with what the AP is sending to the client then we're still back at it being an Intel issue anyway.  It just doesn't involve Cisco in that case.

I cannot run 16.x code as I have some very new hardware which requires 17.3 or newer firmware.

I am happy to help work on this issue if required to debug this.  I've got two machines which crash frequently and a test environment which I can use if this helps.  It looks like the OP has many more though

[AndrewG_Intel]

Hello @Reuben_Farrelly

Thank you for your response.

We appreciate your interest in helping to debug this further. In this case, we kindly recommend you to create a new thread providing all the detailed information that you have shared in this post and the information about your environment and devices experiencing the issue. This will allow us to better assist you and also this is since every environment is different (e.g. computer models, OEM drives in use, etc.) despite sharing common devices like Access point models or Wireless Adapter models.

Thank you for your understanding. We will be looking forward to hearing back from you.

Best regards,

Andrew G.

Intel Customer Support Technician

[AndrewG_Intel]

Hello @MondelezINT

We are checking this thread and we would like to know if you were able to check this further with CISCO* and if they provided you with any feedback.

Best regards,

Andrew G.

Intel Customer Support Technician

[MondelezINT]

Morning @AndrewG_Intel , we are currently engaged with HP Engineering teams in the US, they are also bringing Intel and Cisco into the discussion aswell.

Currently in the process of setting up a seperate lab at one of our offices in the UK to replicate this issue and share necessary debugs to the technical teams.

Thank you for your interest in this case, i'll leave this thread open and post updates as/when we find an RCA for this.

Ultimately i suspect that it will result in a rebuild of the current driver and/or update to the IOS-XE code on the WLC.

Regards

[AndrewG_Intel]

Hello @MondelezINT

Thank you for your response and for the details.

We have released a new driver for Intel® Wi-Fi 6 AX200, version 22.0.0. Could you please try it and let us know if the behavior persists? Here is the download link: https://downloadcenter.intel.com/download/29902/Windows-10-Wi-Fi-Drivers-for-Intel-Wireless-Adapters?product=189347

Also, we understand you are running Windows® versions 1809 and 1909. By any chance are you able to test with a more updated version? For instance: "2004" or "20H2".

Best regards,

Andrew G.

Intel Customer Support Technician

[AndrewG_Intel]

Hello @MondelezINT

We would like to know if you were able to review our previous post and try the latest wireless driver version 22.0.0. Please do not hesitate to contact us back if you have additional inquiries.

Best Regards,

Andrew G.

Intel Customer Support Technician

[MondelezINT]

@AndrewG_Intel Evening, next earliest opportunity to test this again will be Thursday, i have samples of laptops with both the AX200 and AX201 chipset, so will be able to validate both.

Will let you know how we get on.

[AndrewG_Intel]

Hello @MondelezINT

Perfect! We will be looking forward to hearing back from you about the outcome of those tests.

Best regards,

Andrew G.

Intel Customer Support Technician

[AndrewG_Intel]

Hello @MondelezINT

We are checking this thread and we would like to know if you were able to test using the latest drivers recommended. If you need more days to test this and/or if you have a preferred follow-up schedule please let us know.

Best regards,

Andrew G.

Intel Customer Support Technician

[MondelezINT]

@AndrewG_Intel Evening Andrew, sorry for delayed response, unfortunately still seeing the same issues on 22.00 on ProBook G5, Elitebook G6 with AX200 chipsets.

I also have a HP Elitebook 840 G7 with the AX201 chipset that ill be testing later this week.

Our HP account team has a case opened with Intel direct to address this issue. PI Id: SIP102071.

Will be raising a Cisco TAC incident soon, to allow for collaborative troubleshooting effort between all parties.

Regards

[AndrewG_Intel]

Hello @MondelezINT

Thank you very much for your response. We are sorry to know that the issue persists with driver 22.0.0 but it is good to know that your OEM HP* has a case opened directly with Intel and Cisco will be joining soon. We will check this internally to see if there is something additional that we can assist you with. Also, please feel free to post back in case you have any new or different outcomes from additional tests.

Best regards,

Andrew G.

Intel Customer Support Technician

[AndrewG_Intel]

Hello @MondelezINT

Since you mentioned that your HP* account team has a case opened with Intel, we would like to gather contact information and more details about your environment in order to check this further. We will send you an email to the email account associated with your Profile.

Best regards,

Andrew G.

Intel Customer Support Technician

[AndrewG_Intel]

Hello MondelezINT

We are checking this thread and we would like to know if you receive the email we sent on November 11th? Please do not hesitate to contact us back if you have additional inquiries or feedback.

Best regards,

Andrew G.

Intel Customer Support Technician

[AndrewG_Intel]

Hello MondelezINT

We have not heard back from you so we will proceed to close this thread soon. Please do not hesitate to contact us back if you have additional inquiries.

Best regards,

Andrew G.

Intel Customer Support Technician

[MondelezINT]

Afternoon @AndrewG_Intel Sorry for the delay, we continue to progress this case internally between Intel, HP, Microsoft and Intel.

Cisco have just released new code for these controllers (17.3.2a) which we will be testing this week.

Please feel free to close this thread though, thanks again for your time.

Regards