Intel AX210 Firmware Reset under Load

[KevinA1]

Hello,

I am having trouble with an Intel AX210 card in a Framework laptop that resets when performing network I/O for any reasonable amount of time. When the card resets, WiFi performance drops to around ~500Kb/s till the interface is restarted or the kernel module is unloaded and reloaded.

I can pretty easily reproduce it by running iperf while connected on a 5GHz WiFi network and get a crash in about 5 minutes. On a 2.4GHz network I can reproduce it but it takes a little bit longer.

Some system information is below:

Fedora 34

Linux 5.13.4-200.fc34.x86_64  : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 10 crashes
Linux 5.13.6-200.fc34.x86_64  : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 4 crashes
Linux 5.13.7-200.fc34.x86_64  : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 32 crashes
Linux 5.13.10-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 2 crashes
Linux 5.13.12-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 2 crashes
Linux 5.13.13-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 9 crashes 
Linux 5.13.14-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 7 crashes
Linux 5.13.16-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 6 crashes
Linux 5.13.19-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 13 crashes
Linux 5.14.9-200.fc34.x86_64  : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 2 crashes
Linux 5.14.10-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 3 crashes 
Linux 5.14.11-200.fc34.x86_64 : 63.c04f3485.0 ty-a0-gf-a0-63.ucode : 5 crashes

I would appreciate any help with resolving the problem.

Here is some dmesg output during the crash/reset:

Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: regular scan timed out
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Microcode SW error detected. Restarting 0x0.
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Start IWL Error Log Dump:
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Transport status: 0x0000004A, valid: 6
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Loaded firmware version: 63.c04f3485.0 ty-a0-gf-a0-63.ucode
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00808201 | trm_hw_status0
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000000 | trm_hw_status1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x004DA004 | branchlink2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00008370 | interruptlink1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00008370 | interruptlink2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00008FEE | data1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x01000000 | data2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000000 | data3
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x69006F4E | beacon time
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0xECCF6084 | tsf low
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000013 | tsf hi
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00006EEB | time gp1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x73C84F8B | time gp2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000001 | uCode revision type
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x0000003F | uCode version major
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0xC04F3485 | uCode version minor
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000420 | hw version
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00C89002 | board version
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x058A001C | hcmd
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0xE6F2B000 | isr0
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x09448000 | isr1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x48F0011A | isr2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00C37FCC | isr3
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00200000 | isr4
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x0565001C | last cmd Id
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00008FEE | wait_event
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x000000C4 | l2p_control
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x0001CC14 | l2p_duration
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000007 | l2p_mhvalid
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00810048 | l2p_addr_match
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000009 | lmpm_pmg_sel
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000000 | timestamp
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00005858 | flow_handler
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Start IWL Error Log Dump:
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Transport status: 0x0000004A, valid: 7
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x20000066 | NMI_INTERRUPT_HOST
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000000 | umac branchlink1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x8045CF40 | umac branchlink2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x8047E3DE | umac interruptlink1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x8047E3DE | umac interruptlink2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x01000000 | umac data1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x8047E3DE | umac data2
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000000 | umac data3
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x0000003F | umac major
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0xC04F3485 | umac minor
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x73C854BD | frame pointer
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0xC0886264 | stack pointer
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x0084010D | last host cmd
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000400 | isr status reg
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: IML/ROM dump:
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000B03 | IML/ROM error/state
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x000077A7 | IML/ROM data1
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: Fseq Registers:
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x60000100 | FSEQ_ERROR_CODE
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00440003 | FSEQ_TOP_INIT_VERSION
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00080009 | FSEQ_CNVIO_INIT_VERSION
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x0000A652 | FSEQ_OTP_VERSION
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000002 | FSEQ_TOP_CONTENT_VERSION
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00400410 | FSEQ_CNVI_ID
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00400410 | FSEQ_CNVR_ID
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00400410 | CNVI_AUX_MISC_CHIP
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00400410 | CNVR_AUX_MISC_CHIP
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00009061 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: 0x00000061 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
Oct 21 18:58:33 redacted kernel: iwlwifi 0000:aa:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms).
Oct 21 18:58:33 redacted kernel: ieee80211 phy0: Hardware restart was requested

As a side note I reported this to the Fedora bugzilla instance: https://bugzilla.redhat.com/show_bug.cgi?id=1991752

[tknuc]

This is also happening on Lenovo Thinkpad X1 and P1 Gen 4 laptops running Ubuntu 21.10

Kernel 5.13.0-21-generic (as provided by Ubuntu 21.10 latest updates)

Tried it with both these firmware files, not sure how to identify their version better than their md5:

d78be163ec3864dd34cc6e4abcd70a20 iwlwifi-ty-a0-gf-a0-59.ucode (downloaded directly from Intel)
61d01edc2709f52b58899d748c8c754b iwlwifi-ty-a0-gf-a0-59.ucode (part of the Ubuntu 21.10 install/updates)

We have confirmed reports now on Dell, Lenovo, Framework ... this is clearly a problem with the Intel firmware that should not be difficult for Intel to reproduce in their own labs and provide a fix for.

Running a simple iperf3 test causes the crash and disconnect.

In my setup I use AC mode, 5GHz band, channel 149, 80MHz, WPA3, if any of these matter.

[MCrank]

Can confirm.  I have a Thinkpad P1 Gen 4 and the wireless speeds are VERY slow with the AX210.  Running PopOS 21.04

[KevinA1]

Hi Deivid,

Thank you. I have opened a support request with Framework to look in to this some more. Additionally it appears some of the Intel kernel developers are aware of the problem upstream and have a preliminary patch to at least reduce the impact of the firmware restart. BZ #214693 

Thanks,

Kevin

[DeividA_Intel]

Hello KevinA1, 

I am glad to hear that you opened a case with Framework, and also thanks for the information about the "preliminary patch". I will close this thread since you are working with Framework.

@tknuc , I recommend you create a post, so you can get proper support of=n this matter or you can follow the steps and recommendations provided in this thread.

Regards,  

Deivid A.  

Intel Customer Support Technician  

[tknuc]

Does it really make sense to start a new thread for the exact same issue instead of consolidating all reports into this thread?

Like KevinA1 mentioned, it appears the kernel developers are aware of the issue, so fingers crossed for a fix:

https://bugzilla.kernel.org/show_bug.cgi?id=214693