AX200 Hardware error detected on Linux Mint 20

[gameldar]

Hi,

I have an issue with my Intel® Wi-Fi 6 AX200 160MHz on my laptop using the iwlwifi driver on Linux Mint 20 where it fails with a hardware error detected message (in dmesg). This occurs when transferring large amounts of data (doing a rsync backup to a machine in the local network). It appears to only happen after a sleep/resume has occurred as after a fresh boot the rsync succeeds.

I've installed (downgraded from iwlwifi-cc-a0-50.ucode as provided with the kernel) the recommended firmware version as recommended here: https://www.intel.com/content/www/us/en/support/articles/000005511/wireless.html but the problem persists.

The full message from dmesg is:

[93689.432867] iwlwifi 0000:04:00.0: Hardware error detected. Restarting.
[93689.432984] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[93689.432986] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 6
[93689.432988] iwlwifi 0000:04:00.0: Loaded firmware version: 46.3cfab8da.0
[93689.432991] iwlwifi 0000:04:00.0: 0x00000066 | NMI_INTERRUPT_HOST 
[93689.432992] iwlwifi 0000:04:00.0: 0x0000A200 | trm_hw_status0
[93689.432994] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[93689.432995] iwlwifi 0000:04:00.0: 0x004FC308 | branchlink2
[93689.432997] iwlwifi 0000:04:00.0: 0x0001E2D4 | interruptlink1
[93689.432998] iwlwifi 0000:04:00.0: 0x0001E2D4 | interruptlink2
[93689.432999] iwlwifi 0000:04:00.0: 0x000151BE | data1
[93689.433001] iwlwifi 0000:04:00.0: 0x00000001 | data2
[93689.433002] iwlwifi 0000:04:00.0: 0x40000008 | data3
[93689.433003] iwlwifi 0000:04:00.0: 0x27408575 | beacon time
[93689.433005] iwlwifi 0000:04:00.0: 0x4DED3A8F | tsf low
[93689.433006] iwlwifi 0000:04:00.0: 0x00000068 | tsf hi
[93689.433007] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[93689.433009] iwlwifi 0000:04:00.0: 0x076A5FF7 | time gp2
[93689.433010] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[93689.433011] iwlwifi 0000:04:00.0: 0x0000002E | uCode version major
[93689.433013] iwlwifi 0000:04:00.0: 0x3CFAB8DA | uCode version minor
[93689.433014] iwlwifi 0000:04:00.0: 0x00000340 | hw version
[93689.433016] iwlwifi 0000:04:00.0: 0x18C89000 | board version
[93689.433017] iwlwifi 0000:04:00.0: 0x0556001C | hcmd
[93689.433018] iwlwifi 0000:04:00.0: 0x20028000 | isr0
[93689.433019] iwlwifi 0000:04:00.0: 0x00400000 | isr1
[93689.433021] iwlwifi 0000:04:00.0: 0x00F0000A | isr2
[93689.433022] iwlwifi 0000:04:00.0: 0x00C0FFCC | isr3
[93689.433023] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[93689.433025] iwlwifi 0000:04:00.0: 0x054B001C | last cmd Id
[93689.433026] iwlwifi 0000:04:00.0: 0x000151BE | wait_event
[93689.433027] iwlwifi 0000:04:00.0: 0x00004288 | l2p_control
[93689.433029] iwlwifi 0000:04:00.0: 0x00018034 | l2p_duration
[93689.433030] iwlwifi 0000:04:00.0: 0x00000000 | l2p_mhvalid
[93689.433031] iwlwifi 0000:04:00.0: 0x000000E7 | l2p_addr_match
[93689.433033] iwlwifi 0000:04:00.0: 0x00000009 | lmpm_pmg_sel
[93689.433034] iwlwifi 0000:04:00.0: 0x16042142 | timestamp
[93689.433035] iwlwifi 0000:04:00.0: 0x0AD478F8 | flow_handler
[93689.433081] iwlwifi 0000:04:00.0: 0x00000000 | ADVANCED_SYSASSERT
[93689.433083] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[93689.433084] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink2
[93689.433085] iwlwifi 0000:04:00.0: 0x00000000 | umac interruptlink1
[93689.433087] iwlwifi 0000:04:00.0: 0x00000000 | umac interruptlink2
[93689.433088] iwlwifi 0000:04:00.0: 0x00000000 | umac data1
[93689.433089] iwlwifi 0000:04:00.0: 0x00000000 | umac data2
[93689.433090] iwlwifi 0000:04:00.0: 0x00000000 | umac data3
[93689.433092] iwlwifi 0000:04:00.0: 0x00000000 | umac major
[93689.433093] iwlwifi 0000:04:00.0: 0x00000000 | umac minor
[93689.433094] iwlwifi 0000:04:00.0: 0x00000000 | frame pointer
[93689.433096] iwlwifi 0000:04:00.0: 0x00000000 | stack pointer
[93689.433097] iwlwifi 0000:04:00.0: 0x00000000 | last host cmd
[93689.433098] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[93689.433107] iwlwifi 0000:04:00.0: Fseq Registers:
[93689.433111] iwlwifi 0000:04:00.0: 0xE0000101 | FSEQ_ERROR_CODE
[93689.433116] iwlwifi 0000:04:00.0: 0x00290021 | FSEQ_TOP_INIT_VERSION
[93689.433120] iwlwifi 0000:04:00.0: 0x80050008 | FSEQ_CNVIO_INIT_VERSION
[93689.433125] iwlwifi 0000:04:00.0: 0x0000A503 | FSEQ_OTP_VERSION
[93689.433129] iwlwifi 0000:04:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
[93689.433133] iwlwifi 0000:04:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
[93689.433137] iwlwifi 0000:04:00.0: 0x00100530 | FSEQ_CNVI_ID
[93689.433142] iwlwifi 0000:04:00.0: 0x00000532 | FSEQ_CNVR_ID
[93689.433146] iwlwifi 0000:04:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
[93689.433152] iwlwifi 0000:04:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
[93689.433159] iwlwifi 0000:04:00.0: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[93689.433165] iwlwifi 0000:04:00.0: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[93689.433353] iwlwifi 0000:04:00.0: Collecting data: trigger 2 fired.
[93689.433358] ieee80211 phy0: Hardware restart was requested
[93689.433378] iwlwifi 0000:04:00.0: Microcode SW error detected. Restarting 0x0.
[93690.407511] iwlwifi 0000:04:00.0: Failed to send recovery cmd blob was invalid 1

The issue doesn't recover unless I manually disconnect and reconnect, or unload/reload the kernel module.

The issue occurs on both 2.4GHz and 5GHz networks with two different routers (a fritzbox and a technicolor).

The following is the details of the system:

hostnamectl
Chassis: laptop
Operating System: Linux Mint 20
Kernel: Linux 5.4.0-88-generic
Architecture: x86-64

lspci | grep Network
04:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)

modinfo iwlwifi
filename: /lib/modules/5.4.0-88-generic/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
license: GPL
author: Copyright(c) 2003- 2015 Intel Corporation <linuxwifi@intel.com>
description: Intel(R) Wireless WiFi driver for Linux
srcversion: 4737C02B82ECA037690796F
depends: cfg80211
retpoline: Y
intree: Y
name: iwlwifi
vermagic: 5.4.0-88-generic SMP mod_unload modversions 
sig_id: PKCS#7
signer: Build time autogenerated kernel key
sig_key: 2D:2D:71:A0:22:44:6D:60:C8:49:CB:0E:D7:43:D0:D2:7A:5C:0E:F1
sig_hashalgo: sha512
parm: swcrypto:using crypto in software (default 0 [hardware]) (int)
parm: 11n_disable:disable 11n functionality, bitmap: 1: full, 2: disable agg TX, 4: disable agg RX, 8 enable 
agg TX (uint)
parm: amsdu_size:amsdu size 0: 12K for multi Rx queue devices, 2K for AX210 devices, 4K for other devices 1:4
K 2:8K 3:12K 4: 2K (default 0) (int)
parm: fw_restart:restart firmware in case of error (default true) (bool)
parm: antenna_coupling:specify antenna coupling in dB (default: 0 dB) (int)


parm: nvm_file:NVM file name (charp)
parm: lar_disable:disable LAR functionality (default: N) (bool)
parm: uapsd_disable:disable U-APSD functionality bitmap 1: BSS 2: P2P Client (default: 3) (uint)
parm: enable_ini:Enable debug INI TLV FW debug infrastructure (default: 0 (bool)
parm: bt_coex_active:enable wifi/bt co-exist (default: enable) (bool)
parm: led_mode:0=system default, 1=On(RF On)/Off(RF Off), 2=blinking, 3=Off (default: 0) (int)
parm: power_save:enable WiFi power management (default: disable) (bool)
parm: power_level:default power save level (range from 1 - 5, default: 1) (int)
parm: fw_monitor:firmware monitor - to debug FW (default: false - needs lots of memory) (bool)
parm: disable_11ac:Disable VHT capabilities (default: false) (bool)
parm: remove_when_gone:Remove dev from PCIe bus if it is deemed inaccessible (default: false) (bool)
parm: disable_11ax:Disable HE capabilities (default: false) (bool)

lsmod | grep iwlwifi
iwlwifi 331776 1 iwlmvm
cfg80211 708608 3 iwlmvm,iwlwifi,mac80211

dmesg | grep iwlwifi

Returns the above - plus the following is from where I unloaded/reloaded the kernel module (you can see it looking for the other firmware versions that I've moved out of the way)

[93700.000426] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-cc-a0-50.ucode failed with error -2
[93700.000432] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-cc-a0-49.ucode failed with error -2
[93700.000438] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-cc-a0-48.ucode failed with error -2
[93700.000443] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-cc-a0-47.ucode failed with error -2
[93700.000903] iwlwifi 0000:04:00.0: loaded firmware version 46.3cfab8da.0 op_mode iwlmvm
[93700.019341] iwlwifi 0000:04:00.0: Detected Killer(R) Wi-Fi 6 AX1650x 160MHz Wireless Network Adapter (200NGW), REV=0x340

Thanks!

[Alberto_R_Intel]

gameldar, Thank you for posting in the Intel® Communities Support.

In order for us to provide the most accurate assistance on this scenario, we just wanted to confirm a few details about your system:

Is this a new computer?

Was it working fine before?

When did the issue start?

Did you make any recent hardware/software changes?

The wireless card, did you purchased it separately or did it came installed on the computer?

Does the problem happen at home or in the work environment?

Please attach the SSU report so we can verify further details about the components in your platform, check all the options in the report including the one that says "3rd party software logs":

https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-?product=91600

https://www.intel.com/content/www/us/en/download/18895/26735/intel-system-support-utility-for-the-linux-operating-system.html

WiFi troubleshooting steps

Step 1: Check whether the kernel version customer uses supports this device:

The reference table of Wireless adapters and corresponding kernel versions is available by the link: https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi

The latest FWs are available by the following link:​ ​https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/

If the kernel version corresponds to the Wireless adapter, proceed with step 2.

Step 2: Check whether the Wireless adapter is connected correctly:

Type the following command to the terminal: # lspci​ 

Verify that the Intel network controller ​is identified, as on a picture below. If not, then the system doesn't identify the Intel network adapter and you will need to make sure it's properly integrated. You may need to request help from the system manufacturer. If the wireless adapter is properly identified - proceed with the step 3.

Step 3: Check whether this is an iwlwifi issue:

Type the following command to the terminal: # modinfo iwlwifi​ (example of the command output is on the picture below). If there is no output, then the module isn’t installed this is the distribution installation issue.

Please type: # lsmod | grep iwlwifi​   

If the result is empty, the driver hasn't been loaded. Most likely to be a distribution issue.

To verify, please type: # dmesg | grep iwlwifi​   

If the result is empty, the driver wasn’t initialized and this is a distribution issue. If the result is populated as on the above picture, proceeed with the step 4.

Step 4: Check whether the suitable firmware is installed:

Please type: # dmesg | grep “no suitable firmware found!”​

If the result is not empty, it means you don't have a suitable FW (api mentioned in the dmesg).

Please visit here​ h​ttps://wireless.wiki.kernel.org/en/users/drivers/iwlwifi ​for supported FW, or here​ ​https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/ for the latest versions.

If these steps didn't reveal the issue or didn't solve the problem, proceed with the Step 5: 

Step 5: Collect the debug data:

Please type: # dmesg | grep ASSERT​ 

If the output is empty, please follow this wiki: https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi/debugging ​to extract dmesg and trace-cmd.

If the output has value, please collect from the wiki FW dump too.

Any questions, please let me know.

Regards,

Albert R.

Intel Customer Support Technician

[gameldar]

Is this a new computer?

It is about 14 months old now

Was it working fine before?

It hasn't been working for a while (since March) - I don't know if it was always this way but I just never tried to push as much across the network. I had originally assumed it was a problem with the destination end, but noticed the logs in dmesg and realised the failure was happening on this machine.

When did the issue start?

I first noticed this happening about 6 months ago, but haven't really investigated it until now as I can work around it by plugging in an ethernet cable

Did you make any recent hardware/software changes?

No

The wireless card, did you purchased it separately or did it came installed on the computer?

It came installed on the laptop

Does the problem happen at home or in the work environment?

Home

I've attached the SSU report, and the other troubleshooting information has already been provided.

Thanks

[Alberto_R_Intel]

gameldar, Thank you very much for sharing that information and the SSU report.

We will now do further research on this matter, as soon as I get any updates I will post all the details on this thread.

Regards,

Albert R.

Intel Customer Support Technician

[Alberto_R_Intel · Accepted solution]

Hello gameldar, I just received an update on this matter.

In order to try to fix this problem, please test a newer wireless firmware version 64 by updating to the latest & stable kernel 5.14.

If the issue remains with the latest kernel, please generate all logs again and the full dmesg log using command: # dmesg > dmesg.log.

Regards,

Albert R.

Intel Customer Support Technician

[Alberto_R_Intel]

Hello gameldar, I just wanted to check if the information posted previously was useful for you and if you need further assistance on this matter?

Regards,

Albert R.

Intel Customer Support Technician

[gameldar]

Hi,

I have some deadlines to meet first before I play with my kernel on this machine - so I'm hoping I'll have a chance to test it next week, but I haven't managed to try out the latest kernel yet.

Thanks

[Alberto_R_Intel]

Hi gameldar, Thank you very much for letting us know those updates.

No problem at all, go ahead and take your time in doing the testings, the thread will remain open for as long as you need it and once you get the chance, please let us know the results.

Regards,

Albert R.

Intel Customer Support Technician

[Alberto_R_Intel]

Hello gameldar, I just wanted to check if you had the chance to do the tests and if you have the results?

Regards,

Albert R.

Intel Customer Support Technician

[gameldar]

sorry for the delay - I've managed to upgrade my kernel and test briefly and this appears to have addressed the issue, but I haven't done a lot of testing (as I need to stay on my old kernel for other reasons). But we can assume this is the fix and I'll reopen it if the problem persists when I get to permanently move to the newer kernel.

Thanks

[Alberto_R_Intel]

Hello gameldar, No problem at all, thank you very much for sharing those results.

Perfect, excellent, it is great to hear that you were able to upgrade the kernel and that seems to fix the problem in reference to the hardware error that you were seeing in your computer.

"I'll reopen it if the problem persists when I get to permanently move to the newer kernel.", Sure, we are glad to help.

Any other inquiries, do not hesitate to contact us again.

Regards,

Albert R.

Intel Customer Support Technician