Intel WiFi 6E AX210 getting microcode errors

[mmzxl]

We are repeatedly getting microcode error dumps on many different devices:

we are on an nvidia kernel based off 5.15.148

iwlwifi 0001:01:00.0: api flags index 2 larger than supported by driver
iwlwifi 0001:01:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.63.2.2
iwlwifi 0001:01:00.0: loaded firmware version 66.f1c864e0.0 ty-a0-gf-a0-66.ucode op_mode iwlmvm
iwlwifi 0001:01:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
iwlwifi 0001:01:00.0: loaded PNVM version 0x35148b80
iwlwifi 0001:01:00.0: Detected RF GF, rfid=0x10d000
iwlwifi 0001:01:00.0: base HW address: 00:00:00:00:00:00

kernel: iwlwifi 0001:01:00.0: Microcode SW error detected. Restarting 0x0.
kernel: iwlwifi 0001:01:00.0: Start IWL Error Log Dump:
kernel: iwlwifi 0001:01:00.0: Transport status: 0x0000004B, valid: 6
kernel: iwlwifi 0001:01:00.0: Loaded firmware version: 66.f1c864e0.0 ty-a0-gf-a0-66.ucode
kernel: iwlwifi 0001:01:00.0: 0x00000034 | NMI_INTERRUPT_WDG           
kernel: iwlwifi 0001:01:00.0: 0x05B082F0 | trm_hw_status0
kernel: iwlwifi 0001:01:00.0: 0x00000000 | trm_hw_status1
kernel: iwlwifi 0001:01:00.0: 0x004DA722 | branchlink2
kernel: iwlwifi 0001:01:00.0: 0x0000B70A | interruptlink1
kernel: iwlwifi 0001:01:00.0: 0x0000B70A | interruptlink2
kernel: iwlwifi 0001:01:00.0: 0x0000C05E | data1
kernel: iwlwifi 0001:01:00.0: 0x00000000 | data2
kernel: iwlwifi 0001:01:00.0: 0x00000000 | data3
kernel: iwlwifi 0001:01:00.0: 0x070037D6 | beacon time
kernel: iwlwifi 0001:01:00.0: 0x00000000 | tsf low
kernel: iwlwifi 0001:01:00.0: 0x00000000 | tsf hi
kernel: iwlwifi 0001:01:00.0: 0x00000000 | time gp1
kernel: iwlwifi 0001:01:00.0: 0x6B5CA4F3 | time gp2
kernel: iwlwifi 0001:01:00.0: 0x00000001 | uCode revision type
kernel: iwlwifi 0001:01:00.0: 0x00000042 | uCode version major
kernel: iwlwifi 0001:01:00.0: 0xF1C864E0 | uCode version minor
kernel: iwlwifi 0001:01:00.0: 0x00000420 | hw version
kernel: iwlwifi 0001:01:00.0: 0x00489002 | board version
kernel: iwlwifi 0001:01:00.0: 0x0690001C | hcmd
kernel: iwlwifi 0001:01:00.0: 0xE6F21C10 | isr0
kernel: iwlwifi 0001:01:00.0: 0x09440000 | isr1
kernel: iwlwifi 0001:01:00.0: 0x48F80002 | isr2
kernel: iwlwifi 0001:01:00.0: 0x44C3201F | isr3
kernel: iwlwifi 0001:01:00.0: 0x00200000 | isr4
kernel: iwlwifi 0001:01:00.0: 0x0409001C | last cmd Id
kernel: iwlwifi 0001:01:00.0: 0x0000C05E | wait_event
kernel: iwlwifi 0001:01:00.0: 0x00000024 | l2p_control
kernel: iwlwifi 0001:01:00.0: 0x00000020 | l2p_duration
kernel: iwlwifi 0001:01:00.0: 0x0000000F | l2p_mhvalid
kernel: iwlwifi 0001:01:00.0: 0x08C61898 | l2p_addr_match
kernel: iwlwifi 0001:01:00.0: 0x00000009 | lmpm_pmg_sel
kernel: iwlwifi 0001:01:00.0: 0x00000000 | timestamp
kernel: iwlwifi 0001:01:00.0: 0x0000E808 | flow_handler
kernel: iwlwifi 0001:01:00.0: Start IWL Error Log Dump:
kernel: iwlwifi 0001:01:00.0: Transport status: 0x0000004B, valid: 7
kernel: iwlwifi 0001:01:00.0: 0x20003523 | ADVANCED_SYSASSERT
kernel: iwlwifi 0001:01:00.0: 0x00000000 | umac branchlink1
kernel: iwlwifi 0001:01:00.0: 0x8045DFC6 | umac branchlink2
kernel: iwlwifi 0001:01:00.0: 0xC008185C | umac interruptlink1
kernel: iwlwifi 0001:01:00.0: 0x00000000 | umac interruptlink2
kernel: iwlwifi 0001:01:00.0: 0x00000000 | umac data1
kernel: iwlwifi 0001:01:00.0: 0x6B5A5C43 | umac data2
kernel: iwlwifi 0001:01:00.0: 0x6B5CA521 | umac data3
kernel: iwlwifi 0001:01:00.0: 0x00000042 | umac major
kernel: iwlwifi 0001:01:00.0: 0xF1C864E0 | umac minor
kernel: iwlwifi 0001:01:00.0: 0x6B5CA528 | frame pointer
kernel: iwlwifi 0001:01:00.0: 0xC0885DD4 | stack pointer
kernel: iwlwifi 0001:01:00.0: 0x00F1010C | last host cmd
kernel: iwlwifi 0001:01:00.0: 0x00000000 | isr status reg
kernel: iwlwifi 0001:01:00.0: IML/ROM dump:
kernel: iwlwifi 0001:01:00.0: 0x00000B03 | IML/ROM error/state
kernel: iwlwifi 0001:01:00.0: 0x00007FD5 | IML/ROM data1
kernel: iwlwifi 0001:01:00.0: 0x00000090 | IML/ROM WFPM_AUTH_KEY_0
kernel: iwlwifi 0001:01:00.0: Fseq Registers:
kernel: iwlwifi 0001:01:00.0: 0x60000000 | FSEQ_ERROR_CODE
kernel: iwlwifi 0001:01:00.0: 0x80440003 | FSEQ_TOP_INIT_VERSION
kernel: iwlwifi 0001:01:00.0: 0x00080009 | FSEQ_CNVIO_INIT_VERSION
kernel: iwlwifi 0001:01:00.0: 0x0000A652 | FSEQ_OTP_VERSION
kernel: iwlwifi 0001:01:00.0: 0x00000002 | FSEQ_TOP_CONTENT_VERSION
kernel: iwlwifi 0001:01:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
kernel: iwlwifi 0001:01:00.0: 0x00400410 | FSEQ_CNVI_ID
kernel: iwlwifi 0001:01:00.0: 0x00400410 | FSEQ_CNVR_ID
kernel: iwlwifi 0001:01:00.0: 0x00400410 | CNVI_AUX_MISC_CHIP
kernel: iwlwifi 0001:01:00.0: 0x00400410 | CNVR_AUX_MISC_CHIP
kernel: iwlwifi 0001:01:00.0: 0x00009061 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
kernel: iwlwifi 0001:01:00.0: 0x00000061 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
kernel: iwlwifi 0001:01:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms).
kernel: ieee80211 phy0: Hardware restart was requested
kernel: iwlwifi 0001:01:00.0: FW error in SYNC CMD CMD_DTS_MEASUREMENT_TRIGGER_WIDE
kernel: CPU: 3 PID: 46981 Comm: python3 Tainted: G           O      5.15.148-l4t-r36.4-1012.12+g717c798c2e8f #1
kernel: Hardware name: NVIDIA CTI Rogue-Orin + Orin AGX/Jetson, BIOS v36.4.0 10/01/2024
kernel: Call trace:
kernel:  dump_backtrace+0x0/0x1d0
kernel:  show_stack+0x34/0x44
kernel:  dump_stack_lvl+0x64/0x80
kernel:  dump_stack+0x18/0x34
kernel:  iwl_trans_txq_send_hcmd+0x4f4/0x50c [iwlwifi]
kernel:  iwl_trans_send_cmd+0x78/0x13c [iwlwifi]
kernel:  iwl_mvm_send_cmd+0x30/0x60 [iwlmvm]
kernel:  iwl_mvm_send_temp_cmd+0x14c/0x1c0 [iwlmvm]
kernel:  iwl_mvm_get_temp+0x108/0x130 [iwlmvm]
kernel:  iwl_mvm_tzone_get_temp+0x6c/0xd0 [iwlmvm]
kernel:  thermal_zone_get_temp+0x80/0x170
kernel:  temp_show+0x48/0x94
kernel:  dev_attr_show+0x44/0x8c
kernel:  sysfs_kf_seq_show+0x98/0x150
kernel:  kernfs_seq_show+0x4c/0x60
kernel:  seq_read_iter+0xec/0x490
kernel:  kernfs_fop_read_iter+0x198/0x1e0
kernel:  new_sync_read+0x110/0x1c0
kernel:  vfs_read+0x110/0x1c0
kernel:  ksys_read+0x88/0x11c
kernel:  __arm64_sys_read+0x2c/0x40
kernel:  invoke_syscall+0x5c/0x130
kernel:  el0_svc_common.constprop.0+0xf0/0x110
kernel:  do_el0_svc+0x3c/0xa0
kernel:  el0_svc+0x20/0x60
kernel:  el0t_64_sync_handler+0x108/0x114
kernel:  el0t_64_sync+0x1a4/0x1a8

 are these errors connected to reading the temperature from the dev (CMD_DTS_MEASUREMENT_TRIGGER_WIDE).

We have already tried to work with disabling powersaving by setting iwlmvm power_scheme to 'active' which did not make any difference.

Any idea how to fix this?

[Mike_Intel]

Hello mmzxl,

Thank you for posting in Intel community Forum.

For me to further check this, please help provide the following details:

1. What is the brand and model of your laptop/system?
2. What is your OS?
3. Was it working fine before?
4. What was the last change made before the issue started?
5. Did you manually upgrade the wireless controller of your system or this is embedded in your system?

If you have questions, please let us know. Thank you.

Best regards,

Michael L.

Intel Customer Support Technician

[Mike_Intel]

Hello mmzxl,

I hope this message finds you well. 

Were you able to check the previous post?

Please let us know if you still need assistance.

Best regards,

Michael L.

[Mike_Intel]

Hello mmzxl,

I hope you are having a good day.

I am sending another follow up on the information that I requested. 

Since we have not heard back from you, I need to close this inquiry. 

If you need further assistance, please post a new question as this thread will no longer be monitored. 

Thank you and have a great day. 

Best regards,

Michael L.

Intel Customer Support Technician

[mmzxl]

Dear Michael,

Thanks for getting back. Regarding your questions:

1. What is the brand and model of your laptop/system?
   We're running a nvidia jetson orin agx machine.
2. What is your OS?
   OS is from from nvidia, based on Ubuntu 22.04 with a Kernel based on 5.15.148.
   
3. Was it working fine before?
   Well system is actually working. We just repeatedly get these microcode errors, that can lead to massive drop outs in wifi connectivity. This has always been the case. (Is there anything you can say w.r.t. what the reason for these micro-code errors could be?)
4. What was the last change made before the issue started?
   Issue has always been there
5. Did you manually upgrade the wireless controller of your system or this is embedded in your system?
   With wireless controller you mean iwlwifi (the driver is shipped with the kernel)? Or wpa-supplicant?

Thank you for your support!

[fouge]

hey

we have the same error, same OS, kernel, Orin NX. It's not reproducible every time, and the bug appeared more frequently once we added support for the RTS5264 in EDK2.

we were able to resolve the issue by forcing iwlwifi to load ucode `iwlwifi-ty-a0-gf-a0-64.ucode` by removing `/lib/firmware/iwlwifi-ty-a0-gf-a0-66.ucode`

Here are message from `dmesg`:

[ 51.081425][ T59] iwlwifi 0004:01:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-66.ucode failed with error -2
[ 51.081433][ T59] iwlwifi 0004:01:00.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-66.ucode
[ 51.088589][ T59] iwlwifi 0004:01:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-65.ucode failed with error -2
[ 51.088597][ T59] iwlwifi 0004:01:00.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-65.ucode
[ 51.098413][ T59] iwlwifi 0004:01:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-64.ucode failed with error -2
[ 51.098420][ T59] iwlwifi 0004:01:00.0: Falling back to sysfs fallback for: iwlwifi-ty-a0-gf-a0-64.ucode

if people from Intel can share a link to the ucode release notes, we would like to understand more (I can't find the release notes).

hope this help

best