Re:AX210 System Crash after iwlwifi Queue Stuck Error - Issue Persists

chrisx · ‎01-11-2024

Hi Intel Community,

I would like to revisit the issue I previously discussed in this thread: https://community.intel.com/t5/Wireless/AX210-system-crash-after-iwlwifi-Queue-stuck-error/m-p/1550753. Despite implementing the suggested solution and updating to even newer kernel and firmware versions, the problem persists. I'm still experiencing system crashes similar to the ones reported initially.

Here are the updated general details of my system:

```
$ hostnamectl
Static hostname: nixos
Operating System: NixOS 24.05 (Uakari)
Kernel: Linux 6.7.0
Architecture: x86-64
Hardware Vendor: ASUS
Hardware Model: ROG STRIX X670E-E GAMING WIFI
Firmware Version: 1602
Firmware Date: Tue 2023-08-15
```

I've also confirmed that I'm running a much newer firmware for the Wi-Fi adapter than the one during the time of the "accepted solution":

```
$ journalctl -b 0 | grep "firmware version"
Jan 12 05:03:44 nixos kernel: iwlwifi 0000:08:00.0: loaded firmware version 86.fb5c9aeb.0 ty-a0-gf-a0-86.ucode op_mode iwlmvm
```

One of the recent crashes is documented below, indicating the same error pattern:

```
Jan 12 05:00:55 nixos kernel: iwlwifi 0000:08:00.0: Queue 4 is stuck 29100 29144
Jan 12 05:01:02 nixos org.freedesktop.Akonadi.Control[9701]: org.kde.pim.imapresource: "metafont" : "Connection to server lost."
Jan 12 05:01:02 nixos org.freedesktop.Akonadi.Control[9695]: org.kde.pim.imapresource: "[Gmail]/Important" : "Connection to server lost."
Jan 12 05:01:06 nixos kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Jan 12 05:01:06 nixos kernel: rcu: 10-....: (2 ticks this GP) idle=d1ac/1/0x4000000000000002 softirq=1972394/1972394 fqs=5036
Jan 12 05:01:06 nixos kernel: rcu: (detected by 6, t=21002 jiffies, g=10564093, q=7055 ncpus=12)
Jan 12 05:01:06 nixos kernel: Sending NMI from CPU 6 to CPUs 10:
```

The system becomes completely unresponsive, requiring a hard reboot via the power button.

I look forward to your continued support in resolving this persistent issue.

Best regards,
Chris

P.S.: Previously, @IsaacQ_Intel kindly provided assistance with this matter.

_adam_ · ‎01-15-2024

I have also been experiencing the same behavior for quite a while now. I'm patiently awaiting kernel 6.7 from Fedora, and I seem to be on an older iwlwifi firmware, but I figured I would add my log here in case it is of any help. Please let me know if it is considered unhelpful due to my older firmware. I'd rather keep relevant information here for anyone willing to look into the issue. ```

$ hostnamectl
     Static hostname: (unset)                             
  Transient hostname: fedora
    Operating System: Fedora Linux 39.20240112.0 (Kinoite)
         CPE OS Name: cpe:/o:fedoraproject:fedora:39
              Kernel: Linux 6.6.9-200.fc39.x86_64
        Architecture: x86-64
     Hardware Vendor: ASUS
      Hardware Model: ROG STRIX X670E-E GAMING WIFI
    Firmware Version: 1602
       Firmware Date: Tue 2023-08-15

``` ```

$ journalctl -b -3 | grep "firmware version"
Jan 15 10:19:48 fedora kernel: iwlwifi 0000:0a:00.0: loaded firmware version 83.e8f84e98.0 ty-a0-gf-a0-83.ucode op_mode iwlmvm

``` ```

$ journalctl -b -3
Jan 15 10:31:50 fedora kernel: iwlwifi 0000:0a:00.0: Error sending STATISTICS_CMD: time out after 2000ms.
Jan 15 10:31:50 fedora kernel: iwlwifi 0000:0a:00.0: Current CMD queue read_ptr 1405 write_ptr 1406
Jan 15 10:31:58 fedora kernel: iwlwifi 0000:0a:00.0: Queue 4 is stuck 12684 12704
Jan 15 10:32:17 fedora kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [NetworkManager:1502]
Jan 15 10:32:17 fedora kernel: Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink qrtr bnep sunrpc binfmt_misc iwlmvm mac80211 libarc4 snd_hda_codec_hdmi snd_hda_intel snd_usb_audio snd_intel_dspcfg snd_intel_sdw_acpi intel_rapl_msr snd_hda_codec intel_rapl_common snd_usbmidi_lib snd_ump iwlwifi edac_mce_amd snd_hda_core snd_rawmidi btusb mc snd_hwdep btrtl kvm_amd btintel snd_seq btbcm vfat fat btmtk kvm snd_seq_device irqbypass cfg80211 bluetooth snd_pcm eeepc_wmi asus_nb_wmi asus_wmi ledtrig_audio snd_timer sparse_keymap snd platform_profile rapl wmi_bmof k10temp igc pcspkr i2c_piix4 joydev rfkill soundcore gpio_amdpt gpio_generic loop zram nvme nvme_core nvme_common amdgpu drm_exec amdxcp drm_buddy gpu_sched crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
Jan 15 10:32:17 fedora kernel:  polyval_generic i2c_algo_bit drm_suballoc_helper drm_display_helper ghash_clmulni_intel sha512_ssse3 cec sha256_ssse3 sha1_ssse3 drm_ttm_helper ccp ttm sp5100_tco video wmi ip6_tables ip_tables fuse
Jan 15 10:32:17 fedora kernel: CPU: 2 PID: 1502 Comm: NetworkManager Not tainted 6.6.9-200.fc39.x86_64 #1
Jan 15 10:32:17 fedora kernel: Hardware name: ASUS System Product Name/ROG STRIX X670E-E GAMING WIFI, BIOS 1602 08/15/2023
Jan 15 10:32:17 fedora kernel: RIP: 0010:iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:17 fedora kernel: Code: 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 89 f6 48 03 b7 48 35 00 00 8b 06 <e9> c7 9d ba f5 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90
Jan 15 10:32:17 fedora kernel: RSP: 0018:ffffc90001abf438 EFLAGS: 00000286
Jan 15 10:32:17 fedora kernel: RAX: 00000000ffffffff RBX: 00000000000004ba RCX: 0000000000000002
Jan 15 10:32:17 fedora kernel: RDX: 000000000000056a RSI: ffffc90001758024 RDI: ffff888107078028
Jan 15 10:32:17 fedora kernel: RBP: ffff888107078028 R08: 0000000000003a98 R09: 0000000000000101
Jan 15 10:32:17 fedora kernel: R10: ffffffffb8206100 R11: 0000000000034140 R12: 0000000000000011
Jan 15 10:32:17 fedora kernel: R13: 0000000000000001 R14: 0000000000000024 R15: 0000000000003a98
Jan 15 10:32:17 fedora kernel: FS:  00007f007be73540(0000) GS:ffff889018280000(0000) knlGS:0000000000000000
Jan 15 10:32:17 fedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 10:32:17 fedora kernel: CR2: 00003f4073362180 CR3: 000000010d00a000 CR4: 0000000000f50ee0
Jan 15 10:32:17 fedora kernel: PKRU: 55555554
Jan 15 10:32:17 fedora kernel: Call Trace:
Jan 15 10:32:17 fedora kernel:  <IRQ>
Jan 15 10:32:17 fedora kernel:  ? watchdog_timer_fn+0x1b8/0x220
Jan 15 10:32:17 fedora kernel:  ? __pfx_watchdog_timer_fn+0x10/0x10
Jan 15 10:32:17 fedora kernel:  ? __hrtimer_run_queues+0x10f/0x2b0
Jan 15 10:32:17 fedora kernel:  ? hrtimer_interrupt+0xf8/0x230
Jan 15 10:32:17 fedora kernel:  ? __sysvec_apic_timer_interrupt+0x4d/0x140
Jan 15 10:32:17 fedora kernel:  ? sysvec_apic_timer_interrupt+0x6d/0x90
Jan 15 10:32:17 fedora kernel:  </IRQ>
Jan 15 10:32:17 fedora kernel:  <TASK>
Jan 15 10:32:17 fedora kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 15 10:32:17 fedora kernel:  ? iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:17 fedora kernel:  iwl_poll_bit+0x4b/0x80 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  __iwl_trans_pcie_grab_nic_access+0xb7/0x150 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  iwl_trans_pcie_grab_nic_access+0x1a/0x40 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  iwl_force_nmi+0xdf/0x110 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  iwl_trans_sync_nmi_with_addr+0x14b/0x170 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:17 fedora kernel:  iwl_trans_txq_send_hcmd+0x401/0x450 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Jan 15 10:32:17 fedora kernel:  iwl_trans_send_cmd+0x98/0x100 [iwlwifi]
Jan 15 10:32:17 fedora kernel:  iwl_mvm_send_cmd+0x1f/0x60 [iwlmvm]
Jan 15 10:32:17 fedora kernel:  iwl_mvm_request_statistics+0x124/0x1b0 [iwlmvm]
Jan 15 10:32:17 fedora kernel:  ? __pfx_iwl_wait_stats_complete+0x10/0x10 [iwlmvm]
Jan 15 10:32:17 fedora kernel:  iwl_mvm_mac_sta_statistics+0x18b/0x3b0 [iwlmvm]
Jan 15 10:32:17 fedora kernel:  sta_set_sinfo+0xd9/0xc00 [mac80211]
Jan 15 10:32:17 fedora kernel:  ieee80211_dump_station+0x6f/0x90 [mac80211]
Jan 15 10:32:17 fedora kernel:  nl80211_dump_station+0x13f/0x290 [cfg80211]
Jan 15 10:32:17 fedora kernel:  genl_dumpit+0x33/0x90
Jan 15 10:32:17 fedora kernel:  netlink_dump+0x126/0x320
Jan 15 10:32:17 fedora kernel:  __netlink_dump_start+0x1d6/0x290
Jan 15 10:32:17 fedora kernel:  genl_family_rcv_msg_dumpit+0x9c/0x100
Jan 15 10:32:17 fedora kernel:  ? free_unref_page+0xf6/0x180
Jan 15 10:32:17 fedora kernel:  ? __pfx_genl_start+0x10/0x10
Jan 15 10:32:17 fedora kernel:  ? __pfx_genl_dumpit+0x10/0x10
Jan 15 10:32:17 fedora kernel:  ? __pfx_genl_done+0x10/0x10
Jan 15 10:32:17 fedora kernel:  genl_rcv_msg+0x146/0x2c0
Jan 15 10:32:17 fedora kernel:  ? __pfx_nl80211_dump_station+0x10/0x10 [cfg80211]
Jan 15 10:32:17 fedora kernel:  ? __pfx_genl_rcv_msg+0x10/0x10
Jan 15 10:32:17 fedora kernel:  netlink_rcv_skb+0x58/0x110
Jan 15 10:32:17 fedora kernel:  genl_rcv+0x28/0x40
Jan 15 10:32:17 fedora kernel:  netlink_unicast+0x1a3/0x290
Jan 15 10:32:17 fedora kernel:  netlink_sendmsg+0x254/0x4d0
Jan 15 10:32:17 fedora kernel:  ____sys_sendmsg+0x396/0x3d0
Jan 15 10:32:17 fedora kernel:  ? copy_msghdr_from_user+0x7d/0xc0
Jan 15 10:32:17 fedora kernel:  ___sys_sendmsg+0x9a/0xe0
Jan 15 10:32:17 fedora kernel:  ? __pfx_pollwake+0x10/0x10
Jan 15 10:32:17 fedora kernel:  __sys_sendmsg+0x7a/0xd0
Jan 15 10:32:17 fedora kernel:  do_syscall_64+0x5d/0x90
Jan 15 10:32:17 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:17 fedora kernel:  ? __x64_sys_epoll_wait+0x73/0x110
Jan 15 10:32:17 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:17 fedora kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
Jan 15 10:32:17 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:17 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:17 fedora kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
Jan 15 10:32:17 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:17 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:17 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:17 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:17 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:17 fedora kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Jan 15 10:32:17 fedora kernel: RIP: 0033:0x7f007cfb4a5b
Jan 15 10:32:17 fedora kernel: Code: 48 89 e5 48 83 ec 20 89 55 ec 48 89 75 f0 89 7d f8 e8 c9 2f f7 ff 8b 55 ec 48 8b 75 f0 41 89 c0 8b 7d f8 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 45 f8 e8 21 30 f7 ff 48 8b
Jan 15 10:32:17 fedora kernel: RSP: 002b:00007fff4a639fa0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
Jan 15 10:32:17 fedora kernel: RAX: ffffffffffffffda RBX: 000055bb4779c2a0 RCX: 00007f007cfb4a5b
Jan 15 10:32:17 fedora kernel: RDX: 0000000000000000 RSI: 00007fff4a639fe0 RDI: 000000000000000b
Jan 15 10:32:17 fedora kernel: RBP: 00007fff4a639fc0 R08: 0000000000000000 R09: 00007f007d075b80
Jan 15 10:32:17 fedora kernel: R10: 0000000000000080 R11: 0000000000000293 R12: 000055bb478153c0
Jan 15 10:32:17 fedora kernel: R13: 00007fff4a63a070 R14: 000055bb4779c2a0 R15: 00007fff4a63a2c4
Jan 15 10:32:17 fedora kernel:  </TASK>
Jan 15 10:32:21 fedora kernel: watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [swapper/21:0]
Jan 15 10:32:21 fedora kernel: Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink qrtr bnep sunrpc binfmt_misc iwlmvm mac80211 libarc4 snd_hda_codec_hdmi snd_hda_intel snd_usb_audio snd_intel_dspcfg snd_intel_sdw_acpi intel_rapl_msr snd_hda_codec intel_rapl_common snd_usbmidi_lib snd_ump iwlwifi edac_mce_amd snd_hda_core snd_rawmidi btusb mc snd_hwdep btrtl kvm_amd btintel snd_seq btbcm vfat fat btmtk kvm snd_seq_device irqbypass cfg80211 bluetooth snd_pcm eeepc_wmi asus_nb_wmi asus_wmi ledtrig_audio snd_timer sparse_keymap snd platform_profile rapl wmi_bmof k10temp igc pcspkr i2c_piix4 joydev rfkill soundcore gpio_amdpt gpio_generic loop zram nvme nvme_core nvme_common amdgpu drm_exec amdxcp drm_buddy gpu_sched crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
Jan 15 10:32:21 fedora kernel:  polyval_generic i2c_algo_bit drm_suballoc_helper drm_display_helper ghash_clmulni_intel sha512_ssse3 cec sha256_ssse3 sha1_ssse3 drm_ttm_helper ccp ttm sp5100_tco video wmi ip6_tables ip_tables fuse
Jan 15 10:32:21 fedora kernel: CPU: 21 PID: 0 Comm: swapper/21 Tainted: G             L     6.6.9-200.fc39.x86_64 #1
Jan 15 10:32:21 fedora kernel: Hardware name: ASUS System Product Name/ROG STRIX X670E-E GAMING WIFI, BIOS 1602 08/15/2023
Jan 15 10:32:21 fedora kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x71/0x2b0
Jan 15 10:32:21 fedora kernel: Code: 0f ba 2b 08 0f 92 c2 8b 03 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 77 53 85 c0 74 10 0f b6 03 84 c0 74 09 f3 90 0f b6 03 <84> c0 75 f7 b8 01 00 00 00 66 89 03 5b 5d 41 5c 41 5d e9 f8 11 00
Jan 15 10:32:21 fedora kernel: RSP: 0018:ffffc900006fce28 EFLAGS: 00000202
Jan 15 10:32:21 fedora kernel: RAX: 0000000000000001 RBX: ffff88810707b5e4 RCX: ffffffffc16bd540
Jan 15 10:32:21 fedora kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810707b5e4
Jan 15 10:32:21 fedora kernel: RBP: 0000000000d05c04 R08: 0000000000000000 R09: ffffc900006fcbc0
Jan 15 10:32:21 fedora kernel: R10: 0000000000000003 R11: ffffffffb8346508 R12: ffffffffc145e520
Jan 15 10:32:21 fedora kernel: R13: ffff8881158a0330 R14: ffff88810707b5e4 R15: ffff889018762280
Jan 15 10:32:21 fedora kernel: FS:  0000000000000000(0000) GS:ffff889018740000(0000) knlGS:0000000000000000
Jan 15 10:32:21 fedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 10:32:21 fedora kernel: CR2: 00007f7457956000 CR3: 0000000494222000 CR4: 0000000000f50ee0
Jan 15 10:32:21 fedora kernel: PKRU: 55555554
Jan 15 10:32:21 fedora kernel: Call Trace:
Jan 15 10:32:21 fedora kernel:  <IRQ>
Jan 15 10:32:21 fedora kernel:  ? watchdog_timer_fn+0x1b8/0x220
Jan 15 10:32:21 fedora kernel:  ? __pfx_watchdog_timer_fn+0x10/0x10
Jan 15 10:32:21 fedora kernel:  ? __hrtimer_run_queues+0x10f/0x2b0
Jan 15 10:32:21 fedora kernel:  ? hrtimer_interrupt+0xf8/0x230
Jan 15 10:32:21 fedora kernel:  ? __sysvec_apic_timer_interrupt+0x4d/0x140
Jan 15 10:32:21 fedora kernel:  ? sysvec_apic_timer_interrupt+0x39/0x90
Jan 15 10:32:21 fedora kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 15 10:32:21 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  ? native_queued_spin_lock_slowpath+0x71/0x2b0
Jan 15 10:32:21 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  _raw_spin_lock+0x29/0x30
Jan 15 10:32:21 fedora kernel:  __iwl_trans_pcie_grab_nic_access+0x25/0x150 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  iwl_trans_pcie_grab_nic_access+0x1a/0x40 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  iwl_force_nmi+0xdf/0x110 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  call_timer_fn+0x24/0x130
Jan 15 10:32:21 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:21 fedora kernel:  __run_timers+0x222/0x2c0
Jan 15 10:32:21 fedora kernel:  run_timer_softirq+0x1d/0x40
Jan 15 10:32:21 fedora kernel:  __do_softirq+0xd1/0x2c8
Jan 15 10:32:21 fedora kernel:  __irq_exit_rcu+0xa6/0xc0
Jan 15 10:32:21 fedora kernel:  sysvec_apic_timer_interrupt+0x72/0x90
Jan 15 10:32:21 fedora kernel:  </IRQ>
Jan 15 10:32:21 fedora kernel:  <TASK>
Jan 15 10:32:21 fedora kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 15 10:32:21 fedora kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x440
Jan 15 10:32:21 fedora kernel: Code: ea be 1a ff e8 c5 f3 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 73 c2 19 ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d
Jan 15 10:32:21 fedora kernel: RSP: 0018:ffffc9000025fe90 EFLAGS: 00000246
Jan 15 10:32:21 fedora kernel: RAX: ffff889018774140 RBX: ffff88810e71b000 RCX: 000000000000001f
Jan 15 10:32:21 fedora kernel: RDX: 0000000000000015 RSI: 000000001e79f308 RDI: 0000000000000000
Jan 15 10:32:21 fedora kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000288
Jan 15 10:32:21 fedora kernel: R10: 0000000000000006 R11: ffff889018772aa4 R12: ffffffffb8447200
Jan 15 10:32:21 fedora kernel: R13: 000000aba6eddf17 R14: 0000000000000002 R15: 0000000000000000
Jan 15 10:32:21 fedora kernel:  cpuidle_enter+0x2d/0x40
Jan 15 10:32:21 fedora kernel:  do_idle+0x20d/0x270
Jan 15 10:32:21 fedora kernel:  cpu_startup_entry+0x2a/0x30
Jan 15 10:32:21 fedora kernel:  start_secondary+0x11e/0x140
Jan 15 10:32:21 fedora kernel:  secondary_startup_64_no_verify+0x18f/0x19b
Jan 15 10:32:21 fedora kernel:  </TASK>
Jan 15 10:32:45 fedora kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 52s! [NetworkManager:1502]
Jan 15 10:32:45 fedora kernel: Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink qrtr bnep sunrpc binfmt_misc iwlmvm mac80211 libarc4 snd_hda_codec_hdmi snd_hda_intel snd_usb_audio snd_intel_dspcfg snd_intel_sdw_acpi intel_rapl_msr snd_hda_codec intel_rapl_common snd_usbmidi_lib snd_ump iwlwifi edac_mce_amd snd_hda_core snd_rawmidi btusb mc snd_hwdep btrtl kvm_amd btintel snd_seq btbcm vfat fat btmtk kvm snd_seq_device irqbypass cfg80211 bluetooth snd_pcm eeepc_wmi asus_nb_wmi asus_wmi ledtrig_audio snd_timer sparse_keymap snd platform_profile rapl wmi_bmof k10temp igc pcspkr i2c_piix4 joydev rfkill soundcore gpio_amdpt gpio_generic loop zram nvme nvme_core nvme_common amdgpu drm_exec amdxcp drm_buddy gpu_sched crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
Jan 15 10:32:45 fedora kernel:  polyval_generic i2c_algo_bit drm_suballoc_helper drm_display_helper ghash_clmulni_intel sha512_ssse3 cec sha256_ssse3 sha1_ssse3 drm_ttm_helper ccp ttm sp5100_tco video wmi ip6_tables ip_tables fuse
Jan 15 10:32:45 fedora kernel: CPU: 2 PID: 1502 Comm: NetworkManager Tainted: G             L     6.6.9-200.fc39.x86_64 #1
Jan 15 10:32:45 fedora kernel: Hardware name: ASUS System Product Name/ROG STRIX X670E-E GAMING WIFI, BIOS 1602 08/15/2023
Jan 15 10:32:45 fedora kernel: RIP: 0010:iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:45 fedora kernel: Code: 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 89 f6 48 03 b7 48 35 00 00 8b 06 <e9> c7 9d ba f5 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90
Jan 15 10:32:45 fedora kernel: RSP: 0018:ffffc90001abf438 EFLAGS: 00000286
Jan 15 10:32:45 fedora kernel: RAX: 00000000ffffffff RBX: 00000000000009ba RCX: 0000000000000002
Jan 15 10:32:45 fedora kernel: RDX: 000000000000056a RSI: ffffc90001758024 RDI: ffff888107078028
Jan 15 10:32:45 fedora kernel: RBP: ffff888107078028 R08: 0000000000003a98 R09: 0000000000000101
Jan 15 10:32:45 fedora kernel: R10: ffffffffb8206100 R11: 0000000000034140 R12: 0000000000000011
Jan 15 10:32:45 fedora kernel: R13: 0000000000000001 R14: 0000000000000024 R15: 0000000000003a98
Jan 15 10:32:45 fedora kernel: FS:  00007f007be73540(0000) GS:ffff889018280000(0000) knlGS:0000000000000000
Jan 15 10:32:45 fedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 10:32:45 fedora kernel: CR2: 00003f4073362180 CR3: 000000010d00a000 CR4: 0000000000f50ee0
Jan 15 10:32:45 fedora kernel: PKRU: 55555554
Jan 15 10:32:45 fedora kernel: Call Trace:
Jan 15 10:32:45 fedora kernel:  <IRQ>
Jan 15 10:32:45 fedora kernel:  ? watchdog_timer_fn+0x1b8/0x220
Jan 15 10:32:45 fedora kernel:  ? __pfx_watchdog_timer_fn+0x10/0x10
Jan 15 10:32:45 fedora kernel:  ? __hrtimer_run_queues+0x10f/0x2b0
Jan 15 10:32:45 fedora kernel:  ? hrtimer_interrupt+0xf8/0x230
Jan 15 10:32:45 fedora kernel:  ? __sysvec_apic_timer_interrupt+0x4d/0x140
Jan 15 10:32:45 fedora kernel:  ? sysvec_apic_timer_interrupt+0x6d/0x90
Jan 15 10:32:45 fedora kernel:  </IRQ>
Jan 15 10:32:45 fedora kernel:  <TASK>
Jan 15 10:32:45 fedora kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 15 10:32:45 fedora kernel:  ? iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:45 fedora kernel:  iwl_poll_bit+0x4b/0x80 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  __iwl_trans_pcie_grab_nic_access+0xb7/0x150 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  iwl_trans_pcie_grab_nic_access+0x1a/0x40 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  iwl_force_nmi+0xdf/0x110 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  iwl_trans_sync_nmi_with_addr+0x14b/0x170 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:45 fedora kernel:  iwl_trans_txq_send_hcmd+0x401/0x450 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Jan 15 10:32:45 fedora kernel:  iwl_trans_send_cmd+0x98/0x100 [iwlwifi]
Jan 15 10:32:45 fedora kernel:  iwl_mvm_send_cmd+0x1f/0x60 [iwlmvm]
Jan 15 10:32:45 fedora kernel:  iwl_mvm_request_statistics+0x124/0x1b0 [iwlmvm]
Jan 15 10:32:45 fedora kernel:  ? __pfx_iwl_wait_stats_complete+0x10/0x10 [iwlmvm]
Jan 15 10:32:45 fedora kernel:  iwl_mvm_mac_sta_statistics+0x18b/0x3b0 [iwlmvm]
Jan 15 10:32:45 fedora kernel:  sta_set_sinfo+0xd9/0xc00 [mac80211]
Jan 15 10:32:45 fedora kernel:  ieee80211_dump_station+0x6f/0x90 [mac80211]
Jan 15 10:32:45 fedora kernel:  nl80211_dump_station+0x13f/0x290 [cfg80211]
Jan 15 10:32:45 fedora kernel:  genl_dumpit+0x33/0x90
Jan 15 10:32:45 fedora kernel:  netlink_dump+0x126/0x320
Jan 15 10:32:45 fedora kernel:  __netlink_dump_start+0x1d6/0x290
Jan 15 10:32:45 fedora kernel:  genl_family_rcv_msg_dumpit+0x9c/0x100
Jan 15 10:32:45 fedora kernel:  ? free_unref_page+0xf6/0x180
Jan 15 10:32:45 fedora kernel:  ? __pfx_genl_start+0x10/0x10
Jan 15 10:32:45 fedora kernel:  ? __pfx_genl_dumpit+0x10/0x10
Jan 15 10:32:45 fedora kernel:  ? __pfx_genl_done+0x10/0x10
Jan 15 10:32:45 fedora kernel:  genl_rcv_msg+0x146/0x2c0
Jan 15 10:32:45 fedora kernel:  ? __pfx_nl80211_dump_station+0x10/0x10 [cfg80211]
Jan 15 10:32:45 fedora kernel:  ? __pfx_genl_rcv_msg+0x10/0x10
Jan 15 10:32:45 fedora kernel:  netlink_rcv_skb+0x58/0x110
Jan 15 10:32:45 fedora kernel:  genl_rcv+0x28/0x40
Jan 15 10:32:45 fedora kernel:  netlink_unicast+0x1a3/0x290
Jan 15 10:32:45 fedora kernel:  netlink_sendmsg+0x254/0x4d0
Jan 15 10:32:45 fedora kernel:  ____sys_sendmsg+0x396/0x3d0
Jan 15 10:32:45 fedora kernel:  ? copy_msghdr_from_user+0x7d/0xc0
Jan 15 10:32:45 fedora kernel:  ___sys_sendmsg+0x9a/0xe0
Jan 15 10:32:45 fedora kernel:  ? __pfx_pollwake+0x10/0x10
Jan 15 10:32:45 fedora kernel:  __sys_sendmsg+0x7a/0xd0
Jan 15 10:32:45 fedora kernel:  do_syscall_64+0x5d/0x90
Jan 15 10:32:45 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:45 fedora kernel:  ? __x64_sys_epoll_wait+0x73/0x110
Jan 15 10:32:45 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:45 fedora kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
Jan 15 10:32:45 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:45 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:45 fedora kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
Jan 15 10:32:45 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:45 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:45 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:45 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:45 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:45 fedora kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Jan 15 10:32:45 fedora kernel: RIP: 0033:0x7f007cfb4a5b
Jan 15 10:32:45 fedora kernel: Code: 48 89 e5 48 83 ec 20 89 55 ec 48 89 75 f0 89 7d f8 e8 c9 2f f7 ff 8b 55 ec 48 8b 75 f0 41 89 c0 8b 7d f8 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 45 f8 e8 21 30 f7 ff 48 8b
Jan 15 10:32:45 fedora kernel: RSP: 002b:00007fff4a639fa0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
Jan 15 10:32:45 fedora kernel: RAX: ffffffffffffffda RBX: 000055bb4779c2a0 RCX: 00007f007cfb4a5b
Jan 15 10:32:45 fedora kernel: RDX: 0000000000000000 RSI: 00007fff4a639fe0 RDI: 000000000000000b
Jan 15 10:32:45 fedora kernel: RBP: 00007fff4a639fc0 R08: 0000000000000000 R09: 00007f007d075b80
Jan 15 10:32:45 fedora kernel: R10: 0000000000000080 R11: 0000000000000293 R12: 000055bb478153c0
Jan 15 10:32:45 fedora kernel: R13: 00007fff4a63a070 R14: 000055bb4779c2a0 R15: 00007fff4a63a2c4
Jan 15 10:32:45 fedora kernel:  </TASK>
Jan 15 10:32:49 fedora kernel: watchdog: BUG: soft lockup - CPU#21 stuck for 49s! [swapper/21:0]
Jan 15 10:32:49 fedora kernel: Modules linked in: uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink qrtr bnep sunrpc binfmt_misc iwlmvm mac80211 libarc4 snd_hda_codec_hdmi snd_hda_intel snd_usb_audio snd_intel_dspcfg snd_intel_sdw_acpi intel_rapl_msr snd_hda_codec intel_rapl_common snd_usbmidi_lib snd_ump iwlwifi edac_mce_amd snd_hda_core snd_rawmidi btusb mc snd_hwdep btrtl kvm_amd btintel snd_seq btbcm vfat fat btmtk kvm snd_seq_device irqbypass cfg80211 bluetooth snd_pcm eeepc_wmi asus_nb_wmi asus_wmi ledtrig_audio snd_timer sparse_keymap snd platform_profile rapl wmi_bmof k10temp igc pcspkr i2c_piix4 joydev rfkill soundcore gpio_amdpt gpio_generic loop zram nvme nvme_core nvme_common amdgpu drm_exec amdxcp drm_buddy gpu_sched crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
Jan 15 10:32:49 fedora kernel:  polyval_generic i2c_algo_bit drm_suballoc_helper drm_display_helper ghash_clmulni_intel sha512_ssse3 cec sha256_ssse3 sha1_ssse3 drm_ttm_helper ccp ttm sp5100_tco video wmi ip6_tables ip_tables fuse
Jan 15 10:32:49 fedora kernel: CPU: 21 PID: 0 Comm: swapper/21 Tainted: G             L     6.6.9-200.fc39.x86_64 #1
Jan 15 10:32:49 fedora kernel: Hardware name: ASUS System Product Name/ROG STRIX X670E-E GAMING WIFI, BIOS 1602 08/15/2023
Jan 15 10:32:49 fedora kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x71/0x2b0
Jan 15 10:32:49 fedora kernel: Code: 0f ba 2b 08 0f 92 c2 8b 03 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 77 53 85 c0 74 10 0f b6 03 84 c0 74 09 f3 90 0f b6 03 <84> c0 75 f7 b8 01 00 00 00 66 89 03 5b 5d 41 5c 41 5d e9 f8 11 00
Jan 15 10:32:49 fedora kernel: RSP: 0018:ffffc900006fce28 EFLAGS: 00000202
Jan 15 10:32:49 fedora kernel: RAX: 0000000000000001 RBX: ffff88810707b5e4 RCX: ffffffffc16bd540
Jan 15 10:32:49 fedora kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810707b5e4
Jan 15 10:32:49 fedora kernel: RBP: 0000000000d05c04 R08: 0000000000000000 R09: ffffc900006fcbc0
Jan 15 10:32:49 fedora kernel: R10: 0000000000000003 R11: ffffffffb8346508 R12: ffffffffc145e520
Jan 15 10:32:49 fedora kernel: R13: ffff8881158a0330 R14: ffff88810707b5e4 R15: ffff889018762280
Jan 15 10:32:49 fedora kernel: FS:  0000000000000000(0000) GS:ffff889018740000(0000) knlGS:0000000000000000
Jan 15 10:32:49 fedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 10:32:49 fedora kernel: CR2: 00007f7457956000 CR3: 0000000494222000 CR4: 0000000000f50ee0
Jan 15 10:32:49 fedora kernel: PKRU: 55555554
Jan 15 10:32:49 fedora kernel: Call Trace:
Jan 15 10:32:49 fedora kernel:  <IRQ>
Jan 15 10:32:49 fedora kernel:  ? watchdog_timer_fn+0x1b8/0x220
Jan 15 10:32:49 fedora kernel:  ? __pfx_watchdog_timer_fn+0x10/0x10
Jan 15 10:32:49 fedora kernel:  ? __hrtimer_run_queues+0x10f/0x2b0
Jan 15 10:32:49 fedora kernel:  ? hrtimer_interrupt+0xf8/0x230
Jan 15 10:32:49 fedora kernel:  ? __sysvec_apic_timer_interrupt+0x4d/0x140
Jan 15 10:32:49 fedora kernel:  ? sysvec_apic_timer_interrupt+0x39/0x90
Jan 15 10:32:49 fedora kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 15 10:32:49 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  ? native_queued_spin_lock_slowpath+0x71/0x2b0
Jan 15 10:32:49 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  _raw_spin_lock+0x29/0x30
Jan 15 10:32:49 fedora kernel:  __iwl_trans_pcie_grab_nic_access+0x25/0x150 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  iwl_trans_pcie_grab_nic_access+0x1a/0x40 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  iwl_force_nmi+0xdf/0x110 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  call_timer_fn+0x24/0x130
Jan 15 10:32:49 fedora kernel:  ? __pfx_iwl_txq_stuck_timer+0x10/0x10 [iwlwifi]
Jan 15 10:32:49 fedora kernel:  __run_timers+0x222/0x2c0
Jan 15 10:32:49 fedora kernel:  run_timer_softirq+0x1d/0x40
Jan 15 10:32:49 fedora kernel:  __do_softirq+0xd1/0x2c8
Jan 15 10:32:49 fedora kernel:  __irq_exit_rcu+0xa6/0xc0
Jan 15 10:32:49 fedora kernel:  sysvec_apic_timer_interrupt+0x72/0x90
Jan 15 10:32:49 fedora kernel:  </IRQ>
Jan 15 10:32:49 fedora kernel:  <TASK>
Jan 15 10:32:49 fedora kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
Jan 15 10:32:49 fedora kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x440
Jan 15 10:32:49 fedora kernel: Code: ea be 1a ff e8 c5 f3 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 73 c2 19 ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d
Jan 15 10:32:49 fedora kernel: RSP: 0018:ffffc9000025fe90 EFLAGS: 00000246
Jan 15 10:32:49 fedora kernel: RAX: ffff889018774140 RBX: ffff88810e71b000 RCX: 000000000000001f
Jan 15 10:32:49 fedora kernel: RDX: 0000000000000015 RSI: 000000001e79f308 RDI: 0000000000000000
Jan 15 10:32:49 fedora kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000288
Jan 15 10:32:49 fedora kernel: R10: 0000000000000006 R11: ffff889018772aa4 R12: ffffffffb8447200
Jan 15 10:32:49 fedora kernel: R13: 000000aba6eddf17 R14: 0000000000000002 R15: 0000000000000000
Jan 15 10:32:49 fedora kernel:  cpuidle_enter+0x2d/0x40
Jan 15 10:32:49 fedora kernel:  do_idle+0x20d/0x270
Jan 15 10:32:49 fedora kernel:  cpu_startup_entry+0x2a/0x30
Jan 15 10:32:49 fedora kernel:  start_secondary+0x11e/0x140
Jan 15 10:32:49 fedora kernel:  secondary_startup_64_no_verify+0x18f/0x19b
Jan 15 10:32:49 fedora kernel:  </TASK>
Jan 15 10:32:50 fedora kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Jan 15 10:32:50 fedora kernel: rcu:         2-....: (275 ticks this GP) idle=19dc/1/0x4000000000000000 softirq=76024/76024 fqs=13556
Jan 15 10:32:50 fedora kernel: rcu:         (detected by 7, t=60002 jiffies, g=336801, q=28891 ncpus=32)
Jan 15 10:32:50 fedora kernel: Sending NMI from CPU 7 to CPUs 2:
Jan 15 10:32:50 fedora kernel: NMI backtrace for cpu 2
Jan 15 10:32:50 fedora kernel: CPU: 2 PID: 1502 Comm: NetworkManager Tainted: G             L     6.6.9-200.fc39.x86_64 #1
Jan 15 10:32:50 fedora kernel: Hardware name: ASUS System Product Name/ROG STRIX X670E-E GAMING WIFI, BIOS 1602 08/15/2023
Jan 15 10:32:50 fedora kernel: RIP: 0010:iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:50 fedora kernel: Code: 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 89 f6 48 03 b7 48 35 00 00 8b 06 <e9> c7 9d ba f5 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90
Jan 15 10:32:50 fedora kernel: RSP: 0018:ffffc90001abf438 EFLAGS: 00000286
Jan 15 10:32:50 fedora kernel: RAX: 00000000ffffffff RBX: 0000000000000a96 RCX: 0000000000000002
Jan 15 10:32:50 fedora kernel: RDX: 000000000000056a RSI: ffffc90001758024 RDI: ffff888107078028
Jan 15 10:32:50 fedora kernel: RBP: ffff888107078028 R08: 0000000000003a98 R09: 0000000000000101
Jan 15 10:32:50 fedora kernel: R10: ffffffffb8206100 R11: 0000000000034140 R12: 0000000000000011
Jan 15 10:32:50 fedora kernel: R13: 0000000000000001 R14: 0000000000000024 R15: 0000000000003a98
Jan 15 10:32:50 fedora kernel: FS:  00007f007be73540(0000) GS:ffff889018280000(0000) knlGS:0000000000000000
Jan 15 10:32:50 fedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 15 10:32:50 fedora kernel: CR2: 00003f4073362180 CR3: 000000010d00a000 CR4: 0000000000f50ee0
Jan 15 10:32:50 fedora kernel: PKRU: 55555554
Jan 15 10:32:50 fedora kernel: Call Trace:
Jan 15 10:32:50 fedora kernel:  <NMI>
Jan 15 10:32:50 fedora kernel:  ? nmi_cpu_backtrace+0x99/0x110
Jan 15 10:32:50 fedora kernel:  ? nmi_cpu_backtrace_handler+0x11/0x20
Jan 15 10:32:50 fedora kernel:  ? nmi_handle+0x5e/0x150
Jan 15 10:32:50 fedora kernel:  ? default_do_nmi+0x40/0x100
Jan 15 10:32:50 fedora kernel:  ? exc_nmi+0x139/0x1c0
Jan 15 10:32:50 fedora kernel:  ? end_repeat_nmi+0x16/0x67
Jan 15 10:32:50 fedora kernel:  ? iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  ? iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  ? iwl_trans_pcie_read32+0x14/0x20 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  </NMI>
Jan 15 10:32:50 fedora kernel:  <TASK>
Jan 15 10:32:50 fedora kernel:  iwl_poll_bit+0x4b/0x80 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  __iwl_trans_pcie_grab_nic_access+0xb7/0x150 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  iwl_trans_pcie_grab_nic_access+0x1a/0x40 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  iwl_force_nmi+0xdf/0x110 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  iwl_trans_sync_nmi_with_addr+0x14b/0x170 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:50 fedora kernel:  iwl_trans_txq_send_hcmd+0x401/0x450 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Jan 15 10:32:50 fedora kernel:  iwl_trans_send_cmd+0x98/0x100 [iwlwifi]
Jan 15 10:32:50 fedora kernel:  iwl_mvm_send_cmd+0x1f/0x60 [iwlmvm]
Jan 15 10:32:50 fedora kernel:  iwl_mvm_request_statistics+0x124/0x1b0 [iwlmvm]
Jan 15 10:32:50 fedora kernel:  ? __pfx_iwl_wait_stats_complete+0x10/0x10 [iwlmvm]
Jan 15 10:32:50 fedora kernel:  iwl_mvm_mac_sta_statistics+0x18b/0x3b0 [iwlmvm]
Jan 15 10:32:50 fedora kernel:  sta_set_sinfo+0xd9/0xc00 [mac80211]
Jan 15 10:32:50 fedora kernel:  ieee80211_dump_station+0x6f/0x90 [mac80211]
Jan 15 10:32:50 fedora kernel:  nl80211_dump_station+0x13f/0x290 [cfg80211]
Jan 15 10:32:50 fedora kernel:  genl_dumpit+0x33/0x90
Jan 15 10:32:50 fedora kernel:  netlink_dump+0x126/0x320
Jan 15 10:32:50 fedora kernel:  __netlink_dump_start+0x1d6/0x290
Jan 15 10:32:50 fedora kernel:  genl_family_rcv_msg_dumpit+0x9c/0x100
Jan 15 10:32:50 fedora kernel:  ? free_unref_page+0xf6/0x180
Jan 15 10:32:50 fedora kernel:  ? __pfx_genl_start+0x10/0x10
Jan 15 10:32:50 fedora kernel:  ? __pfx_genl_dumpit+0x10/0x10
Jan 15 10:32:50 fedora kernel:  ? __pfx_genl_done+0x10/0x10
Jan 15 10:32:50 fedora kernel:  genl_rcv_msg+0x146/0x2c0
Jan 15 10:32:50 fedora kernel:  ? __pfx_nl80211_dump_station+0x10/0x10 [cfg80211]
Jan 15 10:32:50 fedora kernel:  ? __pfx_genl_rcv_msg+0x10/0x10
Jan 15 10:32:50 fedora kernel:  netlink_rcv_skb+0x58/0x110
Jan 15 10:32:50 fedora kernel:  genl_rcv+0x28/0x40
Jan 15 10:32:50 fedora kernel:  netlink_unicast+0x1a3/0x290
Jan 15 10:32:50 fedora kernel:  netlink_sendmsg+0x254/0x4d0
Jan 15 10:32:50 fedora kernel:  ____sys_sendmsg+0x396/0x3d0
Jan 15 10:32:50 fedora kernel:  ? copy_msghdr_from_user+0x7d/0xc0
Jan 15 10:32:50 fedora kernel:  ___sys_sendmsg+0x9a/0xe0
Jan 15 10:32:50 fedora kernel:  ? __pfx_pollwake+0x10/0x10
Jan 15 10:32:50 fedora kernel:  __sys_sendmsg+0x7a/0xd0
Jan 15 10:32:50 fedora kernel:  do_syscall_64+0x5d/0x90
Jan 15 10:32:50 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:50 fedora kernel:  ? __x64_sys_epoll_wait+0x73/0x110
Jan 15 10:32:50 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:50 fedora kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
Jan 15 10:32:50 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:50 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:50 fedora kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
Jan 15 10:32:50 fedora kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 15 10:32:50 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:50 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:50 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:50 fedora kernel:  ? do_syscall_64+0x6c/0x90
Jan 15 10:32:50 fedora kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Jan 15 10:32:50 fedora kernel: RIP: 0033:0x7f007cfb4a5b
Jan 15 10:32:50 fedora kernel: Code: 48 89 e5 48 83 ec 20 89 55 ec 48 89 75 f0 89 7d f8 e8 c9 2f f7 ff 8b 55 ec 48 8b 75 f0 41 89 c0 8b 7d f8 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 45 f8 e8 21 30 f7 ff 48 8b
Jan 15 10:32:50 fedora kernel: RSP: 002b:00007fff4a639fa0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
Jan 15 10:32:50 fedora kernel: RAX: ffffffffffffffda RBX: 000055bb4779c2a0 RCX: 00007f007cfb4a5b
Jan 15 10:32:50 fedora kernel: RDX: 0000000000000000 RSI: 00007fff4a639fe0 RDI: 000000000000000b
Jan 15 10:32:50 fedora kernel: RBP: 00007fff4a639fc0 R08: 0000000000000000 R09: 00007f007d075b80
Jan 15 10:32:50 fedora kernel: R10: 0000000000000080 R11: 0000000000000293 R12: 000055bb478153c0
Jan 15 10:32:50 fedora kernel: R13: 00007fff4a63a070 R14: 000055bb4779c2a0 R15: 00007fff4a63a2c4
Jan 15 10:32:50 fedora kernel:  </TASK>
```

chrisx · ‎01-16-2024

Hi Adam,

Thank you for sharing your experience and logs. This additional information is certainly helpful as it appears we are dealing with a similar issue. Despite the difference in firmware versions, the errors you've encountered seem to align closely with what I've been experiencing.

For context, I was facing this exact problem while using firmware version `83.e8f84e98.0 ty-a0-gf-a0-83.ucode` as well, and updating to the latest firmware `86.fb5c9aeb.0 ty-a0-gf-a0-86.ucode` has not led to any noticeable improvements. Our issues may indeed be related, regardless of the firmware version.

Here is what I believe is a signature of the crash I've been experiencing, but extracted from your `journalctl` output:

```
$ egrep -i 'queue.*is stuck|rcu_preempt' fedora_journalctl_20240116
Jan 15 10:31:58 fedora kernel: iwlwifi 0000:0a:00.0: Queue 4 is stuck 12684 12704
Jan 15 10:32:50 fedora kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
```

And then the system crashes -- at least, it always does in my case.

To provide an update, I experienced another crash just yesterday. This reaffirms that the issue is still active with kernel 6.7 and the latest firmware.

Best,
Chris

Jose_Intel · ‎01-22-2024

Hello @chrisx

Thank you for posting on the Intel️® communities.

We sincerely regret you are still experiencing issues with your Intel® Wi-Fi 6E AX210 (Gig+), we will be more than happy to assist you.

According to the information you have provided, we noticed there’s a new BIOS update (1807), please keep it up to date and let us know if there is any difference.

Best regards,

Jose B.

Intel Customer Support Technician

chrisx · ‎01-25-2024

Hi Jose,

Following your recommendation, I've updated the BIOS on my system to the latest version (1807, with a firmware date of Thu 2023-12-14). I've also informed Adam, who has been experiencing the same `iwlwifi` issue, about this BIOS update.

Unfortunately, the issue persists despite the BIOS update. Below are the system details post-update and a snippet from the `journalctl` log that captures the recurring crash:

```
$ hostnamectl
Static hostname: nixos
Icon name: computer-desktop
Chassis: desktop
Operating System: NixOS 24.05 (Uakari)
Kernel: Linux 6.7.1
Architecture: x86-64
Hardware Vendor: ASUS
Hardware Model: ROG STRIX X670E-E GAMING WIFI
Firmware Version: 1807
Firmware Date: Thu 2023-12-14
```

```
$ journalctl -b -1
...
Jan 26 03:24:55 nixos kernel: iwlwifi 0000:08:00.0: Queue 4 is stuck 39108 39109
Jan 26 03:25:00 nixos kernel: sched: RT throttling activated
Jan 26 03:25:11 nixos bluetoothd[1059]: profiles/audio/avdtp.c:cancel_request() Suspend: Connection timed out (110)
Jan 26 03:25:16 nixos kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Jan 26 03:25:16 nixos kernel: rcu: 0-....: (8 GPs behind) idle=d6bc/1/0x4000000000000002 softirq=3077933/3077934 fqs=5080
Jan 26 03:25:16 nixos kernel: rcu: (detected by 3, t=21002 jiffies, g=14745593, q=886 ncpus=12)
```

As before, the system was completely unresponsive, and I had to resort to using the power button to reboot. I'll continue to observe and report back with any additional occurrences or findings.

Best regards,
Chris

IsaacQ_Intel · ‎01-26-2024

Hello chrisx

Thank you for the information provided.

I will proceed to check the issue internally and post back soon with more details.

Best regards,

Isaac Q.

Intel Customer Support Technician

IsaacQ_Intel · ‎02-05-2024

Hello chrisx

Thank you for your patience.

There may be a compatibility issue on this AMD system under Linux OS. The compatibility and limitation reply on the computer manufacturer's design. Please try the following troubleshooting steps:

Possible workarounds:

Option #1). Consider to disable PCIe ASPM on AX210

According to customer's logs:

$ egrep -i 'queue.*is stuck|rcu_preempt' fedora_journalctl_20240116

Jan 15 10:31:58 fedora kernel: iwlwifi 0000:0a:00.0: Queue 4 is stuck 12684 12704

Thus customer can try to disable ASPM using the command:

# Disables ASPM, enables only L0

sudo setpci -s 0a:00.0 0x50.B=0x40

(Note. The ASPM value will be automatically re-enabled after system reboot.)

A brief introduction about ASPM can be found here: https://wireless.wiki.kernel.org/en/users/documentation/aspm

Option #2). If the setpci method doesn't work, try to access BIOS and disable settings related to 'PCIe power save' or 'PCIe ASPM (Active State Power Management)'

Option #3). Load the branch core83 from backport-iwlwifi https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/, and switch to the branch

Best regards,

Isaac Q.

Intel Customer Support Technician

IsaacQ_Intel · ‎02-08-2024

Hello chrisx,

We hope you are doing fine.

Were you able to check the previous post?

Let us know if you still need assistance.

Best regards,

Isaac Q.

Intel Customer Support Technician

chrisx · ‎02-08-2024

Hi Isaac,

I've dedicated considerable effort to understanding whether your recommended backport contains updates beyond what is covered in my current setup, which includes kernel 6.7.1 and the `loaded firmware version 86.fb5c9aeb.0 ty-a0-gf-a0-86.ucode op_mode iwlmvm`. From what I can ascertain, my firmware appears to be on par with or perhaps ahead of the backport you suggested. However, I must concede that my grasp of the nuances within kernel and iwlwifi commit logs is rather limited, so there's a chance I could be missing finer details.

Kernel and iwlwifi changelogs are notably compact and technical, making it challenging for me to discern the practical implications of each update or modification.

Regarding Adam's configuration and kernel version, he might indeed find the suggested backport more advantageous. Unfortunately, he has mentioned that he won't have access to his machine for the next 10 days.

Given these circumstances, I propose we allow some time for Adam to return and assess how he might proceed with the backport in relation to his setup.

Thank you for your understanding and patience as we navigate these technical challenges. I'll ensure to keep you updated with our progress.

Best regards,
Chris

IsaacQ_Intel · ‎02-09-2024

Hello chrisx

Thank you for your reply.

Take all the time you need; we'll be patiently awaiting the results.

Best regards,

Isaac Q.

Intel Customer Support Technician

_adam_ · ‎02-11-2024

Hello All,

I'm sorry for the delay, I was able to spend some time with this yesterday and have some results to report back on.

I setup my test case in a way that I was hoping I would be able to confirm that I not only could reproduce the issue, but that I could be fairly certain of my results after applying one of the options laid out by Issac above.

In order to reproduce, I chose a game on Steam that was >100GB to download. I started the download and started tailing the journal log. After ~20GB of data downloaded, I saw our "queue stuck" error, and the system quickly degraded to a hang and other "soft lockup" messages started reporting to the log. Let me know if these logs would be helpful. I can provide details if you'd like.

After a forced reboot, I decided to start with Option #1 above, which is to disable ASPM leaving only L0 enabled. After confirming 0x50.B was indeed the Link Control Register (by using the algorithm in the documentation Issac sent) I ran the following command to disable ASPM:

`sudo setpci -s 0a:00.0 0x50.B=0x40`

After confirming that ASPM was indeed disabled via `lspci -s 0a:00.0 -vvv`, I restarted my download, and began tailing the journal log once again. I was able to download all 112GB of data without any "queue stuck" error! Just to be sure though, I repeated the entire download a second time with the same results, no issues. I will continue to run with ASPM disabled in case I just "got lucky" through those 224GB of data. I'll surely report back if I see any issues.

I assume with this outcome, Option #2 would produce the same results, given I would be disabling ASPM via the BIOS. Is that a safe assumption?

Is Option #3 still worth exploring? As Chris stated above, I am not familiar with the details of the iwlwifi release management process, how exactly changes from backport-iwlwifi are merged, etc. So it's tough for me to tell if the version of iwlwifi Chris is running contains all the relevant commits from the core83 branch that Issac mentions.

Please let me know if there are any other test cases you'd like me run! Hope this outcome helps!

Thanks

Adam

Alberto_R_Intel · ‎02-12-2024

Hello _adam_, Thank you very much for your reply.

In reference to your questions related to the options above, I will double-check those details and provide a response as soon as possible.

Regards,

Albert R.

Intel Customer Support Technician

Alberto_R_Intel · ‎02-13-2024

Hello _adam_ and chrisx, I just received an update on this matter.

After further review of this case, we would like to let you know that in this scenario the next thing to do will be to get in contact directly with ASUS Support for further assistance as AMD is not compatible with the Operating System, therefore there is a limitation we can't address:

https://www.asus.com/support/

Any other inquiries, do not hesitate to contact us again.

Regards,

Albert R.

Intel Customer Support Technician