AX210 + Linux - stability issues

[DmitryKh]

Hello, 

I got two machines with exactly the same hardware and software - Debian 12, kernel 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux. Parts of dmesg log related to BT/WiFi can be found below. 

I use these machines as a BT gateways - receiving advertisements from a sensors and reporting them via network to the remote DB. The software is written with Python bleak package on top of BlueZ 5.66.

Unfortunately, when both BT and WiFi are in use, after couple of hours BT adapter (hci0) begins to report "network is down" and becomes unusable. If WiFi is not connected - it will take longer (from one day to one week), but will always end with the same result. There is a "workaround" - unloading btusb module and loading it again - but it takes up to 30 sec and causes loss of data. 

The logs are full with similar messages in the changing order (please see below). As you can mention, sometimes the issue "cures itself", sometimes even multiple times - but at some point in time it breaks for good. 

Please advice.

[ 1319.372812] Bluetooth: hci0: command 0x2042 tx timeout
[ 1319.372812] Bluetooth: hci0: Opcode 0x2042 failed: -110
[ 1319.386983] Bluetooth: hci0: Hardware error 0x0c
[ 1319.413745] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[249543.134914] Bluetooth: hci0: Hardware error 0x0c
[249543.158194] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[253776.345142] Bluetooth: hci0: Opcode 0x2042 failed: -110
[253776.345147] Bluetooth: hci0: command 0x2042 tx timeout
[253776.347387] Bluetooth: hci0: Hardware error 0x0c
[253776.372160] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[254795.305474] Bluetooth: hci0: Malformed LE Event: 0x0d
[257246.632146] Bluetooth: hci0: Opcode 0x2042 failed: -110
[257246.632154] Bluetooth: hci0: command 0x2042 tx timeout
[257246.635542] Bluetooth: hci0: Hardware error 0x0c
[257246.660624] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[257935.553975] Bluetooth: hci0: Hardware error 0x0c
[257935.577973] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[258160.034219] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258160.034239] Bluetooth: hci0: command 0x2042 tx timeout
[258160.034262] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258160.034274] Bluetooth: hci0: failed to restart LE scan: status -110
[258160.034287] Bluetooth: hci0: Hardware error 0x0c
[258160.061931] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[258886.627649] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258886.627668] Bluetooth: hci0: command 0x2042 tx timeout
[258886.627693] Bluetooth: hci0: No way to reset. Ignoring and continuing
[258886.627695] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258886.627711] Bluetooth: hci0: failed to restart LE scan: status -110
[258886.651782] Bluetooth: hci0: Hardware error 0x0c
[258886.676886] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[259707.937451] Bluetooth: hci0: command 0x2042 tx timeout
[259707.937478] Bluetooth: hci0: Opcode 0x2042 failed: -110
[259707.937483] Bluetooth: hci0: No way to reset. Ignoring and continuing

Dmesg log from the boot:

[ 3.571579] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[ 3.583165] iwlwifi 0000:03:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0-72.ucode
[ 3.583181] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[ 3.583204] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.36
[ 3.583815] iwlwifi 0000:03:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 3.583837] iwlwifi 0000:03:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 3.583843] iwlwifi 0000:03:00.0: loaded firmware version 72.daa05125.0 ty-a0-gf-a0-72.ucode op_mode iwlmvm
[ 3.720192] Bluetooth: Core ver 2.22
[ 3.720229] Bluetooth: HCI device and connection manager initialized
[ 3.720234] Bluetooth: HCI socket layer initialized
[ 3.720236] Bluetooth: L2CAP socket layer initialized
[ 3.720242] Bluetooth: SCO socket layer initialized
[ 3.721159] iwlwifi 0000:03:00.0: BIOS contains WGDS but no WRDS
[ 3.721631] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 3.896912] iwlwifi 0000:03:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0.pnvm
[ 3.896957] iwlwifi 0000:03:00.0: loaded PNVM version 64acdc51
[ 3.921121] Bluetooth: hci0: Device revision is 0
[ 3.921126] Bluetooth: hci0: Secure boot is enabled
[ 3.921127] Bluetooth: hci0: OTP lock is enabled
[ 3.921128] Bluetooth: hci0: API lock is enabled
[ 3.921129] Bluetooth: hci0: Debug lock is disabled
[ 3.921130] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[ 3.921132] Bluetooth: hci0: Bootloader timestamp 2019.40 buildtype 1 build 38
[ 3.923009] Bluetooth: hci0: Found device firmware: intel/ibt-0041-0041.sfi
[ 3.923107] Bluetooth: hci0: Boot Address: 0x100800
[ 3.923110] Bluetooth: hci0: Firmware Version: 107-51.22
[ 4.050039] iwlwifi 0000:03:00.0: Detected RF GF, rfid=0x10d000
[ 4.119554] iwlwifi 0000:03:00.0: base HW address: 50:28:4a:ae:c1:67
[ 4.151495] iwlwifi 0000:03:00.0 wlp3s0: renamed from wlan0
[ 5.262840] Bluetooth: hci0: Waiting for firmware download to complete
[ 5.263080] Bluetooth: hci0: Firmware loaded in 1308662 usecs
[ 5.263129] Bluetooth: hci0: Waiting for device to boot
[ 5.280728] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 5.280734] Bluetooth: BNEP filters: protocol multicast
[ 5.280738] Bluetooth: BNEP socket layer initialized
[ 5.289087] Bluetooth: hci0: Device booted in 25372 usecs
[ 5.289297] Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-0041-0041.ddc
[ 5.291087] Bluetooth: hci0: Applying Intel DDC parameters completed
[ 5.294154] Bluetooth: hci0: Firmware timestamp 2022.51 buildtype 1 build 56683
[ 5.367378] Bluetooth: MGMT ver 1.22

[AlfredoS_Intel]

Hi Dmitrykh,

We are just following up if the brilliant workaround that you ingeniously found is still working.

I will wait for your update.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

Unfortunately it was a false impression, caused by reduced data amount (some sensors went off during a long holiday).

We do not have a resources to "play" with kernels and drivers, and very close to the decision to abandon Intel-based cards in favour of another vendor. 

Thanks for your effort!

[AlfredoS_Intel]

Hi DmitryKh,

Thank you for your update.

I understand your decision regarding on how you will move forward with your project or planned configuration.

Thank you for appreciating our effort and for choosing Intel.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

In the meanwhile I did some experiments with the alternative Bluetooth stack (Bumble) - and it gave me some information that could help to resolve the issue. 

Just to recap:
Debian Linux (amd64) - #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01)
BT firmware - 107-51.22
WiFi firmware - 72.daa05125.0 
WiFi adapter not connected to any network (wired Ethernet link is used)

BT adapter is set to Coded PHY and runs an infinite LE scan. 

After ~1 hour of successful receiving of the packets (each one contains 22-25 bytes of "manufacturer-specific data") this is what I can see in the logs:

HCI_VENDOR_EVENT[0x87]: 878003010100070107061c00000000741611005e000880000207007416110000000000000000000514005f0180761611001ec91300da3414001ec9130043104ac11300c0ed110038203dc141204220

HCI_HARDWARE_ERROR_EVENT: 0c

From this point on, hci0 device disappears and appears again only after removing and re-loading the btusb module. 

Could you shed some light on this vendor event meaning?

Thanks!

[DmitryKh]

And one more update - upgrading to a very up-to-date kernel (6.11.9-1) and firmware 120-18.24 did not change much:

HCI_VENDOR_EVENT[0x87]: 878003010100070107061c00000000722111001e06088000020700722111000000000000000000051409d50000322111004eec13006ee413004a7e1500431009d5000024fe11005920500141204220

HCI_HARDWARE_ERROR_EVENT: 0c