Wireless
Participate in insightful discussions regarding issues related to Intel® Wireless Adapters and technologies
8711 Discussions

AX210 + Linux - stability issues

DmitryKh
Novice
11,932 Views

Hello, 

I got two machines with exactly the same hardware and software - Debian 12, kernel 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux. Parts of dmesg log related to BT/WiFi can be found below. 

I use these machines as a BT gateways - receiving advertisements from a sensors and reporting them via network to the remote DB. The software is written with Python bleak package on top of BlueZ 5.66.

Unfortunately, when both BT and WiFi are in use, after couple of hours BT adapter (hci0) begins to report "network is down" and becomes unusable. If WiFi is not connected - it will take longer (from one day to one week), but will always end with the same result. There is a "workaround" - unloading btusb module and loading it again - but it takes up to 30 sec and causes loss of data. 

The logs are full with similar messages in the changing order (please see below). As you can mention, sometimes the issue "cures itself", sometimes even multiple times - but at some point in time it breaks for good. 

Please advice.

[ 1319.372812] Bluetooth: hci0: command 0x2042 tx timeout
[ 1319.372812] Bluetooth: hci0: Opcode 0x2042 failed: -110
[ 1319.386983] Bluetooth: hci0: Hardware error 0x0c
[ 1319.413745] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[249543.134914] Bluetooth: hci0: Hardware error 0x0c
[249543.158194] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[253776.345142] Bluetooth: hci0: Opcode 0x2042 failed: -110
[253776.345147] Bluetooth: hci0: command 0x2042 tx timeout
[253776.347387] Bluetooth: hci0: Hardware error 0x0c
[253776.372160] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[254795.305474] Bluetooth: hci0: Malformed LE Event: 0x0d
[257246.632146] Bluetooth: hci0: Opcode 0x2042 failed: -110
[257246.632154] Bluetooth: hci0: command 0x2042 tx timeout
[257246.635542] Bluetooth: hci0: Hardware error 0x0c
[257246.660624] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[257935.553975] Bluetooth: hci0: Hardware error 0x0c
[257935.577973] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[258160.034219] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258160.034239] Bluetooth: hci0: command 0x2042 tx timeout
[258160.034262] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258160.034274] Bluetooth: hci0: failed to restart LE scan: status -110
[258160.034287] Bluetooth: hci0: Hardware error 0x0c
[258160.061931] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[258886.627649] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258886.627668] Bluetooth: hci0: command 0x2042 tx timeout
[258886.627693] Bluetooth: hci0: No way to reset. Ignoring and continuing
[258886.627695] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258886.627711] Bluetooth: hci0: failed to restart LE scan: status -110
[258886.651782] Bluetooth: hci0: Hardware error 0x0c
[258886.676886] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[259707.937451] Bluetooth: hci0: command 0x2042 tx timeout
[259707.937478] Bluetooth: hci0: Opcode 0x2042 failed: -110
[259707.937483] Bluetooth: hci0: No way to reset. Ignoring and continuing

 

Dmesg log from the boot:

[ 3.571579] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[ 3.583165] iwlwifi 0000:03:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0-72.ucode
[ 3.583181] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[ 3.583204] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.36
[ 3.583815] iwlwifi 0000:03:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 3.583837] iwlwifi 0000:03:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 3.583843] iwlwifi 0000:03:00.0: loaded firmware version 72.daa05125.0 ty-a0-gf-a0-72.ucode op_mode iwlmvm
[ 3.720192] Bluetooth: Core ver 2.22
[ 3.720229] Bluetooth: HCI device and connection manager initialized
[ 3.720234] Bluetooth: HCI socket layer initialized
[ 3.720236] Bluetooth: L2CAP socket layer initialized
[ 3.720242] Bluetooth: SCO socket layer initialized
[ 3.721159] iwlwifi 0000:03:00.0: BIOS contains WGDS but no WRDS
[ 3.721631] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 3.896912] iwlwifi 0000:03:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0.pnvm
[ 3.896957] iwlwifi 0000:03:00.0: loaded PNVM version 64acdc51
[ 3.921121] Bluetooth: hci0: Device revision is 0
[ 3.921126] Bluetooth: hci0: Secure boot is enabled
[ 3.921127] Bluetooth: hci0: OTP lock is enabled
[ 3.921128] Bluetooth: hci0: API lock is enabled
[ 3.921129] Bluetooth: hci0: Debug lock is disabled
[ 3.921130] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[ 3.921132] Bluetooth: hci0: Bootloader timestamp 2019.40 buildtype 1 build 38
[ 3.923009] Bluetooth: hci0: Found device firmware: intel/ibt-0041-0041.sfi
[ 3.923107] Bluetooth: hci0: Boot Address: 0x100800
[ 3.923110] Bluetooth: hci0: Firmware Version: 107-51.22
[ 4.050039] iwlwifi 0000:03:00.0: Detected RF GF, rfid=0x10d000
[ 4.119554] iwlwifi 0000:03:00.0: base HW address: 50:28:4a:ae:c1:67
[ 4.151495] iwlwifi 0000:03:00.0 wlp3s0: renamed from wlan0
[ 5.262840] Bluetooth: hci0: Waiting for firmware download to complete
[ 5.263080] Bluetooth: hci0: Firmware loaded in 1308662 usecs
[ 5.263129] Bluetooth: hci0: Waiting for device to boot
[ 5.280728] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 5.280734] Bluetooth: BNEP filters: protocol multicast
[ 5.280738] Bluetooth: BNEP socket layer initialized
[ 5.289087] Bluetooth: hci0: Device booted in 25372 usecs
[ 5.289297] Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-0041-0041.ddc
[ 5.291087] Bluetooth: hci0: Applying Intel DDC parameters completed
[ 5.294154] Bluetooth: hci0: Firmware timestamp 2022.51 buildtype 1 build 56683
[ 5.367378] Bluetooth: MGMT ver 1.22

Labels (1)
0 Kudos
29 Replies
AlfredoS_Intel
Moderator
2,063 Views

Hi Dmitrykh,


We are just following up if the brilliant workaround that you ingeniously found is still working.



I will wait for your update.




Best Regards,

Alfred S

Intel Customer Support Technician


0 Kudos
DmitryKh
Novice
2,026 Views

Hi Alfred, 

Unfortunately it was a false impression, caused by reduced data amount (some sensors went off during a long holiday).

We do not have a resources to "play" with kernels and drivers, and very close to the decision to abandon Intel-based cards in favour of another vendor. 

Thanks for your effort!

0 Kudos
AlfredoS_Intel
Moderator
1,989 Views

Hi DmitryKh,


Thank you for your update.


I understand your decision regarding on how you will move forward with your project or planned configuration.


Thank you for appreciating our effort and for choosing Intel.



Best Regards,

Alfred S

Intel Customer Support Technician


0 Kudos
DmitryKh
Novice
1,586 Views

Hi Alfred, 

In the meanwhile I did some experiments with the alternative Bluetooth stack (Bumble) - and it gave me some information that could help to resolve the issue. 

Just to recap:
Debian Linux (amd64) - #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01)
BT firmware - 107-51.22
WiFi firmware - 72.daa05125.0 
WiFi adapter not connected to any network (wired Ethernet link is used)

BT adapter is set to Coded PHY and runs an infinite LE scan. 

After ~1 hour of successful receiving of the packets (each one contains 22-25 bytes of "manufacturer-specific data") this is what I can see in the logs:

HCI_VENDOR_EVENT[0x87]: 878003010100070107061c00000000741611005e000880000207007416110000000000000000000514005f0180761611001ec91300da3414001ec9130043104ac11300c0ed110038203dc141204220

HCI_HARDWARE_ERROR_EVENT: 0c

From this point on, hci0 device disappears and appears again only after removing and re-loading the btusb module. 

Could you shed some light on this vendor event meaning?

Thanks!

0 Kudos
DmitryKh
Novice
1,568 Views

And one more update - upgrading to a very up-to-date kernel (6.11.9-1) and firmware 120-18.24 did not change much:

HCI_VENDOR_EVENT[0x87]: 878003010100070107061c00000000722111001e06088000020700722111000000000000000000051409d50000322111004eec13006ee413004a7e1500431009d5000024fe11005920500141204220

HCI_HARDWARE_ERROR_EVENT: 0c

0 Kudos
mllarena
Beginner
135 Views

@DmitryKh Trying to revive this over a year later. I am experiencing the exact same issue. We are using the AX210 with a debian-based IoT gateway that has to autonomously communicate with dozens of external sensors. Our Zephyr program reports a Hardware Error (HCI event code 10) with a value 0x0c (12). The error most commonly occurs when BLE traffic is heavy.

 

@AlfredoS_Intel The error code 0x0c (12) corresponds to a failure that Intel defines and deliberately returns. Can you please find a way to contact an engineer with the relevant AX210 firmware experience so we can figure out what "12" means and act accordingly? This issue is blocking a major release at my company.

0 Kudos
DmitryKh
Novice
109 Views

Hi @mllarena ,

I regret to write it at Intel's forum, but after spending quite a few hours mixing and matching kernels, firmware versions and physical cards (as the error reproduces on AX201 and if I remember right - AX211 as well) - we end up designing similar M.2 card using the chip (module) by other vendor and communicating to it directly (bypassing Linux BT infrastructure and drivers).

Best regards, 
Dmitry

0 Kudos
mllarena
Beginner
88 Views

Thank you, @DmitryKh. I've since learned a bit more about the issue by observing the btmon output when the error occurred. In one of your earlier posts, you mentioned this:


After ~1 hour of successful receiving of the packets (each one contains 22-25 bytes of "manufacturer-specific data") this is what I can see in the logs:

HCI_VENDOR_EVENT[0x87]: 878003010100070107061c00000000741611005e000880000207007416110000000000000000000514005f0180761611001ec91300da3414001ec9130043104ac11300c0ed110038203dc141204220


I'm getting the exact same event, but btmon decoded it better. As you can see in the picture I've pasted below, it looks like the "0x87" isn't actually the vendor event, but the first byte in the two-byte vendor prefix for intel (0x8780). The next byte, 0x03, apparently represents "Intel Extended Telemetry", followed by an Extended Event Type (0x01), an unknown 0x01 (maybe a length field?), then a 0x00 which means "System Exception", some sub-event 0x07, and so on. The chain of events went as so: Read request on characteristic, 2 seconds later the Intel chip reported a "Soft software reset" with the Reset reason being a "System Exception", followed by this "Extended telemetry" which only Intel can likely decode, followed by the Hardware Error 0x0c.

mllarena_0-1768335326425.png

Unlike in your case, where you're only listening for advertisements, our sensors require actual connections. But our sensors switch between LE 1M and LE CODED PHY, perhaps that's a common factor?

After reading an attached output in one of your earlier posts, I noticed you were using a Compulab gateway. Coincidentally, we use the Compulab IOT-GATE-IMX8PLUS, which has the AX210. If you don't mind me asking, did you request the new M.2 card in the same Compulab device? If so, how difficult/expensive was it to make this change? (Feel free to email me at mbllarena@simbex.com - it seems our projects are similar)

@AlfredoS_Intel I'm a bit disappointed, though not surprised, with how apathetic and unconcerned Intel's support seems to be. It took 8 days to review a trivial hciconfig and dmesg output and the only outcome was a suggestion to check rfkill and try a Bluetooth mouse? The original poster's first message stated that "after a couple of hours BT adapter (hci0) begins to report "network is down" and becomes unusable" - if rfkill was in effect, the adapter never would've worked for hours in the first place, and the reported hardware error wouldn't be received from the AX210. The suggestion to try a Bluetooth mouse is also irrelevant - OP stated that the error occurs while scanning: no connections are active. All other suggestions: "update firmware, update OS, update BIOS" - it's all just throwing things at the wall and seeing what sticks. No direction towards an actionable solution, such as a documented bug or patch that explicitly involves this issue. But perhaps the most disappointing part of this thread is when OP stated that they were "very close to the decision to abandon Intel-based cards in favour of another vendor", which was immediately met with the uncaring concession:

"I understand your decision regarding on how you will move forward with your project or planned configuration. Thank you for appreciating our effort and for choosing Intel."

 

The irony is lost here. They are quite literally not choosing Intel... Because their issue cannot be resolved.... Most vendors would do everything in their power to prevent losing customers, apparently Intel just does not care and would even agree with a customer's decision to use another vendor because it's not worth their time as a multi-billion dollar company.

 

I'd like to believe that Intel can prove me wrong, starting by responding to the same, unanswered question asked by Dmitry in November 2024. What does this vendor-specific error mean and how can we prevent it? 

mllarena_1-1768341203116.png

0 Kudos
DmitryKh
Novice
42 Views

Hi @mllarena ,

Your analysis adds some details to the problem, and I think it can be now generalised as "receiving BLE advertisements containing "manufacturer-specific data" (as per BT spec) - sooner or later will cause an exception in the AX201/210/211 adapter firmware". 

We use Compulab Fitlet3 devices + Debian 12, and I'd like to state as clearly as possible that Compulab has absolutely nothing with this issue, as it reproduces on my ASUS TUF laptop with Ubuntu 24, and computers by other vendors. 

I'll take the replacement question to the email.

BR, 
Dmitry

0 Kudos
Reply