AX210 + Linux - stability issues

[DmitryKh]

Hello, 

I got two machines with exactly the same hardware and software - Debian 12, kernel 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux. Parts of dmesg log related to BT/WiFi can be found below. 

I use these machines as a BT gateways - receiving advertisements from a sensors and reporting them via network to the remote DB. The software is written with Python bleak package on top of BlueZ 5.66.

Unfortunately, when both BT and WiFi are in use, after couple of hours BT adapter (hci0) begins to report "network is down" and becomes unusable. If WiFi is not connected - it will take longer (from one day to one week), but will always end with the same result. There is a "workaround" - unloading btusb module and loading it again - but it takes up to 30 sec and causes loss of data. 

The logs are full with similar messages in the changing order (please see below). As you can mention, sometimes the issue "cures itself", sometimes even multiple times - but at some point in time it breaks for good. 

Please advice.

[ 1319.372812] Bluetooth: hci0: command 0x2042 tx timeout
[ 1319.372812] Bluetooth: hci0: Opcode 0x2042 failed: -110
[ 1319.386983] Bluetooth: hci0: Hardware error 0x0c
[ 1319.413745] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[249543.134914] Bluetooth: hci0: Hardware error 0x0c
[249543.158194] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[253776.345142] Bluetooth: hci0: Opcode 0x2042 failed: -110
[253776.345147] Bluetooth: hci0: command 0x2042 tx timeout
[253776.347387] Bluetooth: hci0: Hardware error 0x0c
[253776.372160] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[254795.305474] Bluetooth: hci0: Malformed LE Event: 0x0d
[257246.632146] Bluetooth: hci0: Opcode 0x2042 failed: -110
[257246.632154] Bluetooth: hci0: command 0x2042 tx timeout
[257246.635542] Bluetooth: hci0: Hardware error 0x0c
[257246.660624] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[257935.553975] Bluetooth: hci0: Hardware error 0x0c
[257935.577973] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[258160.034219] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258160.034239] Bluetooth: hci0: command 0x2042 tx timeout
[258160.034262] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258160.034274] Bluetooth: hci0: failed to restart LE scan: status -110
[258160.034287] Bluetooth: hci0: Hardware error 0x0c
[258160.061931] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[258886.627649] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258886.627668] Bluetooth: hci0: command 0x2042 tx timeout
[258886.627693] Bluetooth: hci0: No way to reset. Ignoring and continuing
[258886.627695] Bluetooth: hci0: Opcode 0x2042 failed: -110
[258886.627711] Bluetooth: hci0: failed to restart LE scan: status -110
[258886.651782] Bluetooth: hci0: Hardware error 0x0c
[258886.676886] Bluetooth: hci0: Retrieving Intel exception info failed (-16)
[259707.937451] Bluetooth: hci0: command 0x2042 tx timeout
[259707.937478] Bluetooth: hci0: Opcode 0x2042 failed: -110
[259707.937483] Bluetooth: hci0: No way to reset. Ignoring and continuing

Dmesg log from the boot:

[ 3.571579] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[ 3.583165] iwlwifi 0000:03:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0-72.ucode
[ 3.583181] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[ 3.583204] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.36
[ 3.583815] iwlwifi 0000:03:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 3.583837] iwlwifi 0000:03:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 3.583843] iwlwifi 0000:03:00.0: loaded firmware version 72.daa05125.0 ty-a0-gf-a0-72.ucode op_mode iwlmvm
[ 3.720192] Bluetooth: Core ver 2.22
[ 3.720229] Bluetooth: HCI device and connection manager initialized
[ 3.720234] Bluetooth: HCI socket layer initialized
[ 3.720236] Bluetooth: L2CAP socket layer initialized
[ 3.720242] Bluetooth: SCO socket layer initialized
[ 3.721159] iwlwifi 0000:03:00.0: BIOS contains WGDS but no WRDS
[ 3.721631] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 3.896912] iwlwifi 0000:03:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0.pnvm
[ 3.896957] iwlwifi 0000:03:00.0: loaded PNVM version 64acdc51
[ 3.921121] Bluetooth: hci0: Device revision is 0
[ 3.921126] Bluetooth: hci0: Secure boot is enabled
[ 3.921127] Bluetooth: hci0: OTP lock is enabled
[ 3.921128] Bluetooth: hci0: API lock is enabled
[ 3.921129] Bluetooth: hci0: Debug lock is disabled
[ 3.921130] Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[ 3.921132] Bluetooth: hci0: Bootloader timestamp 2019.40 buildtype 1 build 38
[ 3.923009] Bluetooth: hci0: Found device firmware: intel/ibt-0041-0041.sfi
[ 3.923107] Bluetooth: hci0: Boot Address: 0x100800
[ 3.923110] Bluetooth: hci0: Firmware Version: 107-51.22
[ 4.050039] iwlwifi 0000:03:00.0: Detected RF GF, rfid=0x10d000
[ 4.119554] iwlwifi 0000:03:00.0: base HW address: 50:28:4a:ae:c1:67
[ 4.151495] iwlwifi 0000:03:00.0 wlp3s0: renamed from wlan0
[ 5.262840] Bluetooth: hci0: Waiting for firmware download to complete
[ 5.263080] Bluetooth: hci0: Firmware loaded in 1308662 usecs
[ 5.263129] Bluetooth: hci0: Waiting for device to boot
[ 5.280728] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 5.280734] Bluetooth: BNEP filters: protocol multicast
[ 5.280738] Bluetooth: BNEP socket layer initialized
[ 5.289087] Bluetooth: hci0: Device booted in 25372 usecs
[ 5.289297] Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-0041-0041.ddc
[ 5.291087] Bluetooth: hci0: Applying Intel DDC parameters completed
[ 5.294154] Bluetooth: hci0: Firmware timestamp 2022.51 buildtype 1 build 56683
[ 5.367378] Bluetooth: MGMT ver 1.22

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for posting in Intel Communities.

Kindly allow me to ask the following information that will be useful in further checking your concern:

1. It would really help us if we can get more information about your system. Please download and run our Intel® System Support Utility from this page.

You can view the instructions on how to run this application on the readme file included on the download.

The steps below are from the readme file:

How to run ssu.sh from the terminal

==========================

a. Open Applications -> Accessories -> Terminal

b. From the current director, navigate to the current folder where the ssu.sh file resides. For example if you saved the ssu.sh to a folder called "SSU File" the command would be: cd "SSU File"

c. Run the ssu.sh file using this command: ./ssu.sh <parameter=0|1>  where 0 = off and 1 = enabled

d. Use the parameter definitions included in the readme to determine which parameter to use.

e. If no parameter is specified and only ./ssu.sh is run, it will capture all system information with the exception of 3rd party logs which must be specified. If no output file is specified, the SSU will write the output to file <sytemname>.txt.

2. Are you using laptops with preinstalled AX210? If yes, kindly provide the brand and model.

Awaiting your reply.

Best Regards,

Alfred S

Intel Customer Support Technician

[AlfredoS_Intel]

Hi Dmitrykh,

I wanted to check if you had the chance to read the questions that we posted. Please let me know at your earliest convenience so that we can determine the best course of action to resolve this matter.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

Sorry for a delay - I was off for a vacation

1. Downloaded the utility, but it said that my system (Debian 12) is not supported. Tried to run it anyway and after some tweaking (it seem to fail when there are 2 Ethernet interfaces are up and running) got the attached file.

2. I am using an industrial computer Fitlet3 by Compulab, having preinstalled AX210. 

Please advise,

Dmitry

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for your response.

I have now begun the initial phases of our investigation on the issue and I would like to confirm if the BIOS of the system is updated to the latest. I checked the website for it and unfortunately, I cannot find any entries about the BIOS update.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

I cannot provide you the precise timestamp of a BIOS - but we recently had an RMA with the supplier, in course of which BIOS was flashed with an up-to-date version. 

Could it be related to Coded PHY encoding we are using to advertise, or to the amount/frequency of the data? BLE advertisement packets, hitting the AX210 card - come from the same sensor up to 10 times per second, and there are currently 10 sensors on site. 
Each packet contains ~50 bytes of a "manufacturer-specific data", as per BLE 5.2 standard...

Thanks,
Dmitry

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for sharing your insights and suspicion on the matter.

I will check if what you mentioned is somehow causing the issue. I would need to get advanced logs from you to further the investigation:

1. Please enter the commands on the terminal and paste the results.

a. hciconfig -a

b. hciconfig -i <interface name> up

c. "lsusb | grep 8087

d. modinfo btintel

e. dmesg | grep Bluetooth

f. # dmesg > dmesg.log

Awaiting your response.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

Here is the output (when the adapter experiences the mentioned issue). dmesg logs are attached. 
Please mention that at about 14:01 GMT, Sep 8 - I've applied a workaround (rmmod btusb and insmod it again) to return the machine to a working state.

$ hciconfig -a
hci0:   Type: Primary  Bus: USB
        BD Address: 50:28:4A:AF:9D:D4  ACL MTU: 1021:4  SCO MTU: 96:6
        DOWN
        RX bytes:95018258 acl:0 sco:0 events:2773475 errors:0
        TX bytes:15480379 acl:0 sco:0 commands:1640346 errors:3
        Features: 0xbf 0xfe 0x0f 0xfe 0xdb 0xff 0x7b 0x87
        Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
        Link policy: RSWITCH SNIFF
        Link mode: PERIPHERAL ACCEPT 

$ hciconfig -i hci0 up
hciconfig: invalid option -- 'i'

$ hciconfig hci0 up
Can't init device hci0: Connection timed out (110)

$ lsusb | grep 8087
Bus 001 Device 002: ID 8087:0032 Intel Corp. AX210 Bluetooth

$ modinfo btintel
filename:       /lib/modules/6.1.0-22-amd64/kernel/drivers/bluetooth/btintel.ko
firmware:       intel/ibt-12-16.ddc
firmware:       intel/ibt-12-16.sfi
firmware:       intel/ibt-11-5.ddc
firmware:       intel/ibt-11-5.sfi
license:        GPL
version:        0.1
description:    Bluetooth support for Intel devices ver 0.1
author:         Marcel Holtmann <marcel@holtmann.org>
srcversion:     D6DCF098C746D34F2A5BD6C
depends:        bluetooth
retpoline:      Y
intree:         Y
name:           btintel
vermagic:       6.1.0-22-amd64 SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Debian Secure Boot CA
sig_key:        32:A0:28:7F:84:1A:03:6F:A3:93:C1:E0:65:C4:3A:E6:B2:42:26:43
sig_hashalgo:   sha256
signature:      81:FA:0C:E6:1B:B1:D6:E5:7C:75:65:29:70:47:4B:41:36:44:EB:86:
                32:82:3C:E6:51:43:53:02:DA:8B:0F:3C:D4:51:37:E2:04:FE:E2:FD:
                18:D2:86:B1:18:30:36:26:6E:84:AB:74:12:20:23:E3:35:99:F3:92:
                96:CC:A9:53:FC:24:58:C2:32:8F:C1:45:12:2B:55:B7:48:45:75:0F:
                DC:78:90:4D:4C:AD:B4:BE:79:40:2D:77:17:7D:48:BE:20:B0:61:A9:
                D0:D9:4C:38:89:63:91:6A:C0:40:A3:98:1E:8C:28:6E:E3:C7:2D:23:
                2C:9D:AC:D4:AF:29:48:82:9A:C6:8A:57:77:0C:C9:ED:8D:0E:31:E3:
                F7:32:FC:3E:18:4B:E3:B9:6A:4E:CC:1D:96:D4:00:35:F5:73:34:05:
                38:D9:B6:8D:E0:F5:CE:0B:5D:77:2B:EC:0A:A8:31:4D:67:CF:F2:06:
                BD:48:29:6A:D9:6A:64:89:78:6A:7D:7D:B1:8B:98:DD:A4:1C:BB:ED:
                4F:AE:3D:CC:6E:64:5A:11:74:9B:79:54:CE:9B:CD:B8:1C:83:3D:76:
                97:FC:99:EC:66:E9:3E:E0:7B:55:34:51:3D:05:43:10:4F:04:FE:55:
                3B:2F:B2:93:88:19:3A:98:CF:97:90:23:D8:F0:44:81

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you providing those information.

Please allow us some time to check on this. 

We will get back to you as soon as we have updates.

Best Regards,

Alfred S

Intel Customer Support Technician

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for waiting for our update.

We reviewed the BT information that you provided and it shows DOWN and it fails to turn up due to “Can't init device hci0: Connection timed out (110)”. We would suggest for you to check if Failed to enable Bluetooth while ‘#rfkill list’ indicates it’s blocked by kernel. If yes, Try # rfkill unblock bluetooth or # rfkill unblock all.

Additionally, how many of Bluetooth sensor devices connected to these machines as BT gateways? If you test with 1 Bluetooth mouse, are there any issues?

We look forward to your feedback regarding our post.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

I will try the suggested solution - but the issue is that these systems shall work in an autonomous mode, and it is challenging even to write a "watchdog" for them - since same error messages (with lower density though) appear even when the adapter seems to work. 

Currently we have 11 sensors and 2 gateways. As I mentioned - we do not use connections, so each gateway can "hear" all sensors. 

For the same reason - I can try to connect a mouse, but not sure what connection-based 1/2M PHY BT mouse shall reveal about non-connected advertising with Coded PHY. 

In parallel - I want to try running Google's autonomous BT stack, Bumble - which supposed to be independent from a kernel. 

Thanks,
Dmitry

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for considering to find time to try our suggestions.

While waiting for the results of our suggestions, we will be taking note of the new information that you shared.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfredo, 

I wasn't able to find the Bluetooth mouse - couple of wireless ones borrowed from a colleagues  use proprietary protocols. 
I got one wireless BT keyboard though - it required pairing and connection, but worked without any issues for ~30 hours. 

Please advise. 

Thanks,
Dmitry

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for finding ways to carry out the testing by our engineers.

I just want to confirm, if you were able to test the “We reviewed the BT information that you provided and it shows DOWN and it fails to turn up due to “Can't init device hci0: Connection timed out (110)”. We would suggest for you to check if Failed to enable Bluetooth while ‘#rfkill list’ indicates it’s blocked by kernel. If yes, Try # rfkill unblock bluetooth or # rfkill unblock all” part.

I will wait for your confirmation.

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

$ sudo hciconfig -a
hci0: Type: Primary Bus: USB
BD Address: 50:28:4A:AF:9D:D4 ACL MTU: 1021:4 SCO MTU: 96:6
DOWN
RX bytes:7771590 acl:0 sco:0 events:230313 errors:0
TX bytes:1301793 acl:0 sco:0 commands:137674 errors:2
Features: 0xbf 0xfe 0x0f 0xfe 0xdb 0xff 0x7b 0x87
Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
Link policy: RSWITCH SNIFF
Link mode: PERIPHERAL ACCEPT

$
$ sudo rfkill list
1: phy0: Wireless LAN
Soft blocked: no
Hard blocked: no
7: hci0: Bluetooth
Soft blocked: no
Hard blocked: no
$
$ sudo rfkill unblock all
$
$ sudo hciconfig -a
hci0: Type: Primary Bus: USB
BD Address: 50:28:4A:AF:9D:D4 ACL MTU: 1021:4 SCO MTU: 96:6
DOWN
RX bytes:7771590 acl:0 sco:0 events:230313 errors:0
TX bytes:1301793 acl:0 sco:0 commands:137674 errors:2
Features: 0xbf 0xfe 0x0f 0xfe 0xdb 0xff 0x7b 0x87
Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
Link policy: RSWITCH SNIFF
Link mode: PERIPHERAL ACCEPT

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for considering our suggestions so far.

Just for confirmation, did the recommendation by our engineers result to any improvements?

Best Regards,

Alfred S

Intel Customer Support Technician

[DmitryKh]

Hi Alfred, 

I am not sure which recommendation you are referring to. 

If it is "We would suggest for you to check if Failed to enable Bluetooth while ‘#rfkill list’ indicates it’s blocked by kernel." - as you can see above, "rfkill list" shows that the adapter is not blocked, and unblocking it has no effect on the "DOWN" state

Please advise,
Dmitry

[AlfredoS_Intel]

Hi Dmitrykh,

Thank you for your response and providing more details regarding the issue,

We will continue checking this and the information that provided is invaluable to us.

Please allow us some time to check on your concern. We hope for your understanding regarding this

We will get back to you as soon as we have updates.

Best Regards,

Alfred S

Intel Customer Support Technician