Stability issues about esp32-ble2mqtt HOT 34 CLOSED

The good (?) news is that I'm able to reproduce this issue with my TI Sensor Tag.
I'm using ESP-IDF 139d49894c473bd34ab6e9515e85d3f6871a30c7.

I'm able to reproduce it even after disabling MQTT subscriptions but then I also get BT errors: BT: bta_gattc_enqueue(), the gattc command queue is full.
It seems that the BLE stack has a hard-coded limit of 30 commands.

I think I need to implement a queue for BLE operations to avoid too many concurrent requests. That should also free CPU0 to do other things such has handle the MQTT subscriptions.

from esp32-ble2mqtt.

Apparently, reason=0x0008 from bta_gattc_conn_cback() corresponds to a connection timeout.

from esp32-ble2mqtt.

I see two issues here, the first is that the MQTT publish fails and the other is that the BLE coneection drops and fails to connect afterwards.

First of all, just so we’re on the same page, could you please let me know which commit of esp-idf you’re on?
Are you on the master branch of this project?

I’m not familiar with the Thingy:52. A few dozen (25? 50?) each second sound like a lot! How many bytes are each notification? ~4?
Are they all read and published correctly when performing service discovery after connecting?
Can you configure the Thingy:52 to send notifications less frequently, at least to see if that’s the issue?

I’ve recently had issues with WiFi / BT coexistence. Maybe the burst in traffic from all the notifications is causing the issues. That would explain the BLE disconnecting along with everything else.
ESP have recently implemented this in software which gave me very bad results so I opted to stay with the HW implementation (and is configured as such in the default configuration in this project). You could try to enable the SW implementation to see if it’s a better fit for you.

I will try to dig into the errors you send and see if anything pops up.

from esp32-ble2mqtt.

Thank you for your help.

I am using espressif/esp-idf@6c44fc7 (master).

I am not sure exactly how many notifications are being sent from the Thingy:52 since there a lot of characteristics with different frequencies. But it sounds like ~25 notifications per second, with some notifications being as big as 24 bytes (for 9-axis motion sensor). I know I can reduce the frequency of the notifications but I still need to figure out how to do that for every characteristic. I don't think there an easy option in the current project for blacklisting notifications. Unfortunately for me, that means I cannot disable some characteristics to reduce the amount of data.

How do you enable BLE software implementation?

from esp32-ble2mqtt.

You can play around with WiFi/BT coexistence in make menuconfig
"Component config" -> "Wi-Fi" -> "Software controls WiFi/Bluetooth coexistence"

from esp32-ble2mqtt.

As for the esp_mqtt: lwmqtt_publish: -1 error, it's indeed due to a small buffer.
Could you please try to increase the 256 here to something else and let me know if it helps?

from esp32-ble2mqtt.

It is stable when decreasing the notification rate. 🎉

I will try both software WiFi/BT coexistance and increasing the buffer and report back.

Would you accept a pull request with characteristics whitelisting/blacklisting? This might take some time as I am not familiar with ESP-IDF.

from esp32-ble2mqtt.

Would you accept a pull request with characteristics whitelisting/blacklisting? This might take some time as I am not familiar with ESP-IDF.

Sure thing, logic should be similar to MAC address white/black-listing, so you can use that as a starting point.

from esp32-ble2mqtt.

I get further problems when whitelisting BLE devices with default configuration:

I (9219) BLE2MQTT: Connected to device: e3:af:9b:f4:a1:c7, scanning
ASSERT_PARAM(-218959118 0), in arch_main.c at line 306
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)
Core 0 register dump:
PC      : 0x40084421  PS      : 0x00060734  A0      : 0x8013cc3c  A1      : 0x3ffc0500
A2      : 0x00000001  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x60008054
A6      : 0x3ffc102c  A7      : 0x60008050  A8      : 0x80084421  A9      : 0x3ffc04e0
A10     : 0x00000004  A11     : 0x00000000  A12     : 0x6000804c  A13     : 0xffffffff
A14     : 0x00000000  A15     : 0xfffffffc  SAR     : 0x00000004  EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000  LBEG    : 0x40084359  LEND    : 0x40084360  LCOUNT  : 0x00000000
Core 0 was running in ISR context:
EPC1    : 0x40009203  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x40084421

Backtrace: 0x40084421:0x3ffc0500 0x4013cc39:0x3ffc0520 0x40019fb5:0x3ffc0540 0x400466c3:0x3ffc0570 0x400484ae:0x3ffc0590 0x40086232:0x3ffc05b0 0x40086fda:0x3ffc05d0 0x40087af7:0x3ffc05f0 0x40082531:0x3ffc0610 0x4000b
fed:0x00000000

Core 1 register dump:
PC      : 0x400d3b2a  PS      : 0x00060834  A0      : 0x8008f56c  A1      : 0x3ffd9860
A2      : 0x00000008  A3      : 0x00000001  A4      : 0x00000001  A5      : 0x3ffd939c
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x3ffc698c  A9      : 0x3ffc6950
A10     : 0x00000000  A11     : 0x80000001  A12     : 0x00000000  A13     : 0x00000001
A14     : 0x00060021  A15     : 0x00000000  SAR     : 0x00000000  EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000

Backtrace: 0x400d3b2a:0x3ffd9860 0x4008f569:0x3ffd9880

Rebooting...

I tried erasing the flash without success. Is there an errata on rev 0 of the ESP32 that can cause this issue? I will try on a rev 1 chip. I will also try to run the BLE stack on the second core.

from esp32-ble2mqtt.

I can't say regarding rev. 0 as I only have rev. 1.
It doesn't seem to be related to white-listing devices as that happens before a connection attempt is even made and the log you shared is after the connection was already established.

Also, the crash seems to be from the WDT on core 0 so something caused it to get stuck somewhere. That core is in charge of WiFi and BT while the app is run on core 1, so I'm not sure it directly relates to the app code.
Is this something you can reproduce repeatedly? Was the device white-listing the only modification you made or did you also change the coexistence settings? I've had a few WDT issues when using SW coexistence (which is one of the reasons I opted to stay with HW).

from esp32-ble2mqtt.

It is mostly reproducible as it doesn't happen all the time. Yes, it happens on vanilla project with stock configuration (I deleted sdkconfig to make sure it takes sdkconfig.defaults). I fixed it by setting the cpu core which bluetooth controller run to core 1 (APP CPU). Not only it is now stable but also seems to be much faster.

I implemented the modifications to enable characteristic whitelisting/blacklisting. The configuration section looks like this:

{
  "ble": {
    "notifications": {
      "whitelist": [
        "ef680201-9b35-4933-9b10-52ffa9740042"
      ]
    }
  }
}

I still have to update the README. Please tell me if you want some changes so it can be merged.

Thank you very much for your help. This project will be very useful for students learning about the Web of Things in one of our university courses.

from esp32-ble2mqtt.

I fixed it by setting the cpu core which bluetooth controller run to core 1 (APP CPU). Not only it is now stable but also seems to be much faster.

That's very interesting! This app isn't very CPU intensive so it might be better to pin the BT task to core 1. If you get the chance to test it on a rev 1 chip, could you let me know if the original issue reproduces on it as well? If it's just a rev. 0 issue, then I don't see a reason to change the default.

I implemented the modifications to enable characteristic whitelisting/blacklisting.

Looks good to me. I'm only wondering if we should disable only notifications or if we should ignore black-listed characteristics altogether, i.e. not read/write to them as well

Thank you very much for your help. This project will be very useful for students learning about the Web of Things in one of our university courses.

Happy to hear others can make use of it!

from esp32-ble2mqtt.

I got this Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1) again, even on core 1. A small change to the config.json triggered it and I can't get rid of it anymore. This looks like a race condition or a segmentation problem. I don't have a rev 1 chip at hand but will try as soon as possible.

from esp32-ble2mqtt.

Since it was on CPU0 before and now on CPU1 (after you moved the BT task to CPU1) it's probably a bug in the BT stack.
Could you tell me what change triggered this? Perhaps even post you config.json (after removing WiFi credentials and other sensitive data). I'd like to try it myself.

Since it seems to be a part of the BT task, could you try to enable debug logs so we have better visibility as to what's going on there? It's a part of make menuconfig:
"Component config" -> "Log output" -> "Default log verbosity"

from esp32-ble2mqtt.

Here are the verbose output before the crash:

I (17195) BLE2MQTT: Connected to device: eb:10:8e:f0:e0:c3, scanning
D (17195) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17205) BLE: GATTC event 7 wasn't handled
D (17205) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17215) BLE: GATTC event 7 wasn't handled
D (17215) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17225) BLE: GATTC event 7 wasn't handled
D (17235) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17235) BLE: GATTC event 7 wasn't handled
D (17245) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17245) BLE: GATTC event 7 wasn't handled
D (17255) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17265) BLE: GATTC event 7 wasn't handled
D (17265) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17275) BLE: GATTC event 7 wasn't handled
D (17275) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17285) BLE: GATTC event 7 wasn't handled
D (17285) BLE: Received GATTC event 7 (ESP_GATTC_SEARCH_RES_EVT), gattc_if 3
D (17295) BLE: GATTC event 7 wasn't handled
D (17295) BLE: Received GATTC event 6 (ESP_GATTC_SEARCH_CMPL_EVT), gattc_if 3
D (17305) BLE2MQTT: Services discovered on device: eb:10:8e:f0:e0:c3
D (17315) BLE2MQTT: Found new characteristic!
D (17315) BLE2MQTT:   Service: 00001800-0000-1000-8000-00805f9b34fb
D (17325) BLE2MQTT:   Characteristic: 00002a00-0000-1000-8000-00805f9b34fb
D (17335) MQTT: Subscribing to eb:10:8e:f0:e0:c3/GenericAccess/DeviceName/Get
D (18255) BLE2MQTT: Found new characteristic!
D (18255) BLE2MQTT:   Service: 00001800-0000-1000-8000-00805f9b34fb
D (18255) BLE2MQTT:   Characteristic: 00002a01-0000-1000-8000-00805f9b34fb
D (18265) MQTT: Subscribing to eb:10:8e:f0:e0:c3/GenericAccess/Appearance/Get
D (18665) BLE2MQTT: Found new characteristic!
D (18665) BLE2MQTT:   Service: 00001800-0000-1000-8000-00805f9b34fb
D (18665) BLE2MQTT:   Characteristic: 00002a04-0000-1000-8000-00805f9b34fb
D (18675) MQTT: Subscribing to eb:10:8e:f0:e0:c3/GenericAccess/PeripheralPreferredConnectionParameters/Get
D (19065) BLE2MQTT: Found new characteristic!
D (19065) BLE2MQTT:   Service: 00001800-0000-1000-8000-00805f9b34fb
D (19075) BLE2MQTT:   Characteristic: 00002aa6-0000-1000-8000-00805f9b34fb
D (19075) MQTT: Subscribing to eb:10:8e:f0:e0:c3/GenericAccess/CentralAddressResolution/Get
D (19485) BLE2MQTT: Found new characteristic!
D (19485) BLE2MQTT:   Service: 00001801-0000-1000-8000-00805f9b34fb
D (19485) BLE2MQTT:   Characteristic: 00002a05-0000-1000-8000-00805f9b34fb
D (19495) BLE2MQTT: Found new characteristic!
D (19495) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (19505) BLE2MQTT:   Characteristic: ef680101-9b35-4933-9b10-52ffa9740042
D (19515) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680101-9b35-4933-9b10-52ffa9740042/Get
D (19885) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680101-9b35-4933-9b10-52ffa9740042/Set
D (20095) BLE2MQTT: Found new characteristic!
D (20095) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (20095) BLE2MQTT:   Characteristic: ef680102-9b35-4933-9b10-52ffa9740042
D (20105) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680102-9b35-4933-9b10-52ffa9740042/Get
D (20405) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680102-9b35-4933-9b10-52ffa9740042/Set
D (20705) BLE2MQTT: Found new characteristic!
D (20705) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (20705) BLE2MQTT:   Characteristic: ef680104-9b35-4933-9b10-52ffa9740042
D (20715) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680104-9b35-4933-9b10-52ffa9740042/Get
D (20815) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680104-9b35-4933-9b10-52ffa9740042/Set
D (21225) BLE2MQTT: Found new characteristic!
D (21225) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (21225) BLE2MQTT:   Characteristic: ef680105-9b35-4933-9b10-52ffa9740042
D (21235) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680105-9b35-4933-9b10-52ffa9740042/Get
D (21635) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680105-9b35-4933-9b10-52ffa9740042/Set
D (21945) BLE2MQTT: Found new characteristic!
D (21945) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (21945) BLE2MQTT:   Characteristic: ef680106-9b35-4933-9b10-52ffa9740042
D (21955) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680106-9b35-4933-9b10-52ffa9740042/Get
D (22455) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680106-9b35-4933-9b10-52ffa9740042/Set
D (23065) BLE2MQTT: Found new characteristic!
D (23065) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (23065) BLE2MQTT:   Characteristic: ef680107-9b35-4933-9b10-52ffa9740042
D (23075) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680107-9b35-4933-9b10-52ffa9740042/Get
D (23375) BLE2MQTT: Found new characteristic!
D (23375) BLE2MQTT:   Service: ef680100-9b35-4933-9b10-52ffa9740042
D (23375) BLE2MQTT:   Characteristic: ef680108-9b35-4933-9b10-52ffa9740042
D (23385) MQTT: Subscribing to eb:10:8e:f0:e0:c3/ef680100-9b35-4933-9b10-52ffa9740042/ef680108-9b35-4933-9b10-52ffa9740042/Get
ASSERT_PARAM(-218959118 0), in arch_main.c at line 306
Guru Meditation Error: Core  1 panic'ed (Interrupt wdt timeout on CPU1)

And here is my config.json file:

{
  "wifi": {
    "ssid": "DurandA",
    "password": ""
  },
  "mqtt": {
    "server": {
      "host": "163.172.165.52",
      "port": 1883
    },
    "publish": {
      "retain": false
    }
  },
  "ble": {
    "whitelist": [
      "e3:af:9b:f4:a1:c7",
      "fd:69:c6:d2:3e:1e",
      "d9:60:eb:5f:df:e3",
      "e3:dc:75:9c:04:6f",
      "eb:10:8e:f0:e0:c3"
    ],
    "services": {
      "ef680200-9b35-4933-9b10-52ffa9740042": {
        "name": "Thingy Environment Service"
      },
      "ef680300-9b35-4933-9b10-52ffa9740042": {
        "name": "Thingy User Interface Service"
      }
    },
    "characteristics": {
      "ef680201-9b35-4933-9b10-52ffa9740042": {
        "name": "Thingy Temperature Characteristic",
        "types": [
          "SFLOAT"
        ]
      },
      "ef680302-9b35-4933-9b10-52ffa9740042": {
        "name": "Thingy Button Characteristic",
        "types": [
          "boolean"
        ]
      }
    },
    "notifications": {
      "whitelist": [
        "ef680201-9b35-4933-9b10-52ffa9740042",
        "ef680202-9b35-4933-9b10-52ffa9740042",
        "ef680203-9b35-4933-9b10-52ffa9740042",
        "ef680204-9b35-4933-9b10-52ffa9740042",
        "ef680205-9b35-4933-9b10-52ffa9740042",
        "ef680302-9b35-4933-9b10-52ffa9740042"
      ]
    }
  }
}

I am not sure what I changed exactly. It crashes now even with a simple configuration. There are a lot of BLE devices around my office so I will also try again in a less crowded place.

from esp32-ble2mqtt.

I have a hunch, could you try to change SFLOAT (uppercase) in your configuration to sfloat (lowercase)?

from esp32-ble2mqtt.

I updated SFLOAT (I used shortname from here) to sfloat. The format is now correct but it keeps crashing.

I modified the source code in order to whitelist characteristics (not only notifications). It seems that the more readable/writable characteristics (excluding notifications) it subscribes to, the more likely it is going to crash. It is not deterministic as it does not always crash when connecting to the same device. If I just subscribe to notifications, it doesn't seems to crash.

from esp32-ble2mqtt.

I will test with rev 1 before inspecting further.

from esp32-ble2mqtt.

I believe that the GATTC API queues read/notification requests. Perhaps if a lot of them are created before the queue has a change to be emptied it can cause the issue you're seeing. Before, the BT task was on CPU 0 so it could remove requests from the queue while the app on CPU 1 handles the new characteristics it discovers. Since the BT task is now on CPU 1 as well, it doesn't get that chance.

I don't have a thingy:52, but I do have a TI SensorTag. I don't know exactly how many characteristics it serves but it should be more than the devices I use daily. I'll try it out and let you know if I find anything interesting.

from esp32-ble2mqtt.

I found a rev 1 board (based on ESP32D0WDQ6). The behavior is exactly the same. I also played with every Bluedroid configuration setting without much success.

from esp32-ble2mqtt.

Btw, what version commit of ESP-IDF are you using?

Thanks again!

from esp32-ble2mqtt.

After further inspection and debugging, the crash seems to be happen while executing this line. If I comment it, it stops crashing so I think you're right with the BT request queue that gets full.

from esp32-ble2mqtt.

That would be great!

In the meantime, I tried updating to 256dpi/esp-mqtt@813d683 but it hangs immediately upon first MQTT connection:

I (5364) MQTT: Connecting MQTT client
Guru Meditation Error: Core  0 panic'ed (LoadProhibited)
. Exception was unhandled.

This is however probably unrelated to this project specific implementation. I will have a closer look at what happens there.

from esp32-ble2mqtt.

I have an initial implementation for the BLE queue. It will probably be a couple of days until I actually commit it, but you're free to try this patch out and see if it also helps you with the Thingy:52.

Let me know.

from esp32-ble2mqtt.

I tried your patch but the ESP continues to crash.

For reference, here is the output log:

I (2126) WiFi: Connected
I (2896) event: sta ip: 192.168.43.12, mask: 255.255.255.0, gw: 192.168.43.1
I (2896) BLE2MQTT: Connected to WiFi, connecting to MQTT
I (2896) MQTT: Connecting MQTT client
I (2896) esp_mqtt: esp_mqtt_start: create task
I (2906) esp_mqtt: esp_mqtt_process: begin connection attempt
I (3256) esp_mqtt: esp_mqtt_process: connection attempt successful
I (3256) MQTT: MQTT client connected
I (3256) BLE2MQTT: Connected to MQTT, scanning for BLE devices
I (3426) BLE2MQTT: Discovered BLE device: 27:2b:b3:dd:3f:f7, not connecting
I (3596) BLE2MQTT: Discovered BLE device: 55:79:1c:23:de:7e, not connecting
I (3946) BLE2MQTT: Discovered BLE device: eb:10:8e:f0:e0:c5, not connecting
I (4056) BLE2MQTT: Discovered BLE device: d9:60:eb:5f:df:e5, not connecting
I (6586) BLE2MQTT: Discovered BLE device: d9:60:eb:5f:df:e3, connecting
I (7416) BLE2MQTT: Discovered BLE device: eb:10:8e:f0:e0:c3, connecting
I (8136) BLE2MQTT: Discovered BLE device: e3:dc:75:9c:04:71, not connecting
I (8536) BLE2MQTT: Discovered BLE device: e3:dc:75:9c:04:6f, connecting
I (9566) BLE2MQTT: Connected to device: d9:60:eb:5f:df:e3, scanning
I (10286) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: 00002a00-0000-1000-8000-00805f9b34fb, len: 0, val: 0x0
I (10336) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: 00002a01-0000-1000-8000-00805f9b34fb, len: 0, val: 0x0
I (10376) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: 00002a04-0000-1000-8000-00805f9b34fb, len: 0, val: 0x0
I (10416) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: 00002aa6-0000-1000-8000-00805f9b34fb, len: 0, val: 0x0
I (10446) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680101-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
I (11286) BLE: Perform: type: 0, device: d9:60:eb:5f:df:e3, char: 00002a00-0000-1000-8000-00805f9b34fb, len: 0, val: 0x0
I (11756) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680102-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
I (11846) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680104-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
I (14376) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680105-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
E (16676) esp_mqtt: lwmqtt_subscribe_one: -9
I (16676) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680106-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
I (17646) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680107-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
I (19186) BLE: Enqueue: type: 0, device: d9:60:eb:5f:df:e3, char: ef680108-9b35-4933-9b10-52ffa9740042, len: 0, val: 0x0
ASSERT_PARAM(-218959118 0), in arch_main.c at line 306
Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)

from esp32-ble2mqtt.

I now always got E (16676) esp_mqtt: lwmqtt_subscribe_one: -9 with your patch. This corresponds to LWMQTT_MISSING_OR_WRONG_PACKET. Again, the application stops crashing if I comment the MQTT subscription call.

from esp32-ble2mqtt.

I think I understand what is happening! 💡

esp_mqtt_init() is initialized with a timeout of 2000 ms.
Upon subscribing, esp_mqtt_subscribe() calls lwmqtt_subscribe() which in turn calls lwmqtt_cycle_until which actively wait for a packet until timeout. If no packet come before the configured Interrupt watchdog timeout (ms) (300 by default), the Interrupt wdt timeout on CPU0 is triggered.

You are probably using a MQTT server which is not too far away so you never encountered a RTT > 300ms. Since my MQTT server is hosted on a remote VPS, I sometimes get longer delays.

from esp32-ble2mqtt.

My MQTT broker resides in my LAN so, yes, it should respond quicker.

However, the MQTT tasks which actively waits for 2000ms is on CPU1 while WiFi processing should be on CPU0 so I’m not sure why it would cause a watchdog timeout on CPU0. I tend to believe that the subscribe times-out because something bad has happened on CPU0 that won’t let it handle network traffic.
I assumed it was due to the BLE requests which occupied CPU0 and starved the WiFi task whch is why I introduced the queue in the first place. It’s also worth mentioning that when I worked on supporting OTA updates, downloading the new image always failed until I disconnected from all BLE devices. It just doesn’t work very well together. Could you try to change the timer I added in the patch to some large number so it would start after all MQTT related traffic was completed?

from esp32-ble2mqtt.

Do you mean increasing 100 from TimerHandle_t handle = xTimerCreate("BLE_Queue", 100, pdFALSE, NULL, ble_queue_timer_cb); line?

As a (temporary?) fix, I increased the Interrupt watchdog timeout (ms) to 950 and reduced the timeout of esp_mqtt_init() to 800 and I don't experience crashes anymore (on the queue-patched version). I will check if it is stable with the timeout modification but without the patch.

from esp32-ble2mqtt.

I went ahead and committed the queue implementation with a minor change that would hopefully make sure all characteristics were processed (MQTT subscriptions) before they're read and registered for notifications with BLE.
Let me know if this still doesn't help you and I'll try mimicking you're setup. To that end, if needed:

• Which MQTT broker are you using?
• How many characteristics are exposed by the Thingy:52?

from esp32-ble2mqtt.

Thank you for the commit!

• I am using mosquitto version 1.4.14.
• There are 17 readable characteristics and 37 characteristics in total. Note that I must use characteristic whitelisting since there is too much data from the notify microphone characteristic. I am using three Thingy:52 simultaneously for testing.

from esp32-ble2mqtt.

Are you still experiencing crashes/watch-dog timeouts?

from esp32-ble2mqtt.

This is awesome, I don't have anymore timeouts even with the interrupt watchdog timeout configured to a low value.

Sorry for the delay, I needed more time to test different configurations. I think that I can close the issue now. 😄

Thanks again for helping me with this issue!

from esp32-ble2mqtt.

Glad to hear it works for you!

from esp32-ble2mqtt.