[Bug]: Device becomes unreachable Heltec & LILYGO

[TheDomanNow]

Category

BLE, WiFi

Hardware

Heltec V3, Other

Firmware Version

2.3.12

Description

Hi,

currently I have 2 "base" nodes in good position & with constant power, ROUTER_CLIENT mode. They are connected via WiFi to internet and have MQTT enabled. After like 2-3 days they become unresponsive via IP address (HTTP or Android app). This is not an network (WiFi) issue, also not a hardware issue (1x Heltec V3.1 & 1x LILYGO T3S3).
In this unreachable state it is possible to send a message via other node that is currently connected, so the radio is working. After power cycling the device all is back to normal, but all messages on router are lost...

I'm guessing this is related with current faulty implementation of sleep (?). Is there any workaround of this??? Is it possible to completely turn off all power saving "features"?

I see 2 solutions (probably both should be added):

1. Adding a toggle, like "Turn off all power saving things", that really turns off all power saving "features".
2. Adding auto power cycle feature, that after X time or at desired hour (internet needed) restarts the device.

Update;
after many tests it turned out that for BLE it was an Android app issue - fixed some time ago...
...as for WiFi... there is a bug that persists in many FW releases (2.4, 2.5...), it just drops WiFi after few hours and doesn't attempt to reconnect - not fixed and ignored. More information below...

BR, Doman.

[madeofstown]

Is the MQTT channel very active? About how many nodes are you seeing on your node list? Busy MQTT channels are know to overload these devices.

[HarukiToreda]

Power save mode settings are not on by default, if you did not set them, the device will not go to lite sleep as you're suspecting. The only thing that will turn off is the screen and it should not be that either. I believe mqtt is flooding the node with too many nodes. To work around this, I suggest using admin to be able to control your node from other nodes via the app.

[TheDomanNow]

Is the MQTT channel very active? About how many nodes are you seeing on your node list? Busy MQTT channels are know to overload these devices.

No it isn't. NodeDB is almost full or full, but there are like 30 nodes active at a time, sending maybe 1x msg/hour. It's not that.

Power save mode settings are not on by default, if you did not set them, the device will not go to lite sleep as you're suspecting. The only thing that will turn off is the screen and it should not be that either. I believe mqtt is flooding the node with too many nodes. To work around this, I suggest using admin to be able to control your node from other nodes via the app.

So, you're saying that settings like:
"Wait for Bluetooth duration"
"Light sleep duration"
"Minimum wake time"
are NOT active, unless "Enable power saving mode" is turned ON? Why aren't they greyed out then?

Like said above, MQTT doesn't have much traffic and nodes, it's not flooded. It's not that.

I still think it's related to some power saving "feature". Or... any other ideas?

[madeofstown]

Is the MQTT channel very active? About how many nodes are you seeing on your node list? Busy MQTT channels are know to overload these devices.

No it isn't. NodeDB is almost full or full, but there are like 30 nodes active at a time, sending maybe 1x msg/hour. It's not that.

Power save mode settings are not on by default, if you did not set them, the device will not go to lite sleep as you're suspecting. The only thing that will turn off is the screen and it should not be that either. I believe mqtt is flooding the node with too many nodes. To work around this, I suggest using admin to be able to control your node from other nodes via the app.

So, you're saying that settings like: "Wait for Bluetooth duration" "Light sleep duration" "Minimum wake time" are NOT active, unless "Enable power saving mode" is turned ON? Why aren't they greyed out then?

Like said above, MQTT doesn't have much traffic and nodes, it's not flooded. It's not that.

I still think it's related to some power saving "feature". Or... any other ideas?

Unless you are able to get some logs I think this is going to stay a mystery as most people are assuming it's MQTT related. (I have issues I'm procrastinating on getting logs for)

[bsacharski]

Having similar issue with my Heltec V3 running 2.5.15.

There is very little traffic on the LoRa side and my node is set to ignore the MQTT messages.
Is there anything I could do to help diagnose the issue?

[TheDomanNow]

Not really, this issue is mostly ignored... try FW 2.5.5, it seems somewhat stable with WiFi.

[JaraLowell]

Have same issue with firmware 2.5.10 and up, where IP connection simply drops after x hours; seems be around six for Station G2, issue not there on 2,5,9 where config more or less same. WiFI, MQTT default channel plus extra one. The extra added to push out the neighbor info but not does so... Nor a topic for here as will just do that by software seeing it seems get fade out. Sleep settings on default for all versions. as in disabled. Blue tooth disabled. Heartbeat let disabled

[TheDomanNow]

This issue persists in most of the FW releases and it's ignored all the time. The workaround would be to restart WiFi module after X time or device restart after X time... but no, it's not in plans. So yes, it's a Schrödingers feature/bug and it is/isn't.

[JaraLowell]

This issue is not apparent in 2.5.5.e182ae7, it started after this that after x hours when connected via a program to monitor the node the wifi cuts out and then completely closes. Not tested if serial then still would work but would presume it does. Nor do i per say know how to catch the issue to gather a log of what might of has happened. But for something that for emergencies should always be accessible this is obviously a no go per my view.

[TheDomanNow]

I'm now running Heltec v3 with 2.5.9.93, it was ok on this FW, right? Or should I roll back to 2.5.5?

[JaraLowell]

i my self had no issue in 2.5.9.93 but we had some nodes in neighborhood that showed an halt in relaying packages after x hours. if that is due local db issues or other is yet unknown. so we for past 8 days went to 2.5.5 to see if that keeps running without interfering and or how long so. Our setup being simple two nodes about 10km apart both with same version and both connected to mqtt so to wifi. And basically having a program connected to it near 24/7 (LoraLog from my creation)

[TheDomanNow]

Great info, thanks. I've got 1 node with 2.5.6.d55c08d running for 26 days and WiFi seems stable, it just works, it's TLORA_T3_S3 tho.

If you'll find any new 2.5 FW working stable with WiFi, please let me know here. It's quite important information for me.

Update;
2.5.6 is also bad - WiFi drops... more below...

[TheDomanNow]

A little update; I did 24h tests on few nodes with few FW. WiFi+MQTT+no_smartphone (standalone, CLIENT role), Heltec V3 + T-LoRa T3-S3.
2.5.15 - WiFi went offline after few hours on 3 nodes
2.5.9 - WiFi went offline after few hours on 3 nodes
2.5.6 - WiFi went offline after few hours on 3 nodes (I was mistaken, I thought it works well)
2.5.5 - WiFi was ok after 24h on 2 nodes, 3rd node acted weirdly so I discard results from this one

Conclusions;

• after WiFi drops the node works fine with LoRa, of course MQTT stops working and there is no way to remote manage it... it is strictly an WiFi issue
• everything after FW 2.5.5 is broken, and the issue persists in latest stable releases (last 2.5.15)
• it's hard to get some logs, because after few hours serial freezes, but from what I've seen the WiFi drops and there is no attempt to reconnect... just like that... I think it might be connected to low WiFi activity and/or some power saving features...

...so... for now I recommend sticking with FW 2.5.5.