unable to write to influxdb due to timeouts

[cgleggett]

I haven't paid attention to my power monitor db for some time, but I tried to look at it and there was no current data. Looking at the log files, I see a lot of timeouts:

{"log":"[httpd] 172.17.0.1 - root [15/May/2023:21:57:01 +0000] \"POST /write?db=power_monitor\u0026precision=ms HTTP/1.1\" 500 20 \"-\" \"python-requests/2.24.0\" 6beee8c2-f36b-11ed-8009-0242ac110002 10001047\n","stream":"stderr","time":"2023-05-15T21:57:11.623995135Z"}
{"log":"ts=2023-05-15T21:57:11.623480Z lvl=error msg=\"[500] - \\\"timeout\\\"\" log_id=0hpM20dW000 service=httpd\n","stream":"stderr","time":"2023-05-15T21:57:11.6248218Z"}
{"log":"ts=2023-05-15T21:57:13.198338Z lvl=info msg=\"failed to store statistics\" log_id=0hpM20dW000 service=monitor error=timeout\n","stream":"stderr","time":"2023-05-15T21:57:13.198717593Z"}
{"log":"ts=2023-05-15T21:57:23.671906Z lvl=info msg=\"failed to store statistics\" log_id=0hpM20dW000 service=monitor error=timeout\n","stream":"stderr","time":"2023-05-15T21:57:23.672456758Z"}

Running the power-monitor.py terminal I see this message:

Failed to write data to Influx. Reason: b'{"error":"timeout"}\n'

The docker logfile was very big (4 GB), so I stopped the container, deleted the log, and started up again, but no joy. Even went so far as to delete the container entirely and start a new one.

Any suggestions?

[cgleggett]

Seems the huge log file kept resurrecting itself, despite several "docker rm influx" and reboots. Was finally able to remove all vestiges of the container, re-created it, and now it's working properly.

Starting the docker container with --restart always makes things very challenging, as when the pi is heavily loaded due to the constant errors from influxdb (the load average was around 10), updating this with --restart=no is often unsuccessful, making the container very hard to stop/kill.

[David00]

Hi @cgleggett, thanks for posting the resolution and closing out the issue.

If I remember correctly, the issue with the logs growing was from the initial release of this project. Since then, it's been resolved, along with a number of improvements through v0.2.0, and now, v0.3.0.

The upgrade process is a little lengthy going from 0.1.0 to 0.2.0, but I would recommend checking it out, and at least doing the backup step so that you have a backup of all of your power monitor data. This project seems to kill non-endurance rated microSD cards.

The upgrade process for v0.1.0 -> v0.2.0 is here:
#87

[cgleggett]

Looks like my SD card is toast - it keeps resurrecting the giant log file.

I will do a fresh install of v0.3.0 from a new image. is the format/data of the config file of the calibrations the same?

[David00]

Looks like my SD card is toast - it keeps resurrecting the giant log file.

Yup, in all of the failures I've seen first hand, the card goes into a "read only" mode, so even though it had probably looked like you deleted the log file, it never actually got removed.

The good news is that you can still likely read your old power monitor data off of it. Getting the data off the bad card is a little tricky though. I have some steps posted here in this comment:
#100 (comment)

Regarding v0.3.0, the configuration is different, so you can't apply the same values from your current config.py file. The good news though is that the calibration process is much simpler, and you can calibrate it in place with only a hand-held AC current meter, as long as there's sufficient load for each sensor to monitor.

New docs for v0.3.0 are here:
https://david00.github.io/rpi-power-monitor/docs/general/index.html