Support for InfluxDB 2.x>

[luisfocosta]

I have already a running a Influxdb instance 2.1.1 and I assume I can configure a remote instance - in config.py
However what is the possibility to support a 2.x version?

[David00]

Hi Luis, @BCNelson has already begun working on adding support for Influx 2.x in PR #47. I haven't approved it yet simply because I haven't tested it, but you're welcome to try it out.

You can pull BCNelson's fork with the changes into a separate directory and test it on your Pi from there. Just make sure to stop the original project from running with sudo systemctl stop power-monitor.service (assuming you've setup the service file). Example:

cd ~/
mkdir rpi-power-monitor-influx2
cd rpi-power-monitor-influx2
git clone https://github.com/BCNelson/rpi-power-monitor.git
cd rpi-power-monitor
git checkout feature/influxdb2

So, from the ~/rpi-power-monitor-influx2/rpi-power-monitor/ directory, you can edit config.py to add your influxDB settings. You'll also need to install the additional influxdb library with pip install -r ~/rpi-power-monitor-influx2/rpi-power-monitor/requrements.txt. Then, you should be able to start the software with python3 power-monitor.py from inside the ~/rpi-power-monitor-influx2/rpi-power-monitor/ directory.

[BCNelson]

My fork also does not setup the database. You will need to do that on your own or give it a token with the necessary permissions and uncomment some code. Although i have not tested that.

[luisfocosta]

Thanks! I will give it a shot

[acabrera2012]

Hello. Has anyone had success with version 2.x? I have a client that would like to use a more recent version of InfluxDB and they are concerned about using 1.8.

[David00]

Sounds good! Would you like to help test any early release candidates?

[acabrera2012]

Yes. I have a working system on v0.3 on a Pi 4 and would love to give the new stuff a try and even do it on a Pi 5, if possible.

[David00]

Great! Also planning to release an updated custom OS image that supports Pi 5 too!

[mholzma]

I am also up for testing on my Pi 3 going to Influx 2.x DB on another server actually. As an interim approach, and to test stuff out, I created a plugin for the power monitor that writes out data to Influx 2.x every second. My use case for this is a bit different as I am just monitoring every time the circuit turns on/off.

[acabrera2012]

Sounds good. I was actually hoping that the final solution would allow for specifying the interval for sending the InfluxDB 2.x or 3.x data to the remote server, like hourly, to reduce processing.

[acabrera2012]

Is there anything I can help with on this, David? Research? Testing? Other? ......

[David00]

Sure! Also, tagging @mholzma.

I just pushed the develop-v0.4.0 branch that has my initial changes to support InfluxDB v2. I've briefly tested it with my Influx Cloud account, but have not tested it with a locally hosted Influx v2 server, or a real power monitor that's measuring power.

(Side note: it looks like Influx v2 requires a 64-bit OS, and my custom Pi OS image is still 32-bit)

Assuming you are on my custom OS image, you can checkout the new branch with the following commands:

(The sed command just updates the local git config file so that it can track all branches from the repo - no harm in running it anyways if you're not sure whether you need it)

cd ~/rpi_power_monitor
sed -i 's|+refs/tags/v0.3.*:refs/tags/v0.3.*|+refs/heads/*:refs/remotes/origin/*|g' .git/config
git fetch
git checkout develop-v0.4.0
pip install -r requirements.txt
pip install .

Next, you'll have to update your config.toml file with a few new settings. Here are the full sections for [database] and [database.influx_v2]. Note that the host/port/username/password/database_name become irrelevant when influx_version = 2.

[database]
enabled = true
host = "localhost"
port = 8086
username = "root"
password = "password"
database_name = "power_monitor"
influx_version = 2

[database.influx_v2]
bucket = "power_monitor"
org = "<your Influx Cloud email or custom defined org>"
token = "<an API token with at least r/w permissions for all buckets and all tasks>"
url = "<your unique Influx Cloud or self-hosted InfluxDB v2 server URL>"

Influx API tokens are setup under the "Load Data" section of the cloud UI.
The URL is just the base Influx URL, like this for example (yours may be different): https://us-east-1-1.aws.cloud2.influxdata.com

The bucket will be created automatically if it doesn't exist, so once you get the settings dialed in, you should be good to launch the power monitor. I'd recommend doing it from the terminal with the verbose flag so you can see some details of the startup, like this:

cd ~/rpi_power_monitor
python3 rpi_power_monitor/power_monitor.py -v

Some things still to do in the code:

• create the Influx v2 Tasks for automatic downsampling
• Ensure a bucket exists for the downsampled data; create it if not.
• create retention rules for both buckets and make sure they exist
• write unit tests to handle config parsing, RPiPowerMonitor class instantiation, and general operation

Other related tasks:

• create new Grafana dashboards to display the data

[acabrera2012]

Ok. I'll give it a shot. Thanks

[acabrera2012]

Followed the instructions but ran into a small hurdle. There's probably something silly happening with my paths but I get this error when I try to run using:

python3 rpi_power_monitor/power_monitor.py -v

Error:

Traceback (most recent call last): File "/home/pi/rpi_power_monitor/rpi_power_monitor/power_monitor.py", line 1601, in <module> from rpi_power_monitor.plugin_handler import Plugin ModuleNotFoundError: No module named 'rpi_power_monitor.plugin_handler'

I don't want to add sys.path to power_monitor.py because I'm sure there's some other explanation for the error. Let me know if this is something simple or if I should just keep researching.

[David00]

Thanks for testing. I forgot a step in the setup instructions!

Can you do the following?

cd ~/rpi_power_monitor
pip install .

(Note the space + period at the end of pip install!)

This should let it work.

Also, I am aware of the issue when trying to use influx version 1, and the warning message from the influx API when using influx version 2 after starting the code for the 2nd time, which looks like this:

WARNING : INFLUX - there was a problem when creating bucket. Message:
WARNING : (403)
Reason: Forbidden
... truncated ...
HTTP response body: {
"code": "forbidden",
"message": "creating bucket would exceed quota"
}

This message is ok and you can still use Influx version 2. However, for Influx v1, I'll have to fix that because it is broken right now in this development version.

[acabrera2012]

Ah. Makes sense on the pip install. I'll give it a shot.....

[acabrera2012]

That worked! I don't have the CTs connected to any live wires but it's still logging idle noise values to my InfluxDB Cloud account. It has been for hours now. So far so good. I'll keep it running overnight just to confirm everything is stable. I'll be in touch....

[mholzma]

I have it up and running and right now it is storing data on a personal influx2 server. I'll see how it runs for a bit and let you know if I see/hear anything. Have to go rebuild a bunch of dashboards but it is pulling from one of the CTs. - Mike

…

On Tue, Dec 17, 2024 at 3:32 PM acabrera2012 ***@***.***> wrote: That worked! I don't have the CTs connected to any live wires but it's still logging idle noise values to my InfluxDB Cloud account. It has been for hours now. So far so good. I'll keep it running overnight just to confirm everything is stable. I'll be in touch.... — Reply to this email directly, view it on GitHub <#48 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH5VFMDTBQBYNCN5Z63CUL2GCJ5FAVCNFSM6AAAAABTN7OKRCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJZHA2DCOI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[mholzma]

I also wrote a very quick/simple downsampling task.. I'll have to make sure it is correct.

…

On Tue, Dec 17, 2024 at 4:25 PM Mike Holzman ***@***.***> wrote: I have it up and running and right now it is storing data on a personal influx2 server. I'll see how it runs for a bit and let you know if I see/hear anything. Have to go rebuild a bunch of dashboards but it is pulling from one of the CTs. - Mike On Tue, Dec 17, 2024 at 3:32 PM acabrera2012 ***@***.***> wrote: > That worked! I don't have the CTs connected to any live wires but it's > still logging idle noise values to my InfluxDB Cloud account. It has been > for hours now. So far so good. I'll keep it running overnight just to > confirm everything is stable. I'll be in touch.... > > — > Reply to this email directly, view it on GitHub > <#48 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAH5VFMDTBQBYNCN5Z63CUL2GCJ5FAVCNFSM6AAAAABTN7OKRCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJZHA2DCOI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.*** > com> >

[acabrera2012]

Ran fairly well for the past 16 hours but stopped in the middle of the night for some reason. I thought I was running it in the background but using a Terminal window instead of running it as a service. The Terminal window probably timed out.

Anyway, not bad for a first test.

What's next?

[David00]

If you were connected remotely via SSH, it's possible the SSH connection was interrupted, which would also have killed the power monitor process you started. If this was a local terminal, maybe a bug.

[acabrera2012]

Yeah. No real concern there about the Terminal. But, I wasn't able to run it as a service at the time even though it gave no error. I didn't see any data flowing. I forgot to check the logs before I shut it down, though, so that's my fault.

[acabrera2012]

Going to try with live connection readings now just to confirm.....

[acabrera2012]

Oddly only showing data for one of the CTs but I'm not convinced it's an issue with the new code....

[mholzma]

It has been running fine in my instance. I only have one CT connected but the other 5 are recording data, just zeros. I was wondering how frequently it sends data to the InfluxDB. Does it batch up some of the data so if sending to a remote server it sends a group of points rather than a point each time? I haven't read through all the code to look at that, but it could be useful to have a configuration setting for managing the frequency for when you don't need full realtime data, in case you want to batch every second, or every minute, etc. This could be a feature request for offboard and cloud servers to reduce the frequency of transactions. - Mike

…

On Wed, Dec 18, 2024 at 8:57 AM acabrera2012 ***@***.***> wrote: Oddly only showing data for one of the CTs but I'm not convinced it's an issue with the new code.... — Reply to this email directly, view it on GitHub <#48 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH5VFO27L5T5GPGG2KLDWD2GGEOTAVCNFSM6AAAAABTN7OKRCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRQGY3DINQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[acabrera2012]

It looks like it takes a number of samples and then sends the data to the DB. Not sure if that can be changed to a time interval, like every hour, but that would be awesome.

[David00]

Glad to hear yours has been running fine, @mholzma!

The database writes are batched, albeit a fairly small batch. A single batch consists of 25 measurements, and once there's 25, the batch is written to Influx. This approach balances the overhead in frequent DB writes with the real-time intent of the project. I wouldn't really want to introduce a time-based approach in the core project, but I could see this being a useful plugin instead. A plugin could do its own batching and writes to Influx.

[mholzma]

I could imagine that given the sample rate and the data there may be in memory constraints as well as risk of data loss. But could still be handy. Might be able to do it within the db also. I offloaded Grafana because my Pi3 was lagging a bit.

…

On Wed, Dec 18, 2024 at 10:52 AM acabrera2012 ***@***.***> wrote: It looks like it takes a number of samples and then sends the data to the DB. Not sure if that can be changed to a time interval, like every hour, but that would be awesome. — Reply to this email directly, view it on GitHub <#48 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH5VFJYF5AENMRXK3NSBVD2GGK4HAVCNFSM6AAAAABTN7OKRCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRQG4ZTKNY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[acabrera2012]

So, what's next? Anything I can help with @David00 or @mholzma that won't slow down your current progress?

[David00]

Sorry, just seeing this! There are some additional things I still need to find time to do, like new dashboards and additional testing. The dashboards would be a great thing to help out with if you're good with Grafana!

[acabrera2012]

Unfortunately, I have zero experience with Grafana at all. I could certainly learn it, from scratch, but I think that would just slow everything down.

Is there any other brainstorming, testing, or research I can help you with?

[acabrera2012]

Happy holidays, everyone!

Fyi. I'm going to continue to run the 0.4 code to see if there are any hourly or overall InfluxDB Cloud limitations on the Free Tier.

If there's anything else I can do or test in the mean time, let me know.

Cheers.

[David00]

Happy Holidays!

Sounds good. The only impacting limitation to the free tier is the 30 day retention. The read/write limits will not be hit with the power monitor.

[mholzma]

I’m okay with Grafana if here is something you want me to work on.. I would need to pull down the default ones I think, since I don’t use them. - Mike

…

On Fri, Dec 27, 2024 at 9:01 PM David Albrecht ***@***.***> wrote: Sorry, just seeing this! There are some additional things I still need to find time to do, like new dashboards and additional testing. The dashboards would be a great thing to help out with if you're good with Grafana! — Reply to this email directly, view it on GitHub <#48 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH5VFMWGUJUJNQIMOGZTYL2HYICLAVCNFSM6AAAAABTN7OKRCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRYGE2TGMA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[acabrera2012]

Fyi. So far, everything appears to be running smoothly with continuous data pushes from Power Monitor.

InfluxDB Cloud does have some other limits for the free tier as follows:

`
Data-in: Rate of 5 MB per 5 minutes (average of 17 kb/s)

Read: Rate of 300 MB per 5 minutes (average of 1000 kb/s)

Storage: 30 days of data retention
`

I didn't see an easy way, within InfluxDB Cloud, to tell if Power Monitor exceeds the Data-in or Read limits. I didn't see any warnings or errors either.

I don't think we're exceeding anything but it is possible that InfluxDB Cloud quietly truncates or discards excess data.

[acabrera2012]

Hello, guys. I hope all is well. Anything new and interesting to report on your efforts?