v0.1.0 to v0.2.0 upgrade path #87
Comments
|
Here is the migration process. This is broken down into two main steps, across two comments (this one, and the next one). This comment/step is for backing up your existing power monitor data. The next comment is for executing the upgrade, installing native InfluxDB, and running the import. Influx Backup ProcedureOn the existing power monitor:
The password prompt (assuming you're not using SSH keys) will be for the login of your Raspberry Pi, which might still be the default value of This concludes the Docker container InfluxDB backup - you should now have a file named powermon_migrate.tar.gz inside of ~/. |
|
Here are the steps to upgrade your Pi running v0.1.0 of this software to v0.2.0. Also note that these steps still apply if you're trying to update your original v0.1.0 to v0.3.0. Note that v0.3.0 includes a new configuration file and calibration mechanism, so the settings in your existing Upgrade and Data Restoration ProcedureThe commands below should be ran one at a time, unless otherwise specified.
docker stop influx
docker rm influx # WARNING: You should only run this command if your InfluxDB container has a mapped volume (step 3 from my previous comment above)
sudo systemctl stop power-monitor.service
sudo systemctl disable power-monitor.service
sudo apt update && sudo apt upgrade -y
sudo reboot
wget -q https://repos.influxdata.com/influxdata-archive_compat.key
cat influxdata-archive_compat.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg > /dev/null
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' | sudo tee /etc/apt/sources.list.d/influxdata.list
sudo rm -f /etc/apt/sources.list.d/influxdb.list
sudo apt update
sudo apt install influxdb
cd ~/
wget https://raw.githubusercontent.com/David00/rpi-power-monitor/upgrade-helpers/rpi_power_monitor/db_migrate.pyIt's unlikely you have changed the database name from rpi-power-monitor/rpi_power_monitor/config.py Lines 22 to 28 in 1f37c8f
Make sure the docker version of InfluxDB has been stopped (if you didn't already stop it from step 1 above). Then, make sure the new native InfluxDB is running with:
Then, ensure that the Python influxdb client library is installed with:
Keep an eye on it until you see the prompt asking you which CT numbering scheme you are migrating from. After that, you should be free to let it do it's thing. There are still a few things to do for the upgrade, like:
I'd like to make sure this process works well with a few people before transcribing what I've written here to the new docs that I'm working on. I will reserve the next comment space to address the remaining items. |
|
Reserved to continue from above |
|
db migrate is failing. did the backup as above, everything going smooth, estracts then bombs out `Backup found in: /home/pi/powermon_migrate.tar.gz Extracting backup to /opt/influxdb/power-monitor-influx-import. This can take up to 20 minutes (or more). Please wait. You will not see any output while it is extracting. |
|
Thanks for trying - I just updated
.. and then rerun it with |
|
Working perfect now! Data in my DB started June 4, 2022. Took 4 hours and 20 minutes!
I'm running a 500GB m.2 drive in a USB3.1 carrier. Boots right up! Running flawlessly. Thank you
Get BlueMail for Android
…On Feb 6, 2023, 08:36, at 08:36, David ***@***.***> wrote:
Thanks for trying - I just updated `db_migrate.py` to fix this. You
can try again from step 3:
> 3. Grab the backup script, which I just pushed to a new branch named
[upgrade-helpers](https://github.com/David00/rpi-power-monitor/tree/upgrade-helpers).
.. and then rerun it with `sudo python3 ~/db_migrate.py`.
--
Reply to this email directly or view it on GitHub:
#87 (comment)
You are receiving this because you are subscribed to this thread.
Message ID: ***@***.***>
|
|
Excellent. So now that you have the right data, you'll want to upgrade the power monitor software itself, and re-enable the service file. I haven't yet decided on the best way to do it, but one way would be to do the following, for the v0.2.0 (November 2022) release: Next, you'll need to manually update the new The important thing to note is that the CT channel labels changed between versions, so as you're updating the settings, everything from ct0 in the old file should be applied to ct1 in the new file. Specifically, just these four variables:
Finally, you'll want to update the service file to point to the new version of the code.
If you followed the steps verbatim above, then you can replace the
Then, run the following to apply the changes, start the service, and check its status: That's it! (Well, except for Grafana still running in a Docker container), but that's not too big of a deal. Once you get your dashboards manually backed up, it's easy to remove. Updating GrafanaIf you have customized dashboards, you'll need to back them up manually, or else they will be removed. The default dashboards to go with this project area available on the Dashboard Wiki page. To backup your dashboards, go into each dashboard, click the settings wheel at the top, and then find the JSON model button on the left. Copy the entire text contents of the JSON model out and save it to a text file for future import.
Docker Removal
Old Data RemovalThe final thing to do, which is completely optional, is to remove your old high resolution power monitor data. As long as the migration script went well, all of your old high-res data has been downsampled and stored into new measurements, meaning your old data is safe to remove from Influx. You should retain the original backup archive just in case, though! To remove your old data, we'll go into the Influx CLI and execute a few drop statements. The following will remove all of the high resolution data that is older than 30 days from the time you execute the command: This concludes the upgrade process to get your environment fairly close to the v0.2.0 image. However, I am right on the brink of releasing v0.3.0, which contains a lot of improvements over v0.2.0, so stay tuned for that one (a beta is already out, but there are still a few issues I need to fix). The upgrade path from v0.2.0 to v0.3.0 will be much easier. |
|
I can confirm that 0.2.0 works without issues under the V6.1.11 Kernel on the pi |
|
Dashboard Changes? Hi David Using the dashboard that existed, I wanted to check the old data in grafana but all I get on any time period is No Data! The influxdb "save and test" shows "datasource is working. 19 measurements found". Do the dashboards need to be changed? I did not see that in the above. |
Excellent. I'm not sure that I can implement the credential and Wi-Fi customizations into the image build process, but sure, I'd be curious to see! The All of your old data should be in the database, but you'll have to add a new panel to the dashboard that looks at the downsampled measurements. In the "show and tell" discussion for dashboards, I've shared a panel that shows the last 60 days worth of production and consumption data. You should be able to deploy the panel I've shared in this comment, but you don't need to do any of the Once you start the power monitor service, you should start to see data populate again in the default dashboard. Edit: To answer your question specifically, the dashboards do not need to be changed, but with the new continuous queries and retention policies, there is room for them to be improved to look at older data. Hopefully my explanation above about how the db_migrate script handled the high resolution data explained why your dashboard is empty. |
|
I still don't understand. My ideal is that I can query any of the old data (that hasn't been deleted by a uSD failue) for any desired day or number of hours. A reason for displaying the data in a similar way is for debugging. Is this the same power and currents that I got before? If not, is it software? A hardware fault? Did I have the same current spike when the spa pump turned on last year or is todays turn-on spike higher? Do I have a new pump problem? I remember my neighbor asking a question about solar output, so I could show him my solar for a day in June and compare it to a day in the winter. I personally prefer to use bigger SSD's and keep all historical data rather than throw it away. This is why I am so annoyed by failing uSD cards. You said: "The script does not carry over any of the high resolution data, which is why the dashboards appear to be empty." I have been meaning to ask what is meant by "high resolution data". Will I still see the same current excursions that I could see before? Can a dashboard be created to see the old data as before? |
The data has been downsampled by the https://david00.github.io/rpi-power-monitor/docs/v0.3.0-beta/database-info.html#overview
Yes, when you start the power monitor again, the dashboard that you're used to should continue to function just like before, but the one thing that changed in v0.3.0 is that the power monitor software now configures InfluxDB to automatically downsample the high resolution data into the 5 minute averages. The downsampling happens in the background and is basically a completely different copy of the high resolution data. For example... with updates coming in every second, there swould be about 300 data points every 5 minutes (per channel!) in the "high res" data. InfluxDB will now automatically average all 300 of those points together and store the resulting average in a new retention policy and measurement. The averaged data will be stored forever, so you can always query it in the future. The "high res" data will be deleted after 30 days, which means, the home power dashboard (as currently configured) can only go back 30 days. The retention policies (RPs) are kind of like different folders that hold data inside individual measurements. The "high res" data goes into the default RP named If you look at one of the continuous query (CQ) creations, you can get an idea of what it's doing:
I've highlighted the key points above. So, for each |
|
My problems are more existential. We cannot assume there is any valid data in the database. There is a lot of disk space being taken by something, but is it valid data? Tried loading in the Overview-Panel.JSON as you suggested. Seems to load but the dashboard shows no panel. When I look at the json model version of the importe dashboard, it has only 48 lines, where the original file has 198 lines. No obvious error messages. Remember this database came from the faulty uSD card that went through a series of operations. On the new V0.1 uSD, the system ran and I could, by randomly picking dates, see that some days had data and some did not. You said there was no easy way to look for gaps "Missing is tough - without writing a custom python script to compare the data, I don't think there's an easy way. Problem now is my original technique of changing dates on the "Home Power Dashboard" no longer works because of the v0.2 changes. I know Jan 23 only has a few days of data, and I think a bit of Dec 22 is gone. Jan 22 is gone, but I think other dates might have data. It was unusual that the "db_migrate.py" marched through EVERY day, since late Nov 21, with no errors. Did it convert anything on those days that the old "Home Power Dashboard" said No Data? If there any way to find out if there is valid data in some date, say in July 22? |
|
Potential excess logging in db_migrate.py? Sorry to be such a helpless pain in influxdb, all the small tricks I know so far have not shown me any results in query. Looking at the big picture, how much data is on this new SSD drive with "Raspberry-Pi-OS-Lite_rpi_power_monitor-0.2.0+release" and my 2.1G powermon_migrate.tar.gz? The whole du seems excessive at 13G, so I spent some time examining the drive. Here are the results: List of sizes of top directories of root Note /var has very large log Note usr has large lib, maybe OK? /var/log ts huge! daemon.log is excessive 1.8G, does this fold into system logs, maybe several times? Here are a few lines from the end of the daemon.log file. Maybe help with debugging too much logging in db_migrate.py? Feb 15 23:55:00 powermon influxd-systemd-start.sh[680]: ts=2023-02-16T07:55:00.305666Z lvl=info msg="Executing query" log_id=0g15OJ20000 service=query query="SELECT mean(power) AS power, mean(current) AS current INTO power_monitor.rp_5min.net_5m FROM power_monitor.autogen.net WHERE time >= '2023-02-16T07:50:00Z' AND time < '2023-02-16T07:55:00Z' GROUP BY time(5m)" |
That sounds right - the Overview-Panel.json is just a single panel that goes into a dashboard.
I was concerned with this impacting this functionality, but I think it can be reasonably solved with an update to the Grafana dashboard. You can still technically pull data by changing the date in Grafana, but only for the previous 30 days. If you look at a date beyond that, the existing dashboard won't show any data, because the data gets downsampled and stored elsewhere permanently, and the original data gets deleted as it "ages out". If you duplicate the panels and edit the queries in each one, and change the query to use the new retention policy that holds the downsampled data, your technique of changing the dates in Grafana will work on the edited panels. The only difference is the data it's pulling is "lower resolution", and by that I simply mean each data point represents a 5 minute average, and not a 1 or 2 second average. Here's how you can edit a panel to tell it to query from the down sampled data:
By making that change, you can then query as far back as your data goes AND also query over long intervals, like several years at a time, if you wanted to. It is not possible to do such a long-term query on the high-resolution data, so there's actually a ton of functionality unlocked by using the down-sampled data set. For example, you can compare monthly power output year over year, or even track the annual costs of your EV charger, etc. You can even override the time frame for a single panel so that the date range selector doesn't interfere with what it shows:
The db_migrate script simply stepped through every day, issued a query for each day, and inserted the result into the database. It did not make any checks to see if the data was legitimate or even existed. You can query manually in InfluxDB to see if there is data there. Just remember that if you're looking at the new version of the database, any data beyond 30 days from now is going to be stored in the I've put a table together here that shows the retention policies and measurement names: https://david00.github.io/rpi-power-monitor/docs/v0.3.0-beta/database-info.html#structure Here's a quick example from the influx shell: Excellent diagnosis of the large log files! It does look like the InfluxDB logging can be turned down significantly. We certainly don't need or want detailed results of all the InfluxDB actions, especially on a Pi. I'll look into that separately. |
|
I made the following changes to InfluxDB's config file at This seems to have reduced the verbosity of the logging going to I will include this as the standard config when I build the v0.3.0 OS image. |
|
On this issue, just had a thought. These huge files will go away after some number of days, right!This may only be a potential bug for someone with a smaller uSD and a large data set.
Asides:1. Added explanatory comments and file on RPi Imager Advance Options to Grafana password frequency and SD Card fail · Issue #36 · David00/rpi-power-monitor
|
|
|
| | |
|
|
|
| |
Grafana password frequency and SD Card fail · Issue #36 · David00/rpi-po...
Hi David! Grafana (v7.1.3) is bugging me much more with password requests than I remember when first installing ...
|
|
|
2. Am working on a new post: "Stop the bus! Can I get off at 0.1.5?" Should I post it to this thread?
On Thursday, February 16, 2023 at 12:14:05 PM PST, David ***@***.***> wrote:
I made the following changes to InfluxDB's config file at /etc/influxdb/influxdb.conf:
[logging]
level = "warn"
[http]
log-enabled = false
This seems to have reduced the verbosity of the logging going to /var/log/daemon.log.
I will include this as the standard config when I build the v0.3.0 OS image.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
I'm confused about the clean up of old high res data. Your directions above include: |
Thanks for pointing this out! Yes, I do believe this is a typo, and it should be The measurements (and retention policies) that will be generated in v0.3.0 are documented here, but feel free to follow up if you have further questions :) https://david00.github.io/rpi-power-monitor/docs/v0.3.0/database-info.html#structure |
This issue is to summarize the upgrade path from v0.1.0 to v0.2.0. There are several changes between the versions that makes upgrading a little tricky, which are:
Getting InfluxDB and Grafana installed natively is fairly easy, as is getting the data out of the Docker environment and into the native environment. The data migration process can take a considerable amount of time (many hours) if you have data spanning a year or more. The data migration process is wrapped up into a script I wrote that you can download and run in the instructions in the following post.
Also, the original v0.1.0 installation instructions (before I had released the very first custom OS image) did not create a mapped path for the Grafana container. This just means that all of your Grafana settings/dashboards might be stored only inside the container, and not mapped to a local folder that's accessible outside Grafana... which means if you remove the Grafana container, you'll delete your dashboards. I will include commands to check on this prior to starting the upgrade procedure, and also provide steps to take backups of the Docker versions of Grafana and Influx.
See below to get started.
The text was updated successfully, but these errors were encountered: