All notable changes to this project will be documented in this file. For each version important additions, changes and removals are listed here.
The format is inspired from Keep a Changelog and the versioning aims to respect Semantic Versioning.
- Add an Issue Template for Software Release #589
- Moved old code artefacts from
scriptsfolder to paper specific repository #561
- Replace values in NetzbetreiberpruefungStatus with their entries from Katalogwerte #583
- Add
deleted_market_actorsto data model and prevent crash on unknown tables #575 - Extended documentation of data cleansing process for bulk download #568
- Add OFFIS eV as partner organization #493
- Fix usercff workflow #545
- Fix docs on user-defined output path for csv, xml, database #549
- Set pandas version to >=2.2.2 for compatibility with numpy v2.0 #553
- Allow to configure model/service port in
soap_api.download.MaStRAPI#556 - Allow CSV export of table
storage_units#565
- Extend documentation section
getting startedbased on the JOSS Review #523 - Extend and repair documentation based on the JOSS Review #527
- Add issue template and workflow to populate USERS.cff #528
- Add imposter syndrom disclaimer to
CONTRIBUTING.md#537
- Change License identifier for pypi #525
- Change header to identify as open-mastr during http request #526
- Fixed missing call to gen_url in case first bulk download fails as xml file for today is not yet available #534
- Repair links in the documentation page #536
- Add new table
changed_dso_assignment#510 - Add deprecation warning for
MaStRMirrorandMaStRDownload#492
- Change from
setup.pytopyproject.tomlbased installation #513
- Remove outdated and deprecated dependencies #506
- Fix and add URLs of example projects in readme #481
- No longer require web scraping for bulk download #488
- Replace deprecated pandas map function #491
- Fix the handling of corrupted xml syntax in the downloaded files #494
- Implement relevant API WSDL Patchnotes V24.1.128 #499
- Remove unused Docker File #501
- Change data type of NetzbetreiberpruefungStatus to string #483
- User-defined output path for csv, xml, database #402
- Add date=existing parameter to Mastr.download #452
- Update documentation and release it on mkdocs #460
- Replace values in ArtDerFlaecheIds with their entries from katalogwerte #464
- Add a Mastr.translate method for english translation ##471
- Using sphinx version <7 to build documentation #454
- Get missing column names in xml download in a robust way #470
- Adapt column names to the API web service update #472
- Delete
on pushfor github workflow #445
- Changed the name of the bulk tables for technology=gsgk #456
- Add new table and new columns to the data model #440
- Add a
workflow_dispatchto run CI pipelines from a button click #389 - Update version number with bump2version #407
- Add usage examples to Readme #419
- Check if the zipped file is corrupted #422
- Add Issue and PR Templates #428
- Repair the csv export #384
- Refactor csv export for better maintainability #401
- Replace numeric value in hydro_extended #392
- Fix DB query errors when using SQLAlchemy v2.0 #405
- New columns introduced #409
- Replace numeric value in combustion #413
- Filenames.yml is created everytime Mastr class is called #415
- Omit connectionless DB query (SQLAlchemy v2.0 compatibility) #421
- Option to filter & export by parameter StatistikFlag #401
- Removed unused package pynodo from dependencies #424
- Removed feature for dockered databases #426
- Add workflows to release on PyPI and Test-PyPI #375
- Change error handling for missing tables in postgres #370
- Handle case for empty list #362
- Fix bug occuring for permit and location download from the bulk download source #358
- Update documentation for readthedocs #356
- Update release procedure #355
- Update README.md to README.rst #354
- Technology parameter is renamed to data for better comprehension #337
- Update ORM and documentation according to wsdl patch V1.2.87 #352
- Date parameters are united into one parameter #353
- Dataversion folder entails export date#347
- Removed code artefacts #335
- Fix dependencies in setup.py #341
- Repair technology paramter in API download #327
- Pre-commit hook for Black formatting #312
- Use exclusively sqlalchemy for dialect-free implementation #289
- API Download: Linked units in table permit are written in comma seperated string #302
- API Download: Repaired Location download #303
- CSV export: Tables with mastr_data about power units are exported in chunks #306
- API Download: User credentials are now tested before the download starts #309
- Delete the MastrDownloadFactory class #292
- Move function cleaned_data to data_io #284
- Add warning message if the bulk download speed falls below a certain limit #256
- Add engine parameter for master class initialisation #270
- Refactor code and restructure modules #273
- Fix combustion mapping #253
- Update bulk parsing order #257
- Apply csv method to all tables #275
- Update readme.md to improve PyPi release #249
- Rename branches
dev->developandmaster->production
- Add files and metadata for PyPi release #237
The code becomes a python package and will be available on pypi.org Additionally, a new datasource was identified and can be used: the xml bulk download. The API was updated to the newest version and the data model was adapted.
- The class :class:
open_mastr.mastr.Matrwas introduced as the entrypoint for users, the API download was included in this entrypoint #203 - A method for downloading and parsing the xml dump from the MaStR website was implemented #202
- New data classes and attributes were introduced to orm.py #209
- The documentation page was updated
- Unit tests were created #207
- A CI pipeline was introduced #208
- The metadata was updated #219
A complete refactoring took place! Downloading data was entirely changed; introducing layers of code and removing duplicated code while more of less following DRY. Moreover, post-processing was changed to be more accessible and easier to execute. For example, docker now helps to spin up a database container. The documention on RTD was extended, update and improved to be more helpful for new users. Read more about the details:
- added more technologies
- added documentation for ReadTheDocs
- improved parallel download
- merged all stale branches
- The class :class:
open_mastr.soap_api.mirror.MaStRMirrorwas introduced for mirroring MaStR data with latest updates #149 - Introduce project home
~/.open-MaStR/config/#120 - Documentation of post-processing #117
- Updated documentation of downloading data #124 which is harmonized with the other parts of docs and with GitHubs README #135
- Local execution of post-processing now possible, optionally in dockered database #116
- Post-processing adapted to CSV data from :class:
open_mastr.soap_api.mirror.MaStRMirror#172 - Tests for changed download code are added #131
- Metadata added for raw data as frictionless data package #160
- Suffix columns instead of deferring in database CSV export #157
- Code examples added for :class:
open_mastr.soap_api.mirror.MaStRMirrorexplaining basic use of mirroring database #164 - CSV file reader for MaStR raw data added #181
- Zenodo data upload #173 and the missing LICENSE file for Zenodo is fixed in #186
- Add postgres database service in CI job for interacting with database in tests #159
- Tests for :class:
open_mastr.soap_api.mirror.MaStRMirror#191 - Download functionality for Lokationen (with focus on :class:
open_mastr.soap_api.mirror.MaStRMirror) #162 - The CHANGELOG is now included in the documentation
- Download of raw data has entirely been refactored. A python wrapper for querying the MaStR API was introduced #83
- Based on that, for bulk data download, MaStRDownload provides handy query functions for power unit data #86. See also #128
- configuration through config filen in
~/.open-MaStR/config/with less hard-coded parameters in source files #120, #112 - move code into one package named
open_mastr#123 - Switch to GitHub Actions for CI instead of Travis #143
- Fixed unexpected line breaks during CSV export that corrupted data #170
- Filtering of duplicates in MaStR data (see
MaStR help <https://www.marktstammdatenregister.de/MaStRHilfe/subpages/statistik.html>_) got changed to filter units by leading three characters and select only directly entered data #180 - Generalize CSV reading function #188
- Most of prior code for downloading data
- docstrings for functions
- tests
- setup.py file
- added update function (based on latest timestamp in powerunits csv)
- added wind functions
- only download power units for wind to avoid massive download
- changed : process units wind ("one-click solution")
- added loop to retry failed power unit downloads, currently one retry
- write failed downloads to file
- rename
import-apisoap_api - update README with instruction for tests
- update README with instruction for setup
- unused imports
- obsolete comments
- power unit update
- filter technologies from power units
- README.md
- CHANGELOG.md
- CONTRIBUTING.md
- LICENSE
- continuous integration with TravisCI (
.travis.yml) - linting tests and their config files (
.pylintrcand.flake8) - requirements.txt
- parallelized download for power units and solar
- utils.py for utility functions
- added storage units download
- added wind permit download
- ontology folder (#46)
- took the code from this repository's subfolder