diff --git a/.readthedocs.yml b/.readthedocs.yml
index f0ec3db5..5944fdae 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -1,15 +1,15 @@
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
version: 2
+
build:
- os: ubuntu-20.04
+ os: ubuntu-22.04
tools:
- python: "3.10"
-formats:
- - htmlzip
-sphinx:
- configuration: docs/conf.py
+ python: "3.11"
+
+mkdocs:
+ configuration: mkdocs.yml
+
python:
install:
- - requirements: docs/requirements.txt
- - method: pip
- path: .
- system_packages: true
\ No newline at end of file
+ - requirements: docs/requirements.txt
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b1f8e177..1324b246 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,7 @@ and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/
### Added
- User-defined output path for csv, xml, database [#402](https://github.com/OpenEnergyPlatform/open-MaStR/pull/402)
- Add date=existing parameter to Mastr.download [#452](https://github.com/OpenEnergyPlatform/open-MaStR/pull/452)
+- Update documentation and release it on mkdocs [#460](https://github.com/OpenEnergyPlatform/open-MaStR/pull/460)
- Replace values in ArtDerFlaecheIds with their entries from katalogwerte [#464](https://github.com/OpenEnergyPlatform/open-MaStR/pull/464)
- Add a Mastr.translate method for english translation [##471](https://github.com/OpenEnergyPlatform/open-MaStR/pull/471)
diff --git a/README.rst b/README.rst
index 0b0b51b7..9d2db869 100644
--- a/README.rst
+++ b/README.rst
@@ -20,11 +20,13 @@ open-mastr
* - Tests
- |badge_ci|
* - Publication
- - |badge_pypi| |badge_zenodo|
+ - |badge_pypi|
+ * - Data Publication
+ - |badge_zenodo|
* - Development
- |badge_issue_open| |badge_issue_closes| |badge_pr_open| |badge_pr_closes|
* - Community
- - |badge_contributing| |badge_contributors| |badge_repo_counts| |PyPI download month|
+ - |badge_contributing| |badge_contributors| |badge_repo_counts| |PyPI download month| |Total PyPI downloads|
.. contents::
@@ -53,7 +55,7 @@ The MaStR data can be
Documentation
=============
-| The documentation is in `sphinx `_ reStructuredText format in the ``doc`` sub-folder of the repository.
+| The documentation is in `Material for Mkdocs `_ markdown format in the ``doc`` sub-folder of the repository.
| Find the `documentation `_ hosted on ReadTheDocs.
| The original API documentation can be found on the `Webhilfe des Marktstammdatenregisters `_.
@@ -97,7 +99,7 @@ Install the package with
.. code-block:: python
- python setup.py install
+ pip install "open_mastr[dev]"
Examples of Usage
@@ -152,7 +154,7 @@ Data
:alt: PyPI
.. |badge_zenodo| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.6807426.svg
- :target: https://doi.org/10.5281/zenodo.6807426
+ :target: https://doi.org/10.5281/zenodo.6807425
:alt: zenodo
.. |badge_issue_open| image:: https://img.shields.io/github/issues-raw/OpenEnergyPlatform/open-MaStR
@@ -177,4 +179,9 @@ Data
:alt: counter
.. |PyPI download month| image:: https://img.shields.io/pypi/dm/open-mastr?label=PyPi%20Downloads
- :target: https://pypi.org/project/open-mastr/
+ :target: https://pypistats.org/packages/open-mastr
+
+.. |Total PyPI downloads| image:: https://static.pepy.tech/badge/open-mastr
+ :target: https://pepy.tech/project/open-mastr
+
+
diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md
index 3ccb2437..79d2d743 100644
--- a/RELEASE_PROCEDURE.md
+++ b/RELEASE_PROCEDURE.md
@@ -41,14 +41,7 @@ It always has the format `YYYY-MM-DD`, e.g. `2022-05-16`.
* [Draft a new release](https://github.com/OpenEnergyPlatform/open-MaStR/releases/new)
* Enter the release version number `v0.12.1` as title
* Summarize key changes in the description
- * `## [v0.12.1] Patch - Name - Date`
- * `### Added`
- * `### Changed`
- * `### Removed`
-* Add a link to compare versions
- * `**Compare versions:** [v0.12.0 - v0.12.1](https://github.com/OpenEnergyPlatform/open-MaStR/compare/v0.12.0...v0.12.1)`
-* Add a link to the `📝CHANGELOG.md`
- * `Also see [**CHANGELOG.md**](https://github.com/OpenEnergyPlatform/open-MaStR/blob/production/CHANGELOG.md)`
+ * Use the `generate release notes` button provided by github
* Save draft
### 4. 🐙 Finish all planned Developments
diff --git a/docs/_static/custom.css b/docs/_static/custom.css
index 0a9e3c5a..d55f0e15 100644
--- a/docs/_static/custom.css
+++ b/docs/_static/custom.css
@@ -1,3 +1,7 @@
.wy-nav-content {
max-width: 80% !important;
+}
+
+:not([data-md-state="blur"]) + nav {
+ display: none;
}
\ No newline at end of file
diff --git a/docs/advanced.md b/docs/advanced.md
new file mode 100644
index 00000000..2e9f55e8
--- /dev/null
+++ b/docs/advanced.md
@@ -0,0 +1,409 @@
+For most users, the functionalites described in [Getting Started](/getting_started) are sufficient. If you want to examine how you can configure the package's behavior for your own needs, check out [Cofiguration](#configuration). Or you can explore the two main functionalities of the package, namely the [Bulk Download](#bulk-download)
+or the [SOAP API download](#soap-api-download).
+
+## Configuration
+### Database settings
+
+
+Configure your database with the `engine` parameter of [`Mastr`][open_mastr.Mastr].
+It defines the engine of the database where the MaStR is mirrored to. Default is 'sqlite'.
+
+The possible databases are:
+
+* **sqlite**: By default the database will be stored in `$HOME/.open-MaStR/data/sqlite/open-mastr.db`.
+* **own database**: The Mastr class accepts a sqlalchemy.engine.Engine object as engine which enables the user to
+ use any other desired database.
+ If you do so, you need to insert the connection parameter into the engine variable. It'll look like this:
+
+```python
+
+ from sqlalchemy import create_engine
+
+ engine_postgres = create_engine("postgresql+psycopg2://open-mastr:open-mastr@localhost:55443/open-mastr")
+ engine_sqlite = create_engine("sqlite:///path/to/sqlite/database.db")
+ db = Mastr(engine=engine_sqlite)
+```
+
+### Project directory
+
+The directory `$HOME/.open-MaStR` is automatically created. It is used to store configuration files and save data.
+Default config files are copied to this directory which can be modified - but with caution.
+The project home directory is structured as follows (files and folders below `data/` just an example).
+
+```bash
+
+ .open-MaStR/
+ ├── config
+ │ ├── credentials.cfg
+ │ ├── filenames.yml
+ │ ├── logging.yml
+ ├── data
+ │ ├── dataversion-
+ │ ├── sqlite
+ │ └── open-mastr.db
+ └── xml_download
+ └── Gesamtdatenexport_.zip
+ └── logs
+ └── open_mastr.log
+```
+
+
+* **config**
+ * `credentials.cfg`
+ Credentials used to access
+ [Marktstammdatenregister (MaStR)](https://www.marktstammdatenregister.de/MaStR) API
+ * `filenames.yml`
+ File names are defined here.
+ * `logging.yml`
+ Logging configuration. For changing the log level to increase or decrease details of log
+ messages, edit the level of the handlers.
+* **data**
+ * `dataversion-`
+ Contains exported data as csv files from method [`to_csv`][open_mastr.Mastr.to_csv]
+ * `sqlite`
+ Contains the sqlite database in `open-mastr.db`
+ * `xml_download`
+ Contains the bulk download in `Gesamtdatenexport_.zip`
+ New bulk download versions overwrite older versions.
+* **logs**
+ * `open_mastr.log`
+ The files stores the logging information from executing open-mastr.
+
+
+
+### Logs
+
+For the download via the API, logs are stored in a single file in `/$HOME//.open-MaStR/logs/open_mastr.log`.
+New logging messages are appended. It is recommended to delete the log file from time to time because of its required disk space.
+
+
+### Data
+
+If the zipped dump of the MaStR is downloaded, it is saved in the folder `$HOME/.open-MaStR/data/xml_download`.
+
+The data can then be written to any sql database supported by [sqlalchemy](https://docs.sqlalchemy.org/). The type of the sql database is determined by the parameter `engine` in the [Mastr][open_mastr.Mastr] class.
+
+For more information regarding the database see [Database settings](Database settings).
+
+
+## Bulk download
+
+On the homepage [MaStR/Datendownload](https://www.marktstammdatenregister.de/MaStR/Datendownload) a zipped folder containing the whole
+MaStR is offered. The data is delivered as xml-files. The official documentation can be found
+on the same page (in german). This data is updated on a daily base.
+
+``` mermaid
+flowchart LR
+ id1("📥 Download raw data
+ from marktstammdatenregister.de") --> id2(Write xml files to database)
+ id2 --> id3[("📗 open-mastr database")]
+ id3 --> id4("🔧 Decode and cleanse data")
+ id4 --> id3
+ id3 --> id5("Merge corresponding tables
+ and save as csv")
+ id5 --> id6>"📜 open-mastr csv files"]
+ click id1 "https://www.marktstammdatenregister.de/MaStR/Datendownload" _blank
+ click id2 "https://github.com/OpenEnergyPlatform/open-MaStR/blob/7b155a9ebdd5204de8ae6ba7a96036775a1f4aec/open_mastr/xml_download/utils_write_to_database.py#L17C6-L17C6" _blank
+ click id4 "https://github.com/OpenEnergyPlatform/open-MaStR/blob/7b155a9ebdd5204de8ae6ba7a96036775a1f4aec/open_mastr/xml_download/utils_cleansing_bulk.py#L10" _blank
+ click id5 "https://github.com/OpenEnergyPlatform/open-MaStR/blob/7b155a9ebdd5204de8ae6ba7a96036775a1f4aec/open_mastr/mastr.py#L288" _blank
+ click id6 "https://doi.org/10.5281/zenodo.6807425" _blank
+```
+
+
+In the following, the process is described that is started when calling the [`Mastr.download`][open_mastr.Mastr.download] function with the parameter `method`="bulk".
+First, the zipped files are downloaded and saved in `$HOME/.open-MaStR/data/xml_download`. The zipped folder contains many xml files,
+which represent the different tables from the MaStR. Those tables are then parsed to a sqlite database. If only some specific
+tables are of interest, they can be specified with the parameter `data`. Every table that is selected in `data` will be deleted from the local database, if existent, and then filled with data from the xml files.
+
+In the next step, a basic data cleansing is performed. Many entries in the MaStR from the bulk download are replaced by numbers.
+As an example, instead of writing the german states where the unit is registered (Saxony, Brandenburg, Bavaria, ...) the MaStR states
+corresponding digits (7, 2, 9, ...). One major step of cleansing is therefore to replace those digits with their original meaning.
+Moreover, the datatypes of different entries are set in the data cleansing process and corrupted files are repaired.
+
+If needed, the tables in the database can be obtained as csv files. Those files are created by first merging corresponding tables (e.g all tables that contain information about solar) and then dumping those tables to `.csv` files with the [`to_csv`][open_mastr.Mastr.to_csv] method.
+
+=== "Advantages"
+ * No registration for an API key is needed
+ * Download of the whole dataset is possible
+
+=== "Disadvantages"
+ * No single tables or entries can be downloaded
+ * Download takes long time
+
+
+## SOAP API download
+
+### MaStR account and credentials
+
+
+For downloading data from the
+[Marktstammdatenregister (MaStR) database](https://www.marktstammdatenregister.de/MaStR)
+via its API a [registration](https://www.marktstammdatenregister.de/MaStRHilfe/files/regHilfen/201108_Handbuch%20f%C3%BCr%20Registrierungen%20durch%20Dienstleister.pdf) is mandatory.
+
+To download data from the MaStR API using the `open-MaStR`, the credentials (MaStR user and token) need to be provided in a certain way. Three options exist:
+
+1. **Credentials file:**
+ Both, user and token, are stored in plain text in the credentials file.
+ For storing the credentials in the credentials file (plus optionally using keyring for the token) simply instantiate
+ [`MaStRDownload`][open_mastr.soap_api.download.MaStRDownload] once and you get asked for a user name and a token. The
+ information you insert will be used to create the credentials file.
+
+ It is also possible to create the credentials file by hand, using this format:
+
+ ```
+ [MaStR]
+ user = SOM123456789012
+ token = msöiöo8u2o29933n31733m§=§1n33§304n... # optional, 540 characters
+ ```
+
+ The `token` should be written in one line, without line breaks.
+
+ The credentials file needs to be stored at: `$HOME/.open-MaStR/config/credentials.cfg`
+
+2. **Credentials file + keyring:**
+ The user is stored in the credentials file, while the token is stored encrypted in the [keyring](https://pypi.org/project/keyring/).
+
+ Read in the documentation of the keyring library how to store your token in the
+ keyring.
+
+3. **Don't store:**
+ Just use the password for one query and forget it
+
+ The latter option is only available when using [`MaStRAPI`][open_mastr.soap_api.download.MaStRAPI].
+ Instantiate with
+
+ ```python
+ MaStRAPI(user='USERNAME', key='TOKEN')
+ ```
+
+ to provide user and token in a script and use these
+ credentials in subsequent queries.
+
+### MaStRAPI
+
+You can access the MaStR data via API by using the class `MaStRAPI` directly if you have the API credentials
+configured correctly. Use the code snippet below for queries.
+
+
+```python
+from open_mastr.soap_api.download import MaStRAPI
+
+if __name__ == "__main__":
+
+ mastr_api = MaStRAPI()
+ print(mastr_api.GetLokaleUhrzeit())
+```
+
+For API calls and their optional parameters refer to [API documentation](https://www.marktstammdatenregister.
+de/MaStRHilfe/subpages/webdienst.html).
+
+???+ example "Example queries and their responses"
+
+ === "mastr_api.GetLokaleUhrzeit()"
+
+ Response:
+ ```python
+ {
+ 'Ergebniscode': 'OK',
+ 'AufrufVeraltet': False,
+ 'AufrufLebenszeitEnde': None,
+ 'AufrufVersion': 1,
+ 'LokaleUhrzeit': datetime.datetime(2023, 9, 25, 8, 1, 46, 24766, tzinfo=)
+ }
+ ```
+
+ API function name: `GetLokaleUhrzeit`
+ Example query: `mastr_api.GetLokaleUhrzeit()`
+ Parameter: `None`
+
+
+
+ === "mastr_api.GetListeAlleEinheiten(limit=1)"
+
+ Response:
+ ```python
+ {
+ "Ergebniscode": "OkWeitereDatenVorhanden",
+ "AufrufVeraltet": False,
+ "AufrufLebenszeitEnde": None,
+ "AufrufVersion": 1,
+ "Einheiten": [
+ {
+ "EinheitMastrNummer": "SEE984033548619",
+ "DatumLetzeAktualisierung": datetime.datetime(
+ 2020, 2, 20, 16, 28, 35, 250812
+ ),
+ "Name": "Photovoltaikanlage ERWin4",
+ "Einheitart": "Stromerzeugungseinheit",
+ "Einheittyp": "Solareinheit",
+ "Standort": "48147 Münster",
+ "Bruttoleistung": Decimal("3.960"),
+ "Erzeugungsleistung": None,
+ "EinheitSystemstatus": "Aktiv",
+ "EinheitBetriebsstatus": "InBetrieb",
+ "Anlagenbetreiber": "ABR949444220202",
+ "EegMastrNummer": "EEG920083771065",
+ "KwkMastrNummer": None,
+ "SpeMastrNummer": None,
+ "GenMastrNummer": None,
+ "BestandsanlageMastrNummer": None,
+ "NichtVorhandenInMigriertenEinheiten": None,
+ }
+ ],
+ }
+ ```
+
+ API function name: `GetEinheitSolar`
+ Example query: `mastr_api.GetListeAlleEinheiten(limit=1)`
+
+ | Parameter | Description |
+ |------------------------|-----------------------------------------------------------------------------------------------------------|
+ | marktakteurMastrNummer | The MaStR number of the requested unit |
+ | startAb | Sets the beginning of the records to be delivered [default value: 1] |
+ | datumAb | Restrict the amount of data to be retrieved to changed data from the specified date [Default value: NULL] |
+ | limit | Limit of the maximum data records to be delivered [default/maximum value: maximum of own limit] |
+ | einheitMastrNummern[] | |
+
+ === "mastr_api.GetEinheitSolar(einheitMastrNummer="SEE984033548619")"
+
+ Response:
+ ```python
+ {
+ "Ergebniscode": "OK",
+ "AufrufVeraltet": False,
+ "AufrufLebenszeitEnde": None,
+ "AufrufVersion": 1,
+ "EinheitMastrNummer": "SEE984033548619",
+ "DatumLetzteAktualisierung": datetime.datetime(2020, 2, 20, 16, 28, 35, 250812),
+ "LokationMastrNummer": "SEL948991715391",
+ "NetzbetreiberpruefungStatus": "Geprueft",
+ "Netzbetreiberzuordnungen": [
+ {
+ "NetzbetreiberMastrNummer": "SNB980883363112",
+ "NetzbetreiberpruefungsDatum": datetime.date(2020, 2, 25),
+ "NetzbetreiberpruefungsStatus": "Geprueft",
+ }
+ ],
+ "NetzbetreiberpruefungDatum": datetime.date(2020, 2, 25),
+ "AnlagenbetreiberMastrNummer": "ABR949444220202",
+ "NetzbetreiberMastrNummer": ["SNB980883363112"],
+ "Land": "Deutschland",
+ "Bundesland": "NordrheinWestfalen",
+ "Landkreis": "Münster",
+ "Gemeinde": "Münster",
+ "Gemeindeschluessel": "05515000",
+ "Postleitzahl": "48147",
+ "Gemarkung": None,
+ "FlurFlurstuecknummern": None,
+ "Strasse": None,
+ "StrasseNichtGefunden": False,
+ "Hausnummer": {"Wert": None, "NichtVorhanden": False},
+ "HausnummerNichtGefunden": False,
+ "Adresszusatz": None,
+ "Ort": "Münster",
+ "Laengengrad": None,
+ "Breitengrad": None,
+ "UtmZonenwert": None,
+ "UtmEast": None,
+ "UtmNorth": None,
+ "GaussKruegerHoch": None,
+ "GaussKruegerRechts": None,
+ "Registrierungsdatum": datetime.date(2019, 2, 1),
+ "GeplantesInbetriebnahmedatum": None,
+ "Inbetriebnahmedatum": datetime.date(2007, 7, 20),
+ "DatumEndgueltigeStilllegung": None,
+ "DatumBeginnVoruebergehendeStilllegung": None,
+ "DatumWiederaufnahmeBetrieb": None,
+ "EinheitSystemstatus": "Aktiv",
+ "EinheitBetriebsstatus": "InBetrieb",
+ "BestandsanlageMastrNummer": None,
+ "NichtVorhandenInMigriertenEinheiten": None,
+ "AltAnlagenbetreiberMastrNummer": None,
+ "DatumDesBetreiberwechsels": None,
+ "DatumRegistrierungDesBetreiberwechsels": None,
+ "NameStromerzeugungseinheit": "Photovoltaikanlage ERWin4",
+ "Weic": {"Wert": None, "NichtVorhanden": False},
+ "WeicDisplayName": None,
+ "Kraftwerksnummer": {"Wert": None, "NichtVorhanden": False},
+ "Energietraeger": "SolareStrahlungsenergie",
+ "Bruttoleistung": Decimal("3.960"),
+ "Nettonennleistung": Decimal("3.960"),
+ "Schwarzstartfaehigkeit": None,
+ "Inselbetriebsfaehigkeit": None,
+ "Einsatzverantwortlicher": None,
+ "FernsteuerbarkeitNb": False,
+ "FernsteuerbarkeitDv": None,
+ "FernsteuerbarkeitDr": None,
+ "Einspeisungsart": "Volleinspeisung",
+ "PraequalifiziertFuerRegelenergie": None,
+ "GenMastrNummer": None,
+ "zugeordneteWirkleistungWechselrichter": Decimal("4.000"),
+ "GemeinsamerWechselrichterMitSpeicher": "KeinStromspeicherVorhanden",
+ "AnzahlModule": 22,
+ "Lage": "BaulicheAnlagen",
+ "Leistungsbegrenzung": "Nein",
+ "EinheitlicheAusrichtungUndNeigungswinkel": True,
+ "Hauptausrichtung": "Sued",
+ "HauptausrichtungNeigungswinkel": "Grad20Bis40",
+ "Nebenausrichtung": "None",
+ "NebenausrichtungNeigungswinkel": "None",
+ "InAnspruchGenommeneFlaeche": None,
+ "ArtDerFlaeche": [],
+ "InAnspruchGenommeneAckerflaeche": None,
+ "Nutzungsbereich": "Haushalt",
+ "Buergerenergie": None,
+ "EegMastrNummer": "EEG920083771065",
+ }
+ ```
+
+ API function name: `GetEinheitSolar`
+ Example query: `mastr_api.GetEinheitSolar(einheitMastrNummer="SEE984033548619")`
+
+ | Parameter | Description |
+ |--------------------------|-------------------------------------------------------------------|
+ | `apiKey` | The web service key for validation |
+ | `marktakteurMastrNummer` | The MaStR number of the market actor used by the web service user |
+ | `einheitMastrNummer` | The MaStR number of the requested unit |
+
+
+??? note "Why can't I just query all information of all units of a specific power plant type?"
+
+ As the example queries above demonstrate, the API is structured so that units of power plants types (e.g. wind
+ turbine, solar PV systems, gas power plant) have to be queried directly by their unique identifier (
+ `EinheitMastrNummer"`) and a distinct API query. To download all unit information of a specific power plant
+ you need to know the "EinheitMastrNummer".
+
+ Firstly, by querying for all units with `mastr_api.GetListeAlleEinheiten()` you'll get all units, their unique
+ identifier (`EinheitMastrNummer`) and their power plant type (`Einheitentyp`). You can then sort them by power
+ plant type and use the power plant type specific API query to retrieve information about it.
+
+ Cumbersome?
+ Luckily, `open-MaStR` has you covered and provides methods to just query for all units of a power
+ plant type.
+
+
+
+
+### MaStRDownload
+
+The class `MaStRDownload` builds upon methods provided in the class `MaStRAPI`.
+
+It provides methods to download power plant unit types and additional information
+for each unit type, such as extended unit data, permit data, chp-specific data, location data
+or eeg-specific data.
+
+The class handles the querying logic and knows which additional data for each unit type is available
+and which SOAP service has to be used to query it.
+
+
+### MaStRMirror
+
+The class `MaStRMirror` builds upon methods provided in the class `MaStRDownload`.
+
+The aim of the class has been to mirror the Marktstammdatenregister database and keep it up-to-date.
+Historically, `open-mastr` has been developed before the owner of the dataset, BNetzA, offered the `bulk` download.
+The class can still be used for use-cases where only the most recent changes to a local database are of interest.
+For downloading the entire MaStR database we recommend the bulk download functionalities by specifying `donwload(method="bulk")`.
+
+
+
diff --git a/docs/advanced.rst b/docs/advanced.rst
deleted file mode 100644
index 966afc0c..00000000
--- a/docs/advanced.rst
+++ /dev/null
@@ -1,446 +0,0 @@
-********************
-Advanced Topics
-********************
-In the following sections, a deeper description of the package and its functionalities is given.
-
-Configuration
-==================
-Project directory
--------------------
-
-The directory `$HOME/.open-MaStR` is automatically created. It is used to store configuration files and save data.
-Default config files are copied to this directory which can be modified - but with caution.
-The project home directory is structured as follows (files and folders below `data/` just an example).
-
-.. code-block:: bash
-
- .open-MaStR/
- ├── config
- │ ├── credentials.cfg
- │ ├── filenames.yml
- │ ├── logging.yml
- │ └── tables.yml
- ├── data
- │ ├── rli_v3.0.0
- │ ├── bnetza_mastr_solar_eeg_fail.csv
- │ ├── bnetza_mastr_solar_extended_fail.csv
- │ └── bnetza_mastr_solar_raw.csv
- │ ├── sqlite
- │ └── open-mastr.db
- └── xml_download
- └── Gesamtdatenexport.zip
- └── logs
- └── open_mastr.log
-
-
-Configuration files
--------------------
-
-* :code:`credentials.cfg`: Credentials used to access
- `Marktstammdatenregister (MaStR) `_ API (read more in
- :ref:`MaStR account and credentials `) and token for Zenodo.
-* :code:`filenames.yml`: File names are defined here.
-* :code:`logging.yml`: Logging configuration. For changing the log level to increase or decrease details of log
- messages, edit the `level` of the handlers.
-* :code:`tables.yml`: Names of tables where data gets imported to during :ref:`Post-processing (Outdated)`
-
-Data
-----
-
-If the zipped dump of the MaStR is downloaded, it is saved in the folder `$HOME/.open-MaStR/data/xml_download`. New versions
-of the dump overwrite older versions.
-
-The data can then be written to an sql database. The type of the sql database is determined
-by the parameter `engine` in the Mastr class (see :ref:`mastr module`).
-
-For more information regarding the database see :ref:`Database settings `.
-
-Exported data is saved under `$HOME/.open-MaStR/data/`.
-Files that are suffixed with `_raw` can contain joined data retrieved during :ref:`downloading `.
-The structure of the data is described in :ref:`Data Description`.
-
-
-Environment variables
-^^^^^^^^^^^^^^^^^^^^^
-
-There are some environment variables to customize open-MaStR:
-
-.. list-table::
- :widths: 5 5 5
- :header-rows: 1
-
- * - variable
- - description
- - example
- * - SQLITE_DATABASE_PATH
- - Path to the SQLite file. This allows to use to use multiple instances of the MaStR database. The database instances exist in parallel and are independent of each other.
- - `/home/mastr-rabbit/.open-MaStR/data/sqlite/your_custom_instance_name.db`
- * - OUTPUT_PATH
- - | Path to user-defined output directory for CSV data, XML file and database.
- | If not specified, output directory defaults to `$HOME/.open-MaStR/`
- - | Linux: `/home/mastr-rabbit/open-mastr-user-defined-output-path`
- | Windows: `C:\\Users\\open-mastr-user-defined-output-path`
-
-MaStR account and credentials
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-For downloading data from the
-`Marktstammdatenregister (MaStR) database `_
-via its API a registration is mandatory (please `read here `_).
-
-To download data using `open-MaStR`, the credentials (MaStR user and token) need to be provided in a certain way.
-Three options exist
-
-1. **Credentials file:** Both, user and token, are stored in plain text in the credentials file
-
-
-For storing the credentials in the credentials file (plus optionally using keyring for the token) simply instantiate
-:py:class:`open_mastr.soap_api.download.MaStRDownload` once and you get asked for a user name and a token. The
-information you insert will be used to create the credentials file.
-
-It is also possible to create the credentials file by hand, using this format:
-
-.. code-block::
-
- [MaStR]
- user = SOM123456789012
- token = msöiöo8u2o29933n31733m§=§1n33§304n... # optional, 540 characters
-
-The `token` should be written in one line, without line breaks.
-
-The credentials file needs to be stored at: `$HOME/.open-MaStR/config/credentials.cfg`
-
-2. **Credentials file + keyring:** The user is stored in the credentials file, while the token is stored encrypted in the `keyring `_.
-
-Read in the documentation of the `keyring library `_ how to store your token in the
-keyring.
-
-3. **Don't store:** Just use the password for one query and forget it
-
-The latter option is only available when using :class:`open_mastr.soap_api.download.MaStRAPI`.
-Instantiate with
-
-.. code-block::
-
- MaStRAPI(user='USERNAME', key='TOKEN')
-
-to provide user and token in a script and use these
-credentials in subsequent queries.
-
-Logs
-----
-
-For the download via the API, logs are stored in a single file in `/$HOME/cwm/.open-MaStR/logs/open_mastr.log`.
-New logging messages are appended. It is recommended to delete the log file from time to time because of its required disk space.
-
-
-Zenodo token
-------------------
-
-Uploading data to `Zenodo `_ requires authentication. When logged in with your account you can
-`create tokens `_ for API requests.
-
-The section in `credentials.cfg` looks like:
-
-.. code-block::
-
- [Zenodo]
- token = voh6Zo2ohbohReith4ec2iezeiJ9Miefohso0DohK9ohtha6mahfame7hohc
-
-
-Download
-=============================
-
-Get data via the bulk download
--------------------------------
-On the homepage `MaStR/Datendownload `_ a zipped folder containing the whole
-MaStR is offered. The data is delivered as xml-files. The official documentation can be found
-`here [in german] `_.
-This data is updated on a daily base.
-
-.. figure:: images/MaStR_bulk_download.svg
- :width: 50%
- :align: center
-
- Overview of the open_mastr bulk download functionality.
-
-In the following, the process is described that is started when calling the download function with the parameter method="bulk".
-First, the zipped files are downloaded and saved in `$HOME/.open-MaStR/data/xml_download`. The zipped folder contains many xml files,
-which represent the different tables from the MaStR. Those tables are then parsed to a sqlite database. If only some specific
-data are of interest, they can be specified with the parameter `data`. Every table that is selected in `data` will be deleted, if existent,
-and then filled with data from the xml files.
-
-In the last step, a basic data cleansing is performed. Many entries in the MaStR from the bulk download are replaced by numbers.
-As an example, instead of writing the german states where the unit is registered (Saxony, Brandenburg, Bavaria, ...) the MaStR states
-corresponding digits. One major step of cleansing is therefore to replace those digits with their original meaning.
-Moreover, the datatypes of different entries are set in the data cleansing process.
-
-Advantages of the bulk download:
- * No registration for an API key is needed
-
-Disadvantages of the bulk download:
- * No single tables or entries can be downloaded
-
-Get data via the MaStR-API
----------------------------
-
-Downloading data with the MaStR-API is more complex than using the :ref:`Bulk download `.
-
-Follow this checklist for configuration:
-
-#. Set up a MaStR account and configure your :ref:`credentials `.
-#. Configure your :ref:`database `.
-#. Configure your :ref:`API download ` settings.
-
-Database settings
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Configure your database with the `engine` parameter of :class:`Mastr`.
-It defines the engine of the database where the MaStR is mirrored to. Default is 'sqlite'.
-
-Choose from: {'sqlite', sqlalchemy.engine.Engine}
-
-The possible databases are:
-
-* **sqlite**: By default the database will be stored in `$HOME/.open-MaStR/data/sqlite/open-mastr.db` (can be customized via the
- :ref:`environment variable ` `$SQLITE_DATABASE_PATH`).
-* **own database**: The Mastr class accepts a sqlalchemy.engine.Engine object as engine which enables the user to
- use any other desired database.
- If you're planning to use docker with a postgresql database, you need to set up the docker container including a postegresql database yourself.
- To do so, you can build upon the 'docker-compose.yml' in the scripts folder.
- If you do so, you need to insert the connection parameter into the engine variable. It'll look like this:
-
-.. code-block::
-
- from sqlalchemy import create_engine
-
- engine = create_engine("postgresql+psycopg2://open-mastr:open-mastr@localhost:55443/open-mastr")
- db = Mastr(engine=engine)
-
-
-.. tabs::
-
- .. code-tab:: py SQLite
-
- from sqlalchemy import create_engine
-
- engine = create_engine("sqlite:///path/to/sqlite/database.db")
- db = Mastr(engine=engine)
-
- .. code-tab:: py PostgreSQL
-
- from sqlalchemy import create_engine
-
- engine = create_engine("postgresql://myusername:mypassword@localhost/mydatabase")
- db = Mastr(engine=engine)
-..
-
-
-
-API download settings
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Prior to starting the download of data from MaStR-API, you might want to adjust parameters in the config file.
-Please read in :ref:`Configuration`.
-For downloading data from Marktstammdatenregister (MaStR) registering an account is required.
-Find more information :ref:`here `.
-
-By using the API (e.g. by setting `method` ="API" in the `Mastr.download()` method)
-additional parameters can be set to define in detail which data should be obtained.
-
-.. list-table:: API-related download arguments and explanation
- :widths: 5 5 5
- :header-rows: 1
-
- * - argument
- - options for specification
- - explanation
- * - data
- - ["wind","biomass","combustion","gsgk","hydro","nuclear","storage","solar"]
- - Select data to download.
- * - date
- - None or :class:`datetime.datetime` or str
- - Specify backfill date from which on data is retrieved. Only data with time stamp greater than `date` will be retrieved. Defaults to `None` which means that todays date is chosen.
- * - api_data_types
- - ["unit_data","eeg_data","kwk_data","permit_data"]
- - Select the type of data to download.
- * - api_location_types
- - ["location_elec_generation","location_elec_consumption","location_gas_generation","location_gas_consumption"]
- - Select location_types to download.
- * - api_processes
- - Number of type int, e.g.: 5
- - Select the number of parallel download processes. Possible number depends on the capabilities of your machine. Defaults to `None`.
- * - api_limit
- - Number of type int, e.g.: 1500
- - Select the number of entries to download. Defaults to 50.
- * - api_chunksize
- - int or None, e.g.: 1000
- - Data is downloaded and inserted into the database in chunks of `api_chunksize`. Defaults to 1000.
-
-.. warning::
- The implementation of parallel processes is currently under construction. Please let the argument `api_processes` at the default value `None`.
-
-
-
-
-
-Three different levels of access to data are offered where the code builds on top of each other.
-
-.. figure:: images/MaStR_downloading.svg
- :width: 70%
- :align: center
-
- Overview of open-mastr API download functionality.
-
-The most fundamental data access is provided by :class:`open_mastr.soap_api.download.MaStRAPI` that simply
-wraps SOAP webservice functions with python methods.
-Using the methods of the aforementioned, :class:`open_mastr.soap_api.download.MaStRDownload` provides
-methods for bulk data download and association of data from different sources.
-If one seeks for an option to store the entire data in a local database,
-:class:`open_mastr.soap_api.mirror.MaStRMirror` is the right choice. It offers complete data download
-and updating latest data changes.
-
-Mirror MaStR database
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. autoclass:: open_mastr.soap_api.mirror.MaStRMirror
- :members:
-
-
-Download large number of units
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Downloading thousands of units is simplified using
-:meth:`open_mastr.soap_api.download.MaStRDownload.download_power_plants`.
-
-.. autoclass:: open_mastr.soap_api.download.MaStRDownload
- :members:
-
-
-MaStR API wrapper
-^^^^^^^^^^^^^^^^^^^^
-
-For simplifying queries against the MaStR API, :class:`open_mastr.soap_api.download.MaStRAPI` wraps around exposed
-SOAP queries.
-This is the low-level MaStR API access.
-
-.. autoclass:: open_mastr.soap_api.download.MaStRAPI
- :members:
-
-
-.. include:: _data/raw_data.rst
-
-
-Post-processing (Outdated)
-===========================
-
-The following chapter on postprocessing is outdated. It refers to jupyter notebooks you can find
-on our `github page `_. It is planned to add those post-processing
-features in later versions to the open_mastr package.
-
-Pre-requisites (Outdated)
--------------------------
-
-Major parts of data cleansing, correction and enrichment are written in SQL. We recommend to run these scripts in a
-dockered PostgreSQL database. If you want to use a native installation of PostgreSQL, help yourself and continue with
-:ref:`Run postprocessing (Outdated)`.
-
-Make sure you have `docker-compose `_ installed. Run
-
-.. code-block:: bash
-
- docker-compose up -d
-
-to start a PostgreSQL database.
-
-You can connect to the database with the following credentials (if not changed in `docker-compose.yml`).
-
-======== ==========
-Field Value
-======== ==========
-host localhost
-port 55443
-database open-mastr
-User open-mastr
-Password open-mastr
-======== ==========
-
-
-Once you're finished with working in/with the database, shut it down with
-
-.. code-block:: bash
-
- docker-compose down
-
-
-Run postprocessing (Outdated)
------------------------------
-
-During post-processing downloaded :ref:`Download ` gets cleaned, imported to a PostgreSQL database,
-and enriched.
-To run the postprocessing, use the following code snippets.
-
-.. code-block:: python
-
- from open_mastr.postprocessing.cleaning import cleaned_data
- from postprocessing.postprocessing import postprocess
-
- cleaned = cleaned_data()
- postprocess(cleaned)
-
-As a result, cleaned data gets saved as CSV files and tables named like `bnetza_mastr__cleaned`
-appear in the schema `model_draft`.
-Use
-
-.. code-block:: python
-
- from postprocessing.postprocessing import to_csv
-
- to_csv()
-
-to export processed data in CSV format.
-
-.. note::
-
- It is assumed raw data resides in `~/.open-MaStR/data//` as explained in :ref:`Configuration`.
-
-.. warning::
-
- Raw data downloaded with :class:`open_mastr.soap_api.download.MaStRDownload` is
- currently not supported.
- Please use raw data from a CSV export(:meth:`open_mastr.soap_api.mirror.MaStRMirror.to_csv`)
- of :class:`open_mastr.soap_api.mirror.MaStRMirror` data.
-
-
-Database import (Outdated)
---------------------------
-
-Where available, geo location data, given in lat/lon (*Breitengrad*, *Längengrad*), is converted into a PostGIS geometry
-data type during database import. This allows spatial data operations in PostgreSQL/PostGIS.
-
-
-Data cleansing (Outdated)
--------------------------
-
-Units inside Germany and inside German offshore regions are selected and get distinguished from units that are (falsely)
-located outside of Germany.
-Data is stored in separate tables.
-
-
-Data enrichment (Outdated)
---------------------------
-
-For units without geo location data, a location is estimated based on the zip code. The centroid of the zip code region
-polygon is used as proxy for the exact location.
-To determine the zip code area, zip code data of OSM is used which is stored in
-`boundaries.osm_postcode `_.
-If a unit originally had correct geo data and the origin of estimated geom data is documented in the column `comment`.
-
-
-Zenodo upload
-=============
-
-Use the function :func:`~.open_mastr.data_io.zenodo_upload` for uploading data to Zenodo.
-
-.. autofunction:: open_mastr.data_io.zenodo_upload
\ No newline at end of file
diff --git a/docs/changelog.rst b/docs/changelog.rst
deleted file mode 100644
index 5ec4a0ec..00000000
--- a/docs/changelog.rst
+++ /dev/null
@@ -1 +0,0 @@
-.. mdinclude:: ../CHANGELOG.md
\ No newline at end of file
diff --git a/docs/dataset.md b/docs/dataset.md
new file mode 100644
index 00000000..30e0290f
--- /dev/null
+++ b/docs/dataset.md
@@ -0,0 +1,100 @@
+## Background
+
+The Marktstammdatenregister (MaStR) has been operated by the German Federal Network Agency (Bundesnetzagentur, abbreviated as BNetzA) since January 31, 2019, as a central online database for data related to the German energy system. Owners of electricity or gas generating plants are obligated to report master data about themselves and their plants. Additionally, industrial facilities consuming large amounts of electricity must register if they are connected to at least a high-voltage electricity grid.
+
+Most unit information is openly accessible and is published under an open data license, the Data licence Germany – attribution – version 2.0 (DL-DE-BY-2.0). This data can be downloaded, used, and republished with no restrictions, provided that proper attribution to the Bundesnetzagentur is given.
+
+For units with a net capacity of less than 30 kW, some location information is restricted from publication. This includes street names, house numbers, parcel designations, and exact coordinates of units. The most detailed location information accessible for all units is the postal code or the municipality.
+
+In our paper titled [Analyzing Data Reliability in Germany's Energy System: A Validation of Unit Locations of the Marktstammdatenregister](https://arxiv.org/abs/2304.10581), we provide further insights into the content and quality of the dataset.
+
+## Content
+
+The German Federal Network Agency regularly updates the dataset and adds new tables and attributes. Hence, the primary resource of information about the dataset should be the original website:
+
+* Get information about the `bulk` data [here](https://www.marktstammdatenregister.de/MaStR/Datendownload) (in german)
+* Get information about the `API` data [here](https://www.marktstammdatenregister.de/MaStRHilfe/subpages/webdienst.html) (in german)
+
+## Difference between `bulk` and `API` dataset
+
+As you may have noticed, we distinguish between `bulk` and `API` datasets. The `bulk` dataset refers to the data obtained from the zipped XML files downloaded from [here](https://www.marktstammdatenregister.de/MaStR/Datendownload) using the [`Mastr.download`][open_mastr.Mastr.download] function. The `API` data is obtained by requesting information via the SOAP-API and the [`soap_api.download.MaStRDownload`][open_mastr.soap_api.download.MaStRDownload] module.
+
+??? question "Why is the table structure in the open-mastr database as it is?"
+
+ The structure of the database is historically determined by the data retrieved via API. (open-mastr existed before the XML-dump was provided).
+ See [MaStR data model](#Mastr-data-model)
+
+
+## Tables in the database
+
+!!! question "Confused by all the tables?"
+ :sparkles: We regularly run the whole download and cleansing pipeline and upload the dataset as csv files at [zenodo](https://doi.org/10.5281/zenodo.6807425)!
+
+After downloading the MaStR, you will find a database with a large number of tables. Here we give a brief overview of what you can find in those tables:
+
+### Tables in the local database
+
+
+=== "_extended tables"
+ The main information about the different technologies lies in the `_extended` tables. You can find the capacity, location, and other technology-specific attributes here.
+
+ | Table name | Comments |
+ |------|------|
+ | biomass_extended | |
+ | combustion_extended | *Conventional powerplants: Gas, Oil, Coal, ...* |
+ | gsgk_extended | *gsgk is short for: Geothermal, Mine gas, and Pressure relaxation* |
+ | hydro_extended | |
+ | nuclear_extended | |
+ | solar_extended | |
+ | storage_extended | |
+ | wind_extended | |
+
+=== "_eeg tables"
+ In germany, renewable energies were subsidized by the state - according to a law called 'EEG'. Relevant information like the 'EEG ID' are in the `_eeg` tables.
+
+ | Table name | Comments |
+ |------|------|
+ | biomass_eeg | |
+ | gsgk_eeg | *gsgk is short for: Geothermal, Mine gas, and Pressure relaxation* |
+ | hydro_eeg | |
+ | solar_eeg | |
+ | storage_eeg | |
+ | wind_eeg | |
+
+=== "Other tables"
+ Other tables contain information about the grid, the energy market, or gas consumers and producers:
+
+ | Table name | Comments |
+ |------|------|
+ | balancing_area | *Related to the energy market* |
+ | electricity_consumer | *Only large consumers* |
+ | gas_consumer | *Only large consumers* |
+ | gas_producer | |
+ | gas_storage | |
+ | gas_storage_extended | |
+ | grid_connections | *Does not contain geoinformation* |
+ | grids | *Does not contain geoinformation* |
+ | locations_extended | *Connects units with grids - to get coordinates of units use the _extended tables*|
+ | market_actors | |
+ | market_roles | |
+ | permit | |
+ | storage_units | |
+ | kwk | *short for: Combined heat and power (CHP)* |
+
+### MaStR data model
+A useful overview of the MaStR data model can be found [here (in german)](https://www.marktstammdatenregister.de/MaStRHilfe/files/webdienst/Objektmodell%20-%20Fachliche%20Ansicht%20V1.2.0.pdf). A translated version using the names from the tables you can find in your local database is presented here:
+
+=== "translated image (english)"
+ 
+
+=== "original image (german)"
+ 
+
+
+## Tables as CSV
+
+Tables from the database can be exported to csv files. By default, all available power plant unit data will be exported
+to csv files.
+
+For exported csv's additional available data is joined on basic unit data. For example: For biomass power plants one csv
+is exported consisting of the join of four database tables (unit data, chp data, permit data, eeg data). We regularly run the whole download and cleansing pipeline and upload the dataset as csv files at [zenodo](https://doi.org/10.5281/zenodo.6807425).
diff --git a/docs/development.rst b/docs/development.rst
deleted file mode 100644
index 720bf29e..00000000
--- a/docs/development.rst
+++ /dev/null
@@ -1,103 +0,0 @@
-***********
-Development
-***********
-
-For now, please read in the
-`CONTRIBUTING `_.
-
-Build docs
-==========
-
-For checking that the documentation builds successfully, run sphinx locally.
-Make sure you've installed docs requirements
-
-.. code-block:: bash
-
- pip install -r docs/requirements.txt
-
-The you can build the docs with
-
-.. code-block:: bash
-
- sphinx-build -E -a docs docs/_build/
-
-If structure of functions (new added/deleted or names changed) changes, code reference needs
-to be updated.
-
-.. code-block:: bash
-
- sphinx-apidoc --no-toc -o docs/reference open_mastr open_mastr/postprocessing
-
-If the SOAP API of MaStR changes, data table metadata documentation needs to be updated as well. From within
-`open-MaStR/docs` execute
-
-.. code-block:: bash
-
- python -c "from open_mastr.utils.docs import generate_data_docs; generate_data_docs()"
-
-
-to recreate the files.
-
-
-Code Formatting
-=================
-
-We use `Black `_ as Python formatter. In order to simplify and guarantee the proper usage,
-Black is added as a pre-commit hook. When you try to commit some changes, the changed files will eventually be formatted
-and your commit fails. After adding the new changes to your commit, you should have a clean formatted code and
-a successful commit.
-
-The first time after installing the package in development mode, you have to install the pre-commit hooks.
-
-.. code-block:: bash
-
- pre-commit install
-
-
-Testing
-=======
-
-Make sure you have test requirements installed (in addition to the package itself).
-
-.. code-block:: bash
-
- pip install -e .[dev]
-
-Some tests query data from `MaStR `_. Therefore, user name and the token must
-be provided by credentials file `credentials.cfg` (with token optionally stored in keyring).
-Further explanation :ref:`here `.
-
-`PyTest `_ is used to run the test suite.
-
-.. code-block:: bash
-
- pytest -vv
-
-Updating open-mastr ORM due to wsdl API change
-=================================
-
-The Markstammdatenregister API is yearly updated on 1 April and 1 October. The updates often entail changes, additions or deletions of columns in the data base.
-This changes have to be reflected in the code by adapting:
-
-1. orm.py - ensuring a faultless download with `method='API'`
-2. mastr_datapackage.json - updating the metadata for correct data description
-3. docs\_data\_raw\ - updating tables for correct online documentation
-
-You need to set up a new database with the adapted ORM structure locally to apply the changes after implementation. Your existing database won't reflect the changes.
-
-The issue `#351 `_ in the open-mastr GitHub repo can serve as a process blueprint for future updates.
-
-
-Validating metadata documentation
-=================================
-
-From with a directory of data (default case: `~/.open-MaStR/data/` execute
-
-.. code-block:: bash
-
- frictionless validate datapackage.json --basepath .
-
-for validating datapackage metadata with
-`Frictionless data specifications
-`_.
-At the moment, there complaints about the format.
diff --git a/docs/development/changelog_mirror.md b/docs/development/changelog_mirror.md
new file mode 100644
index 00000000..a1df8959
--- /dev/null
+++ b/docs/development/changelog_mirror.md
@@ -0,0 +1,7 @@
+---
+hide:
+ - toc
+---
+{%
+ include-markdown "../../CHANGELOG.md"
+%}
\ No newline at end of file
diff --git a/docs/development/contributing_mirror.md b/docs/development/contributing_mirror.md
new file mode 100644
index 00000000..66369d0c
--- /dev/null
+++ b/docs/development/contributing_mirror.md
@@ -0,0 +1,8 @@
+---
+hide:
+ - toc
+---
+
+{%
+ include-markdown "../../CONTRIBUTING.md"
+%}
\ No newline at end of file
diff --git a/docs/development/release_procedure_mirror.md b/docs/development/release_procedure_mirror.md
new file mode 100644
index 00000000..ef87ece1
--- /dev/null
+++ b/docs/development/release_procedure_mirror.md
@@ -0,0 +1,8 @@
+---
+hide:
+ - toc
+---
+
+{%
+ include-markdown "../../RELEASE_PROCEDURE.md"
+%}
\ No newline at end of file
diff --git a/docs/getting_started.md b/docs/getting_started.md
new file mode 100644
index 00000000..1cfd6e37
--- /dev/null
+++ b/docs/getting_started.md
@@ -0,0 +1,71 @@
+# Getting Started
+
+The intention of open-MaStR is to provide tools for receiving a complete as possible and accurate as possible list of
+power plant units based on the public registry Marktstammdatenregister (short: [MaStR](https://www.marktstammdatenregister.de)).
+
+## Downloading the MaStR data
+
+
+The MaStR dataset is updated on a daily basis. To download todays MaStR and save it in a sqlite database, you will use the [`Mastr`][open_mastr.Mastr] class and its [`download`][open_mastr.Mastr.download] method. The [`download`][open_mastr.Mastr.download] method offers two different ways to get the data by changing the `method` parameter (if not specified, `method` defaults to "bulk"):
+
+1. `method` = "bulk": Get data via the bulk download from [MaStR/Datendownload](https://www.marktstammdatenregister.de/MaStR/Datendownload). Use this if you want to download the whole dataset (few Gigabite) or if you want to download all units of a given technology (e.g. all wind turbines in Germany).
+2. `method` = "API": Get data via the MaStR SOAP-API. Use this if you want specific information about single units and if you have registerd to get an API token.
+
+
+### Bulk download
+
+The following code block shows the basic download commands:
+
+```python
+from open_mastr import Mastr
+
+db = Mastr()
+db.download()
+```
+
+When a `Mastr` object is initialized, a sqlite database is created in `$HOME/.open-MaStR/data/sqlite`. With the function `Mastr.download()`, the **whole MaStR is downloaded** in the zipped xml file format. It is then read into the sqlite database and simple data cleansing functions are started.
+
+More detailed information can be found in the section [bulk download](advanced/#bulk-download).
+
+API download
+-----------------------------------
+When using `download(method="API")`, the data is retrieved from the MaStR API. For using the MaStR API, credentials are needed (see [SOAP API download](advanced/#soap-api-download)).
+
+```python
+from open_mastr import Mastr
+
+db = Mastr()
+db.download(method='API')
+```
+
+The default settings will save retrieved data into the sqlite database. The function can be used to mirror the open-MaStR database using the SOAP API. Note that the data provided by the bulk download and the SOAP API is similar, but not exactly the same.
+
+## Accessing the database
+
+For accessing and working with the MaStR database after you have downloaded it, you can use sqlite browsers
+such as [DB Browser for SQLite](https://sqlitebrowser.org/) or any python module
+which can process sqlite data. Pandas, for example, comes with the function
+[read_sql](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html).
+
+
+=== "pandas"
+ ```python
+ import pandas as pd
+
+ table="wind_extended"
+ df = pd.read_sql(sql=table, con=db.engine)
+ ```
+
+=== "DB Browser for sqlite"
+ { loading=lazy }
+
+## Exporting data
+
+The tables in the database can be exported as csv files. While technology-related data is joined for each unit,
+additional tables are mirrored from database to csv as they are. To export the data you can use to method `to_csv`.
+
+```python
+
+ tables=["wind", "grid"]
+ db.to_csv(tables)
+```
\ No newline at end of file
diff --git a/docs/getting_started.rst b/docs/getting_started.rst
deleted file mode 100644
index 5ae76edf..00000000
--- a/docs/getting_started.rst
+++ /dev/null
@@ -1,121 +0,0 @@
-********************
-Getting Started
-********************
-
-The intention of open-MaStR is to provide tools for receiving a complete as possible and accurate as possible list of
-power plant units based on the public registry Marktstammdatenregister (short: `MaStR `_).
-
-Downloading the MaStR data
-=============================
-
-For downloading the MaStR and saving
-it in a sqlite database, you will use the :class:`Mastr` class and its `download` method (For documentation of those methods see
-:ref:`mastr module`)
-
-The :meth:`download` function offers two different ways to get the data by changing the `method` parameter (if not specified, `method` defaults to "bulk"):
- #. `method` = "bulk": Get data via the bulk download from `MaStR/Datendownload `_
- #. `method` = "API": Get data via the MaStR-API
-
-Keep in mind: While the data from both methods is quiet similar, it is not exactly the same!
-
-Bulk download
------------------------------------
-The following code block shows the basic download commands:
-
- .. code-block:: python
-
- from open_mastr import Mastr
-
- db = Mastr()
- db.download()
-
-When a :class:`Mastr` object is initialized, a sqlite database is created
-in `$HOME/.open-MaStR/data/sqlite`. With the function `Mastr.download()`, the **whole MaStR is downloaded** in the zipped xml file
-format. It is then read into the sqlite database and simple data cleansing functions are started.
-
-More detailed information can be found in :ref:`Get data via the bulk download `.
-
-API download
------------------------------------
-When using `download(method="API")`, the data is retrieved from the MaStR API. For using the MaStR API,
-credentials are needed (see :ref:`Get data via the MaStR-API`).
-
- .. code-block:: python
-
- from open_mastr import Mastr
-
- db = Mastr()
- db.download(method='API')
-
-The default settings will save retrieved data into the sqlite database. The function can be used to mirror the open-MaStR database regularly
-without needing to download the `provided dumps `_ daily.
-
-More detailed information can be found in :ref:`Get data via the MaStR-API `.
-
-Accessing the database
-===================================
-
-
-For accessing and working with the MaStR database after you have downloaded it, you can use sqlite browsers
-(such as `DB Browser for SQLite `_) or any python module
-which can process sqlite data. Pandas, for example, comes with the function
-`read_sql `_.
-
- .. code-block:: python
-
- import pandas as pd
-
- table="wind_extended"
- df = pd.read_sql(sql=table, con=db.engine)
-
-
-The tables that exist in the database are listed below. Their relations can be found in the chapter :ref:`Data Description `
-
-.. list-table:: Tables in the sqlite database
- :widths: 5
- :header-rows: 1
-
- * - table
- * - balancing_area
- * - basic_units
- * - biomass_eeg
- * - biomass_extended
- * - combustion_extended
- * - electricity_consumer
- * - gas_consumer
- * - gas_producer
- * - gas_storage
- * - gas_storage_extended
- * - grid_connections
- * - grids
- * - gsgk_eeg
- * - gsgk_extended
- * - hydro_eeg
- * - hydro_extended
- * - kwk
- * - locations_basic
- * - locations_extended
- * - market_actors
- * - market_roles
- * - nuclear_extended
- * - permit
- * - solar_eeg
- * - solar_extended
- * - storage_eeg
- * - storage_extended
- * - storage_units
- * - wind_eeg
- * - wind_extended
-
-Exporting data
-===================================
-
-
-The tables in the database can be exported as csv files. While technology-related data is joined for each unit,
-additional tables are mirrored from database to csv as they are. To export the data you can use to method :meth:`to_csv`
-
- .. code-block:: python
-
- tables=["wind", "grid"]
- db.to_csv(tables)
-
diff --git a/docs/images/DBBrowser.PNG b/docs/images/DBBrowser.PNG
new file mode 100644
index 00000000..e6070baf
Binary files /dev/null and b/docs/images/DBBrowser.PNG differ
diff --git a/docs/images/DetailAnlagen_english.PNG b/docs/images/DetailAnlagen_english.PNG
new file mode 100644
index 00000000..e40f4ec8
Binary files /dev/null and b/docs/images/DetailAnlagen_english.PNG differ
diff --git a/docs/images/MaStR_bulk_download.svg b/docs/images/MaStR_bulk_download.svg
deleted file mode 100644
index ac9aa36c..00000000
--- a/docs/images/MaStR_bulk_download.svg
+++ /dev/null
@@ -1,493 +0,0 @@
-
-
-
-
diff --git a/docs/images/OpenEnergyFamily_Logo_OpenMaStR_no_name_grey.png b/docs/images/OpenEnergyFamily_Logo_OpenMaStR_no_name_grey.png
new file mode 100644
index 00000000..0d8d4cf3
Binary files /dev/null and b/docs/images/OpenEnergyFamily_Logo_OpenMaStR_no_name_grey.png differ
diff --git a/docs/images/OpenEnergyFamily_Logo_OpenMaStR_no_name_white.png b/docs/images/OpenEnergyFamily_Logo_OpenMaStR_no_name_white.png
new file mode 100644
index 00000000..bc5ebae0
Binary files /dev/null and b/docs/images/OpenEnergyFamily_Logo_OpenMaStR_no_name_white.png differ
diff --git a/docs/images/OpenEnergyFamily_Logo_OpenMaStR_white.png b/docs/images/OpenEnergyFamily_Logo_OpenMaStR_white.png
new file mode 100644
index 00000000..ccb77800
Binary files /dev/null and b/docs/images/OpenEnergyFamily_Logo_OpenMaStR_white.png differ
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 00000000..ede972b1
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,29 @@
+# Introduction
+
+
+The [Marktstammdatenregister (MaStR)](https://www.marktstammdatenregister.de/MaStR) is a German register
+provided by the German Federal Network Agency (Bundesnetzagentur / BNetza) that keeps track of all power and gas units located in Germany.
+
+Generally, the MaStR data can be accessed via various options:
+
+ 1. browse, filter and download [online](https://www.marktstammdatenregister.de/MaStR)
+ 1. download [daily provided dumps](https://www.marktstammdatenregister.de/MaStR/Datendownload)
+ 1. access via the [web service](https://www.marktstammdatenregister.de/MaStRHilfe/subpages/webdienst.html)
+
+The python package `open-mastr` provides an interface for accessing the data and contributes to improving the
+usability of the access options above. This repository is intended for people who wish to simply work with the
+MaStR data and do not want to deal with the individual obstacles to data access of the three options above.