Updating all base shapes (country_shapes, europe_shape, nuts3_shapes) (…

…#1479) * Updated shapes to high-res 1M NUTS2024 shapes and geoboundaries for non-NUTS countries (UA, MD, BA). * Updated nuts to 2021 and non-nuts to gadm. * Updated non NUTS country adm 1 boundaries to OSM via Overpass (MD, BA, UA, XK * Minor order update in rule. * Updated eurostat population data * Updated dataset to JRC ARDECO 2021, including UK, RS, CH * Finished new implementation of build_shapes. * Small fix. * Running sector-coupling workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated build_shapes * Added release nots and updated data-bundle docs. * Added sandbox databundle for temporary CI testing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix in build_shapes: Build only for selected countries in config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed jrc-data from repo, use REST API instead. removed params from build_osm_boundaries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Re-added shape/polygon simplification. * Made build_osm_boundaries more robust, if wrong country components are retrieved. Reduced nuts shapes to 03m resolution. * Updated build_shapes with simplifying europe function to remove remote islands. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated back to 01m resolution. takes longer. * Removed test_build_shapes from testing, as outdated and not compatible with shapes PR * Readded test_build_shapes.py, however reduced testing, as 'countries()' does not exist anymore, replaced by create_regions(). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for major Danish islands and Kopenhagen. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * integrate new files into data bundle --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <[email protected]>
PyPSA · Jan 24, 2025 · 1d8bb3d · 1d8bb3d
1 parent 71852a2
commit 1d8bb3d
Show file tree

Hide file tree

Showing 11 changed files with 849 additions and 472 deletions.
diff --git a/doc/data-bundle.rst b/doc/data-bundle.rst
@@ -33,13 +33,6 @@ scope to reduce file size, or are not provided through stable URLs elsewhere.
 - **License:** `custom <https://ec.europa.eu/eurostat/about-us/policies/copyright>`__
 - **Description:** Average annual population to calculate regional GDP data (thousand persons) by NUTS 3 regions.
 
-``data/bundle/nama_10r_3gdp.tsv.gz``
-
-- **Source:** Eurostat
-- **Link:** https://ec.europa.eu/eurostat/databrowser/view/nama_10r_3gdp/default/table?lang=en
-- **License:** `custom <https://ec.europa.eu/eurostat/about-us/policies/copyright>`__
-- **Description:** Gross domestic product (GDP) at current market prices by NUTS 3 regions.
-
 ``data/bundle/corine``
 
 - **Source:** European Environment Agency (EEA)
@@ -94,7 +87,7 @@ scope to reduce file size, or are not provided through stable URLs elsewhere.
 - **License:** CC0 (`reference <https://datadryad.org/stash/dataset/doi:10.5061/dryad.dk1j0>`__)
 - **Description:** Gridded GDP data.
 
-``data/bundle/ppp_2013_1km_Aggregated.tif``
+``data/bundle/ppp_2019_1km_Aggregated.tif``
 
 - **Source:** WorldPop (www.worldpop.org - School of Geography and Environmental
   Science, University of Southampton; Department of Geography and Geosciences,
@@ -104,5 +97,5 @@ scope to reduce file size, or are not provided through stable URLs elsewhere.
   Funded by The Bill and Melinda Gates Foundation (OPP1134076).
   https://dx.doi.org/10.5258/SOTON/WP00647
 - **Link:** https://hub.worldpop.org/doi/10.5258/SOTON/WP00647
-- **License:** CC-BY 4.0 (`reference <https://hub.worldpop.org/geodata/summary?id=24770>`__)
+- **License:** CC-BY 4.0 (`reference <https://hub.worldpop.org/geodata/summary?id=24776>`__)
 - **Description:** Gridded population data.
diff --git a/doc/release_notes.rst b/doc/release_notes.rst
@@ -124,6 +124,8 @@ Upcoming Release
 
 * Update locations and capacities of ammonia plants.
 
+* Updating all base shapes (country_shapes, europe_shape, nuts3_shapes, ...). The workflow has been modified to use higher resolution and more harmonised shapes (NUTS3 2021 01M data and OSM administration level 1 for non-NUTS3 countries, such as BA, MD, UA, and XK). Data sources for population and GDP p.c. have been updated to JRC ARDECO https://urban.jrc.ec.europa.eu/ardeco/ -- 2019 values are used. `build_gdp_pop_non_nuts3` (originally created to build regional GDP p.c. and population data for MD and UA) is now integrated into `build_shapes` and extended to build regional values for all non-NUTS3 countries using cutouts of the updated datasets `GDP_per_capita_PPP_1990_2015_v2.nc` and `ppp_2019_1km_Aggregated.tif`,
+
 
 
 PyPSA-Eur 0.13.0 (13th September 2024)

diff --git a/rules/build_electricity.smk b/rules/build_electricity.smk
@@ -100,17 +100,37 @@ rule base_network:
         "../scripts/base_network.py"
 
 
+rule build_osm_boundaries:
+    input:
+        json="data/osm-boundaries/json/{country}_adm1.json",
+        eez=ancient("data/eez/World_EEZ_v12_20231025_LR/eez_v12_lowres.gpkg"),
+    output:
+        boundary="data/osm-boundaries/build/{country}_adm1.geojson",
+    log:
+        "logs/build_osm_boundaries_{country}.log",
+    threads: 1
+    resources:
+        mem_mb=1500,
+    conda:
+        "../envs/environment.yaml"
+    script:
+        "../scripts/build_osm_boundaries.py"
+
+
 rule build_shapes:
     params:
         countries=config_provider("countries"),
     input:
-        naturalearth=ancient("data/naturalearth/ne_10m_admin_0_countries_deu.shp"),
         eez=ancient("data/eez/World_EEZ_v12_20231025_LR/eez_v12_lowres.gpkg"),
-        nuts3=ancient("data/nuts/NUTS_RG_03M_2013_4326_LEVL_3.geojson"),
-        nuts3pop=ancient("data/bundle/nama_10r_3popgdp.tsv.gz"),
-        nuts3gdp=ancient("data/bundle/nama_10r_3gdp.tsv.gz"),
-        ch_cantons=ancient("data/ch_cantons.csv"),
-        ch_popgdp=ancient("data/bundle/je-e-21.03.02.xls"),
+        nuts3_2021="data/nuts/NUTS_RG_01M_2021_4326_LEVL_3.geojson",
+        ba_adm1="data/osm-boundaries/build/BA_adm1.geojson",
+        md_adm1="data/osm-boundaries/build/MD_adm1.geojson",
+        ua_adm1="data/osm-boundaries/build/UA_adm1.geojson",
+        xk_adm1="data/osm-boundaries/build/XK_adm1.geojson",
+        nuts3_gdp="data/jrc-ardeco/ARDECO-SUVGDP.2021.table.csv",
+        nuts3_pop="data/jrc-ardeco/ARDECO-SNPTD.2021.table.csv",
+        other_gdp="data/bundle/GDP_per_capita_PPP_1990_2015_v2.nc",
+        other_pop="data/bundle/ppp_2019_1km_Aggregated.tif",
     output:
         country_shapes=resources("country_shapes.geojson"),
         offshore_shapes=resources("offshore_shapes.geojson"),
@@ -482,42 +502,10 @@ def input_conventional(w):
     }
 
 
-# Optional input when having Ukraine (UA) or Moldova (MD) in the countries list
-def input_gdp_pop_non_nuts3(w):
-    countries = set(config_provider("countries")(w))
-    if {"UA", "MD"}.intersection(countries):
-        return {"gdp_pop_non_nuts3": resources("gdp_pop_non_nuts3.geojson")}
-    return {}
-
-
-rule build_gdp_pop_non_nuts3:
-    params:
-        countries=config_provider("countries"),
-    input:
-        base_network=resources("networks/base_s.nc"),
-        regions=resources("regions_onshore_base_s.geojson"),
-        gdp_non_nuts3="data/bundle/GDP_per_capita_PPP_1990_2015_v2.nc",
-        pop_non_nuts3="data/bundle/ppp_2013_1km_Aggregated.tif",
-    output:
-        resources("gdp_pop_non_nuts3.geojson"),
-    log:
-        logs("build_gdp_pop_non_nuts3.log"),
-    benchmark:
-        benchmarks("build_gdp_pop_non_nuts3")
-    threads: 1
-    resources:
-        mem_mb=8000,
-    conda:
-        "../envs/environment.yaml"
-    script:
-        "../scripts/build_gdp_pop_non_nuts3.py"
-
-
 rule build_electricity_demand_base:
     params:
         distribution_key=config_provider("load", "distribution_key"),
     input:
-        unpack(input_gdp_pop_non_nuts3),
         base_network=resources("networks/base_s.nc"),
         regions=resources("regions_onshore_base_s.geojson"),
         nuts3=resources("nuts3_shapes.geojson"),

diff --git a/rules/retrieve.smk b/rules/retrieve.smk
@@ -18,7 +18,6 @@ if config["enable"]["retrieve"] and config["enable"].get("retrieve_databundle",
     datafiles = [
         "je-e-21.03.02.xls",
         "nama_10r_3popgdp.tsv.gz",
-        "nama_10r_3gdp.tsv.gz",
         "corine/g250_clc06_V18_5.tif",
         "eea/UNFCCC_v23.csv",
         "emobility/KFZ__count",
@@ -27,7 +26,7 @@ if config["enable"]["retrieve"] and config["enable"].get("retrieve_databundle",
         "natura/natura.tiff",
         "gebco/GEBCO_2014_2D.nc",
         "GDP_per_capita_PPP_1990_2015_v2.nc",
-        "ppp_2013_1km_Aggregated.tif",
+        "ppp_2019_1km_Aggregated.tif",
     ]
 
     rule retrieve_databundle:
@@ -79,7 +78,7 @@ if config["enable"]["retrieve"] and config["enable"].get("retrieve_databundle",
 
 if config["enable"]["retrieve"]:
 
-    rule retrieve_nuts_shapes:
+    rule retrieve_nuts_2013_shapes:
         input:
             shapes=storage(
                 "https://gisco-services.ec.europa.eu/distribution/v2/nuts/download/ref-nuts-2013-03m.geojson.zip"
@@ -103,6 +102,34 @@ if config["enable"]["retrieve"]:
 
 
 
+if config["enable"]["retrieve"]:
+
+    rule retrieve_nuts_2021_shapes:
+        input:
+            shapes=storage(
+                "https://gisco-services.ec.europa.eu/distribution/v2/nuts/download/ref-nuts-2021-01m.geojson.zip"
+            ),
+        output:
+            shapes_level_3="data/nuts/NUTS_RG_01M_2021_4326_LEVL_3.geojson",
+            shapes_level_2="data/nuts/NUTS_RG_01M_2021_4326_LEVL_2.geojson",
+            shapes_level_1="data/nuts/NUTS_RG_01M_2021_4326_LEVL_1.geojson",
+            shapes_level_0="data/nuts/NUTS_RG_01M_2021_4326_LEVL_0.geojson",
+        params:
+            zip_file="data/nuts/ref-nuts-2021-01m.geojson.zip",
+        run:
+            os.rename(input.shapes, params.zip_file)
+            with ZipFile(params.zip_file, "r") as zip_ref:
+                for level in ["LEVL_3", "LEVL_2", "LEVL_1", "LEVL_0"]:
+                    filename = f"NUTS_RG_01M_2021_4326_{level}.geojson"
+                    zip_ref.extract(filename, Path(output.shapes_level_0).parent)
+                    extracted_file = Path(output.shapes_level_0).parent / filename
+                    extracted_file.rename(
+                        getattr(output, f"shapes_level_{level[-1]}")
+                    )
+            os.remove(params.zip_file)
+
+
+
 if config["enable"]["retrieve"] and config["enable"].get("retrieve_cutout", True):
 
     rule retrieve_cutout:
@@ -373,27 +400,6 @@ if config["enable"]["retrieve"]:
 
 
 
-if config["enable"]["retrieve"]:
-
-    # Download directly from naciscdn.org which is a redirect from naturalearth.com
-    # (https://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-admin-0-countries/)
-    # Use point-of-view (POV) variant of Germany so that Crimea is included.
-    rule retrieve_naturalearth_countries:
-        input:
-            storage(
-                "https://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_0_countries_deu.zip"
-            ),
-        params:
-            zip="data/naturalearth/ne_10m_admin_0_countries_deu.zip",
-        output:
-            countries="data/naturalearth/ne_10m_admin_0_countries_deu.shp",
-        run:
-            move(input[0], params["zip"])
-            output_folder = Path(output["countries"]).parent
-            unpack_archive(params["zip"], output_folder)
-            os.remove(params["zip"])
-
-
 if config["enable"]["retrieve"]:
 
     rule retrieve_gem_europe_gas_tracker:
@@ -631,6 +637,20 @@ if config["enable"]["retrieve"] and (
             ),
 
 
+if config["enable"]["retrieve"]:
+
+    rule retrieve_osm_boundaries:
+        output:
+            json="data/osm-boundaries/json/{country}_adm1.json",
+        log:
+            "logs/retrieve_osm_boundaries_{country}_adm1.log",
+        threads: 1
+        conda:
+            "../envs/retrieve.yaml"
+        script:
+            "../scripts/retrieve_osm_boundaries.py"
+
+
 if config["enable"]["retrieve"]:
 
     rule retrieve_heat_source_utilisation_potentials:
@@ -647,3 +667,27 @@ if config["enable"]["retrieve"]:
             "data/heat_source_utilisation_potentials/{heat_source}.gpkg",
         script:
             "../scripts/retrieve_heat_source_utilisation_potentials.py"
+
+
+if config["enable"]["retrieve"]:
+
+    rule retrieve_jrc_ardeco:
+        output:
+            ardeco_gdp="data/jrc-ardeco/ARDECO-SUVGDP.2021.table.csv",
+            ardeco_pop="data/jrc-ardeco/ARDECO-SNPTD.2021.table.csv",
+        run:
+            import requests
+
+            urls = {
+                "ardeco_gdp": "https://urban.jrc.ec.europa.eu/ardeco-api-v2/rest/export/SUVGDP?version=2021&format=csv-table",
+                "ardeco_pop": "https://urban.jrc.ec.europa.eu/ardeco-api-v2/rest/export/SNPTD?version=2021&format=csv-table",
+            }
+
+            for key, url in urls.items():
+                response = requests.get(url)
+                output_path = output[key] if key in urls else None
+                if output_path:
+                    with open(output_path, "wb") as f:
+                        f.write(response.content)
+
+
diff --git a/scripts/build_electricity_demand_base.py b/scripts/build_electricity_demand_base.py
@@ -44,7 +44,6 @@ def upsample_load(
     regions_fn: str,
     load_fn: str,
     nuts3_fn: str,
-    gdp_pop_non_nuts3_fn: str,
     distribution_key: dict[str, float],
 ) -> pd.DataFrame:
     substation_lv_i = n.buses.index[n.buses["substation_lv"]]
@@ -61,19 +60,7 @@ def upsample_load(
     for cntry, group in gdf_regions.geometry.groupby(gdf_regions.country):
         load_ct = load[cntry]
 
-        if cntry in ["UA", "MD"]:
-            # separate handling because nuts3 provides no data for UA+MD
-            gdp_pop_non_nuts3 = gpd.read_file(gdp_pop_non_nuts3_fn).set_index("Bus")
-            gdp_pop_non_nuts3 = gdp_pop_non_nuts3.loc[
-                (gdp_pop_non_nuts3.country == cntry)
-                & (gdp_pop_non_nuts3.index.isin(substation_lv_i))
-            ]
-            factors = normed(
-                gdp_weight * normed(gdp_pop_non_nuts3["gdp"])
-                + pop_weight * normed(gdp_pop_non_nuts3["pop"])
-            )
-
-        elif len(group) == 1:
+        if len(group) == 1:
             factors = pd.Series(1.0, index=group.index)
 
         else:
@@ -116,7 +103,6 @@ def upsample_load(
         regions_fn=snakemake.input.regions,
         load_fn=snakemake.input.load,
         nuts3_fn=snakemake.input.nuts3,
-        gdp_pop_non_nuts3_fn=snakemake.input.get("gdp_pop_non_nuts3"),
         distribution_key=params.distribution_key,
     )
Original file line number	Diff line number	Diff line change
Expand Up		@@ -124,6 +124,8 @@ Upcoming Release

		* Update locations and capacities of ammonia plants.

		* Updating all base shapes (country_shapes, europe_shape, nuts3_shapes, ...). The workflow has been modified to use higher resolution and more harmonised shapes (NUTS3 2021 01M data and OSM administration level 1 for non-NUTS3 countries, such as BA, MD, UA, and XK). Data sources for population and GDP p.c. have been updated to JRC ARDECO https://urban.jrc.ec.europa.eu/ardeco/ -- 2019 values are used. `build_gdp_pop_non_nuts3` (originally created to build regional GDP p.c. and population data for MD and UA) is now integrated into `build_shapes` and extended to build regional values for all non-NUTS3 countries using cutouts of the updated datasets `GDP_per_capita_PPP_1990_2015_v2.nc` and `ppp_2019_1km_Aggregated.tif`,



		PyPSA-Eur 0.13.0 (13th September 2024)
Expand Down