PostGIS-based tile generation, tools

** NOTE ** this was accidentally published ahead of time, the notes are still being written. Maintainers, please help update this page.

Breaking changes

• Use Imposm 0.11.1
• Use generate-vectortiles-pg as the new PostGIS-based tile generation. #310 #317. Thanks @lazaa32 @nyurik
• Deprecated Mapnik-based tile generation (generate-vectortiles docker image is now obsolete)

New features

• mbtiles-tools copy can copy tiles between .mbtiles files. #355. Thanks @nyurik
• layers can assert that required tables/views/functions created by other systems are already present, and fail early if they are not. #370, #373. Thanks @Falke-Design
• It is now possible to import multiple OSM files into the db without clearing / removing the existing data. #365. Thanks @Falke-Design

Miscellaneous

• All python code is now style-checked with flake8. #372, #374. Thanks @nyurik

Improvements of ST_AsMVT

Improve buffer and pixel_width calculation for ST_AsMVT

Bug fix tile_to_bbox(): use pixel_width instead of extent. (#333)
Bugfix st as mvt (#334)

Bugfix and improvements of v5.3

Bugfix

Fix wrong variable name (#328) - thanks @lazaa32

Improvements

1. Include all name tags in slice_language_tags (#315) - thanks @zstadler
2. Implement osmfr poly bbox support (#326) - thanks @nyurik
3. remove slash from name (#330)

Generate tiles by PGquery, layer-stats tool, generate bbox

Main features

1. A script to generate vector tiles from PostGIS' ST_AsMVT using tilelive-pgquery (thanks @nyurik) - Thanks @lazaa32 for implement #317 and @smellman, @zstadler and @nyurik for the original script
2. Remove download-osm make-dc, create a simple text file with bbox value instead (#297) - thanks @nyurik
3. New layer-stats tool to explore data (#293) - Thanks @nyurik for implement based on OpenMapTiles / qa

$ docker-compose run --rm openmaptiles-tools layer-stats frequency openmaptiles.yaml aeroway class
======= Analyzing value occurrence in layer 'aeroway' for field [class] =======
class        z10    z11    z12    z13    z14
---------  -----  -----  -----  -----  -----
aerodrome    151    151    154    154    154
apron         56     56    126    224    308
runway       470    470    515    525    528
taxiway     1075   1075   1156   1219   1245
heliport                     1      2      2
helipad                            11    162

Improvements

• Improve delete_empty_keys() performance, add ps to tools (#295) - thanks @nyurik
• Remove the double set of the header Access-Control-Allow-Origin. (#296) - thanks @francois2metz
• preserve mapping order (#302) - thanks @remi-dupre
• Use print_err in places that mix errors and output on stdout. (#307) - thanks @lseelenbinder
• layers without attributes in yaml mapping (#319) - thanks @frodrigo
• Stop parallel import-sql on error (#322) - thanks @frodrigo
• Use pg_roles view instead of pg_authid table (#323) - thanks @baipi

Fixes

• mbtiles-tools fix broken --show-json param (#304) - thanks @nyurik)
• Fixed render scheme in generate-vectortiles (#305) - thanks @kukulich

Clean ups

• #294 (thanks @nyurik), #299 (thanks @erictheise), #301 (thanks @mattcen)

Bugfix release: download-osm imposm config file

• Ensure trailing '/' at the end of replication_url in imposm config file #286 (#287) -- @frodrigo

Fix download-osm bug with some data src not having timestamps

download-osm make-dc for paris area calls osmconvert --out-statistics, which does not produce timestamp max, possibly due to the missing timestamps in the data. As the result, make-dc would crash. Fixing with #292 (@nyurik)

Fix bug with download-osm make-dc not working

Make sure download-osm make-dc ... runs. (#291) - @nyurik

Examine raw MVT tiles, perf, better .mbtiles support

Examine MVT tile content

debug-mvt and mbtiles-tools can now decode MVT tile content, either as a summary table or in an extended view, with optional language names lists.
The tile can be retrieved from Postgres (using layer definition yaml file), from a url/file/STDIN, or from an .mbtiles file (#273, #274) - @nyurik

$ debug-mvt dump "https://tiles.maps.elastic.co/data/v3/6/17/23.pbf" --summary
Uncompressed tile from https://tiles.maps.elastic.co/data/v3/6/17/23.pbf size=68,765 bytes, gzipped=39,095 bytes, 8 layers
Layer                  Extent    Ver    Features                   GeoType    GeoSize  AVG GeoSize    Fields (percentage only if not all features have it)                                        name:* fields
-------------------  --------  -----  ----------  ------------------------  ---------  -------------  ------------------------------------------------------------------------------------------  ---------------
water                    4096      2          29                   POLYGON     11,427  394.0          class, intermittent
waterway                 4096      2           6                LINESTRING        470  78.3           class
landuse                  4096      2          69                   POLYGON      1,287  18.7           class
park                     4096      2          90  POLYGON(51%), POINT(49%)      3,498  38.9           class, name, name_en, name_de, name_int, rank(49%)                                          21 languages
boundary                 4096      2         149                LINESTRING        924  6.2            admin_level, disputed, maritime
transportation           4096      2         745                LINESTRING      8,656  11.6           class
transportation_name      4096      2         182                LINESTRING      4,136  22.7           name_en, name_de, class, ref(99%), ref_length(99%), network(99%), name(55%), name_int(55%)  1 languages
place                    4096      2          24                     POINT         72  3.0            name, name_en, name_de, name_int, class, rank, capital(8%)                                  60 languages

• mbtiles-tools can now show .mbtiles tile content and metadata:

$ mbtiles-tools tile tiles.mbtiles 0/0/0 [--show-names]
  <output similar to the above>

$ mbtiles-tools meta-all tiles.mbtiles --show-ranges
  attribution <a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
  bounds      -180.0,-85.0511,180.0,85.0511
  center      0.0,0.0,4
  description A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
  filesize    192,512
  format      pbf
  id          openmaptiles
  maxzoom     7
  minzoom     0
  mtime       1590629964868 (2020-05-28T01:39:24.868000)
  name        OpenMapTiles
  pixel_scale 256
  version     3.11.0
  json        (The value is a valid JSON, use --show-json for raw dump)
  
  layer                  minZ    maxZ  fields                                                                                                                             description
  -------------------  ------  ------  ---------------------------------------------------------------------------------------------------------------------------------  -----------------------------------------
  water                     0      14  class, intermittent, brunnel                                                                                                       Water polygons representing oceans and l…
  waterway                  0      14  name, name_en, name_de, name_int, class, brunnel, intermittent, name:* (am,ar,az,be,bg,br,bs…)                                     OpenStreetMap [waterways](https://wiki.o…
  landcover                 0      14  class, subclass                                                                                                                    Landcover is used to describe the physic…
  landuse                   0      14  class                                                                                                                              Landuse is used to describe use of land …
  mountain_peak             0      14  osm_id, name, name_en, name_de, name_int, class, ele, ele_ft, rank, name:* (am,ar,az,be,bg,br,bs…)                                 [Natural peaks](http://wiki.openstreetma…
  park                      0      14  class, name, name_en, name_de, name_int, rank, name:* (am,ar,az,be,bg,br,bs…)                                                      The park layer contains parks from OpenS…
  boundary                  0      14  admin_level, disputed, disputed_name, claimed_by, maritime                                                                         Contains administrative boundaries as li…
  aeroway                   0      14  ref, class                                                                                                                         Aeroway polygons based of OpenStreetMap …
  transportation            0      14  class, subclass, oneway, ramp, brunnel, service, layer, level, indoor, bicycle, foot, horse, mtb_scale, surface                    **transportation** contains roads, railw…
  building                  0      14  osm_id, render_height, render_min_height, colour, hide_3d                                                                          All [OSM Buildings](http://wiki.openstre…
  water_name                0      14  osm_id, name, name_en, name_de, name_int, class, intermittent, name:* (am,ar,az,be,bg,br,bs…)                                      Lake center lines for labelling lake bod…
  transportation_name       0      14  name, name_en, name_de, name_int, ref, ref_length, network, class, subclass, layer, level, indoor, name:* (am,ar,az,be,bg,br,bs…)  This is the layer for labelling the high…
  place                     0      14  osm_id, name, name_en, name_de, name_int, class, rank, capital, iso_a2, name:* (am,ar,az,be,bg,br,bs…)                             The place layer consists out of [countri…
  housenumber               0      14  housenumber                                                                                                                        Everything in OpenStreetMap which contai…
  poi                       0      14  osm_id, name, name_en, name_de, name_int, class, subclass, agg_stop, layer, level, indoor, rank, name:* (am,ar,az,be,bg,br,bs…)    [Points of interests](http://wiki.openst…
  aerodrome_label           0      14  osm_id, name, name_en, name_de, name_int, class, iata, icao, ele, ele_ft, name:* (am,ar,az,be,bg,br,bs…)                           [Aerodrome labels](http://wiki.openstree…
  
    Zoom    Tile count  Found tile ranges
  ------  ------------  -------------------
       0             1  0,0 x 0,0
       1             1  1,1 x 1,1
       2             1  2,2 x 2,2
       3             1  4,5 x 4,5
       4             1  8,10 x 8,10
       5             1  16,20 x 16,20
       6             1  33,40 x 33,40
       7             1  66,81 x 66,81

BREAKING

• generate-metadata script has been removed. Use mbtiles-tools instead (#278) - @nyurik

mbtiles-tools

• New --show-ranges param shows mbtiles tile statistics (#279) - @nyurik
• Comprehensive testing for mbtiles-tools utility (#277) - @nyurik

postgis Docker image

• Cleanup postgis docker, bump osml10n ver (#276) - @nyurik
  • Upgrade osml10n PG extension to the faster v2.5.9 (significant performance improvements merged upstream by @nyurik)
  • parametrize base Postgis image version
  • misc cleanup

Misc

• The latest Imposm 0.10.0 is now packaged as a separate set of Docker images, tagged with -imposm10 suffix (#281) - @nyurik
• Add protoc to go-builder in tools (#272) - @nyurik
• Add less utility to the tools image (#275) - @nyurik
• As part of #273, several minor Python dependencies were bumped to the latest versions.
• download-osm --make-dc now adds openmaptiles-tools env vars in the generated docker-compose file (#283) - @nyurik
• Remove obsolete consts.py -- uses values from layer definition instead #284 - @nyurik
• nano was added to the tools container #285 - @nyurik

Bug fixes

• Unable to delete some metadata values using mbtiles-tools meta-set (#271) - @nyurik
• missing EXPIRETILES_DIR env value (#280) - @nyurik

v5.1 back-ported undeclared EXPIRETILES_DIR bugfix

Bugfix release: #282

v5.0 back-ported undeclared EXPIRETILES_DIR fix & ver

Bugfix release: #282 and incorrect version number was updated to 5.0.1