Skip to content
Carmen Tawalika edited this page Nov 12, 2021 · 9 revisions

concept (70/100)

Decisions for prototypical implementation

  • location in which to calculate is determined by load_collection process from openEO, or user depending on which endpoint is called
  • actinia is responsible for reading in the needed collections, applying filters (space + time) to collect needed items and create the process chain for GRASS GIS
  • see more details for filters here. It is necessary to support the filters first, which reduce data size best
  • no development is needed in GRASS GIS
  • r.in.gdal (r.import) will be used for all imports. Actinia will pass information (data location, bands, extent, ...)
  • single scenes are registered directly as STRDS (no mosaic for prototype) -> no real difference for the user, it might take longer and use more storage space

Implementation in actinia

  • STAC will be a new type of importer (import_descr)
  • this needs to be enabled within actinia-core
  • as soon as possible, a dedicated plugin should be called (actinia-stac-plugin or actinia-stac-importer-plugin)
    • plugin should do as much as possible (as little as possible inside actinia-core)
    • plugin should be called and only return process (or process chain piece) to actinia-core
    • plugin should download to tmp folder / resource storage, where data will be deleted after processing / can be deleted manually -> look at other import options like sentinel
  • for a first draft it is fine to do everything within actinia and move to plugin afterwards

Relevant after prototype

General

  • Reading of data has nothing to do with STAC but depends on where the items are and how they can be retrieved
  • All needed data for retrieving it needs to be in STAC Metadata
  • Type of storage (file mount, S3, copernicus API hub, ...)
    • will be implemented on-demand (first for one example, one after another for different storage types / STAC instances / collections
    • Determined by analysing path to resource (e.g. content of item.assets.B01.href)
  • Use common band names where possible when registering data
    • Implement look-up table for "common not common band names", eg as seen here

Pre-Checks within actinia

  • only needed extent should be downloaded if possible - should be possible with COG
  • pass Bandinformation
  • in later stage, choose special GRASS GIS modules for metadata import (e.g. i.sentinel)

Decision-making

In which location will be calculated?

how it is now:

  • usually determined by which endpoint is called (/api/v1/locations/nc_spm_08/processing_async_export)
  • currently, location is determined by openeo-grassgis-driver depending on location of input dataset
    • what, if multiple inputs are used from different locations?
  • currently, user is not allowed to create location, only mapser. Admin is allowed
    • TODO: enable user to create locations but with limit configurable -> check if temporary location is possible

possible solutions

  • not nice: change projection in current location on-the-fly
  • Analyse input data and decide on this basis, in which location the processing should be done
    • consider projection of assets and filter spatially, so that location is clear before real processing starts
  • in prototypical implementation, reduce complexity:
    • user should decide via openeo process load_collections
      • it is required to do any processing with openEO
      • Default for CRS for BBOX filter is 4326
      • It is meant to define BBOX for filter, not data, so it could confuse user more than it helps
      • OpenEO API allows defining it, but can it be also manipulated with web editor?

Read data by GRASS GIS or actinia?

  • Should GRASS GIS be able to read STAC?
    • -> would be nice to have but currently no real requirement
    • for each data type, a different module would be used -> a lot of metadata analysis is needed
    • could be done by GRASS GIS addon (r.in.stac / t.in.stac ?) input would then be a URL to a Feature / FeatureCollection with bands. Example
  • Should actinia analyse the data?
    • If done via Importer, data would be cleaned
    • actinia is able to do everything that GRASS GIS can do, including forward to dedicated GRASS GIS import modules -> no disadvantage
    • Advantage of actinia: analysis of input data location can be done before processing
  • => Decision for actinia-stac-plugin/actinia-stac-importer-plugin

How will the data be imported into GRASS GIS?

  • r.in.gdal (r.import) / r.external / special modules like i.sentinel / viscurl ?
  • differentiate between storage (mount, local copy for landsat + Sentinel Copernicus, use GDAL for S3)
  • prevent download where possible?
  • where is the data stored? How large are the extents?
  • do we need to determine by pixel limit?
    • little extent => r.exernal
    • if gdalinfo is able to read => r.external, else r.in.gdal
  • => r.in.gdal (r.import) wins for now. If we have problems in certain use cases later, we can use r.external for them

when / how a mosaic will be build?

  • For prototypical implementation, we focus on single scenes/bands
  • actinia downloads everything, location is determined by load_collection (used actinia endpoint) and single scenes are registered directly as STRDS