Koios v1.1.1rc (#1740)

## Description  Update changelogs and setup-grest reference --------- Co-authored-by: Scitz0 <[email protected]>
cardano-community · May 4, 2024 · 22ca664 · 22ca664
1 parent 9c9e2b2
commit 22ca664
Show file tree

Hide file tree

Showing 3 changed files with 56 additions and 16 deletions.
diff --git a/docs/Build/grest-changelog.md b/docs/Build/grest-changelog.md
@@ -1,5 +1,47 @@
 # Koios gRest Changelog
 
+## [1.1.1] - For all networks.
+
+This release primarily focuses on backend performance fixes and work with dbsync 13.2.0.2 - while also, we have started preparing compatibility with upcoming koios lite release, to make it a seamless swap for specific endpoints without any impact to consumers. There are no breaking (impact to existing columns or inputs) changes with this release, but we have retired 2 deprecated endpoints that were almost unused on mainnet. Due to the amount of backend changes in queries, there is a chance that we may have missed some data accuracy checks, and - hence - would like to test in non-mainnet networks first before marking final release. Accordingly, any testing/reports of data inconsistency would be welcome.
+
+### New endpoints added:
+- `/asset_policy_mints` - List of mint/burn count for all assets minted under a policy [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- `/block_tx_info` - Equivalent of tx_info but uses blocks as inputs to fetch tx_info against all tx in the block[s] requested, also contains additional flags to control performance and output [#255](https://github.com/cardano-community/koios-artifacts/pull/255)
+- `/cli_protocol_params` - Return protocl-parameters as returned by `cardano-cli` from Koios servers [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+
+### Data Input/Output Changes:
+- Output - `/reserve_withdrawals` , `/treasury_withdrawals` - Add `earned_epoch` and `spendable_epoch` fields [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Output - `/block` - Add `parent_hash` field [#263](https://github.com/cardano-community/koios-artifacts/pull/263)
+- Output - `/account_list` - Add `stake_address_hex` and `script_hash` fields [#263](https://github.com/cardano-community/koios-artifacts/pull/263)
+- Output - `/asset_list` - Add `script_hash` field [#263](https://github.com/cardano-community/koios-artifacts/pull/263)
+- Output - `/asset_summary` - Add `addresses` field [#263](https://github.com/cardano-community/koios-artifacts/pull/263)
+- Output - `/asset_addresses` , `/asset_nft_address` and `/policy_asset_addresses` - Add `stake_address` field [#262](https://github.com/cardano-community/koios-artifacts/pull/262)
+- Output - Fix `/script_utxos` as it was incorrectly returning object instead of array for asset_list [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Output - `/tx_info` - Add `plutus_contract` -> `spends_input` to `plutus_contracts` to point the input transaction being consumed by the script [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+
+### Deprecations:
+- None
+
+### Retirements:
+- `asset_address_list` and `asset_policy_info` endpoints are now retired, as they were marked as deprecated in Koios 1.0.10 , and we have seen it's usage to be negligible (only a single hit in 48 hours on mainnet while marking this release). [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+
+### Chores:
+- Retire `stake_distribution_new_accounts` and `stake_snapshot_cache` cache, as we directly perform lookup on live tables for newly registered accounts [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Active stake cache no longer reads the logs, but instead relies on newly added `epoch_sync_progress table` [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Reduce asset_info_cache rollback lookup from 1000 to 250 [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Replace `consumed_by_tx_in_id` references in SQL by `consumed_by_tx_id` [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- `pool_history_cache` now breaks into populating 500 epochs at a time (on guildnet, this query used to run for hours against ~20K epochs) [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Accomodate splitting of `reward` table into `instant_reward` [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Add a check in `stake_distribution_cache` to ensure that epoch info cache was run for current - 1 epoch. [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Change return type for internal function `grest.cip67_strip_label` from `text` to `bytea` [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Remove any references to `tx_in` as it is no longer required [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Remove references to `pool_offline_data` with `off_chain_pool_data` [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- Disable running `asset-txo-cache-update` as the endpoints leveraging asset-txo will be moved to koios-lite
+- Convert `block`, `account_list`, `asset_list` `asset_token_registry` from view to function [#263](https://github.com/cardano-community/koios-artifacts/pull/263)
+- `asset_info_cache` - ensure mint tx is only against a positive mint quantity [#262](https://github.com/cardano-community/koios-artifacts/pull/262)
+- Include burnt asset transactions in asset_txs [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+- `tx_info` - Fix spend_redeemers CTE Join condition [#269](https://github.com/cardano-community/koios-artifacts/pull/269)
+
 ## [1.1.0] - For all networks.
 
 This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (`/api/v1`).
@@ -17,7 +59,6 @@ The major work with this release was to start making use of newer flags in dbsyn
 - Bump schema version for koios-1.1.0 [#250](https://github.com/cardano-community/koios-artifacts/pull/250)
 - Minor patch for output data type (`pool_registrations` and `pool_retirements`) [#249](https://github.com/cardano-community/koios-artifacts/pull/249)
 
-
 ## [1.1.0rc] - For all networks.
 
 This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (`/api/v1`).

diff --git a/scripts/grest-helper-scripts/grest-poll.sh b/scripts/grest-helper-scripts/grest-poll.sh
@@ -133,7 +133,6 @@ function chk_cache_status() {
   last_stakedist_block=$(jq -r 'map(select(.key == "stake_distribution_lbh"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}")
   last_poolhist_update=$(jq -r 'map(select(.key == "pool_history_cache_last_updated"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}")
   last_actvstake_epoch=$(jq -r 'map(select(.key == "last_active_stake_validated_epoch"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}")
-  last_epoch_summary=$(jq -r 'map(select(.key == "last_epoch_summary_data_checked"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}")
   if [[ "${last_stakedist_block}" == "" ]] || [[ "${last_stakedist_block}" == "[]" ]] || [[ $(( block_no - last_stakedist_block )) -gt 2000 ]]; then
     log_err "Stake Distribution cache too far from tip !!"
     optexit
@@ -149,14 +148,10 @@ function chk_cache_status() {
     [[ -z "${GENESIS_JSON}" ]] && GENESIS_JSON="${PARENT}"/../files/shelley-genesis.json
     epoch_length=$(jq -r .epochLength "${GENESIS_JSON}" 2>/dev/null)
     if [[ ${epoch_slot} -ge $(( epoch_length / 6 )) ]]; then
-      if [[ "${last_actvstake_epoch}" != "${epoch}" ]]; then
+      if [[ ${last_actvstake_epoch} -lt ${epoch} ]]; then
         log_err "Active Stake cache for epoch ${epoch} still not populated as of ${epoch_slot} slot, maximum tolerance was $(( epoch_length / 6 )) !!"
         optexit
       fi
-      if [[ "${last_epoch_summary}" != "${epoch}" ]]; then
-        log_err "Epoch Summary Cache for epoch ${epoch} still not populated as of ${epoch_slot} slot, maximum tolerance was $(( epoch_length / 6 )) !!"
-        optexit
-      fi
     fi
   fi
 }

diff --git a/scripts/grest-helper-scripts/setup-grest.sh b/scripts/grest-helper-scripts/setup-grest.sh
@@ -113,12 +113,12 @@ SGVERSION=v1.1.1
   set_cron_variables() {
     local job=$1
     [[ ${PGDATABASE} != cexplorer ]] && sed -e "s@DB_NAME=.*@DB_NAME=${PGDATABASE}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh"
-    [[ ${job} == populate-next-epoch-nonce ]] &&
-      sed -e "s@NWMAGIC=.*@NWMAGIC=${NWMAGIC}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" &&
-      sed -e "s@EPOCH_LENGTH=.*@EPOCH_LENGTH=${EPOCH_LENGTH}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" &&
-      sed -e "s@PROM_URL=.*@PROM_URL=http://${PROM_HOST}:${PROM_PORT}/metrics@" -i "${CRON_SCRIPTS_DIR}/${job}.sh"
-      sed -e "s@CCLI=.*@CCLI=${CCLI}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh"
-      sed -e "s@CARDANO_NODE_SOCKET_PATH=.*@CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh"
+    sed -e "s@NWMAGIC=.*@NWMAGIC=${NWMAGIC}@" \
+      -e "s@EPOCH_LENGTH=.*@EPOCH_LENGTH=${EPOCH_LENGTH}@" \
+      -e "s@PROM_URL=.*@PROM_URL=http://${PROM_HOST}:${PROM_PORT}/metrics@" \
+      -e "s@CCLI=.*@CCLI=${CCLI}@" \
+      -e "s@CARDANO_NODE_SOCKET_PATH=.*@CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH}@" \
+      -i "${CRON_SCRIPTS_DIR}/${job}.sh"
     # update last modified date of all json files to trigger cron job to process all
     [[ -d "${HOME}/git/${CNODE_VNAME}-token-registry" ]] && find "${HOME}/git/${CNODE_VNAME}-token-registry" -mindepth 2 -maxdepth 2 -type f -name "*.json" -exec touch {} +
   }
@@ -165,6 +165,10 @@ SGVERSION=v1.1.1
     set_cron_variables "asset-info-cache-update"
     install_cron_job "asset-info-cache-update" "*/2 * * * *"
 
+    get_cron_job_executable "cli-protocol-params-update"
+    set_cron_variables "cli-protocol-params-update"
+    install_cron_job "cli-protocol-params-update" "*/5 * * * *"
+
     # Only (legacy) testnet and mainnet asset registries supported
     # In absence of official messaging, current (soon to be reset) preprod/preview networks use same registry as testnet. TBC - once there is an update from IO on these
     # Possible future addition for the Guild network once there is a guild registry
@@ -303,7 +307,7 @@ SGVERSION=v1.1.1
     printf "\n[Re]Installing HAProxy.."
     pushd ~/tmp >/dev/null || err_exit
     major_v="2.9"
-    minor_v="2"
+    minor_v="6"
     haproxy_url="http://www.haproxy.org/download/${major_v}/src/haproxy-${major_v}.${minor_v}.tar.gz"
     if curl -sL -f -m ${CURL_TIMEOUT} -o haproxy.tar.gz "${haproxy_url}"; then
       tar xf haproxy.tar.gz &>/dev/null && rm -f haproxy.tar.gz
@@ -436,7 +440,7 @@ SGVERSION=v1.1.1
 			
 			backend ogmios
 			  balance first
-			  http-request set-path \"%[path,regsub(^/api/v1/ogmios/,/)]\"
+			  http-request set-path \"%[path,regsub(^/api/v1/ogmios.*,/)]\"
 			  option httpchk GET /health
 			  http-check expect status 200
 			  default-server inter 20s fall 1 rise 2
@@ -485,7 +489,7 @@ SGVERSION=v1.1.1
 			WantedBy=multi-user.target
 			EOF"
     printf "\n  HAProxy Service"
-    [[ ! -f /usr/sbin/haproxy ]] && sudo bash -c "cat <<-EOF > /etc/systemd/system/${CNODE_VNAME}-haproxy.service
+    [[ -f /usr/sbin/haproxy ]] && sudo bash -c "cat <<-EOF > /etc/systemd/system/${CNODE_VNAME}-haproxy.service
 			[Unit]
 			Description=HAProxy Load Balancer
 			After=network-online.target