diff --git a/docs/Build/grest-changelog.md b/docs/Build/grest-changelog.md index d831bdcc9..a1c250658 100644 --- a/docs/Build/grest-changelog.md +++ b/docs/Build/grest-changelog.md @@ -1,5 +1,47 @@ # Koios gRest Changelog +## [1.1.1] - For all networks. + +This release primarily focuses on backend performance fixes and work with dbsync 13.2.0.2 - while also, we have started preparing compatibility with upcoming koios lite release, to make it a seamless swap for specific endpoints without any impact to consumers. There are no breaking (impact to existing columns or inputs) changes with this release, but we have retired 2 deprecated endpoints that were almost unused on mainnet. Due to the amount of backend changes in queries, there is a chance that we may have missed some data accuracy checks, and - hence - would like to test in non-mainnet networks first before marking final release. Accordingly, any testing/reports of data inconsistency would be welcome. + +### New endpoints added: +- `/asset_policy_mints` - List of mint/burn count for all assets minted under a policy [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- `/block_tx_info` - Equivalent of tx_info but uses blocks as inputs to fetch tx_info against all tx in the block[s] requested, also contains additional flags to control performance and output [#255](https://github.com/cardano-community/koios-artifacts/pull/255) +- `/cli_protocol_params` - Return protocl-parameters as returned by `cardano-cli` from Koios servers [#269](https://github.com/cardano-community/koios-artifacts/pull/269) + +### Data Input/Output Changes: +- Output - `/reserve_withdrawals` , `/treasury_withdrawals` - Add `earned_epoch` and `spendable_epoch` fields [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Output - `/block` - Add `parent_hash` field [#263](https://github.com/cardano-community/koios-artifacts/pull/263) +- Output - `/account_list` - Add `stake_address_hex` and `script_hash` fields [#263](https://github.com/cardano-community/koios-artifacts/pull/263) +- Output - `/asset_list` - Add `script_hash` field [#263](https://github.com/cardano-community/koios-artifacts/pull/263) +- Output - `/asset_summary` - Add `addresses` field [#263](https://github.com/cardano-community/koios-artifacts/pull/263) +- Output - `/asset_addresses` , `/asset_nft_address` and `/policy_asset_addresses` - Add `stake_address` field [#262](https://github.com/cardano-community/koios-artifacts/pull/262) +- Output - Fix `/script_utxos` as it was incorrectly returning object instead of array for asset_list [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Output - `/tx_info` - Add `plutus_contract` -> `spends_input` to `plutus_contracts` to point the input transaction being consumed by the script [#269](https://github.com/cardano-community/koios-artifacts/pull/269) + +### Deprecations: +- None + +### Retirements: +- `asset_address_list` and `asset_policy_info` endpoints are now retired, as they were marked as deprecated in Koios 1.0.10 , and we have seen it's usage to be negligible (only a single hit in 48 hours on mainnet while marking this release). [#269](https://github.com/cardano-community/koios-artifacts/pull/269) + +### Chores: +- Retire `stake_distribution_new_accounts` and `stake_snapshot_cache` cache, as we directly perform lookup on live tables for newly registered accounts [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Active stake cache no longer reads the logs, but instead relies on newly added `epoch_sync_progress table` [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Reduce asset_info_cache rollback lookup from 1000 to 250 [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Replace `consumed_by_tx_in_id` references in SQL by `consumed_by_tx_id` [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- `pool_history_cache` now breaks into populating 500 epochs at a time (on guildnet, this query used to run for hours against ~20K epochs) [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Accomodate splitting of `reward` table into `instant_reward` [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Add a check in `stake_distribution_cache` to ensure that epoch info cache was run for current - 1 epoch. [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Change return type for internal function `grest.cip67_strip_label` from `text` to `bytea` [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Remove any references to `tx_in` as it is no longer required [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Remove references to `pool_offline_data` with `off_chain_pool_data` [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- Disable running `asset-txo-cache-update` as the endpoints leveraging asset-txo will be moved to koios-lite +- Convert `block`, `account_list`, `asset_list` `asset_token_registry` from view to function [#263](https://github.com/cardano-community/koios-artifacts/pull/263) +- `asset_info_cache` - ensure mint tx is only against a positive mint quantity [#262](https://github.com/cardano-community/koios-artifacts/pull/262) +- Include burnt asset transactions in asset_txs [#269](https://github.com/cardano-community/koios-artifacts/pull/269) +- `tx_info` - Fix spend_redeemers CTE Join condition [#269](https://github.com/cardano-community/koios-artifacts/pull/269) + ## [1.1.0] - For all networks. This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (`/api/v1`). @@ -17,7 +59,6 @@ The major work with this release was to start making use of newer flags in dbsyn - Bump schema version for koios-1.1.0 [#250](https://github.com/cardano-community/koios-artifacts/pull/250) - Minor patch for output data type (`pool_registrations` and `pool_retirements`) [#249](https://github.com/cardano-community/koios-artifacts/pull/249) - ## [1.1.0rc] - For all networks. This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (`/api/v1`). diff --git a/scripts/grest-helper-scripts/grest-poll.sh b/scripts/grest-helper-scripts/grest-poll.sh index 9b4b611a2..f97321a25 100755 --- a/scripts/grest-helper-scripts/grest-poll.sh +++ b/scripts/grest-helper-scripts/grest-poll.sh @@ -133,7 +133,6 @@ function chk_cache_status() { last_stakedist_block=$(jq -r 'map(select(.key == "stake_distribution_lbh"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}") last_poolhist_update=$(jq -r 'map(select(.key == "pool_history_cache_last_updated"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}") last_actvstake_epoch=$(jq -r 'map(select(.key == "last_active_stake_validated_epoch"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}") - last_epoch_summary=$(jq -r 'map(select(.key == "last_epoch_summary_data_checked"))[0].last_value' 2>/dev/null <<< "${ctrl_tbl}") if [[ "${last_stakedist_block}" == "" ]] || [[ "${last_stakedist_block}" == "[]" ]] || [[ $(( block_no - last_stakedist_block )) -gt 2000 ]]; then log_err "Stake Distribution cache too far from tip !!" optexit @@ -149,14 +148,10 @@ function chk_cache_status() { [[ -z "${GENESIS_JSON}" ]] && GENESIS_JSON="${PARENT}"/../files/shelley-genesis.json epoch_length=$(jq -r .epochLength "${GENESIS_JSON}" 2>/dev/null) if [[ ${epoch_slot} -ge $(( epoch_length / 6 )) ]]; then - if [[ "${last_actvstake_epoch}" != "${epoch}" ]]; then + if [[ ${last_actvstake_epoch} -lt ${epoch} ]]; then log_err "Active Stake cache for epoch ${epoch} still not populated as of ${epoch_slot} slot, maximum tolerance was $(( epoch_length / 6 )) !!" optexit fi - if [[ "${last_epoch_summary}" != "${epoch}" ]]; then - log_err "Epoch Summary Cache for epoch ${epoch} still not populated as of ${epoch_slot} slot, maximum tolerance was $(( epoch_length / 6 )) !!" - optexit - fi fi fi } diff --git a/scripts/grest-helper-scripts/setup-grest.sh b/scripts/grest-helper-scripts/setup-grest.sh index 10946ee2c..38cf8d5b7 100755 --- a/scripts/grest-helper-scripts/setup-grest.sh +++ b/scripts/grest-helper-scripts/setup-grest.sh @@ -113,12 +113,12 @@ SGVERSION=v1.1.1 set_cron_variables() { local job=$1 [[ ${PGDATABASE} != cexplorer ]] && sed -e "s@DB_NAME=.*@DB_NAME=${PGDATABASE}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" - [[ ${job} == populate-next-epoch-nonce ]] && - sed -e "s@NWMAGIC=.*@NWMAGIC=${NWMAGIC}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" && - sed -e "s@EPOCH_LENGTH=.*@EPOCH_LENGTH=${EPOCH_LENGTH}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" && - sed -e "s@PROM_URL=.*@PROM_URL=http://${PROM_HOST}:${PROM_PORT}/metrics@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" - sed -e "s@CCLI=.*@CCLI=${CCLI}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" - sed -e "s@CARDANO_NODE_SOCKET_PATH=.*@CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH}@" -i "${CRON_SCRIPTS_DIR}/${job}.sh" + sed -e "s@NWMAGIC=.*@NWMAGIC=${NWMAGIC}@" \ + -e "s@EPOCH_LENGTH=.*@EPOCH_LENGTH=${EPOCH_LENGTH}@" \ + -e "s@PROM_URL=.*@PROM_URL=http://${PROM_HOST}:${PROM_PORT}/metrics@" \ + -e "s@CCLI=.*@CCLI=${CCLI}@" \ + -e "s@CARDANO_NODE_SOCKET_PATH=.*@CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH}@" \ + -i "${CRON_SCRIPTS_DIR}/${job}.sh" # update last modified date of all json files to trigger cron job to process all [[ -d "${HOME}/git/${CNODE_VNAME}-token-registry" ]] && find "${HOME}/git/${CNODE_VNAME}-token-registry" -mindepth 2 -maxdepth 2 -type f -name "*.json" -exec touch {} + } @@ -165,6 +165,10 @@ SGVERSION=v1.1.1 set_cron_variables "asset-info-cache-update" install_cron_job "asset-info-cache-update" "*/2 * * * *" + get_cron_job_executable "cli-protocol-params-update" + set_cron_variables "cli-protocol-params-update" + install_cron_job "cli-protocol-params-update" "*/5 * * * *" + # Only (legacy) testnet and mainnet asset registries supported # In absence of official messaging, current (soon to be reset) preprod/preview networks use same registry as testnet. TBC - once there is an update from IO on these # Possible future addition for the Guild network once there is a guild registry @@ -303,7 +307,7 @@ SGVERSION=v1.1.1 printf "\n[Re]Installing HAProxy.." pushd ~/tmp >/dev/null || err_exit major_v="2.9" - minor_v="2" + minor_v="6" haproxy_url="http://www.haproxy.org/download/${major_v}/src/haproxy-${major_v}.${minor_v}.tar.gz" if curl -sL -f -m ${CURL_TIMEOUT} -o haproxy.tar.gz "${haproxy_url}"; then tar xf haproxy.tar.gz &>/dev/null && rm -f haproxy.tar.gz @@ -436,7 +440,7 @@ SGVERSION=v1.1.1 backend ogmios balance first - http-request set-path \"%[path,regsub(^/api/v1/ogmios/,/)]\" + http-request set-path \"%[path,regsub(^/api/v1/ogmios.*,/)]\" option httpchk GET /health http-check expect status 200 default-server inter 20s fall 1 rise 2 @@ -485,7 +489,7 @@ SGVERSION=v1.1.1 WantedBy=multi-user.target EOF" printf "\n HAProxy Service" - [[ ! -f /usr/sbin/haproxy ]] && sudo bash -c "cat <<-EOF > /etc/systemd/system/${CNODE_VNAME}-haproxy.service + [[ -f /usr/sbin/haproxy ]] && sudo bash -c "cat <<-EOF > /etc/systemd/system/${CNODE_VNAME}-haproxy.service [Unit] Description=HAProxy Load Balancer After=network-online.target