From e996aeece38abebcd84358acc836b30240bc78d5 Mon Sep 17 00:00:00 2001 From: illuminatus Date: Sat, 4 May 2024 08:55:09 -0700 Subject: [PATCH] [mithril] Fix bug in checkUpdate / Fix export for signer / Fix PARTY_ID for verification / Add mithril client download to non container docs (#1761) 1. Includes a syntax fix for #1757 2. Removes unecessary UPDATE_CHECK=N exports in the workflow now that it is a default value in the Dockerfile ENV / Image. 3. Exports G_ACCOUNT before calls to `./guild-deploy.sh` to ensure if images are built in a fork that the files come from the fork, not **cardano-community**. 4. Sets default G_ACCOUNT and GUILD_DEPLOY_BRANCH so manual `docker build` commands do not require passing `--build-arg`'s except to alter default values. 5. Fixes a bug leftover in the update logic and abstracts common functions from mithril scripts into a new mithril.library sourced by the scripts, reduces maintenance overhead and code duplication. 6. Prepares for #1744 by getting the minimum versions for cardano node from mithrils new `networks.json` file. 7. Implements changes suggested by @Fuma419 since branch node-8.10.0 will merge into alpha, but not node-8.9.0. 8. Fixes a bug leftover in the update logic ### 1756 update bug * This changes the call to use `$(basename "$0")` for any instance of a mithril script calling checkUpdate for itself, should future proof against script renames. * Also runs checkUpdate for `env` and `mithril.library`. ### 1759 changes to mithril-client * The cardano-db is now the base command * snapshot is relocated inside cardano-db * download is relocated from snapshot directly into cardano-db * Implemented cleanup on crashes as well as user interupts. A new node should either have a full copy of the most recent snapshot, or be empty so `cnode.sh` has no corruption or incomplete files preventing a good sync. * This also changes `mithril-signer.sh` * Existing flags in `mithril-signer.sh` changed to have a better overlap in mithril scripts using similar OPTARGS (-u always skip updates, etc.). * New flags implemented that integrates the upstream mithril scripts to Verify signer registration and signer signatures. Documentation has been updated to account for changes in the scripts usage, flags, commands and subcommands. Also the MITHRIL_DOWNLOAD variable is documented into the `node-cli.md`. closes #1756 closes #1757 closes #1759 --- docs/Build/node-cli.md | 2 +- docs/Scripts/mithril-client.md | 41 ++- docs/Scripts/mithril-relay.md | 11 +- docs/Scripts/mithril-signer.md | 17 +- files/docker/node/addons/entrypoint.sh | 39 +-- files/docker/node/dockerfile_bin | 2 - scripts/cnode-helper-scripts/cnode.sh | 4 +- scripts/cnode-helper-scripts/guild-deploy.sh | 1 + .../cnode-helper-scripts/mithril-client.sh | 256 ++++++----------- scripts/cnode-helper-scripts/mithril-relay.sh | 179 +++++++----- .../cnode-helper-scripts/mithril-signer.sh | 272 ++++++++++-------- scripts/cnode-helper-scripts/mithril.library | 196 +++++++++++++ 12 files changed, 612 insertions(+), 408 deletions(-) create mode 100644 scripts/cnode-helper-scripts/mithril.library diff --git a/docs/Build/node-cli.md b/docs/Build/node-cli.md index 8ec0f5718..c41075c4b 100644 --- a/docs/Build/node-cli.md +++ b/docs/Build/node-cli.md @@ -63,7 +63,7 @@ POOL_NAME="GUILD" #### Start the node -To test starting the node in interactive mode, we will make use of pre-built script `cnode.sh`. This script automatically determines whether to start the node as a relay or block producer (if the required pool keys are present in the `$CNODE_HOME/priv/pool/` as mentioned above). The script contains a user-defined variable `CPU_CORES` which determines the number of CPU cores the node will use upon start-up: +To test starting the node in interactive mode, we will make use of pre-built script `cnode.sh`. This script automatically determines whether to start the node as a relay or block producer (if the required pool keys are present in the `$CNODE_HOME/priv/pool/` as mentioned above). If the `` variable is set to 'Y' it will download the latest snapshot from a Mithril aggregator to speed up the blockchain synchronization. The script contains a user-defined variable `CPU_CORES` which determines the number of CPU cores the node will use upon start-up: ```bash ###################################### diff --git a/docs/Scripts/mithril-client.md b/docs/Scripts/mithril-client.md index 2e4b1f960..32beab383 100644 --- a/docs/Scripts/mithril-client.md +++ b/docs/Scripts/mithril-client.md @@ -1,10 +1,37 @@ `mithril-client.sh` is a script to manage the Mithril client, a tool used to set up the Mithril client environment and manage downloading Mithril snapshots and stake distributions. The main features include: - **environment** - Creates a new `mithril.env` file with all the necessary environment variables for the Mithril client. -- **snapshot** - Download, list all or show a specific available Mithril snapshot. +- **cardano-db** - Download, list all or show a specific available Mithril snapshot. - **stake-distribution** - Download or list available Mithril stake distributions. - **-u** - Skip script update check. +## Usage + +```bash +Usage: bash [-u] [] +A script to run Cardano Mithril Client + +-u Skip script update check overriding UPDATE_CHECK value in env (must be first argument to script) + +Commands: +environment Manage mithril environment file + setup Setup mithril environment file + override Override default variable in the mithril environment file + update Update mithril environment file +cardano-db Interact with Cardano DB + download Download Cardano DB from Mithril snapshot + snapshot Interact with Mithril snapshots + list List available Mithril snapshots + json List availble Mithril snapshots in JSON format + show Show details of a Mithril snapshot + json Show details of a Mithril snapshot in JSON format +stake-distribution Interact with Mithril stake distributions + download Download latest stake distribution + list List available stake distributions + json Output latest Mithril snapshot in JSON format + +``` + ## Preparing a Relay or Block Producer Node To prepare a relay or block producer node, you should follow these steps: @@ -18,7 +45,7 @@ To prepare a relay or block producer node, you should follow these steps: 2. **Download the latest Mithril snapshot:** Once the environment file is set up, you can download the latest Mithril snapshot by running the script with the `snapshot download` command. This snapshot contains the latest state of the Cardano blockchain db from a Mithril Aggregator. ```bash - ./mithril-client.sh snapshot download + ./mithril-client.sh cardano-db download ``` ## Investigating Available Snapshots @@ -28,16 +55,16 @@ You can investigate the available snapshots by using the `snapshot list` and `sn - **List all available Mithril snapshots:** You can list all available Mithril snapshots by running the script with the `snapshot list` command. Add `json` at the end to get the output in JSON format. ```bash - ./mithril-client.sh snapshot list - ./mithril-client.sh snapshot list json + ./mithril-client.sh cardano-dbsnapshot list + ./mithril-client.sh cardano-dbsnapshot list json ``` - **Show details of a specific Mithril snapshot:** You can show details of a specific Mithril snapshot by running the script with the `snapshot show ` command, where `` is the digest of the snapshot. Add `json` at the end to get the output in JSON format. ```bash - ./mithril-client.sh snapshot show - ./mithril-client.sh snapshot show json - ./mithril-client.sh snapshot show json + ./mithril-client.sh cardano-dbsnapshot show + ./mithril-client.sh cardano-dbsnapshot show json + ./mithril-client.sh cardano-dbsnapshot show json ``` ## Managing Stake Distributions diff --git a/docs/Scripts/mithril-relay.md b/docs/Scripts/mithril-relay.md index a7713274a..01454779f 100644 --- a/docs/Scripts/mithril-relay.md +++ b/docs/Scripts/mithril-relay.md @@ -7,12 +7,13 @@ loadbalancer. It provides functionalities such as: ## Usage ```bash -Usage: mithril-relay.sh [-d] [-l] +bash [-d] [-l] [-u] [-h] +A script to setup Cardano Mithril relays -Options: - -d Install squid and configure as a relay - -l Install nginx and configure as a load balancer - -h Show this help text +-d Install squid and configure as a relay +-l Install nginx and configure as a load balancer +-u Skip update check +-h Show this help text ``` # Description diff --git a/docs/Scripts/mithril-signer.md b/docs/Scripts/mithril-signer.md index 1c846d433..4e28d4e6c 100644 --- a/docs/Scripts/mithril-signer.md +++ b/docs/Scripts/mithril-signer.md @@ -5,12 +5,17 @@ environment file to contain variables specific to the Mithril Signer. ## Usage ```bash -Usage: mithril-signer.sh [-d] [-u] - -Options: - -d Deploy mithril-signer as a systemd service - -u Update mithril environment file - -h Show this help text +Usage: bash [-d] [-D] [-e] [-k] [-r] [-s] [-u] [-h] +A script to setup, run and verify Cardano Mithril Signer + +-d Deploy mithril-signer as a systemd service +-D Run mithril-signer as a daemon +-e Update mithril environment file +-k Stop signer using SIGINT +-r Verify signer registration +-s Verify signer signature +-u Skip update check +-h Show this help text ``` # Description diff --git a/files/docker/node/addons/entrypoint.sh b/files/docker/node/addons/entrypoint.sh index e338c0f0a..5915223f7 100755 --- a/files/docker/node/addons/entrypoint.sh +++ b/files/docker/node/addons/entrypoint.sh @@ -6,6 +6,7 @@ trap 'killall -s SIGINT cardano-node' SIGINT SIGTERM head -n 8 ~/.scripts/banner.txt +# shellcheck disable=SC1090 . ~/.bashrc > /dev/null 2>&1 echo "NETWORK: $NETWORK $POOL_NAME $TOPOLOGY"; @@ -17,29 +18,29 @@ echo "NODE: $HOSTNAME - Port:$CNODE_PORT - $POOL_NAME"; cardano-node --version; if [[ "${ENABLE_BACKUP}" == "Y" ]] || [[ "${ENABLE_RESTORE}" == "Y" ]]; then - [[ ! -d "${CNODE_HOME}"/backup/$NETWORK-db ]] && mkdir -p $CNODE_HOME/backup/$NETWORK-db - dbsize=$(du -s $CNODE_HOME/db | awk '{print $1}') - bksizedb=$(du -s $CNODE_HOME/backup/$NETWORK-db 2>/dev/null | awk '{print $1}') - if [[ "${ENABLE_RESTORE}" == "Y" ]] && [[ "$dbsize" -lt "$bksizedb" ]]; then - echo "Backup Started" - cp -rf "${CNODE_HOME}"/backup/"${NETWORK}"-db/* "${CNODE_HOME}"/db 2>/dev/null - echo "Backup Finished" - fi - - if [[ "${ENABLE_BACKUP}" == "Y" ]] && [[ "$dbsize" -gt "$bksizedb" ]]; then - echo "Restore Started" - cp -rf "${CNODE_HOME}"/db/* "${CNODE_HOME}"/backup/"${NETWORK}"-db/ 2>/dev/null - echo "Restore Finished" - fi + [[ ! -d "${CNODE_HOME}"/backup/$NETWORK-db ]] && mkdir -p $CNODE_HOME/backup/$NETWORK-db + dbsize=$(du -s $CNODE_HOME/db | awk '{print $1}') + bksizedb=$(du -s $CNODE_HOME/backup/$NETWORK-db 2>/dev/null | awk '{print $1}') + if [[ "${ENABLE_RESTORE}" == "Y" ]] && [[ "$dbsize" -lt "$bksizedb" ]]; then + echo "Backup Started" + cp -rf "${CNODE_HOME}"/backup/"${NETWORK}"-db/* "${CNODE_HOME}"/db 2>/dev/null + echo "Backup Finished" + fi + + if [[ "${ENABLE_BACKUP}" == "Y" ]] && [[ "$dbsize" -gt "$bksizedb" ]]; then + echo "Restore Started" + cp -rf "${CNODE_HOME}"/db/* "${CNODE_HOME}"/backup/"${NETWORK}"-db/ 2>/dev/null + echo "Restore Finished" + fi fi # Customisation customise () { -find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/127.0.0.1/0.0.0.0/g' > /dev/null 2>&1 -grep -i ENABLE_CHATTR /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_CHATTR=(true|false)?/ENABLE_CHATTR=false/g' /opt/cardano/cnode/scripts/cntools.sh > /dev/null 2>&1 -grep -i ENABLE_DIALOG /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_DIALOG=(true|false)?/ENABLE_DIALOG=false/' /opt/cardano/cnode/scripts/cntools.sh >> /opt/cardano/cnode/scripts/cntools.sh -find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/\"hasEKG\": 12788,/\"hasEKG\": [\n \"0.0.0.0\",\n 12788\n],/g' > /dev/null 2>&1 -return 0 + find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/127.0.0.1/0.0.0.0/g' > /dev/null 2>&1 + grep -i ENABLE_CHATTR /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_CHATTR=(true|false)?/ENABLE_CHATTR=false/g' /opt/cardano/cnode/scripts/cntools.sh > /dev/null 2>&1 + grep -i ENABLE_DIALOG /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_DIALOG=(true|false)?/ENABLE_DIALOG=false/' /opt/cardano/cnode/scripts/cntools.sh > /dev/null 2>&1 + find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/\"hasEKG\": 12788,/\"hasEKG\": [\n \"0.0.0.0\",\n 12788\n],/g' > /dev/null 2>&1 + return 0 } load_configs () { diff --git a/files/docker/node/dockerfile_bin b/files/docker/node/dockerfile_bin index b9aec97a4..9782d4c14 100644 --- a/files/docker/node/dockerfile_bin +++ b/files/docker/node/dockerfile_bin @@ -44,7 +44,6 @@ RUN set -x && apt update \ && mkdir -p /root/.local/bin \ && wget https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators/${GUILD_DEPLOY_BRANCH}/scripts/cnode-helper-scripts/guild-deploy.sh \ && export SUDO='N' \ - && export UPDATE_CHECK='N' \ && export SKIP_DBSYNC_DOWNLOAD='Y' \ && export G_ACCOUNT=${G_ACCOUNT} \ && chmod +x ./guild-deploy.sh && ./guild-deploy.sh -b ${GUILD_DEPLOY_BRANCH} -s p \ @@ -57,7 +56,6 @@ RUN set -x && apt update \ && rm -rf /var/lib/apt/lists/* RUN set -x && export SUDO='N' \ - && export UPDATE_CHECK='N' \ && export SKIP_DBSYNC_DOWNLOAD='Y' \ && export G_ACCOUNT=${G_ACCOUNT} \ && ./guild-deploy.sh -b ${GUILD_DEPLOY_BRANCH} -s dcmowx \ diff --git a/scripts/cnode-helper-scripts/cnode.sh b/scripts/cnode-helper-scripts/cnode.sh index 865bdb2b9..720f88ade 100755 --- a/scripts/cnode-helper-scripts/cnode.sh +++ b/scripts/cnode-helper-scripts/cnode.sh @@ -65,9 +65,9 @@ pre_startup_sanity() { mithril_snapshot_download() { [[ -z "${MITHRIL_CLIENT}" ]] && MITHRIL_CLIENT="${CNODE_HOME}"/scripts/mithril-client.sh if [[ ! -f "${MITHRIL_CLIENT}" ]] || [[ ! -e "${MITHRIL_CLIENT}" ]]; then - echo "ERROR: Could not locate mithril-client.sh script or script is not executable. Skipping mithril snapshot download!!" + echo "ERROR: Could not locate mithril-client.sh script or script is not executable. Skipping mithril cardano-db snapshot download!!" else - "${MITHRIL_CLIENT}" snapshot download + "${MITHRIL_CLIENT}" cardano-db download fi } diff --git a/scripts/cnode-helper-scripts/guild-deploy.sh b/scripts/cnode-helper-scripts/guild-deploy.sh index a746257b2..d69a8c6b5 100755 --- a/scripts/cnode-helper-scripts/guild-deploy.sh +++ b/scripts/cnode-helper-scripts/guild-deploy.sh @@ -638,6 +638,7 @@ populate_cnode() { updateWithCustomConfig "mithril-client.sh" updateWithCustomConfig "mithril-relay.sh" updateWithCustomConfig "mithril-signer.sh" + updateWithCustomConfig "mithril.library" find "${CNODE_HOME}/scripts" -name '*.sh' -exec chmod 755 {} \; 2>/dev/null chmod -R 700 "${CNODE_HOME}"/priv 2>/dev/null diff --git a/scripts/cnode-helper-scripts/mithril-client.sh b/scripts/cnode-helper-scripts/mithril-client.sh index 54c9a140c..9c59716b0 100755 --- a/scripts/cnode-helper-scripts/mithril-client.sh +++ b/scripts/cnode-helper-scripts/mithril-client.sh @@ -2,7 +2,7 @@ # shellcheck disable=SC2086 #shellcheck source=/dev/null -. "$(dirname $0)"/env offline +. "$(dirname $0)"/mithril.library ###################################### # User Variables - Change as desired # @@ -15,9 +15,6 @@ # Do NOT modify code below # ###################################### -U_ID=$(id -u) -G_ID=$(id -g) - ##################### # Functions # ##################### @@ -25,8 +22,8 @@ G_ID=$(id -g) usage() { cat <<-EOF - Usage: $(basename "$0") - Script to run Cardano Mithril Client + Usage: $(basename "$0") [-u] [] + A script to run Cardano Mithril Client -u Skip script update check overriding UPDATE_CHECK value in env (must be first argument to script) @@ -35,12 +32,13 @@ usage() { setup Setup mithril environment file override Override default variable in the mithril environment file update Update mithril environment file - snapshot Interact with Mithril snapshots - download Download latest Mithril snapshot - list List available Mithril snapshots - json List availble Mithril snapshots in JSON format - show Show details of a Mithril snapshot - json Show details of a Mithril snapshot in JSON format + cardano-db Interact with Cardano DB + download Download Cardano DB from Mithril snapshot + snapshot Interact with Mithril snapshots + list List available Mithril snapshots + json List availble Mithril snapshots in JSON format + show Show details of a Mithril snapshot + json Show details of a Mithril snapshot in JSON format stake-distribution Interact with Mithril stake distributions download Download latest stake distribution list List available stake distributions @@ -50,62 +48,10 @@ EOF } SKIP_UPDATE=N -[[ $1 = "-u" ]] && SKIP_UPDATE=Y && shift +[[ $1 = "-u" ]] && export SKIP_UPDATE=Y && shift ## mithril environment subcommands -environment_setup() { - local env_file="${CNODE_HOME}/mithril/mithril.env" - - if [[ -f "$env_file" ]]; then - if [[ "$UPDATE_ENVIRONMENT" != "Y" ]]; then - echo "Error: $env_file already exists. To update it, set UPDATE_ENVIRONMENT to 'Y'." >&2 - return 1 - else - echo "Updating $env_file..." - fi - else - echo "Creating $env_file..." - fi - - if [[ ! -d "${CNODE_HOME}/mithril/data-stores" ]]; then - sudo mkdir -p "${CNODE_HOME}"/mithril/data-stores - sudo chown -R "$U_ID":"$G_ID" "${CNODE_HOME}"/mithril 2>/dev/null - fi - if [[ -n "${POOL_NAME}" ]] && [[ "${POOL_NAME}" != "CHANGE_ME" ]]; then - export ERA_READER_ADDRESS=https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/era.addr - export ERA_READER_VKEY=https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/era.vkey - bash -c "cat <<-'EOF' > ${CNODE_HOME}/mithril/mithril.env - KES_SECRET_KEY_PATH=${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME} - OPERATIONAL_CERTIFICATE_PATH=${POOL_DIR}/${POOL_OPCERT_FILENAME} - NETWORK=${NETWORK_NAME,,} - RELEASE=${RELEASE} - AGGREGATOR_ENDPOINT=https://aggregator.${RELEASE}-${NETWORK_NAME,,}.api.mithril.network/aggregator - RUN_INTERVAL=60000 - DB_DIRECTORY=${CNODE_HOME}/db - CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH} - CARDANO_CLI_PATH=${HOME}/.local/bin/cardano-cli - DATA_STORES_DIRECTORY=${CNODE_HOME}/mithril/data-stores - STORE_RETENTION_LIMITS=5 - ERA_READER_ADAPTER_TYPE=cardano-chain - ERA_READER_ADAPTER_PARAMS=$(jq -nc --arg address "$(wget -q -O - "${ERA_READER_ADDRESS}")" --arg verification_key "$(wget -q -O - "${ERA_READER_VKEY}")" '{"address": $address, "verification_key": $verification_key}') - GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/genesis.vkey) - PARTY_ID=$(cat ${POOL_DIR}/${POOL_ID_FILENAME}) - SNAPSHOT_DIGEST=latest - EOF" - else - bash -c "cat <<-'EOF' > ${CNODE_HOME}/mithril/mithril.env - NETWORK=${NETWORK_NAME,,} - RELEASE=${RELEASE} - AGGREGATOR_ENDPOINT=https://aggregator.${RELEASE}-${NETWORK_NAME,,}.api.mithril.network/aggregator - DB_DIRECTORY=${CNODE_HOME}/db - GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/genesis.vkey) - SNAPSHOT_DIGEST=latest - EOF" - fi - chown $USER:$USER "${CNODE_HOME}"/mithril/mithril.env -} - environment_override() { local var_to_override="$1" local new_value="$2" @@ -121,96 +67,36 @@ environment_override() { sed -i "s|^${var_to_override}=.*|${var_to_override}=${new_value}|" "$env_file" } -pre_startup_sanity() { - if [[ ${UPDATE_CHECK} = Y && ${SKIP_UPDATE} != Y ]]; then - - echo "Checking for script updates..." - - # Check availability of checkUpdate function - if [[ ! $(command -v checkUpdate) ]]; then - echo -e "\nCould not find checkUpdate function in env, make sure you're using official guild docos for installation!" - exit 1 - fi - - # check for env update - ENV_UPDATED=${BATCH_AUTO_UPDATE} - checkUpdate "${PARENT}"/env N N N - case $? in - 1) ENV_UPDATED=Y ;; - 2) exit 1 ;; - esac - - # check for cncli.sh update - checkUpdate "${PARENT}"/cncli.sh ${ENV_UPDATED} - case $? in - 1) $0 "-u" "$@"; exit 0 ;; # re-launch script with same args skipping update check - 2) exit 1 ;; - esac - fi - - REQUIRED_PARAMETERS="Y" - if [[ ! -f "${CNODE_HOME}"/mithril/mithril.env ]]; then - echo "INFO: Mithril environment file not found, creating environment file.." - environment_setup && echo "INFO: Mithril environment file created successfully!!" - elif [[ "${UPDATE_ENVIRONMENT}" == "Y" ]]; then - echo "INFO: Updating mithril environment file.." - environment_setup && echo "INFO: Mithril environment file updated successfully!!" - fi +mithril_init() { + [[ ! -f "${CNODE_HOME}"/mithril/mithril.env ]] && generate_environment_file . "${CNODE_HOME}"/mithril/mithril.env - [[ -z "${NETWORK}" ]] && echo "ERROR: The NETWORK must be set before calling mithril-client!!" && REQUIRED_PARAMETERS="N" - [[ -z "${RELEASE}" ]] && echo "ERROR: Failed to set RELEASE variable, please check NETWORK variable in env file!!" && REQUIRED_PARAMETERS="N" - [[ -z "${CNODE_HOME}" ]] && echo "ERROR: The CNODE_HOME must be set before calling mithril-client!!" && REQUIRED_PARAMETERS="N" - [[ ! -d "${CNODE_HOME}" ]] && echo "ERROR: The CNODE_HOME directory does not exist, please check CNODE_HOME variable in env file!!" && REQUIRED_PARAMETERS="N" - [[ -z "${AGGREGATOR_ENDPOINT}" ]] && echo "ERROR: The AGGREGATOR_ENDPOINT must be set before calling mithril-client!!" && REQUIRED_PARAMETERS="N" - [[ -z "${GENESIS_VERIFICATION_KEY}" ]] && echo "ERROR: The GENESIS_VERIFICATION_KEY must be set before calling mithril-client!!" && REQUIRED_PARAMETERS="N" - [[ ! -x "${MITHRILBIN}" ]] && echo "ERROR: The MITHRILBIN variable does not contain an executable file, please check MITHRILBIN variable in env file!!" && REQUIRED_PARAMETERS="N" - [[ "${REQUIRED_PARAMETERS}" != "Y" ]] && exit 1 - export GENESIS_VERIFICATION_KEY - DOWNLOAD_SNAPSHOT="N" - REMOVE_DB_DIR="N" } -set_defaults() { - [[ -z "${MITHRILBIN}" ]] && MITHRILBIN="${HOME}"/.local/bin/mithril-client - if [[ -z "${NETWORK_NAME}" ]]; then - echo "ERROR: The NETWORK_NAME must be set before mithril-client can download snapshots!!" - exit 1 - else - case "${NETWORK_NAME,,}" in - mainnet|preprod|guild) - RELEASE="release" - ;; - preview) - RELEASE="pre-release" - ;; - *) - echo "ERROR: The NETWORK_NAME must be set to Mainnet, PreProd, Preview, Guild before mithril-client can download snapshots!!" - exit 1 - esac - fi - pre_startup_sanity -} check_db_dir() { # If the DB directory does not exist then set DOWNLOAD_SNAPSHOT to Y if [[ ! -d "${DB_DIRECTORY}" ]]; then echo "INFO: The db directory does not exist.." DOWNLOAD_SNAPSHOT="Y" - # If the DB directory is empty then set DOWNLOAD_SNAPSHOT to Y and REMOVE_DB_DIR to Y + # If the DB directory is empty then set DOWNLOAD_SNAPSHOT to Y elif [[ -d "${DB_DIRECTORY}" ]] && [[ -z "$(ls -A "${DB_DIRECTORY}")" ]] && [[ $(du -cs "${DB_DIRECTORY}"/* 2>/dev/null | awk '/total$/ {print $1}') -eq 0 ]]; then echo "INFO: The db directory is empty.." - REMOVE_DB_DIR="Y" DOWNLOAD_SNAPSHOT="Y" else - echo "INFO: The db directory is not empty.." + echo "INFO: The db directory is not empty, skipping Cardano DB download.." fi } -remove_db_dir() { - # Mithril client errors if the db folder already exists, so remove it if it is empty - if [[ "${REMOVE_DB_DIR}" == "Y" ]]; then - echo "INFO: Removing empty db directory to prepare for snapshot download.." - rmdir "${DB_DIRECTORY}" +cleanup_db_directory() { + echo "WARNING: Download failure, cleaning up DB directory.." + # Safety check to prevent accidental deletion of system files + if [[ -z "${DB_DIRECTORY}" ]]; then + echo "ERROR: DB_DIRECTORY is unset or null." + elif [[ -n "${DB_DIRECTORY}" && "${DB_DIRECTORY}" != "/" && "${DB_DIRECTORY}" != "${CNODE_HOME}" ]]; then + # :? Safety check to prevent accidental deletion of system files, even though initial if block should already prevent this + rm -rf "${DB_DIRECTORY:?}/"* + else + echo "INFO: Skipping cleanup of DB directory: ${DB_DIRECTORY}." fi } @@ -219,18 +105,26 @@ remove_db_dir() { download_snapshot() { if [[ "${DOWNLOAD_SNAPSHOT}" == "Y" ]]; then echo "INFO: Downloading latest mithril snapshot.." - "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} snapshot download --download-dir ${CNODE_HOME} ${SNAPSHOT_DIGEST} + trap 'cleanup_db_directory' INT + if ! "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db download --download-dir ${CNODE_HOME} --genesis-verification-key ${GENESIS_VERIFICATION_KEY} ${SNAPSHOT_DIGEST} ; then + cleanup_db_directory + exit 1 + fi else - echo "INFO: Skipping snapshot download.." + echo "INFO: Skipping Cardano DB download.." fi } list_snapshots() { - if [[ $1 == "json" ]]; then - "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} snapshot list --json - else - "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} snapshot list - fi + local json_flag="" + + for arg in "$@"; do + if [[ $arg == "json" ]]; then + json_flag="--json" + fi + done + + "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db snapshot list $json_flag } show_snapshot() { @@ -250,7 +144,7 @@ show_snapshot() { exit 1 fi - "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} snapshot show $digest $json_flag + "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} cardano-db snapshot show $digest $json_flag } ## mithril-stake-distribution subcommands @@ -265,31 +159,39 @@ download_stake_distribution() { } list_stake_distributions() { - if [[ $1 == "json" ]]; then - "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution list --json - else - "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution list - fi + local json_flag="" + + for arg in "$@"; do + if [[ $arg == "json" ]]; then + json_flag="--json" + fi + done + + "${MITHRILBIN}" -v --aggregator-endpoint ${AGGREGATOR_ENDPOINT} mithril-stake-distribution list $json_flag + } ##################### -# Execution # +# Execution/Main # ##################### +update_check "$@" + +set_defaults + # Parse command line options case $1 in environment) - set_defaults case $2 in setup) - environment_setup + generate_environment_file ;; override) environment_override $3 $4 ;; update) - UPDATE_ENVIRONMENT="Y" - environment_setup + export UPDATE_ENVIRONMENT="Y" + generate_environment_file ;; *) echo "Invalid environment subcommand: $2" >&2 @@ -298,36 +200,38 @@ case $1 in ;; esac ;; - snapshot) - set_defaults + cardano-db) + mithril_init case $2 in download) check_db_dir - remove_db_dir download_snapshot ;; - list) - case $3 in - json) - list_snapshots json + snapshot) + case $3 in + list) + case $4 in + json) + list_snapshots json + ;; + *) + list_snapshots + ;; + esac + ;; + show) + show_snapshot $4 $5 ;; *) - list_snapshots + echo "Invalid snapshot subcommand: $3" >&2 + usage + exit 1 ;; esac - ;; - show) - show_snapshot $3 $4 - ;; - *) - echo "Invalid snapshot subcommand: $2" >&2 - usage - exit 1 - ;; esac ;; stake-distribution) - set_defaults + mithril_init case $2 in download) download_stake_distribution @@ -350,7 +254,7 @@ case $1 in esac ;; *) - echo "Invalid command: $1" >&2 + echo "Invalid $(basename "$0") command: $1" >&2 usage exit 1 ;; diff --git a/scripts/cnode-helper-scripts/mithril-relay.sh b/scripts/cnode-helper-scripts/mithril-relay.sh index 6c104fddf..0207856f4 100755 --- a/scripts/cnode-helper-scripts/mithril-relay.sh +++ b/scripts/cnode-helper-scripts/mithril-relay.sh @@ -1,4 +1,8 @@ #!/bin/bash +# shellcheck disable=SC2086 +#shellcheck source=/dev/null + +. "$(dirname $0)"/mithril.library ###################################### # User Variables - Change as desired # @@ -26,16 +30,18 @@ RELAY_LISTENING_IP=() usage() { cat <<-EOF - $(basename "$0") [-d] [-l] + $(basename "$0") [-d] [-l] [-u] [-h] + A script to setup Cardano Mithril relays - Cardano Mithril relay wrapper script!! -d Install squid and configure as a relay -l Install nginx and configure as a load balancer + -u Skip update check -h Show this help text EOF } + generate_nginx_conf() { sudo bash -c "cat > /etc/nginx/nginx.conf <<'EOF' worker_processes 1; @@ -123,76 +129,103 @@ generate_squid_conf() { EOF" } +deploy_nginx_load_balancer() { + # Install nginx and configure load balancing + echo -e "\nInstalling nginx load balancer" + sudo apt-get update + sudo apt-get install -y nginx + + # Read the listening IP addresses from user input + while true; do + read -r -p "Enter the IP address of a relay: " ip + RELAY_LISTENING_IP+=("${ip}") + read -r -p "Are there more relays? (y/n) " yn + case ${yn} in + [Nn]*) break ;; + *) continue ;; + esac + done + + # Read the listening IP for the load balancer + read -r -p "Enter the IP address of the load balancer (press Enter to use default 127.0.0.1): " SIDECAR_LISTENING_IP + SIDECAR_LISTENING_IP=${SIDECAR_LISTENING_IP:-127.0.0.1} + echo "Using IP address ${SIDECAR_LISTENING_IP} for the load balancer configuration." + + # Read the listening port from user input + read -r -p "Enter the relay's listening port (press Enter to use default 3132): " RELAY_LISTENING_PORT + RELAY_LISTENING_PORT=${RELAY_LISTENING_PORT:-3132} + echo "Using port ${RELAY_LISTENING_PORT} for relay's listening port." + + # Generate the nginx configuration file + generate_nginx_conf + # Restart nginx and check status + echo -e "\nStarting Mithril relay sidecar (nginx load balancer)" + sudo systemctl restart nginx + sudo systemctl status nginx + +} + +deploy_squid_proxy() { + # Install squid and make a backup of the config file + echo -e "\nInstalling squid proxy" + sudo apt-get update + sudo apt-get install -y squid + sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.bak + + # Read the listening IP addresses from user input + while true; do + read -r -p "Enter the IP address of your Block Producer: " ip + BLOCK_PRODUCER_IP+=("${ip}") + read -r -p "Are there more block producers? (y/n) " yn + case ${yn} in + [Nn]*) break ;; + *) continue ;; + esac + done + + # Read the listening port from user input + read -r -p "Enter the relay's listening port (press Enter to use default 3132): " RELAY_LISTENING_PORT + RELAY_LISTENING_PORT=${RELAY_LISTENING_PORT:-3132} + echo "Using port ${RELAY_LISTENING_PORT} for relay's listening port." + generate_squid_conf + + # Restart squid and check status + echo -e "\nStarting Mithril relay (squid proxy)" + sudo systemctl restart squid + sudo systemctl status squid + + # Inform the user to create the appropriate firewall rule + for ip in "${RELAY_LISTENING_IP[@]}"; do + echo "Create the appropriate firewall rule: sudo ufw allow from ${ip} to any port ${RELAY_LISTENING_PORT} proto tcp" + done +} + +stop_relays() { + echo " Stopping squid proxy and nginx load balancers.." + sudo systemctl stop squid 2>/dev/null + sudo systemctl stop nginx 2>/dev/null + sleep 5 + exit 0 +} + +##################### +# Execution/Main # +##################### + # Parse command line arguments -while getopts :dlh opt; do +while getopts :dlsuh opt; do case ${opt} in d) - # Install squid and make a backup of the config file - echo -e "\nInstalling squid proxy" - sudo apt-get update - sudo apt-get install -y squid - sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.bak - - # Read the listening IP addresses from user input - while true; do - read -r -p "Enter the IP address of your Block Producer: " ip - BLOCK_PRODUCER_IP+=("${ip}") - read -r -p "Are there more block producers? (y/n) " yn - case ${yn} in - [Nn]*) break ;; - *) continue ;; - esac - done - - # Read the listening port from user input - read -r -p "Enter the relay's listening port (press Enter to use default 3132): " RELAY_LISTENING_PORT - RELAY_LISTENING_PORT=${RELAY_LISTENING_PORT:-3132} - echo "Using port ${RELAY_LISTENING_PORT} for relay's listening port." - generate_squid_conf - - # Restart squid and check status - echo -e "\nStarting Mithril relay (squid proxy)" - sudo systemctl restart squid - sudo systemctl status squid - - # Inform the user to create the appropriate firewall rule - for ip in "${RELAY_LISTENING_IP[@]}"; do - echo "Create the appropriate firewall rule: sudo ufw allow from ${ip} to any port ${RELAY_LISTENING_PORT} proto tcp" - done + INSTALL_SQUID_PROXY=Y ;; l) - # Install nginx and configure load balancing - echo -e "\nInstalling nginx load balancer" - sudo apt-get update - sudo apt-get install -y nginx - - # Read the listening IP addresses from user input - while true; do - read -r -p "Enter the IP address of a relay: " ip - RELAY_LISTENING_IP+=("${ip}") - read -r -p "Are there more relays? (y/n) " yn - case ${yn} in - [Nn]*) break ;; - *) continue ;; - esac - done - - # Read the listening IP for the load balancer - read -r -p "Enter the IP address of the load balancer (press Enter to use default 127.0.0.1): " SIDECAR_LISTENING_IP - SIDECAR_LISTENING_IP=${SIDECAR_LISTENING_IP:-127.0.0.1} - echo "Using IP address ${SIDECAR_LISTENING_IP} for the load balancer configuration." - - # Read the listening port from user input - read -r -p "Enter the relay's listening port (press Enter to use default 3132): " RELAY_LISTENING_PORT - RELAY_LISTENING_PORT=${RELAY_LISTENING_PORT:-3132} - echo "Using port ${RELAY_LISTENING_PORT} for relay's listening port." - - # Generate the nginx configuration file - generate_nginx_conf - # Restart nginx and check status - echo -e "\nStarting Mithril relay sidecar (nginx load balancer)" - sudo systemctl restart nginx - sudo systemctl status nginx + INSTALL_NGINX_LOAD_BALANCER=Y + ;; + u) + export SKIP_UPDATE='Y' + ;; + s) + STOP_RELAYS=Y ;; h) usage @@ -220,3 +253,15 @@ if [[ ${OPTIND} -eq 1 ]]; then usage exit 1 fi + +[[ "${STOP_RELAYS}" == "Y" ]] && stop_relays + +update_check "$@" + +if [[ ${INSTALL_SQUID_PROXY} = Y ]]; then + deploy_squid_proxy +fi + +if [[ ${INSTALL_NGINX_LOAD_BALANCER} = Y ]]; then + deploy_nginx_load_balancer +fi diff --git a/scripts/cnode-helper-scripts/mithril-signer.sh b/scripts/cnode-helper-scripts/mithril-signer.sh index 9670460c3..82c0f27c0 100755 --- a/scripts/cnode-helper-scripts/mithril-signer.sh +++ b/scripts/cnode-helper-scripts/mithril-signer.sh @@ -2,7 +2,7 @@ # shellcheck disable=SC2086 #shellcheck source=/dev/null -. "$(dirname $0)"/env offline +. "$(dirname $0)"/mithril.library ###################################### # User Variables - Change as desired # @@ -16,9 +16,6 @@ # Do NOT modify code below # ###################################### -U_ID=$(id -u) -G_ID=$(id -g) - ##################### # Functions # ##################### @@ -26,102 +23,36 @@ G_ID=$(id -g) usage() { cat <<-EOF - Usage: $(basename "$0") [-d] [-u] + Usage: $(basename "$0") [-d] [-D] [-e] [-k] [-r] [-s] [-u] [-h] + A script to setup, run and verify Cardano Mithril Signer - Cardano Mithril signer wrapper script !! -d Deploy mithril-signer as a systemd service - -u Update mithril environment file + -D Run mithril-signer as a daemon + -e Update mithril environment file + -k Stop signer using SIGINT + -r Verify signer registration + -s Verify signer signature + -u Skip update check -h Show this help text EOF } -set_defaults() { - [[ -z "${MITHRILBIN}" ]] && MITHRILBIN="${HOME}"/.local/bin/mithril-signer - if [[ -z "${POOL_NAME}" ]] || [[ "${POOL_NAME}" == "CHANGE_ME" ]]; then - echo "ERROR: The POOL_NAME must be set before deploying mithril-signer as a systemd service!!" - exit 1 - else - case "${NETWORK_NAME,,}" in - mainnet|preprod|guild) - RELEASE="release" - ;; - preview) - RELEASE="pre-release" - ;; - *) - echo "ERROR: The NETWORK_NAME must be set to Mainnet, PreProd, Preview, or Guild before mithril-signer can be deployed!!" - exit 1 - esac - fi -} - -pre_startup_sanity() { - [[ ! -f "${MITHRILBIN}" ]] && MITHRILBIN="$(command -v mithril-signer)" - if [[ ! -S "${CARDANO_NODE_SOCKET_PATH}" ]]; then - echo "ERROR: Could not locate socket file at ${CARDANO_NODE_SOCKET_PATH}, the node may not have completed startup !!" - exit 1 - fi +mithril_init() { + [[ ! -f "${CNODE_HOME}"/mithril/mithril.env ]] && generate_environment_file + for line in $(cat "${CNODE_HOME}"/mithril/mithril.env); do + export "${line}" + done # Move logs to archive - [[ -f "${LOG_DIR}"/mithril-signer.log ]] && mv "${LOG_DIR}"/mithril-signer.log "${LOG_DIR}"/archive/ -} - -get_relay_endpoint() { - read -r -p "Enter the IP address of the relay endpoint: " RELAY_ENDPOINT_IP - read -r -p "Enter the port of the relay endpoint (press Enter to use default 3132): " RELAY_PORT - RELAY_PORT=${RELAY_PORT:-3132} - echo "Using RELAY_ENDPOINT=${RELAY_ENDPOINT_IP}:${RELAY_PORT} for the Mithril signer relay endpoint." -} - -generate_environment_file() { - if [[ ! -d "${CNODE_HOME}/mithril/data-stores" ]]; then - sudo mkdir -p "${CNODE_HOME}"/mithril/data-stores - sudo chown -R "$U_ID":"$G_ID" "${CNODE_HOME}"/mithril 2>/dev/null - fi - # Inquire about the relay endpoint - read -r -p "Are you using a relay endpoint? (y/n, press Enter to use default y): " ENABLE_RELAY_ENDPOINT - ENABLE_RELAY_ENDPOINT=${ENABLE_RELAY_ENDPOINT:-y} - if [[ "${ENABLE_RELAY_ENDPOINT}" == "y" ]]; then - get_relay_endpoint - else - echo "Using a naive Mithril configuration without a mithril relay." - fi - - # Generate the full set of environment variables required by Mithril signer use case - export ERA_READER_ADDRESS=https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/era.addr - export ERA_READER_VKEY=https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/era.vkey - sudo bash -c "cat <<-'EOF' > ${CNODE_HOME}/mithril/mithril.env - KES_SECRET_KEY_PATH=${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME} - OPERATIONAL_CERTIFICATE_PATH=${POOL_DIR}/${POOL_OPCERT_FILENAME} - NETWORK=${NETWORK_NAME,,} - RELEASE=${RELEASE} - AGGREGATOR_ENDPOINT=https://aggregator.${RELEASE}-${NETWORK_NAME,,}.api.mithril.network/aggregator - RUN_INTERVAL=60000 - DB_DIRECTORY=${CNODE_HOME}/db - CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH} - CARDANO_CLI_PATH=${HOME}/.local/bin/cardano-cli - DATA_STORES_DIRECTORY=${CNODE_HOME}/mithril/data-stores - STORE_RETENTION_LIMITS=5 - ERA_READER_ADAPTER_TYPE=cardano-chain - ERA_READER_ADAPTER_PARAMS=$(jq -nc --arg address "$(wget -q -O - "${ERA_READER_ADDRESS}")" --arg verification_key "$(wget -q -O - "${ERA_READER_VKEY}")" '{"address": $address, "verification_key": $verification_key}') - GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/genesis.vkey) - PARTY_ID=$(cat ${POOL_DIR}/${POOL_ID_FILENAME}) - SNAPSHOT_DIGEST=latest - EOF" && sudo chown $USER:$USER "${CNODE_HOME}"/mithril/mithril.env - - if [[ "${ENABLE_RELAY_ENDPOINT}" == "y" ]]; then - sudo bash -c "echo RELAY_ENDPOINT=http://${RELAY_ENDPOINT_IP}:${RELAY_PORT} >> ${CNODE_HOME}/mithril/mithril.env" - fi + [[ -d "${LOG_DIR}"/archive ]] || mkdir -p "${LOG_DIR}"/archive + [[ -f "${LOG_DIR}"/$(basename "${0::-3}").log ]] && mv "${LOG_DIR}/$(basename "${0::-3}")".log "${LOG_DIR}"/archive/ ; touch "${LOG_DIR}/$(basename "${0::-3}")".log } deploy_systemd() { - echo "Creating ${CNODE_VNAME}-mithril-signer systemd service environment file.." - if [[ ! -f "${CNODE_HOME}"/mithril/mithril.env ]]; then - generate_environment_file && echo "Mithril environment file created successfully!!" - fi + echo "Creating ${CNODE_VNAME}-$(basename "${0::-3}") systemd service environment file.." - echo "Deploying ${CNODE_VNAME}-mithril-signer as systemd service.." - sudo bash -c "cat <<-'EOF' > /etc/systemd/system/${CNODE_VNAME}-mithril-signer.service + echo "Deploying ${CNODE_VNAME}-$(basename "${0::-3}") as systemd service.." + sudo bash -c "cat <<-'EOF' > /etc/systemd/system/${CNODE_VNAME}-$(basename "${0::-3}").service [Unit] Description=Cardano Mithril signer service StartLimitIntervalSec=0 @@ -136,35 +67,116 @@ deploy_systemd() { RestartSec=60 User=${USER} EnvironmentFile=${CNODE_HOME}/mithril/mithril.env - ExecStart=/bin/bash -l -c \"exec ${HOME}/.local/bin/mithril-signer -vv\" + ExecStart=/bin/bash -l -c \"exec ${HOME}/.local/bin/$(basename "${0::-3}") -vv\" KillSignal=SIGINT SuccessExitStatus=143 StandardOutput=syslog StandardError=syslog - SyslogIdentifier=${CNODE_VNAME}-mithril-signer + SyslogIdentifier=${CNODE_VNAME}-$(basename "${0::-3}") TimeoutStopSec=5 KillMode=mixed [Install] WantedBy=multi-user.target - EOF" && echo "${CNODE_VNAME}-mithril-signer.service deployed successfully!!" && sudo systemctl daemon-reload && sudo systemctl enable ${CNODE_VNAME}-mithril-signer.service + EOF" && echo "${CNODE_VNAME}-$(basename "${0::-3}").service deployed successfully!!" && sudo systemctl daemon-reload && sudo systemctl enable ${CNODE_VNAME}-"$(basename "${0::-3}")".service +} + +stop_signer() { + CNODE_PID=$(pgrep -fn "$(basename ${CNODEBIN}).*.--port ${CNODE_PORT}" 2>/dev/null) # env was only called in offline mode + kill -2 ${CNODE_PID} 2>/dev/null + # touch clean "${CNODE_HOME}"/db/clean # Disabled as it's a bit hacky, but only runs when SIGINT is passed to node process. Should not be needed if node does it's job + echo " Sending SIGINT to $(basename "${0::-3}") process.." + sleep 5 + exit 0 } -################### -# Execution # -################### + +user_interrupt_received() { + echo " SIGINT received, stopping $(basename "${0::-3}").." |tee -a "${LOG_DIR}/$(basename "${0::-3}")".log 2>&1 + stop_signer + +} + +verify_signer_registration() { + set -e + + if [ -z "$AGGREGATOR_ENDPOINT" ] || [ -z "$PARTY_ID" ]; then + echo ">> ERROR: Required environment variables AGGREGATOR_ENDPOINT and/or PARTY_ID are not set." + exit 1 + fi + + CURRENT_EPOCH=$(curl -s "$AGGREGATOR_ENDPOINT/epoch-settings" -H 'accept: application/json' | jq -r '.epoch') + SIGNERS_REGISTERED_RESPONSE=$(curl -s "$AGGREGATOR_ENDPOINT/signers/registered/$CURRENT_EPOCH" -H 'accept: application/json') + + if echo "$SIGNERS_REGISTERED_RESPONSE" | grep -q "$PARTY_ID"; then + echo ">> Congrats, your signer node is registered!" + else + echo ">> Oops, your signer node is not registered. Party ID not found among the signers registered at epoch ${CURRENT_EPOCH}." + fi + +} + +verify_signer_signature() { + set -e + + if [ -z "$AGGREGATOR_ENDPOINT" ] || [ -z "$PARTY_ID" ]; then + echo ">> ERROR: Required environment variables AGGREGATOR_ENDPOINT and/or PARTY_ID are not set." + exit 1 + fi + + CERTIFICATES_RESPONSE=$(curl -s "$AGGREGATOR_ENDPOINT/certificates" -H 'accept: application/json') + CERTIFICATES_COUNT=$(echo "$CERTIFICATES_RESPONSE" | jq '. | length') + + echo "$CERTIFICATES_RESPONSE" | jq -r '.[] | .hash' | while read -r HASH; do + RESPONSE=$(curl -s "$AGGREGATOR_ENDPOINT/certificate/$HASH" -H 'accept: application/json') + SIGNER_COUNT=$(echo "$RESPONSE" | jq '.metadata.signers | length') + for (( i=0; i < SIGNER_COUNT; i++ )); do + PARTY_ID_RESPONSE=$(echo "$RESPONSE" | jq -r ".metadata.signers[$i].party_id") + if [[ "$PARTY_ID_RESPONSE" == "$PARTY_ID" ]]; then + echo ">> Congrats, you have signed this certificate: $AGGREGATOR_ENDPOINT/certificate/$HASH !" + exit 1 + fi + done + done + + echo ">> Oops, your party id was not found in the last ${CERTIFICATES_COUNT} certificates. Please try again later." + +} + + +##################### +# Execution / Main # +##################### # Parse command line options -while getopts :duh opt; do +while getopts :dDekrsuh opt; do case ${opt} in - d ) DEPLOY_SYSTEMD="Y" ;; - u ) UPDATE_ENVIRONMENT="Y" ;; + d ) + DEPLOY_SYSTEMD="Y" ;; + D ) + SIGNER_DAEMON="Y" + ;; + e ) + export UPDATE_ENVIRONMENT="Y" + ;; + k ) + STOP_SIGNER="Y" + ;; + r ) + VERIFY_REGISTRATION="Y" + ;; + s ) + VERIFY_SIGNATURE="Y" + ;; + u ) + export SKIP_UPDATE="Y" + ;; h) usage exit 0 ;; \?) - echo "Invalid option: -${OPTARG}" >&2 + echo "Invalid $(basename "$0") option: -${OPTARG}" >&2 usage exit 1 ;; @@ -176,32 +188,46 @@ while getopts :duh opt; do esac done -# Check if env file is missing in current folder (no update checks as will mostly run as daemon), source env if present -[[ ! -f "$(dirname $0)"/env ]] && echo -e "\nCommon env file missing, please ensure latest guild-deploy.sh was run and this script is being run from ${CNODE_HOME}/scripts folder! \n" && exit 1 -. "$(dirname $0)"/env -case $? in - 1) echo -e "ERROR: Failed to load common env file\nPlease verify set values in 'User Variables' section in env file or log an issue on GitHub" && exit 1;; - 2) clear ;; -esac +[[ "${STOP_SIGNER}" == "Y" ]] && stop_signer + +# Check for updates +update_check "$@" # Set defaults and do basic sanity checks set_defaults + + #Deploy systemd if -d argument was specified -if [[ "${UPDATE_ENVIRONMENT}" == "Y" && "${DEPLOY_SYSTEMD}" == "Y" ]]; then - generate_environment_file && echo "Environment file updated successfully" && deploy_systemd && echo "Mithril signer service successfully deployed" && exit 0 - exit 2 -elif [[ "${UPDATE_ENVIRONMENT}" == "Y" ]]; then - generate_environment_file && echo "Environment file updated successfully" && exit 0 - exit 2 -elif [[ "${DEPLOY_SYSTEMD}" == "Y" ]]; then - deploy_systemd && echo "Mithril signer service successfully deployed" && exit 0 - exit 2 -fi - -pre_startup_sanity - -# Run Mithril Signer Server -echo "Sourcing the Mithril Signer environment file.." -. "${CNODE_HOME}"/mithril/mithril.env -echo "Starting Mithril Signer Server.." -"${MITHRILBIN}" -vvv >> "${LOG_DIR}"/mithril-signer.log 2>&1 +if [[ "${UPDATE_ENVIRONMENT}" == "Y" ]]; then + generate_environment_file + exit 0 +else + mithril_init + if [[ "${DEPLOY_SYSTEMD}" == "Y" ]]; then + if deploy_systemd ; then + echo "Mithril signer Systemd service successfully deployed" + exit 0 + else + echo "Failed to deploy Mithril signer Systemd service" + exit 2 + fi + elif [[ "${VERIFY_REGISTRATION}" == "Y" ]]; then + # Verify signer registration + echo "Verifying Mithril Signer registration.." + verify_signer_registration + exit 0 + elif [[ "${VERIFY_SIGNATURE}" == "Y" ]]; then + # Verify signer signature + echo "Verifying Mithril Signer signature.." + verify_signer_signature + exit 0 + elif [[ "${SIGNER_DAEMON}" == "Y" ]]; then + # Run Mithril Signer Server + echo "Starting Mithril Signer Server.." + trap 'user_interrupt_received' INT + if ! "${MITHRILBIN}" -vv | tee -a "${LOG_DIR}/$(basename "${0::-3}")".log 2>&1 ; then + echo "Failed to start Mithril Signer Server" | tee -a "${LOG_DIR}/$(basename "${0::-3}")".log 2>&1 + exit 1 + fi + fi +fi \ No newline at end of file diff --git a/scripts/cnode-helper-scripts/mithril.library b/scripts/cnode-helper-scripts/mithril.library new file mode 100644 index 000000000..4a471ef18 --- /dev/null +++ b/scripts/cnode-helper-scripts/mithril.library @@ -0,0 +1,196 @@ +#!/usr/bin/env bash +# shellcheck disable=SC2034,SC2086,SC2230,SC2206,SC2140,SC2059,SC2154 +#shellcheck source=/dev/null + +###################################### +# Do NOT modify code below # +###################################### + +. "$(dirname $0)"/env offline + +U_ID=$(id -u) +G_ID=$(id -g) +MITHRILBIN="${HOME}"/.local/bin/$(basename "${0::-3}") + +compare_versions() { + local min_version=$1 + local test_version=$2 + if [[ $(printf '%s\n' "$min_version" "$test_version" | sort -V | head -n1) = "$min_version" ]]; then + return 0 + else + return 1 + fi +} + +set_node_minimum_version() { + response_file=$(mktemp) + status_code=$(curl -s -o "$response_file" -w "%{http_code}" https://raw.githubusercontent.com/input-output-hk/mithril/${MITHRIL_LATEST_VERSION}/networks.json) + + if [[ "$status_code" -eq 404 ]]; then + NODE_MINIMUM_VERSION="" + else + NODE_MINIMUM_VERSION=$(jq -r ".${NETWORK}.\"cardano-minimum-version\".\"mithril-signer\"" "$response_file") + fi + rm -f "$response_file" +} + +update_check() { + # Check availability of checkUpdate function + if [[ ! $(command -v checkUpdate) ]]; then + echo -e "\nCould not find checkUpdate function in env, make sure you're using official guild docos for installation!" + exit 1 + fi + # Check if flag is set by script to skip update check + [[ ${SKIP_UPDATE} == Y ]] && return 0 + # Check if flag is set by user as a global (container environments etc.) to skip update check + if [[ ${UPDATE_CHECK} = Y ]]; then + echo "Checking for script updates..." + # check for env update + ENV_UPDATED=${BATCH_AUTO_UPDATE} + checkUpdate "${PARENT}"/env N N N + case $? in + 1) ENV_UPDATED=Y ;; + 2) exit 1 ;; + esac + # check for mithril.library update + checkUpdate "${PARENT}"/mithril.library N N N + # borrow ENV_UPDATED for mithril.library updates + case $? in + 1) ENV_UPDATED=Y ;; + 2) exit 1 ;; + esac + # check the script update + checkUpdate "${PARENT}"/"$(basename "$0")" ${ENV_UPDATED} + case $? in + 1) echo ""; $0 "-u" "$@"; exit 0 ;; # re-launch script with same args skipping update check + 2) exit 1 ;; + esac + fi +} + + +set_defaults() { + MITHRIL_LATEST_VERSION=$(curl -s https://raw.githubusercontent.com/cardano-community/guild-operators/alpha/files/docker/node/release-versions/mithril-latest.txt) + set_node_minimum_version + NODE_CURRENT_VERSION=$(cardano-node --version | awk 'NR==1{print $2}') + + [[ -z "${MITHRILBIN}" ]] && MITHRILBIN="${HOME}"/.local/bin/"$(basename "${0::-3}")" + if [[ $(basename "${0::-3}") == "mithril-signer" ]] && { [[ -z "${POOL_NAME}" ]] || [[ "${POOL_NAME}" == "CHANGE_ME" ]]; }; then + echo "ERROR: The POOL_NAME must be set before deploying mithril-signer as a systemd service!!" + exit 1 + else + case "${NETWORK_NAME,,}" in + mainnet|preprod|guild) + RELEASE="release" + ;; + preview|sanchonet) + RELEASE="pre-release" + ;; + *) + echo "ERROR: The NETWORK_NAME must be set to mainnet, preprod, preview, or sanchonet before $(basename "${0::-3}") can be deployed!!" + exit 1 + esac + fi + AGGREGATOR_ENDPOINT=https://aggregator.${RELEASE}-${NETWORK_NAME,,}.api.mithril.network/aggregator + GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/genesis.vkey) +} + +create_data_stores_directory() { + if [[ ! -d "${CNODE_HOME}/mithril/data-stores" ]]; then + sudo mkdir -p "${CNODE_HOME}"/mithril/data-stores + sudo chown -R "$U_ID":"$G_ID" "${CNODE_HOME}"/mithril 2>/dev/null + fi +} + +set_env_file_ownership() { + chown $USER:$USER "${CNODE_HOME}"/mithril/mithril.env +} + + +check_mithril_environment_file_exists() { + local env_file="${CNODE_HOME}/mithril/mithril.env" + + if [[ -f "$env_file" ]]; then + if [[ "$UPDATE_ENVIRONMENT" != "Y" ]]; then + echo "Error: $env_file already exists. To update it, set UPDATE_ENVIRONMENT to 'Y'." >&2 + return 1 + else + echo "Updating $env_file..." + fi + else + echo "Creating $env_file..." + fi +} + +get_relay_endpoint() { + read -r -p "Enter the IP address of the relay endpoint: " RELAY_ENDPOINT_IP + read -r -p "Enter the port of the relay endpoint (press Enter to use default 3132): " RELAY_PORT + RELAY_PORT=${RELAY_PORT:-3132} + echo "Using RELAY_ENDPOINT=${RELAY_ENDPOINT_IP}:${RELAY_PORT} for the Mithril signer relay endpoint." +} + +update_mithril_environment_for_signer() { + echo "Info: Setting all environment variables, supporting the Mithril signer use case." + # Inquire about the relay endpoint + read -r -p "Are you using a relay endpoint? (y/n, press Enter to use default y): " ENABLE_RELAY_ENDPOINT + ENABLE_RELAY_ENDPOINT=${ENABLE_RELAY_ENDPOINT:-y} + if [[ "${ENABLE_RELAY_ENDPOINT}" == "y" ]]; then + get_relay_endpoint + else + echo "Using a naive Mithril configuration without a mithril relay." + fi + # Generate the full set of environment variables required by Mithril signer use case + export ERA_READER_ADDRESS=https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/era.addr + export ERA_READER_VKEY=https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/era.vkey + sudo bash -c "cat <<-'EOF' > ${CNODE_HOME}/mithril/mithril.env + KES_SECRET_KEY_PATH=${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME} + OPERATIONAL_CERTIFICATE_PATH=${POOL_DIR}/${POOL_OPCERT_FILENAME} + NETWORK=${NETWORK_NAME,,} + RELEASE=${RELEASE} + AGGREGATOR_ENDPOINT=https://aggregator.${RELEASE}-${NETWORK_NAME,,}.api.mithril.network/aggregator + RUN_INTERVAL=60000 + DB_DIRECTORY=${CNODE_HOME}/db + CARDANO_NODE_SOCKET_PATH=${CARDANO_NODE_SOCKET_PATH} + CARDANO_CLI_PATH=${HOME}/.local/bin/cardano-cli + DATA_STORES_DIRECTORY=${CNODE_HOME}/mithril/data-stores + STORE_RETENTION_LIMITS=5 + ERA_READER_ADAPTER_TYPE=cardano-chain + ERA_READER_ADAPTER_PARAMS=$(jq -nc --arg address "$(wget -q -O - "${ERA_READER_ADDRESS}")" --arg verification_key "$(wget -q -O - "${ERA_READER_VKEY}")" '{"address": $address, "verification_key": $verification_key}') + GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/genesis.vkey) + PARTY_ID=$(cat ${POOL_DIR}/${POOL_ID_FILENAME}-bech32) + SNAPSHOT_DIGEST=latest + EOF" + + if [[ "${ENABLE_RELAY_ENDPOINT}" == "y" ]]; then + sudo bash -c "echo RELAY_ENDPOINT=http://${RELAY_ENDPOINT_IP}:${RELAY_PORT} >> ${CNODE_HOME}/mithril/mithril.env" + fi +} + +update_mithril_environment_for_client() { + echo "Info: Setting minimal environment variables supporting only the Mithril client use case." + bash -c "cat <<-'EOF' > ${CNODE_HOME}/mithril/mithril.env + NETWORK=${NETWORK_NAME,,} + RELEASE=${RELEASE} + AGGREGATOR_ENDPOINT=https://aggregator.${RELEASE}-${NETWORK_NAME,,}.api.mithril.network/aggregator + DB_DIRECTORY=${CNODE_HOME}/db + GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/${RELEASE}-${NETWORK_NAME,,}/genesis.vkey) + SNAPSHOT_DIGEST=latest + EOF" +} + +component_environment_setup() { + check_mithril_environment_file_exists + + if [[ -n "${POOL_NAME}" ]] && [[ "${POOL_NAME}" != "CHANGE_ME" ]] && [[ "$(basename "$0")" == "mithril-signer.sh" ]]; then + update_mithril_environment_for_signer + else + update_mithril_environment_for_client + fi +} + +generate_environment_file() { + create_data_stores_directory + component_environment_setup + set_env_file_ownership +} +