[mithril] Fix bug in checkUpdate / Fix export for signer / Fix PARTY_…

…ID for verification / Add mithril client download to non container docs (#1761)

1. Includes a syntax fix for #1757
2. Removes unecessary UPDATE_CHECK=N exports in the workflow now that it
is a default value in the Dockerfile ENV / Image.
3. Exports G_ACCOUNT before calls to `./guild-deploy.sh` to ensure if
images are built in a fork that the files come from the fork, not
**cardano-community**.
4. Sets default G_ACCOUNT and GUILD_DEPLOY_BRANCH so manual `docker
build` commands do not require passing `--build-arg`'s except to alter
default values.
5. Fixes a bug leftover in the update logic and abstracts common
functions from mithril scripts into a new mithril.library sourced by the
scripts, reduces maintenance overhead and code duplication.
6. Prepares for #1744 by getting the minimum versions for cardano node
from mithrils new `networks.json` file.
7. Implements changes suggested by @Fuma419 since branch node-8.10.0
will merge into alpha, but not node-8.9.0.
8. Fixes a bug leftover in the update logic

### 1756 update bug
* This changes the call to use `$(basename "$0")` for any instance of a
mithril script calling checkUpdate for itself, should future proof
against script renames.
  * Also runs checkUpdate for `env` and `mithril.library`.

### 1759 changes to mithril-client
* The cardano-db is now the base command
  * snapshot is relocated inside cardano-db
  * download is relocated from snapshot directly into cardano-db
* Implemented cleanup on crashes as well as user interupts. A new node
should either have a full copy of the most recent snapshot, or be empty
so `cnode.sh` has no corruption or incomplete files preventing a good
sync.

* This also changes `mithril-signer.sh`
* Existing flags in `mithril-signer.sh` changed to have a better overlap
in mithril scripts using similar OPTARGS (-u always skip updates, etc.).
* New flags implemented that integrates the upstream mithril scripts to
Verify signer registration and signer signatures.

Documentation has been updated to account for changes in the scripts
usage, flags, commands and subcommands. Also the MITHRIL_DOWNLOAD
variable is documented into the `node-cli.md`.

closes #1756
closes #1757
closes #1759

docs/Build/node-cli.md

@@ -63,7 +63,7 @@ POOL_NAME="GUILD"
 
 #### Start the node
 
-To test starting the node in interactive mode, we will make use of pre-built script `cnode.sh`. This script automatically determines whether to start the node as a relay or block producer (if the required pool keys are present in the `$CNODE_HOME/priv/pool/<POOL_NAME>` as mentioned above). The script contains a user-defined variable `CPU_CORES` which determines the number of CPU cores the node will use upon start-up:
+To test starting the node in interactive mode, we will make use of pre-built script `cnode.sh`. This script automatically determines whether to start the node as a relay or block producer (if the required pool keys are present in the `$CNODE_HOME/priv/pool/<POOL_NAME>` as mentioned above). If the `<MITHRIL_DOWNLOAD>` variable is set to 'Y' it will download the latest snapshot from a Mithril aggregator to speed up the blockchain synchronization. The script contains a user-defined variable `CPU_CORES` which determines the number of CPU cores the node will use upon start-up:
 
 ```bash
 ######################################

docs/Scripts/mithril-client.md

@@ -1,10 +1,37 @@
 `mithril-client.sh` is a script to manage the Mithril client, a tool used to set up the Mithril client environment and manage downloading Mithril snapshots and stake distributions. The main features include:
 
 - **environment** - Creates a new `mithril.env` file with all the necessary environment variables for the Mithril client.
-- **snapshot** - Download, list all or show a specific available Mithril snapshot.
+- **cardano-db** - Download, list all or show a specific available Mithril snapshot.
 - **stake-distribution** - Download or list available Mithril stake distributions.
 - **-u** - Skip script update check.
 
+## Usage
+
+```bash
+Usage: bash [-u] <command> <subcommand> [<sub arg>]
+A script to run Cardano Mithril Client
+
+-u          Skip script update check overriding UPDATE_CHECK value in env (must be first argument to script)
+
+Commands:
+environment           Manage mithril environment file
+  setup               Setup mithril environment file
+  override            Override default variable in the mithril environment file
+  update              Update mithril environment file
+cardano-db            Interact with Cardano DB
+  download            Download Cardano DB from Mithril snapshot
+  snapshot            Interact with Mithril snapshots
+    list              List available Mithril snapshots
+      json            List availble Mithril snapshots in JSON format
+    show              Show details of a Mithril snapshot
+      json            Show details of a Mithril snapshot in JSON format
+stake-distribution    Interact with Mithril stake distributions
+  download            Download latest stake distribution
+  list                List available stake distributions
+    json              Output latest Mithril snapshot in JSON format
+
+```
+
 ## Preparing a Relay or Block Producer Node
 
 To prepare a relay or block producer node, you should follow these steps:
@@ -18,7 +45,7 @@ To prepare a relay or block producer node, you should follow these steps:
 2. **Download the latest Mithril snapshot:** Once the environment file is set up, you can download the latest Mithril snapshot by running the script with the `snapshot download` command. This snapshot contains the latest state of the Cardano blockchain db from a Mithril Aggregator.
 
    ```bash
-   ./mithril-client.sh snapshot download
+   ./mithril-client.sh cardano-db download
    ```
 
 ## Investigating Available Snapshots
@@ -28,16 +55,16 @@ You can investigate the available snapshots by using the `snapshot list` and `sn
 - **List all available Mithril snapshots:** You can list all available Mithril snapshots by running the script with the `snapshot list` command. Add `json` at the end to get the output in JSON format.
 
   ```bash
-  ./mithril-client.sh snapshot list
-  ./mithril-client.sh snapshot list json
+  ./mithril-client.sh cardano-dbsnapshot list
+  ./mithril-client.sh cardano-dbsnapshot list json
   ```
 
 - **Show details of a specific Mithril snapshot:** You can show details of a specific Mithril snapshot by running the script with the `snapshot show <DIGEST>` command, where `<DIGEST>` is the digest of the snapshot. Add `json` at the end to get the output in JSON format.
 
   ```bash
-  ./mithril-client.sh snapshot show <DIGEST>
-  ./mithril-client.sh snapshot show <DIGEST> json
-  ./mithril-client.sh snapshot show json <DIGEST>
+  ./mithril-client.sh cardano-dbsnapshot show <DIGEST>
+  ./mithril-client.sh cardano-dbsnapshot show <DIGEST> json
+  ./mithril-client.sh cardano-dbsnapshot show json <DIGEST>
   ```
 
 ## Managing Stake Distributions

docs/Scripts/mithril-relay.md

@@ -7,12 +7,13 @@ loadbalancer. It provides functionalities such as:
 ## Usage
 
 ```bash
-Usage: mithril-relay.sh [-d] [-l]
+bash [-d] [-l] [-u] [-h]
+A script to setup Cardano Mithril relays
 
-Options:
-    -d  Install squid and configure as a relay
-    -l  Install nginx and configure as a load balancer
-    -h  Show this help text
+-d  Install squid and configure as a relay
+-l  Install nginx and configure as a load balancer
+-u  Skip update check
+-h  Show this help text
 ```
 
 # Description

docs/Scripts/mithril-signer.md

@@ -5,12 +5,17 @@ environment file to contain variables specific to the Mithril Signer.
 ## Usage
 
 ```bash
-Usage: mithril-signer.sh [-d] [-u]
-
-Options:
-    -d    Deploy mithril-signer as a systemd service
-    -u    Update mithril environment file
-    -h    Show this help text
+Usage: bash [-d] [-D] [-e] [-k] [-r] [-s] [-u] [-h]
+A script to setup, run and verify Cardano Mithril Signer
+
+-d    Deploy mithril-signer as a systemd service
+-D    Run mithril-signer as a daemon
+-e    Update mithril environment file
+-k    Stop signer using SIGINT
+-r    Verify signer registration
+-s    Verify signer signature
+-u    Skip update check
+-h    Show this help text
 ```
 
 # Description

files/docker/node/addons/entrypoint.sh

@@ -6,6 +6,7 @@ trap 'killall -s SIGINT cardano-node' SIGINT SIGTERM
 
 head -n 8 ~/.scripts/banner.txt
 
+# shellcheck disable=SC1090
 . ~/.bashrc > /dev/null 2>&1
 
 echo "NETWORK: $NETWORK $POOL_NAME $TOPOLOGY";
@@ -17,29 +18,29 @@ echo "NODE: $HOSTNAME - Port:$CNODE_PORT - $POOL_NAME";
 cardano-node --version;
 
 if [[ "${ENABLE_BACKUP}" == "Y" ]] || [[ "${ENABLE_RESTORE}" == "Y" ]]; then
-    [[ ! -d "${CNODE_HOME}"/backup/$NETWORK-db ]] && mkdir -p $CNODE_HOME/backup/$NETWORK-db
-    dbsize=$(du -s $CNODE_HOME/db | awk '{print $1}')
-    bksizedb=$(du -s $CNODE_HOME/backup/$NETWORK-db 2>/dev/null | awk '{print $1}')
-    if [[ "${ENABLE_RESTORE}" == "Y" ]] && [[ "$dbsize" -lt "$bksizedb" ]]; then
-        echo "Backup Started"
-        cp -rf "${CNODE_HOME}"/backup/"${NETWORK}"-db/* "${CNODE_HOME}"/db 2>/dev/null
-        echo "Backup Finished"
-    fi
-
-    if [[ "${ENABLE_BACKUP}" == "Y" ]] && [[ "$dbsize" -gt "$bksizedb" ]]; then
-        echo "Restore Started"
-        cp -rf "${CNODE_HOME}"/db/* "${CNODE_HOME}"/backup/"${NETWORK}"-db/ 2>/dev/null
-        echo "Restore Finished"
-    fi
+  [[ ! -d "${CNODE_HOME}"/backup/$NETWORK-db ]] && mkdir -p $CNODE_HOME/backup/$NETWORK-db
+  dbsize=$(du -s $CNODE_HOME/db | awk '{print $1}')
+  bksizedb=$(du -s $CNODE_HOME/backup/$NETWORK-db 2>/dev/null | awk '{print $1}')
+  if [[ "${ENABLE_RESTORE}" == "Y" ]] && [[ "$dbsize" -lt "$bksizedb" ]]; then
+    echo "Backup Started"
+    cp -rf "${CNODE_HOME}"/backup/"${NETWORK}"-db/* "${CNODE_HOME}"/db 2>/dev/null
+    echo "Backup Finished"
+  fi
+
+  if [[ "${ENABLE_BACKUP}" == "Y" ]] && [[ "$dbsize" -gt "$bksizedb" ]]; then
+    echo "Restore Started"
+    cp -rf "${CNODE_HOME}"/db/* "${CNODE_HOME}"/backup/"${NETWORK}"-db/ 2>/dev/null
+    echo "Restore Finished"
+  fi
 fi
 
 # Customisation 
 customise () {
-find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/127.0.0.1/0.0.0.0/g' > /dev/null 2>&1 
-grep -i ENABLE_CHATTR /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_CHATTR=(true|false)?/ENABLE_CHATTR=false/g' /opt/cardano/cnode/scripts/cntools.sh > /dev/null 2>&1
-grep -i ENABLE_DIALOG /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_DIALOG=(true|false)?/ENABLE_DIALOG=false/' /opt/cardano/cnode/scripts/cntools.sh >> /opt/cardano/cnode/scripts/cntools.sh
-find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/\"hasEKG\": 12788,/\"hasEKG\": [\n    \"0.0.0.0\",\n    12788\n],/g' > /dev/null 2>&1
-return 0
+  find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/127.0.0.1/0.0.0.0/g' > /dev/null 2>&1 
+  grep -i ENABLE_CHATTR /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_CHATTR=(true|false)?/ENABLE_CHATTR=false/g' /opt/cardano/cnode/scripts/cntools.sh > /dev/null 2>&1
+  grep -i ENABLE_DIALOG /opt/cardano/cnode/scripts/cntools.sh >/dev/null && sed -E -i 's/^#?ENABLE_DIALOG=(true|false)?/ENABLE_DIALOG=false/' /opt/cardano/cnode/scripts/cntools.sh > /dev/null 2>&1
+  find /opt/cardano/cnode/files -name "*config*.json" -print0 | xargs -0 sed -i 's/\"hasEKG\": 12788,/\"hasEKG\": [\n    \"0.0.0.0\",\n    12788\n],/g' > /dev/null 2>&1
+  return 0
 }
 
 load_configs () {

files/docker/node/dockerfile_bin

@@ -44,7 +44,6 @@ RUN set -x && apt update \
     && mkdir -p /root/.local/bin \
     && wget https://raw.githubusercontent.com/${G_ACCOUNT}/guild-operators/${GUILD_DEPLOY_BRANCH}/scripts/cnode-helper-scripts/guild-deploy.sh \
     && export SUDO='N' \
-    && export UPDATE_CHECK='N' \
     && export SKIP_DBSYNC_DOWNLOAD='Y' \
     && export G_ACCOUNT=${G_ACCOUNT} \
     && chmod +x ./guild-deploy.sh && ./guild-deploy.sh -b ${GUILD_DEPLOY_BRANCH} -s p \
@@ -57,7 +56,6 @@ RUN set -x && apt update \
     && rm -rf /var/lib/apt/lists/*
 
 RUN set -x && export SUDO='N' \
-    && export UPDATE_CHECK='N' \
     && export SKIP_DBSYNC_DOWNLOAD='Y' \
     && export G_ACCOUNT=${G_ACCOUNT} \
     && ./guild-deploy.sh -b ${GUILD_DEPLOY_BRANCH} -s dcmowx \

scripts/cnode-helper-scripts/cnode.sh

@@ -65,9 +65,9 @@ pre_startup_sanity() {
 mithril_snapshot_download() {
   [[ -z "${MITHRIL_CLIENT}" ]] && MITHRIL_CLIENT="${CNODE_HOME}"/scripts/mithril-client.sh
   if [[ ! -f "${MITHRIL_CLIENT}" ]] || [[ ! -e "${MITHRIL_CLIENT}" ]]; then 
-    echo "ERROR: Could not locate mithril-client.sh script or script is not executable. Skipping mithril snapshot download!!"
+    echo "ERROR: Could not locate mithril-client.sh script or script is not executable. Skipping mithril cardano-db snapshot download!!"
   else
-    "${MITHRIL_CLIENT}" snapshot download
+    "${MITHRIL_CLIENT}" cardano-db download
   fi
 }
 

scripts/cnode-helper-scripts/guild-deploy.sh

@@ -638,6 +638,7 @@ populate_cnode() {
   updateWithCustomConfig "mithril-client.sh"
   updateWithCustomConfig "mithril-relay.sh"
   updateWithCustomConfig "mithril-signer.sh"
+  updateWithCustomConfig "mithril.library"
 
   find "${CNODE_HOME}/scripts" -name '*.sh' -exec chmod 755 {} \; 2>/dev/null
   chmod -R 700 "${CNODE_HOME}"/priv 2>/dev/null