CI 📈: Add workflow for checking changes to weights in all runtimes

[pLabarta]

What does it do?

This PR adds a new CI workflow which checks if files inside the weights folder changed for each runtime and generates a report of the changes sorted by Change %. It will only run for PRs that change the files generated by benchmarks.

Reports look like the following:

What important points reviewers should know?

Is there something left for follow-up PRs?

Currently, big changes might generate massive reports. If they end up being unreadable, this workflow could be changed to show only a summary in the PR discussion and upload full reports as artifacts or publish them as HTML pages.

What alternative implementations were considered?

Are there relevant PRs or issues in other repositories (Substrate, Polkadot, Frontier, Cumulus)?

What value does it bring to the blockchain users?

[github-actions]

WASM runtime size check:

Compared to target branch

Moonbase runtime: 2268 KB (no changes) ✅

Moonbeam runtime: 2252 KB (no changes) ✅

Moonriver runtime: 2260 KB (no changes) ✅

Compared to latest release (runtime-3400)

Moonbase runtime: 2268 KB (+240 KB compared to latest release) ⚠️

Moonbeam runtime: 2252 KB (+240 KB compared to latest release) ⚠️

Moonriver runtime: 2260 KB (+248 KB compared to latest release) ⚠️

[github-actions]

Coverage Report

@@                       Coverage Diff                       @@
##           master   pablo/ext-weight-diff-report     +/-   ##
===============================================================
  Coverage   74.40%                         74.40%   0.00%     
  Files         376                            376             
  Lines       95432                          95432             
===============================================================
  Hits        70998                          70998             
  Misses      24434                          24434             

Files Changed	Coverage

Coverage generated Thu Jan 9 18:58:55 UTC 2025

[RomarQ]

We can move repeated code into a single function and reuse it for every runtime. Asked github Copilot and it gave me the following suggestion, not sure if it will work, but should be close.

Suggested change

	- name: Moonbase Weight Difference Report
	if: steps.changed-files-yaml.outputs.moonbase_any_changed == 'true'
	env:
	ALL_CHANGED_FILES: ${{ steps.changed-files-yaml.outputs.moonbase_all_changed_files }}
	run: |
	for file in ${ALL_CHANGED_FILES}; do
	subweight compare commits --threshold 1 --strip-path-prefix ".*/" --format csv --no-color --method exact-worst --path-pattern $file ${{github.base_ref}} ${{github.head_ref}} | sed 1,1d | sed '$d' >> moonbase_diffs.csv
	done
	jq -R 'split(",") | {File: .[0], Extrinsic: .[1], Old: .[2], New: .[3], "Change Percent": (.[4] | tonumber)}' moonbase_diffs.csv | jq -s '.' > moonbase_diffs.json
	jq 'sort_by(.["Change Percent"] | abs ) | reverse' moonbase_diffs.json > moonbase_diffs_sorted.json
	echo "## Moonbase Weight Difference Report" >> weight_diff_report.md
	echo "| File | Extrinsic | Old | New | Change Percent |" >> weight_diff_report.md
	echo "|------|-----------|-----|-----|----------------|" >> weight_diff_report.md
	jq -r '.[] | "| \(.File) | \(.Extrinsic) | \(.Old) | \(.New) | \(.["Change Percent"])% |"' moonbase_diffs_sorted.json >> weight_diff_report.md
	- name: Moonriver Weight Difference Report
	if: steps.changed-files-yaml.outputs.moonriver_any_changed == 'true'
	env:
	ALL_CHANGED_FILES: ${{ steps.changed-files-yaml.outputs.moonriver_all_changed_files }}
	run: |
	for file in ${ALL_CHANGED_FILES}; do
	subweight compare commits --threshold 1 --strip-path-prefix ".*/" --format csv --no-color --method exact-worst --path-pattern $file ${{github.base_ref}} ${{github.head_ref}} | sed 1,1d | sed '$d' >> moonriver_diffs.csv
	done
	jq -R 'split(",") | {File: .[0], Extrinsic: .[1], Old: .[2], New: .[3], "Change Percent": (.[4] | tonumber)}' moonriver_diffs.csv | jq -s '.' > moonriver_diffs.json
	jq 'sort_by(.["Change Percent"] | abs ) | reverse' moonriver_diffs.json > moonriver_diffs_sorted.json
	echo "## Moonriver Weight Difference Report" >> weight_diff_report.md
	echo "| File | Extrinsic | Old | New | Change Percent |" >> weight_diff_report.md
	echo "|------|-----------|-----|-----|----------------|" >> weight_diff_report.md
	jq -r '.[] | "| \(.File) | \(.Extrinsic) | \(.Old) | \(.New) | \(.["Change Percent"])% |"' moonriver_diffs_sorted.json >> weight_diff_report.md
	- name: Moonbeam Weight Difference Report
	if: steps.changed-files-yaml.outputs.moonbeam_any_changed == 'true'
	env:
	ALL_CHANGED_FILES: ${{ steps.changed-files-yaml.outputs.moonbeam_all_changed_files }}
	run: |
	for file in ${ALL_CHANGED_FILES}; do
	subweight compare commits --threshold 1 --strip-path-prefix ".*/" --format csv --no-color --method exact-worst --path-pattern $file ${{github.base_ref}} ${{github.head_ref}} | sed 1,1d | sed '$d' >> moonbeam_diffs.csv
	done
	jq -R 'split(",") | {File: .[0], Extrinsic: .[1], Old: .[2], New: .[3], "Change Percent": (.[4] | tonumber)}' moonbeam_diffs.csv | jq -s '.' > moonbeam_diffs.json
	jq 'sort_by(.["Change Percent"] | abs ) | reverse' moonbeam_diffs.json > moonbeam_diffs_sorted.json
	echo "## Moonbeam Weight Difference Report" >> weight_diff_report.md
	echo "| File | Extrinsic | Old | New | Change Percent |" >> weight_diff_report.md
	echo "|------|-----------|-----|-----|----------------|" >> weight_diff_report.md
	jq -r '.[] | "| \(.File) | \(.Extrinsic) | \(.Old) | \(.New) | \(.["Change Percent"])% |"' moonbeam_diffs_sorted.json >> weight_diff_report.md
	- name: Generate Weight Difference Report
	run: |
	generate_report() {
	local runtime=$1
	local changed_files=$2
	local diffs_csv="${runtime}_diffs.csv"
	local diffs_json="${runtime}_diffs.json"
	local diffs_sorted_json="${runtime}_diffs_sorted.json"
	for file in ${changed_files}; do
	subweight compare commits --threshold 1 --strip-path-prefix ".*/" --format csv --no-color --method exact-worst --path-pattern $file ${{github.base_ref}} ${{github.head_ref}} | sed 1,1d | sed '$d' >> ${diffs_csv}
	done
	jq -R 'split(",") | {File: .[0], Extrinsic: .[1], Old: .[2], New: .[3], "Change Percent": (.[4] | tonumber)}' ${diffs_csv} | jq -s '.' > ${diffs_json}
	jq 'sort_by(.["Change Percent"] | abs ) | reverse' ${diffs_json} > ${diffs_sorted_json}
	echo "## ${runtime^} Weight Difference Report" >> weight_diff_report.md
	echo "| File | Extrinsic | Old | New | Change Percent |" >> weight_diff_report.md
	echo "|------|-----------|-----|-----|----------------|" >> weight_diff_report.md
	jq -r '.[] | "| \(.File) | \(.Extrinsic) | \(.Old) | \(.New) | \(.["Change Percent"])% |"' ${diffs_sorted_json} >> weight_diff_report.md
	}
	if [[ "${{ steps.changed-files-yaml.outputs.moonbase_any_changed }}" == "true" ]]; then
	generate_report "moonbase" "${{ steps.changed-files-yaml.outputs.moonbase_all_changed_files }}"
	fi
	if [[ "${{ steps.changed-files-yaml.outputs.moonriver_any_changed }}" == "true" ]]; then
	generate_report "moonriver" "${{ steps.changed-files-yaml.outputs.moonriver_all_changed_files }}"
	fi
	if [[ "${{ steps.changed-files-yaml.outputs.moonbeam_any_changed }}" == "true" ]]; then
	generate_report "moonbeam" "${{ steps.changed-files-yaml.outputs.moonbeam_all_changed_files }}"
	fi