Add Standard Data Release YAML to GitHub Actions #1329

afontani · 2024-12-06T17:34:02Z

Pull Request Description

This pull request introduces the Standard Data Release (SDR) YAML file into the "develop" branch of ResStock and into GitHub Actions integration-tests. This PR will give contributors on the SDR measures a place to commit changes and aid in completing an end-to-end workflow in CI.

Workflow changes

National Upgrades and Testing Upgrades have been removed

Path upload errors added to .github/workflows/config.yml. Previously was defaulted to "warn" and wasn't catching that a file was missing. This is done by adding if-no-files-found: error to the actions/upload-artifact@v4 action. Example:

resstock/.github/workflows/config.yml

Lines 121 to 126 in 037a9df

    
                 - name: Upload precomputed buildstocks 
        
                   uses: actions/upload-artifact@v4 
        
                   with: 
        
                     name: precomputed_buildstocks 
        
                     path: test/tests_yml_files/yml_precomputed*/buildstock*.csv 
        
                     if-no-files-found: error

The SDR project file tests have been added similarly to the tests for the project national and project testing upgrade files.

Test Modifications

The tests have been updated to remove the tests around the National Upgrades and Testing Upgrades and switch the tests to the SDR project file.

In test/test_analysis_tools.rb: BuildStockBatch sometimes has some extra columns that are blank. Removing these columns from the test as they are blank.

resstock/test/test_analysis_tools.rb

Lines 49 to 54 in 0d7fb07

    
           # Test if BuildStockBatch has extra columns that are not empty 
        
           buildstockbatch_extras = buildstockbatch.headers - run_analysis.headers 
        
           buildstockbatch_extras -= ['simulation_output_report.applicable'] # buildstockbatch contains simulation_output_report.applicable (old workflow) 
        
           empty_columns.each do |col| # Remove empty columns from the diff. BuildStockBatch sometimes has empty blank columns 
        
             buildstockbatch_extras -= [col] 
        
           end

Area for improved testing

This PR is meant to introduce the SDR project file and do some elementary testing (similar to project national, project testing, and national upgrades). At this point, the tests make sure there is some data and columns. Improving the tests that exist today, building out functional tests, and regression testing are important areas for improvement moving forward. Example: using only 2 datapoints being simulated some upgrades do not get run because the upgrade is invalid for the two units.

Related Pull Requests

None.

Related Issues

Closes #1261 -- "Commit ResStock 2024.2 data release yaml to develop"

Checklist

Required:

Technical reference documentation has been updated
Add to the changelog_dev.rst file
No unexpected regression test changes on CI (comparison artifacts have been checked)

Optional (not all items may apply):

Tests (and test files) have been updated
Supporting resource files have been updated (related to resstock-estimation)
openstudio tasks.rb update_measures has been run

joseph-robertson · 2024-12-16T20:27:05Z

Remove some tests? For example, remove national/testing upgrade ymls in favor of SDR upgrade yml.

Update changelog Add workflow generator version update number of allowable upgrades Add files to .gitignore update input and output directories Update options lookup arguments

rename sdr yaml file Unzip and upload sdr upgrade artifact Add testing for sdr upgrade run with bsb Reduce upgrade datapoints on national and testing upgrades ymls revert data point changes and reduce sdr datapoints. Remove testing and national upgrades Update process_bsb_analysis Updates. test_bsb_analysis passing Fix test_run_analysis Update testing yml_valid Update expected warnings Remove testing_upgrades from gitignore Update buildstock_missing.csv Make sdr tests an upgrade Add the upgrade osw and xml files as requested from SDR Update the package name load only headers for comparison Add commented out production number of data points remove Zone People Occupant Count: Conditioned Space from timeseries contents New tsvs with updated well pump update changelog update tsvs

Increase n_datapoints Fix option name change from well pump PR Typo reduce datapoints

Update test_upgrade_results Update test_upgrade_columns Fix paths Fix number of expected files Fix run_analysis file path Fix missing upgrade results file in run_analysis results Fix missing upgrade results file in run_analysis results, try 2 Add path errors during artifact upload Cleanup Update the expected number of expected results Switch upgrade to one that applies to all units

Fix bsb sdr results csv

joseph-robertson · 2025-01-09T03:23:59Z

project_national/sdr_upgrades_tmy3.yml

+  args:
+    # Annual simulation and timestep
+    build_existing_model:
+      simulation_control_timestep: 15


Is it necessary to have 15 instead of 60 here?

We might need to for CI to get a couple more datapoints simulated. 15 min is what we run for the dataset publication.

joseph-robertson · 2025-01-09T03:25:01Z

project_national/sdr_upgrades_tmy3.yml

+
+    # Outputs to include
+    simulation_output_report:
+      timeseries_frequency: timestep


Can this be hourly?

This is what we run for the dataset publication. I would like this to be as close as possible to the simulated version. If we forget to change this then we have to re-simulate the entire dataset. Yes, the downside is limiting the number of data points. Can we make the change later if we figure out a solution to the memory limitations?

joseph-robertson · 2025-01-15T16:49:51Z

project_national/sdr_upgrades_tmy3.yml

@@ -0,0 +1,1149 @@
+schema_version: '0.4'


I'd suggest making the following updates to the baseline yml files for consistency with the new sdr_upgrades_tmy3.yml file:

order the yml blocks the same (e.g., move baseline up)

remove the eagle blocks

joseph-robertson · 2025-01-15T16:56:23Z

.github/workflows/config.yml

-            project_national/national_upgrades/results_csvs/results_up17.csv
-            project_testing/testing_upgrades/results_csvs/results_up17.csv
-          name: buildstockbatch_results_csvs
+            project_national/sdr_upgrades_tmy3/results_csvs/results_up16.csv


For the old upgrade yml files, "17" referred to the final "Package Upgrade" upgrade scenario. I believe the thinking here was that the simulation output would be the most comprehensive for testing purposes. I'm not sure that the final "16" has the same significance here for the new yml file.

Also, you may want to avoid hardcoding "16" here if the new yml will expand over time.

joseph-robertson · 2025-01-15T16:58:36Z

docs/read_the_docs/source/changelog/changelog_dev.rst

@@ -36,3 +36,18 @@ Development Changelog
        resstock-estimation: `pull request 436 <https://github.com/NREL/resstock-estimation/pull/436>`_

        Assignees: Janet Reyna


Outside of scope of this PR, but this heat pump pool heaters entry is missing a .. change:: line above.

joseph-robertson · 2025-01-15T17:01:32Z

measures/ApplyUpgrade/resources/constants.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true

 module Constants
-  NumApplyUpgradeOptions = 25
+  NumApplyUpgradeOptions = 100


This requires an update to RTD at docs/read_the_docs/advanced_tutorial/increasing_upgrade_options.rst.

joseph-robertson · 2025-01-15T17:07:59Z

docs/read_the_docs/source/workflow_inputs/characteristics.rst

@@ -40097,6 +40097,12 @@ Arguments
     - Double
     -
     - The maximum capacity limit applied to the auto-sizing methodology. If not provided, no limit is used.
+   * - ``heating_system_2_fraction_heat_load_served``


This argument is being added here under "HVAC Secondary Heating Efficiency" because you've added new "Fuel Wall/Floor Furnace" options which assign heating_system_2_fraction_heat_load_served. But that argument is already assigned in "HVAC Secondary Heating Partial Space Conditioning"...

joseph-robertson · 2025-01-15T17:11:55Z

resources/data/utility_bills/sdr_rates/README.md

+In the YML file's "simple_filepath" field for utility bill scenario definitions, enter a relative file path to a TSV lookup file (e.g., "data/simple_rates/State.tsv") containing user-defined values corresponding to arguments for fixed costs, marginal rates, and PV.
+The first column of the TSV lookup file contains the name of a chosen parameter for which sets of argument values are assigned according to its options.
+Any blank fields, or missing options for a parameter not specified in the TSV lookup file, will be defaulted.
+
+See the Simple section of OpenStudio-HPXML's documentation for [Electricity Rates](https://openstudio-hpxml.readthedocs.io/en/latest/workflow_inputs.html#electricity-rates), and [Fuel Rates](https://openstudio-hpxml.readthedocs.io/en/latest/workflow_inputs.html#fuel-rates), for more information about arguments for fixed charges, marginal rates, and PV.
+Refer to BuildStockBatch's documentation for [Residential HPXML Workflow Generator](https://buildstockbatch.readthedocs.io/en/stable/workflow_generators/residential_hpxml.html) for more information about YML file arguments.


This appears to be copied from resources/data/utility_bills/README.md. Instead, cite the source of the data contained in sdr_rates/State.tsv?

joseph-robertson · 2025-01-15T17:13:24Z

test/test_bsb_analysis.rb

    @testing_upgrades = 'project_testing/testing_upgrades'
    @national_upgrades = 'project_national/national_upgrades'


Delete these.

joseph-robertson · 2025-01-15T17:16:35Z

test/test_run_analysis.rb

@@ -15,14 +15,13 @@ def before_setup
    buildstock_directory = File.join(File.dirname(__FILE__), '..')

    @testing_baseline = File.join(buildstock_directory, 'testing_baseline')
-    @national_baseline = File.join(buildstock_directory, 'national_baseline')
    @testing_upgrades = File.join(buildstock_directory, 'testing_upgrades')


I was going to suggest deleting this too, but appears its used in some tests below. Suggest a name change so that it's not confused with the "testing project"? Maybe that's overkill; your call.

Initial commit of the srd yaml file

32f3e32

afontani self-assigned this Dec 6, 2024

afontani added this to the ResStock v3.5.0 milestone Dec 6, 2024

afontani added the feature label Dec 6, 2024

afontani changed the title ~~Add Standard Data Release YAML to Integration Tests~~ Add Standard Data Release YAML to GitHub Actions Jan 6, 2025

afontani and others added 3 commits January 7, 2025 22:47

Add SRD yaml into integration tests

17a07fc

Update changelog Add workflow generator version update number of allowable upgrades Add files to .gitignore update input and output directories Update options lookup arguments

Add utility bills

b7ca9f9

Latest results.

416ea9a

afontani force-pushed the sdr_yml branch from dec5462 to f6505c7 Compare January 8, 2025 05:56

afontani and others added 11 commits January 8, 2025 11:30

Latest results.

d043d64

Well pump merge changes

b029136

Increase n_datapoints Fix option name change from well pump PR Typo reduce datapoints

Latest results.

4edd694

Update README.md.

4c9ed32

Fix path

8628fd9

Fix bsb sdr results csv

New tsvs with updated well pump

1234f76

Update test analysis tools

851e4bf

Latest results.

f22c262

Latest results.

122df88

afontani force-pushed the sdr_yml branch from 09e210c to 122df88 Compare January 8, 2025 18:56

joseph-robertson reviewed Jan 9, 2025

View reviewed changes

afontani added 2 commits January 9, 2025 09:50

Merge remote-tracking branch 'origin/develop' into sdr_yaml_test

32f78ed

Update changelog date

fe36207

afontani requested a review from joseph-robertson January 9, 2025 16:59

GitHub Action and others added 3 commits January 9, 2025 18:17

Latest results.

b859e80

Update new results to the base_results

3eec898

remove files

5613ee5

joseph-robertson reviewed Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Standard Data Release YAML to GitHub Actions #1329

Add Standard Data Release YAML to GitHub Actions #1329

afontani commented Dec 6, 2024 •

edited by joseph-robertson

Loading

joseph-robertson commented Dec 16, 2024

joseph-robertson Jan 9, 2025

afontani Jan 9, 2025

joseph-robertson Jan 9, 2025

afontani Jan 9, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

joseph-robertson Jan 15, 2025

	- name: Upload precomputed buildstocks
	uses: actions/upload-artifact@v4
	with:
	name: precomputed_buildstocks
	path: test/tests_yml_files/yml_precomputed/buildstock.csv
	if-no-files-found: error

	# Test if BuildStockBatch has extra columns that are not empty
	buildstockbatch_extras = buildstockbatch.headers - run_analysis.headers
	buildstockbatch_extras -= ['simulation_output_report.applicable'] # buildstockbatch contains simulation_output_report.applicable (old workflow)
	empty_columns.each do \|col\| # Remove empty columns from the diff. BuildStockBatch sometimes has empty blank columns
	buildstockbatch_extras -= [col]
	end

		@@ -36,3 +36,18 @@ Development Changelog
		resstock-estimation: `pull request 436 <https://github.com/NREL/resstock-estimation/pull/436>`_

		Assignees: Janet Reyna

		@testing_upgrades = 'project_testing/testing_upgrades'
		@national_upgrades = 'project_national/national_upgrades'

Add Standard Data Release YAML to GitHub Actions #1329

Are you sure you want to change the base?

Add Standard Data Release YAML to GitHub Actions #1329

Conversation

afontani commented Dec 6, 2024 • edited by joseph-robertson Loading

Pull Request Description

Workflow changes

Test Modifications

Area for improved testing

Related Pull Requests

Related Issues

Checklist

Required:

Optional (not all items may apply):

joseph-robertson commented Dec 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afontani commented Dec 6, 2024 •

edited by joseph-robertson

Loading