CTest: split checksum analysis from test analysis, expose arguments (#…

…5456) Prototype of implementation to see if this can achieve goals such as: 1. run test and analysis locally (possibly without CTest) without worrying about checksums 2. avoid duplicate code for default regression analysis with custom parameters (e.g., tolerance, output format) 3. minimize work needed to implement checksum regression analysis for new tests This PR replaces #5447, see #5447 (comment). ## Old usage 1. Add this to the test analysis script: ```python import os import sys ... ... sys.path.insert(1, "../../../../warpx/Regression/Checksum/") from checksumAPI import evaluate_checksum ... ... # compare checksums evaluate_checksum( test_name=os.path.split(os.getcwd())[1], output_file=sys.argv[1], ) ``` 2. Add this to the CMakeLists.txt file: ```cmake add_warpx_test( test_1d_laser_acceleration_fluid_boosted # name 1 # dims 2 # nprocs inputs_test_1d_laser_acceleration_fluid_boosted # inputs analysis_1d_fluid_boosted.py # analysis diags/diag1000001 # output OFF # dependency ) ``` ## New usage 1. Create a soft link to the default regression analysis script from the test directory (we already do this when there isn't a custom analysis script, see comment in the follow-up section below): ```console ln -s ../../analysis_default_regression.py analysis_default_regression.py ``` 2. Add this to the CMakeLists.txt file: ```cmake add_warpx_test( test_1d_laser_acceleration_fluid_boosted # name 1 # dims 2 # nprocs inputs_test_1d_laser_acceleration_fluid_boosted # inputs "analysis_1d_fluid_boosted.py diags/diag1000001" # analysis "analysis_default_regression.py --path diags/diag1000001" # checksum OFF # dependency ) ``` ## Notes - The updated default regression analysis script has the following usage: ```console usage: analysis_default_regression.py [-h] [--path PATH] [--rtol RTOL] [--skip-fields] [--skip-particles] options: -h, --help show this help message and exit --path PATH path to output file(s) --rtol RTOL relative tolerance to compare checksums --skip-fields skip fields when comparing checksums --skip-particles skip particles when comparing checksums ``` - The checksum files that changed (as opposed to the ones that were added from scratch) changed because the corresponding analysis scripts were not performing any checksum analysis. The files had been added but they were not used by the test analysis, hence they were outdated. ## To-do - [x] Add missing checksum files or update existing checksum files that were not used - [x] Update documentation ## Follow-up - Improve documentation even more: - do we need all user-facing features of `checksumAPI` or are some obsolete? - can we merge documentation for testing and checksums into one section? - Check/fix custom tolerances (`git grep "# checksum" Examples/ | grep "rtol"`) - Add logic to reset tolerances based on environment variables (e.g., to run all tests in single-precision) - Update custom test analysis scripts so that they do not need to take the output path as argument (would be difficult to maintain as a convention, though, as these scripts are up to each PR author) - Try (again) to make it work with direct path rather than soft link (did not work when we tried in the past)
ECP-WarpX · Dec 19, 2024 · d79fe71 · d79fe71
1 parent b4eebb9
commit d79fe71
Show file tree

Hide file tree

Showing 290 changed files with 1,259 additions and 1,983 deletions.
diff --git a/Docs/source/developers/testing.rst b/Docs/source/developers/testing.rst
@@ -140,8 +140,8 @@ A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as
            1  # dims
            2  # nprocs
            inputs_test_1d_laser_acceleration  # inputs
-           analysis.py  # analysis
-           diags/diag1000100  # output (plotfile)
+           "analysis.py diags/diag1000100"  # analysis
+           "analysis_default_regression.py --path diags/diag1000100"  # checksum
            OFF  # dependency
        )
 
@@ -154,8 +154,8 @@ A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as
            2  # dims
            2  # nprocs
            inputs_test_2d_laser_acceleration_picmi.py  # inputs
-           analysis.py  # analysis
-           diags/diag1000100  # output (plotfile)
+           "analysis.py diags/diag1000100"  # analysis
+           "analysis_default_regression.py --path diags/diag1000100"  # checksum
            OFF  # dependency
        )
 
@@ -168,14 +168,14 @@ A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as
            3  # dims
            2  # nprocs
            inputs_test_3d_laser_acceleration_restart  # inputs
-           analysis_default_restart.py  # analysis
-           diags/diag1000100  # output (plotfile)
+           "analysis_default_restart.py diags/diag1000100"  # analysis
+           "analysis_default_regression.py --path diags/diag1000100"  # checksum
            test_3d_laser_acceleration  # dependency
        )
 
   Note that the restart has an explicit dependency, namely it can run only provided that the original test, from which the restart checkpoint files will be read, runs first.
 
-* A more complex example. Add the **PICMI test** ``test_rz_laser_acceleration_picmi``, with custom command-line arguments ``--test`` and ``dir``, and openPMD time series output:
+* A more complex example. Add the **PICMI test** ``test_rz_laser_acceleration_picmi``, with custom command-line arguments ``--test`` and ``dir``, openPMD time series output, and custom command line arguments for the checksum comparison:
 
   .. code-block:: cmake
 
@@ -184,18 +184,38 @@ A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as
            RZ  # dims
            2   # nprocs
            "inputs_test_rz_laser_acceleration_picmi.py --test --dir 1"  # inputs
-           analysis.py  # analysis
-           diags/diag1/  # output (openPMD time series)
+           "analysis.py diags/diag1/"  # analysis
+           "analysis_default_regression.py --path diags/diag1/ --skip-particles --rtol 1e-7"  # checksum
            OFF  # dependency
        )
 
+The ``analysis`` and ``checksum`` commands passed as arguments to ``add_warpx_test`` can be set to ``OFF`` if the intention is to skip the respective analysis for a given test.
+
 If you need a new Python package dependency for testing, please add it in `Regression/requirements.txt <https://github.com/ECP-WarpX/WarpX/blob/development/Regression/requirements.txt>`__.
 
 Sometimes two or more tests share a large number of input parameters.
 The shared input parameters can be collected in a "base" input file that can be passed as a runtime parameter in the actual test input files through the parameter ``FILE``.
 
 If the new test is added in a new directory that did not exist before, please add the name of that directory with the command ``add_subdirectory`` in `Physics_applications/CMakeLists.txt <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications/CMakeLists.txt>`__ or `Tests/CMakeLists.txt <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests/CMakeLists.txt>`__, depending on where the new test directory is located.
 
+If not already present, the default regression analysis script ``analysis_default_regression.py`` in the examples above must be linked from `Examples/analysis_default_regression.py <https://github.com/ECP-WarpX/WarpX/blob/development/Examples/analysis_default_regression.py>`__, by executing once the following command from the test directory:
+
+  .. code-block:: bash
+
+       ln -s ../../analysis_default_regression.py analysis_default_regression.py
+
+Here is the help message of the default regression analysis script, including usage and list of available options and arguments:
+
+  .. code-block:: bash
+
+       usage: analysis_default_regression.py [-h] [--path PATH] [--rtol RTOL] [--skip-fields] [--skip-particles]
+       options:
+         -h, --help        show this help message and exit
+         --path PATH       path to output file(s)
+         --rtol RTOL       relative tolerance to compare checksums
+         --skip-fields     skip fields when comparing checksums
+         --skip-particles  skip particles when comparing checksums
+
 Naming conventions for automated tests
 --------------------------------------
 

diff --git a/Examples/CMakeLists.txt b/Examples/CMakeLists.txt
@@ -21,8 +21,8 @@ endif()
 # dims: 1,2,RZ,3
 # nprocs: 1 or 2 (maybe refactor later on to just depend on WarpX_MPI)
 # inputs: inputs file or PICMI script, WarpX_MPI decides w/ or w/o MPI
-# analysis: analysis script, always run without MPI
-# output: output file(s) to analyze
+# analysis: custom test analysis command, always run without MPI
+# checksum: default regression analysis command (checksum benchmark)
 # dependency: name of base test that must run first
 #
 function(add_warpx_test
@@ -31,7 +31,7 @@ function(add_warpx_test
     nprocs
     inputs
     analysis
-    output
+    checksum
     dependency
 )
     # cannot run MPI tests w/o MPI build
@@ -72,14 +72,25 @@ function(add_warpx_test
     separate_arguments(ANALYSIS_LIST UNIX_COMMAND "${analysis}")
     list(GET ANALYSIS_LIST 0 ANALYSIS_FILE)
     cmake_path(SET ANALYSIS_FILE "${CMAKE_CURRENT_SOURCE_DIR}/${ANALYSIS_FILE}")
-    # TODO Enable lines below to handle command-line arguments
-    #list(LENGTH ANALYSIS_LIST ANALYSIS_LIST_LENGTH)
-    #if(ANALYSIS_LIST_LENGTH GREATER 1)
-    #    list(SUBLIST ANALYSIS_LIST 1 -1 ANALYSIS_ARGS)
-    #    list(JOIN ANALYSIS_ARGS " " ANALYSIS_ARGS)
-    #else()
-    #    set(ANALYSIS_ARGS "")
-    #endif()
+    list(LENGTH ANALYSIS_LIST ANALYSIS_LIST_LENGTH)
+    if(ANALYSIS_LIST_LENGTH GREATER 1)
+        list(SUBLIST ANALYSIS_LIST 1 -1 ANALYSIS_ARGS)
+        list(JOIN ANALYSIS_ARGS " " ANALYSIS_ARGS)
+    else()
+        set(ANALYSIS_ARGS "")
+    endif()
+
+    # get checksum script and optional command-line arguments
+    separate_arguments(CHECKSUM_LIST UNIX_COMMAND "${checksum}")
+    list(GET CHECKSUM_LIST 0 CHECKSUM_FILE)
+    cmake_path(SET CHECKSUM_FILE "${CMAKE_CURRENT_SOURCE_DIR}/${CHECKSUM_FILE}")
+    list(LENGTH CHECKSUM_LIST CHECKSUM_LIST_LENGTH)
+    if(CHECKSUM_LIST_LENGTH GREATER 1)
+        list(SUBLIST CHECKSUM_LIST 1 -1 CHECKSUM_ARGS)
+        list(JOIN CHECKSUM_ARGS " " CHECKSUM_ARGS)
+    else()
+        set(CHECKSUM_ARGS "")
+    endif()
 
     # Python test?
     set(python OFF)
@@ -175,25 +186,52 @@ function(add_warpx_test
 
     # test analysis
     if(analysis)
+        # for argparse, do not pass command-line arguments as one quoted string
+        separate_arguments(ANALYSIS_ARGS UNIX_COMMAND "${ANALYSIS_ARGS}")
         add_test(
             NAME ${name}.analysis
             COMMAND
-                ${THIS_Python_SCRIPT_EXE} ${ANALYSIS_FILE}
-                ${output}
+                ${THIS_Python_SCRIPT_EXE}
+                ${ANALYSIS_FILE}
+                ${ANALYSIS_ARGS}
             WORKING_DIRECTORY ${THIS_WORKING_DIR}
         )
         # test analysis depends on test run
         set_property(TEST ${name}.analysis APPEND PROPERTY DEPENDS "${name}.run")
         # FIXME Use helper function to handle Windows exceptions
         set(PYTHONPATH "$ENV{PYTHONPATH}:${CMAKE_PYTHON_OUTPUT_DIRECTORY}")
         # add paths for custom Python modules
-        set(PYTHONPATH "${PYTHONPATH}:${WarpX_SOURCE_DIR}/Regression/Checksum")
         set(PYTHONPATH "${PYTHONPATH}:${WarpX_SOURCE_DIR}/Regression/PostProcessingUtils")
         set(PYTHONPATH "${PYTHONPATH}:${WarpX_SOURCE_DIR}/Tools/Parser")
         set(PYTHONPATH "${PYTHONPATH}:${WarpX_SOURCE_DIR}/Tools/PostProcessing")
         set_property(TEST ${name}.analysis APPEND PROPERTY ENVIRONMENT "PYTHONPATH=${PYTHONPATH}")
     endif()
 
+    # checksum analysis
+    if(checksum)
+        # for argparse, do not pass command-line arguments as one quoted string
+        separate_arguments(CHECKSUM_ARGS UNIX_COMMAND "${CHECKSUM_ARGS}")
+        add_test(
+            NAME ${name}.checksum
+            COMMAND
+                ${THIS_Python_SCRIPT_EXE}
+                ${CHECKSUM_FILE}
+                ${CHECKSUM_ARGS}
+            WORKING_DIRECTORY ${THIS_WORKING_DIR}
+        )
+        # test analysis depends on test run
+        set_property(TEST ${name}.checksum APPEND PROPERTY DEPENDS "${name}.run")
+        if(analysis)
+            # checksum analysis depends on test analysis
+            set_property(TEST ${name}.checksum APPEND PROPERTY DEPENDS "${name}.analysis")
+        endif()
+        # FIXME Use helper function to handle Windows exceptions
+        set(PYTHONPATH "$ENV{PYTHONPATH}:${CMAKE_PYTHON_OUTPUT_DIRECTORY}")
+        # add paths for custom Python modules
+        set(PYTHONPATH "${PYTHONPATH}:${WarpX_SOURCE_DIR}/Regression/Checksum")
+        set_property(TEST ${name}.checksum APPEND PROPERTY ENVIRONMENT "PYTHONPATH=${PYTHONPATH}")
+    endif()
+
     # CI: remove test directory after run
     if(WarpX_TEST_CLEANUP)
         add_test(
@@ -206,6 +244,10 @@ function(add_warpx_test
             # test cleanup depends on test analysis
             set_property(TEST ${name}.cleanup APPEND PROPERTY DEPENDS "${name}.analysis")
         endif()
+        if(checksum)
+            # test cleanup depends on test analysis
+            set_property(TEST ${name}.cleanup APPEND PROPERTY DEPENDS "${name}.checksum")
+        endif()
     endif()
 
     # Do we depend on another test?
@@ -215,6 +257,9 @@ function(add_warpx_test
         if(analysis)
             set_property(TEST ${name}.run APPEND PROPERTY DEPENDS "${dependency}.analysis")
         endif()
+        if(checksum)
+            set_property(TEST ${name}.run APPEND PROPERTY DEPENDS "${dependency}.checksum")
+        endif()
         if(WarpX_TEST_CLEANUP)
             # do not clean up dependency test before current test is completed
             set_property(TEST ${dependency}.cleanup APPEND PROPERTY DEPENDS "${name}.cleanup")

diff --git a/Examples/Physics_applications/beam_beam_collision/CMakeLists.txt b/Examples/Physics_applications/beam_beam_collision/CMakeLists.txt
@@ -6,8 +6,8 @@ add_warpx_test(
     3  # dims
     2  # nprocs
     inputs_test_3d_beam_beam_collision  # inputs
-    analysis_default_openpmd_regression.py  # analysis
-    diags/diag1/  # output
+    OFF  # analysis
+    "analysis_default_regression.py --path diags/diag1/"  # checksum
     OFF  # dependency
 )
 label_warpx_test(test_3d_beam_beam_collision slow)
diff --git a/Examples/Physics_applications/beam_beam_collision/analysis_default_openpmd_regression.py b/Examples/Physics_applications/beam_beam_collision/analysis_default_openpmd_regression.py
diff --git a/Examples/Physics_applications/beam_beam_collision/analysis_default_regression.py b/Examples/Physics_applications/beam_beam_collision/analysis_default_regression.py
@@ -0,0 +1 @@
+../../analysis_default_regression.py
diff --git a/Examples/Physics_applications/capacitive_discharge/CMakeLists.txt b/Examples/Physics_applications/capacitive_discharge/CMakeLists.txt
@@ -6,8 +6,8 @@ add_warpx_test(
     1  # dims
     2  # nprocs
     "inputs_base_1d_picmi.py --test --pythonsolver"  # inputs
-    analysis_1d.py  # analysis
-    diags/diag1000050  # output
+    "analysis_1d.py"  # analysis
+    "analysis_default_regression.py --path diags/diag1000050"  # checksum
     OFF  # dependency
 )
 
@@ -16,8 +16,8 @@ add_warpx_test(
     1  # dims
     2  # nprocs
     "inputs_base_1d_picmi.py --test --dsmc"  # inputs
-    analysis_dsmc.py  # analysis
-    diags/diag1000050  # output
+    "analysis_dsmc.py"  # analysis
+    "analysis_default_regression.py --path diags/diag1000050"  # checksum
     OFF  # dependency
 )
 
@@ -26,19 +26,19 @@ add_warpx_test(
     2  # dims
     2  # nprocs
     inputs_test_2d_background_mcc  # inputs
-    analysis_default_regression.py  # analysis
-    diags/diag1000050  # output
+    OFF  # analysis
+    "analysis_default_regression.py --path diags/diag1000050"  # checksum
     OFF  # dependency
 )
 
-# FIXME: can we make this a single precision for now?
+# FIXME: can we make this single precision for now?
 #add_warpx_test(
 #    test_2d_background_mcc_dp_psp  # name
 #    2  # dims
 #    2  # nprocs
-##    inputs_test_2d_background_mcc_dp_psp  # inputs
-#    analysis_default_regression.py  # analysis
-#    diags/diag1000050  # output
+#    inputs_test_2d_background_mcc_dp_psp  # inputs
+#    OFF  # analysis
+#    "analysis_default_regression.py --path diags/diag1000050"  # checksum
 #    OFF  # dependency
 #)
 
@@ -47,7 +47,7 @@ add_warpx_test(
     2  # dims
     2  # nprocs
     inputs_test_2d_background_mcc_picmi.py  # inputs
-    analysis_2d.py  # analysis
-    diags/diag1000050  # output
+    OFF  # analysis
+    "analysis_default_regression.py --path diags/diag1000050 --rtol 5e-3"  # checksum
     OFF  # dependency
 )
diff --git a/Examples/Physics_applications/capacitive_discharge/analysis_1d.py b/Examples/Physics_applications/capacitive_discharge/analysis_1d.py
@@ -2,14 +2,8 @@
 
 # Copyright 2022 Modern Electron, David Grote
 
-import os
-import sys
-
 import numpy as np
 
-sys.path.insert(1, "../../../../warpx/Regression/Checksum/")
-from checksumAPI import evaluate_checksum
-
 # fmt: off
 ref_density = np.array([
        1.27989677e+14, 2.23601330e+14, 2.55400265e+14, 2.55664972e+14,
@@ -51,9 +45,3 @@
 density_data = np.load("ion_density_case_1.npy")
 print(repr(density_data))
 assert np.allclose(density_data, ref_density)
-
-# compare checksums
-evaluate_checksum(
-    test_name=os.path.split(os.getcwd())[1],
-    output_file=sys.argv[1],
-)
diff --git a/Examples/Physics_applications/capacitive_discharge/analysis_2d.py b/Examples/Physics_applications/capacitive_discharge/analysis_2d.py
diff --git a/Examples/Physics_applications/capacitive_discharge/analysis_dsmc.py b/Examples/Physics_applications/capacitive_discharge/analysis_dsmc.py
@@ -2,14 +2,8 @@
 
 # 2023 TAE Technologies
 
-import os
-import sys
-
 import numpy as np
 
-sys.path.insert(1, "../../../../warpx/Regression/Checksum/")
-from checksumAPI import evaluate_checksum
-
 # fmt: off
 ref_density = np.array([
     1.27942709e+14, 2.23579371e+14, 2.55384387e+14, 2.55660663e+14,
@@ -51,9 +45,3 @@
 density_data = np.load("ion_density_case_1.npy")
 print(repr(density_data))
 assert np.allclose(density_data, ref_density)
-
-# compare checksums
-evaluate_checksum(
-    test_name=os.path.split(os.getcwd())[1],
-    output_file=sys.argv[1],
-)
diff --git a/Examples/Physics_applications/free_electron_laser/CMakeLists.txt b/Examples/Physics_applications/free_electron_laser/CMakeLists.txt
@@ -6,7 +6,7 @@ add_warpx_test(
     1  # dims
     2  # nprocs
     inputs_test_1d_fel  # inputs
-    analysis_fel.py  # analysis
-    diags/diag_labframe  # output
+    "analysis_fel.py diags/diag_labframe"  # analysis
+    "analysis_default_regression.py --path diags/diag_labframe"  # checksum
     OFF  # dependency
 )
diff --git a/Examples/Physics_applications/free_electron_laser/analysis_default_regression.py b/Examples/Physics_applications/free_electron_laser/analysis_default_regression.py
@@ -0,0 +1 @@
+../../analysis_default_regression.py
diff --git a/Examples/Physics_applications/free_electron_laser/analysis_fel.py b/Examples/Physics_applications/free_electron_laser/analysis_fel.py
@@ -17,16 +17,12 @@
 lab-frame diagnostics and boosted-frame diagnostics.
 """
 
-import os
 import sys
 
 import numpy as np
 from openpmd_viewer import OpenPMDTimeSeries
 from scipy.constants import c, e, m_e
 
-sys.path.insert(1, "../../../../warpx/Regression/Checksum/")
-from checksumAPI import evaluate_checksum
-
 # Physical parameters of the test
 gamma_bunch = 100.6
 Bu = 0.5
@@ -136,10 +132,3 @@ def extract_peak_E_boost(iteration):
 lambda_radiation_lab = lambda_radiation_boost / (2 * gamma_boost)
 lambda_expected = lambda_u / (2 * gamma_boost**2)
 assert abs(lambda_radiation_lab - lambda_expected) / lambda_expected < 0.01
-
-# compare checksums
-evaluate_checksum(
-    test_name=os.path.split(os.getcwd())[1],
-    output_file=sys.argv[1],
-    output_format="openpmd",
-)