Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: improve "Workflows" section for developers #5525

Open
wants to merge 8 commits into
base: development
Choose a base branch
from
Open

Docs: improve "Workflows" section for developers #5525

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7797e89
Start refactoring of workflows section for developers
EZoni Dec 19, 2024
a796ade
Work on test section
EZoni Dec 20, 2024
2b6b24b
More work on test section
EZoni Dec 21, 2024
1f50a25
Reduce and merge checksums documentation
EZoni Jan 9, 2025
8cc9aa7
Merge `ECP-WarpX:development` into `EZoni:docs_howto`
EZoni Jan 9, 2025
8f5c728
Rename headers of other how-to sections
EZoni Jan 9, 2025
7d9ae40
Clean up how-to section on clang-tidy linter
EZoni Jan 9, 2025
07c18e4
Add info on AMReX's post-processing tools
EZoni Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions Docs/source/developers/workflows.rst → Docs/source/developers/how_to_guides.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
.. _development-workflows:
.. _development-howtoguides:

Workflows
=========
How-to guides
=============

.. toctree::
:maxdepth: 1

profiling
testing
checksum
how_to_profile
how_to_test
run_clang_tidy_locally
local_compile
documentation
4 changes: 2 additions & 2 deletions Docs/source/developers/profiling.rst → Docs/source/developers/how_to_profile.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _developers-profiling:

Profiling the Code
==================
How to profile the code
=======================

Profiling allows us to find the bottle-necks of the code as it is currently implemented.
Bottle-necks are the parts of the code that may delay the simulation, making it more computationally expensive.
Expand Down
171 changes: 56 additions & 115 deletions Docs/source/developers/testing.rst → Docs/source/developers/how_to_test.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,40 @@
.. _developers-testing:

Testing the Code
================
How to test the code
====================

When proposing a code change, you want to make sure that
When you propose code changes, you want to make sure that

* the code change does not break the existing code;
* the code change gives correct results (numerics, physics, etc.).
* the code changes do not break the behavior of the rest of the code;
* the code changes give correct results (numerics, physics, etc.).

WarpX follows the continuous integration (CI) software development practice, where automated builds and tests are run after merging code changes into the main branch.

While the code is tested regularly remotely (on the cloud when commits are pushed to an open PR, and every night on local clusters), it can also be useful to run tests on your custom input file.
Following the continuous integration (CI) software development practice, WarpX runs automated builds and tests after a commit is pushed to an open PR as well as after a PR is merged into the main branch.

How to run pre-commit tests locally
-----------------------------------

First, when proposing a code change, we perform a couple of automated style and correctness checks.
First, WarpX uses `pre-commit <https://pre-commit.com/>`__ to perform automated style and correctness checks.

Here is how to install ``pre-commit`` locally:

#. Install ``pre-commit``:

.. code-block:: sh

If you install the ``pre-commit`` tool on your local machine via
python -m pip install -U pre-commit

.. code-block:: sh
#. Install the git hook scripts:

python -m pip install -U pre-commit
pre-commit install
.. code-block:: sh

the style and correctness checks will run automatically on your local machine, after you commit the change and before you push.
pre-commit install

If you do not install the ``pre-commit`` tool on your local machine, these checks will run automatically as part of our CI workflows and a commit containing style and correctness changes might be added automatically to your branch.
In that case, you will need to pull that automated commit before pushing further changes.
If you install ``pre-commit`` locally, the style and correctness checks will run automatically on your computer, after you commit the code changes and before you push them to the remote repository.

See `pre-commit.com <https://pre-commit.com>`__ and our ``.pre-commit-config.yaml`` file in the repository for more details.
If you do not install ``pre-commit`` locally, these checks will run automatically as part of our CI workflows and a commit containing style and correctness changes might be added automatically to your branch after you have pushed your own commit.
In that case, you will need to pull that automated commit before pushing further commits.

The configuration options for ``pre-commit`` are set in the `pre-commit-config.yaml <https://github.com/ECP-WarpX/WarpX/blob/development/.pre-commit-config.yaml>`__ file.

How to configure the automated tests
------------------------------------
Expand All @@ -43,11 +48,11 @@ A test that requires a build option that was not configured and built will be sk
How to run automated tests locally
----------------------------------

Once your new feature is ready, there are ways to check that you did not break anything.
Once your code changes are ready, there are ways to check that they do not break the rest of the code.
WarpX has automated tests running every time a commit is pushed to an open pull request.
The input files and scripts used by the automated tests can be found in the `Examples <https://github.com/ECP-WarpX/WarpX/tree/development/Examples>`__ directory, either under `Physics_applications <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications>`__ or `Tests <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests>`__.

For easier debugging, it can be convenient to run the tests on your local machine by executing CTest as illustrated in the examples below (where we assume that WarpX was configured and built in the directory ``build``):
For easier debugging, it can be convenient to run the tests on your local computer by executing CTest as illustrated in the examples below (where we assume that WarpX was configured and built in the directory ``build``):

* List tests available for the current build options:

Expand Down Expand Up @@ -101,7 +106,7 @@ For easier debugging, it can be convenient to run the tests on your local machin
ctest --test-dir build -R "test_3d_langmuir_multi\..*"

Note that filtering with ``-R "test_3d_langmuir_multi"`` would include the additional tests that have the same substring in their name and would not be sufficient to isolate a single test.
Note also that the escaping ``\.`` in the regular expression is necessary in order to take into account the fact that each test is automatically appended with the strings ``.run``, ``.analysis`` and possibly ``.cleanup``.
Note also that the escaping ``\.`` in the regular expression is necessary in order to take into account the fact that each test is automatically appended with the strings ``.run``, ``.analysis``, ``.checksum`` and possibly ``.cleanup``.

* Run only tests not labeled with the ``slow`` label:

Expand All @@ -120,74 +125,54 @@ How to add automated tests
An automated test typically consists of the following components:

* input file or PICMI input script;

* analysis script;
* checksum file.

To learn more about how to use checksums in automated tests, please see the corresponding section :ref:`Checksums on Tests <developers-checksum>`.
* checksum file.

As mentioned above, the input files and scripts used by the automated tests can be found in the `Examples <https://github.com/ECP-WarpX/WarpX/tree/development/Examples>`__ directory, under either `Physics_applications <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications>`__ or `Tests <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests>`__.

Each test directory must contain a file named ``CMakeLists.txt`` where all tests associated with the input files and scripts in that directory must be listed.

A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as illustrated in the examples below:
A checksum file is [to-do] ...

* Add the **regular test** ``test_1d_laser_acceleration``:
A new test can be added by calling the function ``add_warpx_test`` in ``CMakeLists.txt``. The function has the following signature:

.. code-block:: cmake
.. code-block:: cmake

add_warpx_test(
test_1d_laser_acceleration # name
1 # dims
2 # nprocs
inputs_test_1d_laser_acceleration # inputs
"analysis.py diags/diag1000100" # analysis
"analysis_default_regression.py --path diags/diag1000100" # checksum
OFF # dependency
)
function(add_warpx_test
name # unique test name:
# test_1d_example, test_2d_example_picmi, etc.
dims # dimensionality: 1, 2, 3, RZ
nprocs # number of processes: 1, 2
inputs # inputs file or PICMI script:
# inputs_test_1d_example, inputs_test_2d_example_picmi.py, "inputs_test_2d_example_picmi.py arg1 arg2", etc.
analysis # custom test analysis command:
# OFF, "analysis.py", "analysis.py arg1 arg2", etc.
checksum # default regression analysis command:
# OFF, "analysis_default_regression.py --path diags/diag1", etc.
dependency # name of base test that must run first (must match name exactly):
# OFF, test_1d_example_prepare, etc.
)

* Add the **PICMI test** ``test_2d_laser_acceleration_picmi``:
Here's how to add an automated test:

.. code-block:: cmake
#. Choose the test directory, either an existing one or a new one.

add_warpx_test(
test_2d_laser_acceleration_picmi # name
2 # dims
2 # nprocs
inputs_test_2d_laser_acceleration_picmi.py # inputs
"analysis.py diags/diag1000100" # analysis
"analysis_default_regression.py --path diags/diag1000100" # checksum
OFF # dependency
)
#. Add an input file or PICMI input script.
The name must follow the naming conventions described in the section :ref:`developers-testing-naming` below.

* Add the **restart test** ``test_3d_laser_acceleration_restart``:
#. Add a Python analysis script to analyze the results of the test.

.. code-block:: cmake
#. Add the test to the ``CMakeLists.txt`` file (add such file if you are adding the test in a new test directory) using the function ``add_warpx_test`` mentioned above.

add_warpx_test(
test_3d_laser_acceleration_restart # name
3 # dims
2 # nprocs
inputs_test_3d_laser_acceleration_restart # inputs
"analysis_default_restart.py diags/diag1000100" # analysis
"analysis_default_regression.py --path diags/diag1000100" # checksum
test_3d_laser_acceleration # dependency
)
#. If the test directory is new, add the directory with the command ``add_subdirectory`` in `Physics_applications/CMakeLists.txt <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications/CMakeLists.txt>`__ or `Tests/CMakeLists.txt <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests/CMakeLists.txt>`__, depending on where the test directory is located.

Note that the restart has an explicit dependency, namely it can run only provided that the original test, from which the restart checkpoint files will be read, runs first.
#. If the test directory is new, make a symbolic link to the default regression analysis script ``analysis_default_regression.py`` from `Examples/analysis_default_regression.py <https://github.com/ECP-WarpX/WarpX/blob/development/Examples/analysis_default_regression.py>`__, by running ``ln -s ../../analysis_default_regression.py analysis_default_regression.py`` from the test directory.

* A more complex example. Add the **PICMI test** ``test_rz_laser_acceleration_picmi``, with custom command-line arguments ``--test`` and ``dir``, openPMD time series output, and custom command line arguments for the checksum comparison:
#. Run the test locally with ``ctest`` by prefixing the ``ctest`` command with ``CHECKSUM_RESET=ON`` in order to generate automatically the checksum file.

.. code-block:: cmake

add_warpx_test(
test_rz_laser_acceleration_picmi # name
RZ # dims
2 # nprocs
"inputs_test_rz_laser_acceleration_picmi.py --test --dir 1" # inputs
"analysis.py diags/diag1/" # analysis
"analysis_default_regression.py --path diags/diag1/ --skip-particles --rtol 1e-7" # checksum
OFF # dependency
)
Once you have added the test, run the test locally again, without ``CHECKSUM_RESET=ON``, to check that everything works as expected.

The ``analysis`` and ``checksum`` commands passed as arguments to ``add_warpx_test`` can be set to ``OFF`` if the intention is to skip the respective analysis for a given test.

Expand All @@ -196,14 +181,6 @@ If you need a new Python package dependency for testing, please add it in `Regre
Sometimes two or more tests share a large number of input parameters.
The shared input parameters can be collected in a "base" input file that can be passed as a runtime parameter in the actual test input files through the parameter ``FILE``.

If the new test is added in a new directory that did not exist before, please add the name of that directory with the command ``add_subdirectory`` in `Physics_applications/CMakeLists.txt <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Physics_applications/CMakeLists.txt>`__ or `Tests/CMakeLists.txt <https://github.com/ECP-WarpX/WarpX/tree/development/Examples/Tests/CMakeLists.txt>`__, depending on where the new test directory is located.

If not already present, the default regression analysis script ``analysis_default_regression.py`` in the examples above must be linked from `Examples/analysis_default_regression.py <https://github.com/ECP-WarpX/WarpX/blob/development/Examples/analysis_default_regression.py>`__, by executing once the following command from the test directory:

.. code-block:: bash

ln -s ../../analysis_default_regression.py analysis_default_regression.py

Here is the help message of the default regression analysis script, including usage and list of available options and arguments:

.. code-block:: bash
Expand All @@ -216,6 +193,8 @@ Here is the help message of the default regression analysis script, including us
--skip-fields skip fields when comparing checksums
--skip-particles skip particles when comparing checksums

.. _developers-testing-naming:

Naming conventions for automated tests
--------------------------------------

Expand All @@ -230,41 +209,3 @@ Note that we currently obey the following snake\_case naming conventions for tes
#. **Test input files** start with the string ``inputs_`` followed by the test name. For example, ``inputs_test_3d_laser_acceleration`` or ``inputs_test_3d_laser_acceleration_picmi.py`` or ``inputs_test_3d_laser_acceleration_restart``.

#. **Base input files** (that is, files collecting input parameters shared between two or more tests) are typically named ``inputs_base_1d``, ``inputs_base_2d``, ``inputs_base_3d`` or ``inputs_base_rz``, possibly followed by additional strings if need be.

Useful tool for plotfile comparison: ``fcompare``
-------------------------------------------------

AMReX provides ``fcompare``, an executable that takes two ``plotfiles`` as input and returns the absolute and relative difference for each field between these two plotfiles. For some changes in the code, it is very convenient to run the same input file with an old and your current version, and ``fcompare`` the plotfiles at the same iteration. To use it:

.. code-block:: sh

# Compile the executable
cd <path to AMReX>/Tools/Plotfile/ # This may change
make -j 8
# Run the executable to compare old and new versions
<path to AMReX>/Tools/Plotfile/fcompare.gnu.ex old/plt00200 new/plt00200

which should return something like

.. code-block:: sh

variable name absolute error relative error
(||A - B||) (||A - B||/||A||)
----------------------------------------------------------------------------
level = 0
jx 1.044455105e+11 1.021651316
jy 4.08631977e+16 7.734299273
jz 1.877301764e+14 1.073458933
Ex 4.196315448e+10 1.253551615
Ey 3.330698083e+12 6.436470137
Ez 2.598167798e+10 0.6804387128
Bx 273.8687473 2.340209782
By 152.3911863 1.10952567
Bz 37.43212767 2.1977289
part_per_cell 15 0.9375
Ex_fp 4.196315448e+10 1.253551615
Ey_fp 3.330698083e+12 6.436470137
Ez_fp 2.598167798e+10 0.6804387128
Bx_fp 273.8687473 2.340209782
By_fp 152.3911863 1.10952567
Bz_fp 37.43212767 2.1977289
EZoni marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion Docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Development
developers/developers
developers/doxygen
developers/gnumake
developers/workflows
developers/how_to_guides
developers/faq
.. good to have in the future:
.. developers/repostructure
Expand Down