-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWICM3 on aleph TCo1279-DART plus rudamentary intel compiler config. #1000
Open
JanStreffing
wants to merge
21
commits into
release
Choose a base branch
from
feat/aleph_intel_openmpi
base: release
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 15 commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
7253113
first shot at getting intel compiler with openmpi to work on aleph
56bdd10
more failed configs
84407dc
cpu specific intel flags for aleph
892b2b8
almost there with mpich
660b5e8
works with cray mpich in the end
6829c8c
Merge remote-tracking branch 'origin/release' into feat/aleph_intel_o…
976069e
working TCo1279-DART with restart on aleph
053d187
namelist.io as used for TCO1279-DART
f9c591c
oifs output for high res sims
677235d
fix issue #1009
mandresm 393fa97
add warnings
JanStreffing d19b3d6
Merge remote-tracking branch 'origin/fix/resolve_symlinks' into feat/…
907e334
adding CTRL runscript
8aa04ff
splitting monthly
a9d2072
adding potential density to default awicm3 frontiers output
e76ffc6
Update configs/machines/aleph.yaml
JanStreffing b4f2104
fix typo
53e3ad4
Merge branch 'feat/aleph_intel_openmpi' of ssh://github.com/esm-tools…
3a3a316
configs for TCO1279 on aleph
972edee
update IO to match req
30291d8
netcdf restarts deactivated via binary
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,18 +1,19 @@ | ||
| add_compiletime_environment_changes: | ||
| choose_computer.fc: | ||
| ftn: | ||
| add_export_vars: | ||
| - 'OASIS_FFLAGS=-emf' | ||
| choose_computer.name: | ||
| levante: | ||
| add_export_vars: | ||
| OASIS_FFLAGS: '"-march=core-avx2 -mtune=core-avx2"' | ||
| albedo: | ||
| choose_computer.compiler_suite: | ||
| gcc: | ||
| add_export_vars: | ||
| FFLAGS: "'-fallow-argument-mismatch -w'" | ||
| FCFLAGS: "'-fallow-argument-mismatch -w'" | ||
| intel-oneapi: | ||
| add_export_vars: | ||
| OASIS_FFLAGS: '"-march=core-avx2"' | ||
| choose_computer.name: | ||
| aleph: | ||
| choose_computer.useMPI: | ||
| cray_mpich: | ||
| add_export_vars: | ||
| OASIS_FFLAGS: '"-emf"' | ||
| levante: | ||
| add_export_vars: | ||
| OASIS_FFLAGS: '"-march=core-avx2 -mtune=core-avx2"' | ||
| albedo: | ||
| choose_computer.compiler_suite: | ||
| gcc: | ||
| add_export_vars: | ||
| FFLAGS: "'-fallow-argument-mismatch -w'" | ||
| FCFLAGS: "'-fallow-argument-mismatch -w'" | ||
| intel-oneapi: | ||
| add_export_vars: | ||
| OASIS_FFLAGS: '"-march=core-avx2"' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,53 +46,135 @@ hyper_flag: "" | |
| # | ||
| # | ||
|
|
||
| useMPI: cray_mpich | ||
|
|
||
| fc: ftn | ||
| f77: ftn | ||
| mpifc: ftn | ||
| mpicc: cc | ||
| cc: cc | ||
| cxx: CC | ||
|
|
||
| module_actions: | ||
| - "unload craype" | ||
| - "load craype/2.6.2" | ||
| - "load PrgEnv-cray/6.0.4" | ||
| - "load pbs" | ||
| - "load cray-mpich/7.7.3" | ||
| - "load craype-x86-skylake" | ||
| - "load cmake/3.14.0" | ||
| - "load cray-hdf5-parallel/1.10.2.0" | ||
| - "load cray-netcdf-hdf5parallel/4.6.1.3" | ||
| - "load cdo/1.9.5" | ||
| - "load fftw/2.1.5.9" | ||
| - "load nco/4.9.4" | ||
| - "load proj4/5.1.0" | ||
| - "load python/3.9.1" | ||
| - "list" | ||
|
|
||
| useMPI: cray_mpich #intel_mpich #intelopenmpi #cray_mpich | ||
|
|
||
| choose_useMPI: | ||
| cray_mpich: | ||
| fc: ftn | ||
| f77: ftn | ||
| mpifc: ftn | ||
| mpicc: cc | ||
| cc: cc | ||
| cxx: CC | ||
| module_actions: | ||
| - "unload craype" | ||
| - "load craype/2.6.2" | ||
| - "load PrgEnv-cray/6.0.4" | ||
| - "load pbs" | ||
| - "load cray-mpich/7.7.3" | ||
| - "load craype-x86-skylake" | ||
| - "load cmake/3.14.0" | ||
| - "load cray-hdf5-parallel/1.10.2.0" | ||
| - "load cray-netcdf-hdf5parallel/4.6.1.3" | ||
| - "load cdo/1.9.5" | ||
| - "load fftw/2.1.5.9" | ||
| - "load nco/4.9.4" | ||
| - "load proj4/5.1.0" | ||
| - "load python/3.9.1" | ||
| - "list" | ||
| add_export_vars: | ||
| ECCODESROOT: /proj/awiiccp/software/ecmwf/eccodes_cce_mpich | ||
| HDF5ROOT: $HDF5_ROOT | ||
|
|
||
| NETCDFFROOT: $NETCDF_DIR | ||
| NETCDFROOT: $NETCDF_DIR | ||
| NETCDF_Fortran_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| NETCDF_CXX_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| NETCDF_CXX_LIBRARIES: $NETCDFROOT/lib | ||
| ZLIBROOT: /usr | ||
|
|
||
| intelopenmpi: | ||
| warning: | ||
| Configuration does not run: | ||
| message: "The intelopenmpi configuration on aleph is experimental. From past experiance it can compile AWI-CM3, but crashes on runtime. Use as your own disgression. Contact [email protected] for further questions" | ||
| ask_user_to_continue: True | ||
JanStreffing marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| fc: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpifort | ||
| f77: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpifort | ||
| mpifc: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpifort | ||
| mpicc: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpicc | ||
| cc: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpicc | ||
| cxx: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpicxx | ||
| module_actions: | ||
| - "unload PrgEnv-cray" | ||
| - "load PrgEnv-intel/6.0.5" | ||
| - "load craype-x86-skylake" | ||
| - "load cray-mpich/7.7.3" | ||
| - "load gcc/7.3.0" | ||
| - "load pbs" | ||
| - "load cmake/3.14.0" | ||
| - "load cdo/1.9.5" | ||
| - "load fftw/2.1.5.9" | ||
| - "load nco/4.9.4" | ||
| - "load proj4/5.1.0" | ||
| - "load python/3.9.1" | ||
| - "list" | ||
| add_export_vars: | ||
| CMAKE_C_COMPILER: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpicc | ||
| CMAKE_CXX_COMPILER: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717/bin/mpicxx | ||
| IO_LIB_ROOT: /home/awiiccp2/software/HPC_libraries/intel-18.0.3.222_ompi_20230717 | ||
| LD_LIBRARY_PATH: $IO_LIB_ROOT/lib:$LD_LIBRARY_PATH | ||
|
|
||
| SZIPROOT: $IO_LIB_ROOT | ||
| HDF5ROOT: $IO_LIB_ROOT | ||
| HDF5_ROOT: $HDF5ROOT | ||
| NETCDFROOT: $IO_LIB_ROOT | ||
| NETCDFFROOT: $IO_LIB_ROOT | ||
| ECCODESROOT: $IO_LIB_ROOT | ||
|
|
||
| HDF5_C_INCLUDE_DIRECTORIES: $HDF5_ROOT/include | ||
| NETCDF_Fortran_INCLUDE_DIRECTORIES: $NETCDFFROOT/include | ||
| NETCDF_C_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| NETCDF_CXX_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| OASIS3MCT_FC_LIB: '"-L$NETCDFFROOT/lib -lnetcdff"' | ||
| PATH: $IO_LIB_ROOT/bin:$PATH | ||
| MPIFC: ftn | ||
|
|
||
| intel_mpich: | ||
| warning: | ||
| Configuration does not run: | ||
| message: "The intel_mpich configuration on aleph is experimental. From past experiance it can compile AWI-CM3, but crashes on runtime. Use as your own disgression. Contact [email protected] for further questions" | ||
| ask_user_to_continue: True | ||
| fc: ftn | ||
| f77: ftn | ||
| mpifc: ftn | ||
| mpicc: cc | ||
| cc: cc | ||
| cxx: CC | ||
| module_actions: | ||
| - "unload PrgEnv-cray" | ||
| - "load PrgEnv-intel/6.0.5" | ||
| - "load craype-x86-skylake" | ||
| - "load cray-mpich/7.7.3" | ||
| - "load gcc/7.3.0" | ||
| - "load pbs" | ||
| - "load cmake/3.14.0" | ||
| - "load cdo/1.9.5" | ||
| - "load fftw/2.1.5.9" | ||
| - "load nco/4.9.4" | ||
| - "load proj4/5.1.0" | ||
| - "load python/3.9.1" | ||
| - "load cray-netcdf-hdf5parallel/4.6.1.3" | ||
| - "load cray-hdf5-parallel/1.10.2.0" | ||
| - "list" | ||
| add_export_vars: | ||
| ECCODESROOT: /home/awiiccp2/software/HPC_libraries/eccodes_intel_mpich/ | ||
|
|
||
| HDF5_C_INCLUDE_DIRECTORIES: $HDF5_ROOT/include | ||
| NETCDF_Fortran_INCLUDE_DIRECTORIES: $NETCDFFROOT/include | ||
| NETCDF_C_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| NETCDF_CXX_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| OASIS3MCT_FC_LIB: '"-L$NETCDFFROOT/lib -lnetcdff"' | ||
| PATH: $IO_LIB_ROOT/bin:$PATH | ||
| MPIFC: ftn | ||
JanStreffing marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| export_vars: | ||
| LC_ALL: en_US.UTF-8 | ||
|
|
||
| HDF5ROOT: $HDF5_ROOT | ||
|
|
||
| NETCDFFROOT: $NETCDF_DIR | ||
| NETCDFROOT: $NETCDF_DIR | ||
| NETCDF_Fortran_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| NETCDF_CXX_INCLUDE_DIRECTORIES: $NETCDFROOT/include | ||
| NETCDF_CXX_LIBRARIES: $NETCDFROOT/lib | ||
|
|
||
| PERL5LIB: /usr/lib64/perl5 | ||
| #LAPACK_LIB: '"-lmkl_intel_lp64 -lmkl_core -mkl=sequential -lpthread -lm -ldl"' | ||
| #LAPACK_LIB_DEFAULT: '"-L/global/AWIsoft/intel/2018/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential"' | ||
| XML2ROOT: /usr | ||
| ZLIBROOT: /usr | ||
| TMPDIR: /tmp | ||
| # Prepend labels to the stdout/sterr lines | ||
| PMI_LABEL_ERROUT: 1 | ||
|
|
||
| ECCODESROOT: /proj/awiiccp/software/ecmwf/eccodes_cce_mpich | ||
JanStreffing marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # enable full MPI thread support level (MPI_THREAD_MULTIPLE) | ||
| MPICH_MAX_THREAD_SAFETY: multiple # to also switch to an alternative (probably with faster locking) multi threading implementation of the cray-mpich library, use the compiler flag -craympich-mt | ||
|
|
@@ -108,7 +190,7 @@ export_vars: | |
| # Important for eccodes to work | ||
| CRAYPE_LINK_TYPE: dynamic | ||
|
|
||
| label_format: '"[@MODEL@]%l:"' | ||
| label_format: '"[@MODEL@]%r:"' | ||
mandresm marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| launcher_flags_per_component: "-n @nproc@ -N @nproc_per_node@ -d @cpus_per_proc@ | ||
| env OMP_NUM_THREADS=@omp_num_threads@ | ||
| env PMI_LABEL_ERROUT_FORMAT=${label_format}" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of this can go outside in a general
module_actions. Here you can place only the differences. It's a matter of taste, I find it easier to find what the difference are in between the options if in the choose blocks there are only differences.If you want to give it a shot I recommend to use here
add_module_actionsand also variable definition and called by placeholders in the generalmodule_actionslike in the Albedo example:https://github.com/esm-tools/esm_tools/blob/release/configs/machines/albedo.yaml#L127-L149
https://github.com/esm-tools/esm_tools/blob/release/configs/machines/albedo.yaml#L103-L125
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I place the defaults first, and then unload the craype, that might not work. The order can matter for module commands. Thus I try to keep them together as much as I can.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is indeed possible to keep the order if you follow the examples I pasted in the comment above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have time to give it a shot, if you have time to test it. Let me know what you prefer, doing it yourself or that I do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might find some time tomorrow. Otherwise it would sit till after I'm back.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm back and would have time to test.