Skip to content

Commit

Permalink
Initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@abozec
abozec committed Feb 20, 2019
1 parent 087c8d3 commit 386dcd8
Show file tree
Hide file tree
Showing 106 changed files with 85,964 additions and 0 deletions.
15 changes: 15 additions & 0 deletions ALT_CODE/README.OpenMP
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
The OpenMP directives allow HYCOM to run on multiple processors of
shared memory machines. They can also be used in conjuction with
MPI domain decomposition (MPI between multi-processor nodes, OpenMP on
each node). This mode of parallelization is typically best for a
relatively modest number of processors (2,3,4,6,8), although more can
profitably be used on large grids.

The PARAMETER mxthrd has been added to mod_dimesnions.F90 and dimensions.h.
Each outer (i or j) loop is diviuded into mxthrd pieces by OpenMP. So
mxthrd should be a multiple of NOMP (i.e. OMP_NUM_THREADS). It is often
best to set mxthrd larger than NOMP, because that tends to give better
land/sea load balance between threads. For example, mxthrd=16 could
be used with 2, 4, 8 or 16 threads. Other good choices are 12, 24, 32
etcetera. Large values of mxthrd are only likely to be optimal for
large idm and jdm.
53 changes: 53 additions & 0 deletions ALT_CODE/README.dimensions
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
If the RELO macro is set, arrays are allocated at run time and the defaults
for the small number of user changeable parameters in mod_dimensions.F90 are
usually appropriate. The most commonly edited parameter in mod_dimensions.F90
is mxthrd which might need changing when parallelizing with OpenMP (TYPE=omp
or TYPE=ompi), see README.OpenMP.

-------------------------------------------------------------------------------

If the RELO macro is not set, arrays are allocated at compile time and
dimensions.h will need customizing for a new region or a different number
of layers or a different TYPE. The user-tunable parameters are:

itdm = total grid dimension in i direction
jtdm = total grid dimension in j direction
kdm = grid dimension in k direction
iqr = maximum number of tiles in i direction
jqr = maximum number of tiles in j direction
idm = maximum single tile grid dimension in i direction
jdm = maximum single tile grid dimension in j direction
mxthrd = maximum number of OpenMP threads
kkwall = grid dimension in k direction for wall relax arrays
kknest = grid dimension in k direction for nest relax arrays
kkmy25 = grid dimension in k direction for M-Y 2.5 arrays

If memory is plentifull, then kkwall,kknest,kkmy25 can all be set to
kdm. However, if memory is in short supply then kwall and/or kknest
can be set to 1 (if wall or nest relaxation is not being used) and
if the Mellor-Yamada mixed layer isn't being used kkmy25 can be set
to -1.

A new source code directory and executable is always required whenever
itdm, or jtdm, or kdm change (i.e. whenever the region size or the
number of layers change). Note that ../topo/regional.grid.b is used
to define the region to setup programs, and needs to be consistent with
dimensions.h.

A separate source code directory and executable is always required for
each parallelization strategy, or TYPE, chosen (TYPE=one, or omp, or ompi,
or mpi, or shmem). The TYPE also effects how dimensions.h is configured.
There are examples of dimensions.h for each TYPE in ALT_CODE.

When running on a shared memory machine (TYPE=one or omp) set:
iqr=jqr=1, idm=itdm, and jdm=jtdm. Note that the same OpenMP executable
(TYPE=omp) can be used for a range of processor counts, provided mxthrd
is chosen appropriately, see README.OpenMP.

When running on a distributed memory machine (TYPE=mpi or ompi or shmem)
set: iqr and jqr to the maximum number of processors used in each
dimension, and idm and jdm to the maximum (worse case) dimensions for
any single tile on any targeted number of processors. Note that the
same executable can be used for a range of processor counts, provided
iqr,jqr,idm,jdm are all large enough for each case. The mxthrd parameter
is only important when using OpenMP, see README.OpenMP.
54 changes: 54 additions & 0 deletions ALT_CODE/dimensions_mpi.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
!-----------------------------------------------------------------------------
! --- START OF REGION AND TILING SPECIFIC PARAMETERS
! --- Static memory version, not used when macro /* RELO */ is set (dynamic memory)
! --- See: README.dimensions and README.OpenMP for more details.
!
! --- itdm = total grid dimension in i direction
! --- jtdm = total grid dimension in j direction
! --- kdm = grid dimension in k direction
integer itdm,jtdm,kdm
parameter (itdm=500,jtdm=382,kdm=41) ! GLBT0.72
!
! --- iqr = maximum number of tiles in i direction
! --- jqr = maximum number of tiles in j direction
integer iqr,jqr
parameter (iqr=10,jqr=10) ! multiple tiles (TYPE=ompi or mpi or shmem)
!
! --- idm = maximum single tile grid dimension in i direction
! --- jdm = maximum single tile grid dimension in j direction
integer idm,jdm
!!!!!!parameter (idm=itdm,jdm=jtdm) ! always works if enough memory
parameter (idm= 250,jdm= 191) ! NMPI=4,8,16,24,32,40,47,64
!
! --- mxthrd= maximum number of OpenMP threads
integer mxthrd
parameter (mxthrd=1) ! NOMP=0,1
!
! --- kkwall= grid dimension in k direction for wall relax arrays
! --- kknest= grid dimension in k direction for nest relax arrays
integer kkwall,kknest
parameter (kkwall= 1) ! must be 1 or kdm
parameter (kknest= 1) ! must be 1 or kdm
!
! --- kkmy25= grid dimension in k direction for M-Y 2.5 arrays
integer kkmy25
parameter (kkmy25= -1) ! must be -1 or kdm
!
! --- nlgiss= size of lookup table for GISS
integer nlgiss
parameter (nlgiss= 1) ! must be 1 (no GISS) or 762
!
! --- mxtrcr= maximum number of tracers
integer mxtrcr
parameter (mxtrcr=1)
!
! --- natm = number of saved atmospheric fields
integer natm
parameter (natm=2) ! must be 2 (high freq.) or 4 (monthly)
!
! --- max_nsteps_batrop = maximum barotropic steps per baroclinic time step
integer max_nsteps_batrop
parameter (max_nsteps_batrop = 128)
!
! --- END OF REGION AND TILING SPECIFIC PARAMETERS
!-----------------------------------------------------------------------------
53 changes: 53 additions & 0 deletions ALT_CODE/dimensions_omp.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
!-----------------------------------------------------------------------------
! --- START OF REGION AND TILING SPECIFIC PARAMETERS
! --- Static memory version, not used when macro /* RELO */ is set (dynamic memory)
! --- See: README.dimensions and README.OpenMP for more details.
!
! --- itdm = total grid dimension in i direction
! --- jtdm = total grid dimension in j direction
! --- kdm = grid dimension in k direction
integer itdm,jtdm,kdm
parameter (itdm=500,jtdm=382,kdm=41) ! GLBT0.72
!
! --- iqr = maximum number of tiles in i direction
! --- jqr = maximum number of tiles in j direction
integer iqr,jqr
parameter (iqr= 1,jqr= 1) ! single tile (TYPE=one or omp)
!
! --- idm = maximum single tile grid dimension in i direction
! --- jdm = maximum single tile grid dimension in j direction
integer idm,jdm
parameter (idm=itdm,jdm=jtdm) ! single tile (TYPE=one or omp)
!
! --- mxthrd= maximum number of OpenMP threads
integer mxthrd
parameter (mxthrd=16) ! NOMP=0,1,2,4,8,16
!
! --- kkwall= grid dimension in k direction for wall relax arrays
! --- kknest= grid dimension in k direction for nest relax arrays
integer kkwall,kknest
parameter (kkwall= 1) ! must be 1 or kdm
parameter (kknest= 1) ! must be 1 or kdm
!
! --- kkmy25= grid dimension in k direction for M-Y 2.5 arrays
integer kkmy25
parameter (kkmy25= -1) ! must be -1 or kdm
!
! --- nlgiss= size of lookup table for GISS
integer nlgiss
parameter (nlgiss= 1) ! must be 1 (no GISS) or 762
!
! --- mxtrcr= maximum number of tracers
integer mxtrcr
parameter (mxtrcr=1)
!
! --- natm = number of saved atmospheric fields
integer natm
parameter (natm=2) ! must be 2 (high freq.) or 4 (monthly)
!
! --- max_nsteps_batrop = maximum barotropic steps per baroclinic time step
integer max_nsteps_batrop
parameter (max_nsteps_batrop = 128)
!
! --- END OF REGION AND TILING SPECIFIC PARAMETERS
!-----------------------------------------------------------------------------
54 changes: 54 additions & 0 deletions ALT_CODE/dimensions_ompi.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
!-----------------------------------------------------------------------------
! --- START OF REGION AND TILING SPECIFIC PARAMETERS
! --- Static memory version, not used when macro /* RELO */ is set (dynamic memory)
! --- See: README.dimensions and README.OpenMP for more details.
!
! --- itdm = total grid dimension in i direction
! --- jtdm = total grid dimension in j direction
! --- kdm = grid dimension in k direction
integer itdm,jtdm,kdm
parameter (itdm=500,jtdm=382,kdm=41) ! GLBT0.72
!
! --- iqr = maximum number of tiles in i direction
! --- jqr = maximum number of tiles in j direction
integer iqr,jqr
parameter (iqr=10,jqr=10) ! multiple tiles (TYPE=ompi or mpi or shmem)
!
! --- idm = maximum single tile grid dimension in i direction
! --- jdm = maximum single tile grid dimension in j direction
integer idm,jdm
!!!!!!parameter (idm=itdm,jdm=jtdm) ! always works if enough memory
parameter (idm= 250,jdm= 191) ! NMPI=4,8,16,24,32,40,47,64
!
! --- mxthrd= maximum number of OpenMP threads
integer mxthrd
parameter (mxthrd=8) ! NOMP=0,1,2,4,8
!
! --- kkwall= grid dimension in k direction for wall relax arrays
! --- kknest= grid dimension in k direction for nest relax arrays
integer kkwall,kknest
parameter (kkwall= 1) ! must be 1 or kdm
parameter (kknest= 1) ! must be 1 or kdm
!
! --- kkmy25= grid dimension in k direction for M-Y 2.5 arrays
integer kkmy25
parameter (kkmy25= -1) ! must be -1 or kdm
!
! --- nlgiss= size of lookup table for GISS
integer nlgiss
parameter (nlgiss= 1) ! must be 1 (no GISS) or 762
!
! --- mxtrcr= maximum number of tracers
integer mxtrcr
parameter (mxtrcr=1)
!
! --- natm = number of saved atmospheric fields
integer natm
parameter (natm=2) ! must be 2 (high freq.) or 4 (monthly)
!
! --- max_nsteps_batrop = maximum barotropic steps per baroclinic time step
integer max_nsteps_batrop
parameter (max_nsteps_batrop = 128)
!
! --- END OF REGION AND TILING SPECIFIC PARAMETERS
!-----------------------------------------------------------------------------
53 changes: 53 additions & 0 deletions ALT_CODE/dimensions_one.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
!-----------------------------------------------------------------------------
! --- START OF REGION AND TILING SPECIFIC PARAMETERS
! --- Static memory version, not used when macro /* RELO */ is set (dynamic memory)
! --- See: README.dimensions and README.OpenMP for more details.
!
! --- itdm = total grid dimension in i direction
! --- jtdm = total grid dimension in j direction
! --- kdm = grid dimension in k direction
integer itdm,jtdm,kdm
parameter (itdm=500,jtdm=382,kdm=41) ! GLBT0.72
!
! --- iqr = maximum number of tiles in i direction
! --- jqr = maximum number of tiles in j direction
integer iqr,jqr
parameter (iqr= 1,jqr= 1) ! single tile (TYPE=one or omp)
!
! --- idm = maximum single tile grid dimension in i direction
! --- jdm = maximum single tile grid dimension in j direction
integer idm,jdm
parameter (idm=itdm,jdm=jtdm) ! single tile (TYPE=one or omp)
!
! --- mxthrd= maximum number of OpenMP threads
integer mxthrd
parameter (mxthrd=1) ! NOMP=0,1
!
! --- kkwall= grid dimension in k direction for wall relax arrays
! --- kknest= grid dimension in k direction for nest relax arrays
integer kkwall,kknest
parameter (kkwall= 1) ! must be 1 or kdm
parameter (kknest= 1) ! must be 1 or kdm
!
! --- kkmy25= grid dimension in k direction for M-Y 2.5 arrays
integer kkmy25
parameter (kkmy25= -1) ! must be -1 or kdm
!
! --- nlgiss= size of lookup table for GISS
integer nlgiss
parameter (nlgiss= 1) ! must be 1 (no GISS) or 762
!
! --- mxtrcr= maximum number of tracers
integer mxtrcr
parameter (mxtrcr=1)
!
! --- natm = number of saved atmospheric fields
integer natm
parameter (natm=2) ! must be 2 (high freq.) or 4 (monthly)
!
! --- max_nsteps_batrop = maximum barotropic steps per baroclinic time step
integer max_nsteps_batrop
parameter (max_nsteps_batrop = 128)
!
! --- END OF REGION AND TILING SPECIFIC PARAMETERS
!-----------------------------------------------------------------------------
79 changes: 79 additions & 0 deletions CESMBETA/Makefile.NUOPC_CESMB
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# ------------------------------------------------------------------------------
# --- Standard rules ----------------------------------------------------------
# ------------------------------------------------------------------------------
include Makefile

# ------------------------------------------------------------------------------
# --- NUOPC additions ----------------------------------------------------------
# ------------------------------------------------------------------------------
ocn_comp_mct.o : ocn_comp_mct.F90
$(ESMF_F90COMPILER) -c $(DEP_FRONTS) $(DEP_INCS) -I$(OBJROOT)/lib/include -I$(OBJROOT)/csm_share -I$(OBJROOT)/pio $(ESMF_F90COMPILEOPTS) $(ESMF_F90COMPILEPATHS) $(ESMF_F90COMPILEFREECPP) $(ESMF_F90COMPILECPPFLAGS) $(USER_DEFS) $<

ocn_comp_nuopc_mod.o : ocn_comp_nuopc_mod.F90
$(ESMF_F90COMPILER) -c $(DEP_FRONTS) $(DEP_INCS) -I$(OBJROOT)/lib/include -I$(OBJROOT)/csm_share -I$(OBJROOT)/pio $(ESMF_F90COMPILEOPTS) $(ESMF_F90COMPILEPATHS) $(ESMF_F90COMPILEFREECPP) $(ESMF_F90COMPILECPPFLAGS) $(USER_DEFS) $<

mod_hycom_nuopc_glue.o : mod_hycom_nuopc_glue.F90
$(ESMF_F90COMPILER) -c $(DEP_FRONTS) $(DEP_INCS) -I$(OBJROOT)/lib/include -I$(OBJROOT)/csm_share -I$(OBJROOT)/pio $(ESMF_F90COMPILEOPTS) $(ESMF_F90COMPILEPATHS) $(ESMF_F90COMPILEFREECPP) $(ESMF_F90COMPILECPPFLAGS) $(USER_DEFS) $<

mod_cb_arrays_nuopc_glue.o : mod_cb_arrays_nuopc_glue.F90
$(ESMF_F90COMPILER) -c $(DEP_FRONTS) $(DEP_INCS) -I$(OBJROOT)/lib/include -I$(OBJROOT)/csm_share -I$(OBJROOT)/pio $(ESMF_F90COMPILEOPTS) $(ESMF_F90COMPILEPATHS) $(ESMF_F90COMPILEFREECPP) $(ESMF_F90COMPILECPPFLAGS) $(USER_DEFS) $<


.PRECIOUS: %.o

%.mk : %.o
@echo "# ESMF self-describing build dependency makefile fragment" > $@
@echo >> $@
@echo "ESMF_DEP_FRONT = hycom" >> $@
@echo "ESMF_DEP_INCPATH = `pwd`" >> $@
@echo "ESMF_DEP_CMPL_OBJS = `pwd`/"$< >> $@
@echo "ESMF_DEP_LINK_OBJS = `pwd`/libhycom_nuopc.a" >> $@
@echo "ESMF_DEP_SHRD_PATH = " >> $@
@echo "ESMF_DEP_SHRD_LIBS = " >> $@
@echo
@echo "Finished generating ESMF self-describing build dependency makefile fragment:" $@
@echo

nuopc: ocn_comp_mct.o ocn_comp_nuopc_mod.mk libhycom_nuopc.a

ocn_comp_mct.o: $(MODS) $(OBJS)

ocn_comp_nuopc_mod.o: mod_hycom_nuopc_glue.o $(MODS) $(OBJS)

mod_hycom_nuopc_glue.o: mod_cb_arrays_nuopc_glue.o $(MODS) $(OBJS)

mod_cb_arrays_nuopc_glue.o: $(MODS) $(OBJS)

libhycom_nuopc.a: ocn_comp_mct.o ocn_comp_nuopc_mod.o mod_hycom_nuopc_glue.o mod_cb_arrays_nuopc_glue.o $(MODS) $(OBJS)
ar cr $@ $^

.PHONY: clean nuopcinstall
clean:
rm -f hycom *.o *.mod *.a *.mk

PWDIR := `pwd`

ifndef DESTDIR
DESTDIR := $(PWDIR)
endif

INSTDATE := $(shell date '+%Y-%m-%d-%H-%M-%S')
ifndef INSTDIR
INSTDIR := HYCOM_$(INSTDATE)
endif

nuopcinstall:
@gmake nuopc
@mkdir -p $(DESTDIR)/$(INSTDIR)
@cp libhycom_nuopc.a hycom.mod $(DESTDIR)/$(INSTDIR)
@sed -e 's;'$(PWDIR)';'$(DESTDIR)/$(INSTDIR)';g' ocn_comp_nuopc_mod.mk > $(DESTDIR)/$(INSTDIR)/ocn_comp_nuopc_mod.mk
@touch VERSION
@if [ -d .svn ]; then \
echo "SVN Repository" > VERSION; \
svn info . | grep URL >> VERSION; \
svn info . | grep "Last Changed Rev" >> VERSION; \
fi
@cp VERSION $(DESTDIR)/$(INSTDIR)/
@echo Installation into \"$(DESTDIR)/$(INSTDIR)\" complete!
@echo
# ------------------------------------------------------------------------------
Loading

0 comments on commit 386dcd8

Please sign in to comment.