Improve WW3 mesh cap to allow using vortex formulation in ocean side

[janahaddad]

Background

[based on @uturuncoglu email] UFS Coastal is working on coupling WW3 and SCHISM. We're able to run 2D configurations of SCHISM + WW3 with mesh cap (or NCAR cap) without any issues, but we are preparing the way for running in 3D mode (https://github.com/oceanmodeling/ufs-coastal/issues/32). This requires additional fields from WW3.

The aim is to add missing fields to the WW3 cap (I think they are already available through OASIS complier) and complete the wiring with the SCHSIM (the interaction with wave with new formulation is already implemented) to enable 3d coupling

Info needed to proceed

• Need clarification of which variables are exported from WW3 and their units (mesh-cap)
  • Mesh-cap import/export variables and their units
• Need modification on WW3 side to get those field exported
• See here (NOAA EMC branch):
  https://github.com/NOAA-EMC/WW3/blob/d9b3172f4197c65d471662c6952a668152d71230/model/src/wav_import_export.F90#L138
• Oceanmodeling branch version:

  WW3/model/src/wav_import_export.F90

  Line 139 in 80a5ad5

  if (.not. unstr_mesh) then

@DeniseWorthen we will follow up for a description of what we're searching for here !

cc @josephzhang8 @uturuncoglu @yunfangsun @saeed-moghimi-noaa @anntsay

[DeniseWorthen]

The cpl_scalars are used when CMEPs write history or other files and allows the fields on the mesh to be written as 2-D fields. This simply removes the cpl_scalars from the export fields list, since in the unstructured case it doesn't make sense.

[janahaddad]

from @DeniseWorthen email:

In wav_import_export there are several "calc" routines which we moved over from the old esmf cap to the new mesh cap. I think some of the fields I see listed (like the stokes-drift spectrum, calcstokes3d) are there in the old cap, but were not added to the new cap. I think it is straightforward to add them to the new cap.

Some of the other fields---well, I have no idea what internal WW3 fields correspond to the needed export. But, if it is calculated as a "history" field in WW3, I think you could make them available by adding additional "calc" routines in wav_import_export. For example, in wav_grdout, I see

varatts( "BHD ", "BHD ", "Bernoulli head (J term) ", "m2 s-2 ", " ", .false.)

and if I look in w3iogomd, I see how that term is calculated on LN 1653. I think you can find the bottom momentum fluxes also.

As for units, I'm not 100% sure I got them right when I wrote the wav_grdout. Ana has found a couple of errors (I still have on my list to fix these!). I think you'll need to rely on wave modelers to be entirely sure of the units.

[janahaddad]

@uturuncoglu @DeniseWorthen transferring conversation on this issue from email to GH! thx

[DeniseWorthen]

For point of clarify, you do want any export fields to be calculated in the cap, and not just use them from the internal fields that WW3 might calculate as an output variable. The reason is that the output variables are only calculated at some set frequency, and you want the export fields to be updated at each modelAdvance. See NOAA-EMC#843

[uturuncoglu]

@DeniseWorthen Yes, it would be nice to have them in the cap and updated with coupling frequency. It would be also nice to have them in the output but we could still output import and export states for debugging purpose.

[janahaddad]

from @DeniseWorthen: w3ounfmetamd.F90 is script with BHD, etc. units as above...

[saeed-moghimi-noaa]

Denise Worthen - NOAA Affiliate
1:51 PM
All the mesh-cap files are named as wav_*F90

[saeed-moghimi-noaa]

Denise Worthen - NOAA Affiliate
1:53 PM
wav_import_export.F90

[saeed-moghimi-noaa]

where to add field to w3:

[saeed-moghimi-noaa]

[saeed-moghimi-noaa]

See in this file (in ufs-weather there is one) in application level: something like this:
Denise Worthen - NOAA Affiliate
1:56 PM
fd_ufs.yaml

[saeed-moghimi-noaa]

This file is not reserved for ufs-weather (coastal):

Panagiotis Velissariou - NOAA Affiliate
2:00 PM
./tests/parm/fd_ufs.yaml

This one is the one we suppose to use the one under:
tests/parm/fd_ufs.yaml

[saeed-moghimi-noaa]

• need to create the variables
• create calculation routine
• create export var
• extort

[saeed-moghimi-noaa]

Just to document chats from the meeting:

Jana Haddad - NOAA Affiliate
1:33 PM
#11
Denise Worthen - NOAA Affiliate
1:44 PM
w3ounfmetamd.F90
Ali Abdolali
1:46 PM
Are we doing 2D or 3D coupling or both? And could you pass the list of variabls you need?
for 2D, we need SXX, XYY and SXY,
but for 3D, we need more
Ali Salimi - NOAA Affiliate
1:47 PM
https://docs.google.com/document/d/140_xAMOb30iljGL09C_S7tNAKGDY2j8GEjPb7Pg3Xxg/edit
Jana Haddad - NOAA Affiliate
1:47 PM
https://docs.google.com/document/d/140_xAMOb30iljGL09C_S7tNAKGDY2j8GEjPb7Pg3Xxg/edit
Ali Salimi - NOAA Affiliate
1:47 PM
Ali I think they are here
Ali Abdolali
1:47 PM
thanks
Ali Abdolali
1:48 PM
SO, as Denise said, for ufs-weather-model, we already have a few variables for coupling. Do you want Ufuk to add the rest to the same location?
I would recommend do not go back and forth between wmesmf, OASIS, cstorm, ... because as Denise said, they do not show up magically.
Ali Abdolali
1:51 PM
the same should apply to import fields to WW3 mesh cap
Denise Worthen - NOAA Affiliate
1:51 PM
All the mesh-cap files are named as wav_*F90
Denise Worthen - NOAA Affiliate
1:53 PM
wav_import_export.F90
Denise Worthen - NOAA Affiliate
1:56 PM
fd_ufs.yaml
Panagiotis Velissariou - NOAA Affiliate
2:00 PM
./tests/parm/fd_ufs.yaml
Denise Worthen - NOAA Affiliate
2:00 PM
can I ask---your coastal oceanmodel is now a fork of ufs-wm?
Panagiotis Velissariou - NOAA Affiliate
2:00 PM
./CMEPS-interface/CMEPS/mediator/fd_cesm.yaml
Denise Worthen - NOAA Affiliate
2:02 PM
tests/parm/fd_ufs.yaml
Ali Abdolali
2:06 PM
where is the caluclation routine? in wave_import?
Ali Abdolali
2:16 PM
FYI, Denise does not consider herself a WW3 developer :)
Ali Salimi - NOAA Affiliate
2:19 PM
We are not seeing
Denise Worthen - NOAA Affiliate
2:22 PM
w3outg is the SR where the output variables are calculated

[josephzhang8]

Thank you all! Here is the list of arrays we need from WW3:

    wave_hs(1:np)=WW3__OHS !Sig wave height [m]
    wave_dir(1:np)=WW3__DIR !mean wave dir [deg]
    wave_tm1(1:np)=WW3_T0M1 !mean wave period [s]
    wave_wnm(1:np)=WW3__WNM !mean wave number [1/m]
    wave_pres(1:np)=WW3__BHD !wave-induced Bernoulli head pressure [N/m or Pa?]
    wave_stokes_x(1:np)=WW3_USSX !Stokes drift [m/s]
    wave_stokes_y(1:np)=WW3_USSY
    wave_ocean_flux_x(1:np)=WW3_TWOX !wave-ocean mom flux [m2/s2]
    wave_ocean_flux_y(1:np)=WW3_TWOY
    wave_flux_friction_x(1:np)=WW3_TBBX !Momentum flux due to bottom friction [m2/s2]
    wave_flux_friction_y(1:np)=WW3_TBBY
    wave_orbu(1:np)=WW3_UBRX !near bed orbital vel [m/s]
    wave_orbv(1:np)=WW3_UBRY

[DeniseWorthen]

Take the bottom momentum fields as an example: wave_flux_friction_x,y, the "momentum flux due to bottom friction" in m2 s-2. I can see this field is exported via the oasis coupler (in w3gocmmd.F90).

It appears to be calculated in w3sbt4md.F90, at LN 556-570:

! 5. Fills output arrays and estimates the energy and momentum loss
!
DO IK=1, NK
  CONST2=DDEN(IK)/CG(IK) &         !Jacobian to get energy in band
       *GRAV/(SIG(IK)/WN(IK))    ! coefficient to get momentum
  DSUM(IK)=-FW*UORB*CBETA(IK)      !*WSUB(ISUB)
  DO ITH=1,NTH
    IS=ITH+(IK-1)*NTH
    D(IS)=DSUM(IK)
    TEMP2=CONST2*D(IS)*A(IS)
    TAUBBL(1) = TAUBBL(1) - TEMP2*ECOS(IS)
    TAUBBL(2) = TAUBBL(2) - TEMP2*ESIN(IS)
    S(IS)=D(IS)*A(IS)
  END DO
END DO

This is the code (and any preceding code on which it depends) which would need to be added in wav_import_export.F90 as a new "calc" routine. So, the steps would be (I've invented a field name here):

1. add the fields to the list of exported fields

call fldlist_add(fldsFrWav_num, fldsFrWav, 'Sw_taubot_x')
call fldlist_add(fldsFrWav_num, fldsFrWav, 'Sw_taubot_y')

2. add in export_fields (a SR in wav_import_export) code similar to the other export variables

a) define pointers to the esmf fields:

    real(r8), pointer :: tauxbot(:)
    real(r8), pointer :: tauybot(:)

b) check that the field is required in the export state using and get pointers to the fields

   if ( state_fldchk(exportState, 'Sw_taubot_x') .and. &
         state_fldchk(exportState, 'Sw_taubot_y') )then
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
      call state_getfldptr(exportState, 'Sw_taubot_x', tauxbot, rc=rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
      call state_getfldptr(exportState, 'Sw_taubot_y', tauybot, rc=rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return

c) Create a "calc" routine to calculate the fields, replicating whatever is required from the w3sbt4md.F90 routine.

call CalcBottomStress(input argument list, tauxbot, tauybot)

At this point, the bottom stress should be ready to be exported by WW3 at every model advance. There is more to be done however to get them sent to another component:

1. the new field names need to be defined in the field-dictionary used by all components (in UFS-WM this is in tests/parm/fd_ufs.yaml)
2. the fields need to be added to the FieldExchange module in CMEP.
3. the fields need to be requested and realized by whatever model is expecting these fields as imports.

[janahaddad]

Summary from meeting 5/28:

• both 2D and 3D coupling need to go through mesh_cap
• Ufuk updated the mesh cap in oceanmodeling/ww3 (wav_*F90) for 2D ocn+wav coupling
  • need to expand this for 3D (can collaborate to prepare PR and send PR to EMC when @uturuncoglu returns)
• there is a discrepancy between oceanmodeling/WW3 and NOAA-EMC/WW3 (by Denise) wrt how the ww3 exported fields are fed into derived fields and shared with SCHISM cap
• all models need to share test/parm/fd_ufs.yml
  • any fields needed exported from ww3 need to be added to this yml file. This yml file is fed to a generic routine that sends these fields to RealizeFields in the cap
  • calculation of each of those fields needs to be added to WW3/model/src/wave_import_export.F90
• from SCHISM interface routine, @josephzhang8 has currently put a placeholder list of field arrays expected from WW3. The names are currently from Oasis cap, and need to be updated -- See Joseph's list above.
  • these fields are in general are availble as output fields in WW3 (but not all of them @aliabdolali ?)

Next steps:

need to eventually merge import_export work that's already done for the 2d coupling:

• Ufuk needs to bring all 2D ocn-wav WW3 variable exports implemented in UFS Coastal into NOAA-EMC/WW3. @uturuncoglu perhaps this was already sent as PR to ufs-wm's WW3?
  • Takis & Joseph: Ufuk has already only added the radiation stresses, nothing else.
  • wait for Ufuk to return before making any PRs to UFS-WM/WW3

in the meantime:

• Jospeh/VIMS team + SSM team + WW3 devs (Ali A., Ali S., Saeideh) can work together to prepare the way for a future PR following these steps outlined by Denise & Ali. Thank you @DeniseWorthen for the example steps outlined above.

1. find where Joseph's list of fields above are calc'd as output fields
2. note the variable name being used & find that in subroutine that calc's the output: w3outg is the SR where the output variables are calculated
3. copy how it's being calc'd in that output routine and paste it into wave_import_export.f90. Then go into the cap and expose/export the variables to WW3

attendees: @DeniseWorthen @saeed-moghimi-noaa @josephzhang8 @pvelissariou1 @aliabdolali @AliS-Noaa Saeideh Banihashemi @yunfangsun

[janahaddad]

Here is the history showing Ufuk's addition of radiation stress components to wav_import_export.f90, on the UFS Coastal side:

He added to advertise_fields SR:

      call fldlist_add(fldsFrWav_num, fldsFrWav, 'Sw_wavsuu')
      call fldlist_add(fldsFrWav_num, fldsFrWav, 'Sw_wavsuv')
      call fldlist_add(fldsFrWav_num, fldsFrWav, 'Sw_wavsvv')

and corrected those names in export_fields SR.

Looks like those changes have not been merged to UFS-WM's WW3 branch. See here.

[DeniseWorthen]

I need to amend what I wrote previously, because it appears that taubbl might be calculated prognostically at each model advance in the w3srcemd routine. In that case, I believe it would not be required to add a new calc routine, but you could simply 'use' the taubbl values as calculated by the model (they're in w3adatmd). Then in export_fields, you'd have instead of calling a new calc routine in step 2c above,

       do jsea=1, nseal_cpl
        call init_get_isea(isea, jsea)
        ix  = mapsf(isea,1)
        iy  = mapsf(isea,2)
        if (mapsta(iy,ix) == 1) then
           tauxbot(jsea) = taubbl(1)
           tauybot(jsea) = taubbl(2)
        end if
      end do

[josephzhang8]

Thx @DeniseWorthen for your instructions!
The calculation of tau*bot you showed above seems to suggest it's from the multi-grid (not UG) version of WW3?

[DeniseWorthen]

@josephzhang8 Thanks, I should have looked more carefully. I think it should actually just be

        do jsea=1, nseal_cpl
           tauxbot(jsea) = taubbl(jsea,1)
           tauybot(jsea) = taubbl(jsea,2)
        end do

since taubbl allocated as (nsealm,2)

[josephzhang8]

That makes sense. Thx @DeniseWorthen

[josephzhang8]

The list of WW3 variables SCHISM needs (from OASIS ):
wave_hs(1:np)=WW3__OHS !Sig wave height [m]
wave_dir(1:np)=WW3__DIR !mean wave dir [deg]
wave_tm1(1:np)=WW3_T0M1 !mean wave period [s]
wave_wnm(1:np)=WW3__WNM !mean wave number [1/m]
wave_pres(1:np)=WW3__BHD !wave-induced Bernoulli head pressure [N/m or Pa?]
wave_stokes_x(1:np)=WW3_USSX !Stokes drift [m/s]
wave_stokes_y(1:np)=WW3_USSY
wave_ocean_flux_x(1:np)=WW3_TWOX !wave-ocean mom flux [m2/s2]
wave_ocean_flux_y(1:np)=WW3_TWOY
wave_flux_friction_x(1:np)=WW3_TBBX !Momentum flux due to bottom friction [m2/s2]
wave_flux_friction_y(1:np)=WW3_TBBY
wave_orbu(1:np)=WW3_UBRX !near bed orbital vel [m/s]
wave_orbv(1:np)=WW3_UBRY

[josephzhang8]

@aliabdolali many thanks for taking time help us set up the test!

You can find the setup for SCHISM-WWM for the analytical case at:
https://columbia.vims.edu/schism/schism_verification_tests/Test_WWM_Analytical/

The web may warn you about potential risk; just accept and you'll see the dir (our IT guy is still looking to upgrade). Let me know if you have any questions. Thx!

[saeed-moghimi-noaa]

@danishyo @yunfangsun

Hi Dan,

Would please plot time series of Hsig for a node in the middle of the open boundary and the other one a few nodes inside the domain (see figue attached here). I believe these wave height time-series from both schism-ww3 and schism-wwm should be almost identical.

If they are not in agreement then we need to bring it up to discuss with @aliabdolali and others.

Thanks,
-Saeed

[danishyo]

Hi @saeed-moghimi-noaa

See the followings:
At middle open boundary (from node 422)

Inside domain (from node 8545)

Please note the above schism-ww3 results actually did not really do the coupling.
After checking the code, we found that although the info are exchanged, schism didn't use these to do the coupling calculation.

Therefore, only WW3 simply receive elevation+current from schism.
That's why current in SCHISM-WW3 shows different behavior.
We're currently testing with coupling part.

[saeed-moghimi-noaa]

Obviously the boundary spectrum definition for the both models are not consistent i.e. the amount of energy we enter to these domains are not the same where it need to be identical.

@aliabdolali
How would you suggest we find out what is going on?

@danishyo
What is the actual wave height time series that we impose at the boundary from the boundary file. Which of these cases are less or more than what it should be? We should be able to calculate wave height from the spectrum that we force as boundary condition, right?

Thanks,
-Saeed

[yunfangsun]

Hi @danishyo ,

Could you please let me know your most recent case configurations and the source codes of the schism-ww3 case and schism-wwm on Hercules? I would like to compare some outputs, spectra, hs etc.

Thank you!

[danishyo]

Due to spec definition in both models are different, original WWM spec nc file has to do interpolation to fit WW3 requirements for boundary input.
After re-run with interpolated spec nc with WWM, results at open boundary are similar to original WWM. Green line in the following figures are results by using interpolated spec nc. I also try to derive spec nc Hs from

but magnitude seems not right (symbol X is from original spec, symbol + is from interpolated spec), maybe I still have some wrong calculations in the formula. However, interpolated is match to original one.
Hs horizontal distribution with interpolated spec is a bit different but still similar to the original one.

Open boundary:

Inside domain:

Hs with interpolated spec nc

[danishyo]

Hi @danishyo ,

Could you please let me know your most recent case configurations and the source codes of the schism-ww3 case and schism-wwm on Hercules? I would like to compare some outputs, spectra, hs etc.

Thank you!

Hi @yunfangsun
You can find schism-ww3 case at /home/feiye/work2_Dan/UFS/YunFang_case/Test_DUCK/ufs_schism_ww3
Source code is from branch feature/schism_3d (51fc37a)
I did schism-wwm case on Frontera, I will sync files to hercules later and let you know the path.

[danishyo]

Hi @yunfangsun
I put schism-wwm result at /home/feiye/work2_Dan/UFS/YunFang_case/Test_DUCK/schism_wwm
It is using schism master branch (97954f6).

[yunfangsun]

Hi @danishyo ,

Thank you

[janahaddad]

From our meeting on Oct 16, 2024:

From meeting on 10/16/2024

• Try constant Hs at the boundary using this script:

• • https://github.com/erdc/wave-tools/blob/main/bin/matlab/stden_2_swden.m
    

• Try the dock Sandy case:

• • Jana Haddad - NOAA Affiliate 3:31 PM https://github.com/erdc/wave-tools/tree/main
• • Jana Haddad - NOAA Affiliate 3:32 PM https://drive.google.com/drive/u/0/folders/1f3XNntGsOZP_1GVGiu2EwEBXILhL8qL

• Try real test cases

• • @aliabdolali would be able to provide some test cases that cover other hurricane cases for this Duck Domain

[aliabdolali]

Hi team, as I mentioned at the meeting, in order to see if boundary condition is set properly, you always can plot mean wave direction field and check it against what was the mean wave direction (th1m) in the forcing BC.
see mean wave direction:

Hs:

and tp:

for Sandy. Note that in this simulation, I used Neumann on the lateral BCs and a uniform water level and wind from nearest gauges in stand alone mode.

[aliabdolali]

@janahaddad Here are inputs for Sandy including all nml and netcdf files:
Duck Case 1.zip

[aliabdolali]

Hi @danishyo
Here is the guide on how to use wave-tools to generate BC from hs, tp, th1m
L07_boundary_condition.pdf

[aliabdolali]

@janahaddad I emailed you the Sandy-Duck guide tutorial, please add it to GD and share with the team

[saeed-moghimi-noaa]

Hi @danishyo
Would you please share your presentation? You can drag and drop it in this issue of you convert it to pdf.

@josephzhang8
Acording to Dan's presentation, the direction convention for wave spectra boundary condition for wwm is not following cartesian / mathematical direction convention which to me is a bit confusing. I strongly suggest to at least give the option (if it is not already there) to be able to define the wave spectra in mathematical convention. I would be happy to discuss this further.

@danishyo would you please add the graphics here?

Thanks to All

[saeed-moghimi-noaa]

@janahaddad I emailed you the Sandy-Duck guide tutorial, please add it to GD and share with the team

@aliabdolali
Highly appreciated.

[yunfangsun]

Hi @aliabdolali ,

Thank you for the files, could you please also share the switch file for the Duck case?

Thank you again!

[aliabdolali]

Hi @aliabdolali ,

Thank you for the files, could you please also share the switch file for the Duck case?

Thank you again!

Try the same you used for Ian

[josephzhang8]

Hi @danishyo Would you please share your presentation? You can drag and drop it in this issue of you convert it to pdf.

@josephzhang8 Acording to Dan's presentation, the direction convention for wave spectra boundary condition for wwm is not following cartesian / mathematical direction convention which to me is a bit confusing. I strongly suggest to at least give the option (if it is not already there) to be able to define the wave spectra in mathematical convention. I would be happy to discuss this further.

@danishyo would you please add the graphics here?

Thanks to All

@danishyo: to output WWM directions in maths convention, you just need to change (in &PROC)

LNAUTOUT=F

[saeed-moghimi-noaa]

Hi @josephzhang8

Is this also the case when you want to force spectra boundary condition to model (as input to force) in maths convention? Thanks.

[josephzhang8]

Yes, another flag LNAUTIN controls that.

My understanding is that compass convention is the default in wave modeling and few people use the math convention

[danishyo]

Hi @danishyo Would you please share your presentation? You can drag and drop it in this issue of you convert it to pdf.

@josephzhang8 Acording to Dan's presentation, the direction convention for wave spectra boundary condition for wwm is not following cartesian / mathematical direction convention which to me is a bit confusing. I strongly suggest to at least give the option (if it is not already there) to be able to define the wave spectra in mathematical convention. I would be happy to discuss this further.

@danishyo would you please add the graphics here?

Thanks to All

Hi @saeed-moghimi-noaa
Here is the presentation.
Shared_with NOAA.pdf

[yunfangsun]

Hi @aliabdolali ,

I would like to reproduce Duck Case 1 within WW3. The sandy_2012_spec.nc file is missing in the Duck Case 1 folder, and dx_cart_mesh.m is missing in the wave-tools folder. Could you please share it in the Google Drive folder?

Thank you very much!

[aliabdolali]

untitled folder.zip
Here is a zip file including the sandy input and the matlab script you asked for. you really do not need the matlab script though.

[danishyo]

Hi @uturuncoglu, I am testing Ali's Sandy case with feature/schism_3d branch (51fc37a).
I had a successful compile process and fv3_coastal.exe generated on hercules about 1 month ago without problem.
When I try to re-compile with the same code today, previous weird "hercules" ESMF compiling error happened again.
Like the following:

/home/feiye/work2_Dan/UFS/YunFang_case/ufs-coastal_schism_3d_20240911/ufs-weather-model/WW3/model/src/wav_shr_mod.F90(11): error #7013: This module file was not generated by any release of this compiler. [ESMF]
use ESMF , only : operator(<), operator(/=), operator(+)
------^
/home/feiye/work2_Dan/UFS/YunFang_case/ufs-coastal_schism_3d_20240911/ufs-weather-model/WW3/model/src/wav_shr_mod.F90(147): internal error: Please visit 'http://www.intel.com/software/products/support' for assistance.
rc = ESMF_SUCCESS
^
[ Aborting due to internal error. ]
compilation aborted for /home/feiye/work2_Dan/UFS/YunFang_case/ufs-coastal_schism_3d_20240911/ufs-weather-model/WW3/model/src/wav_shr_mod.F90 (code 1)
make[2]: *** [WW3/model/src/CMakeFiles/ww3_lib.dir/build.make:933: WW3/model/src/CMakeFiles/ww3_lib.dir/wav_shr_mod.F90.o] Error 1

And this is not happened on Frontera.
Do "hercules" update some spack-tool modules and cause the problem?
I already purge all the modules before doing compile.sh

[uturuncoglu]

@danishyo it could be and the branch that you are using might need to sync with upstream. The current head of UFS Coastal is working fine. Do you want me to go ahead and sync feature/schism_3d?

[danishyo]

Hi @uturuncoglu , now I am able to compile without problem on Hercules.
Guessing you already made some changes about modules.
I can do Sandy tests without problem now.

However, when I tried another larger case (smaller domain in SCHISM, larger domain in WW3 ) with similar settings and the same exec file.
After initialization (which took almost 2 hours), CMEPS has errors from ESMCI_Mesh_Glue.C in the beginning, I got exactly same errors on Frontera as well.
I have no idea how to fix this, could you help me to have a look?
Path on Hercules is /work2/noaa/nosofs/Dan/UFS/SECOFS_couple_mesh/couple_test

PET log shows the following:

20241204 003754.850 INFO PET0000 (esmFldsExchange_coastal_attr): called
20241204 003754.850 INFO PET0000 (esmFldsExchange_coastal_attr): done
20241204 003754.850 INFO PET0000 (esmFldsExchange_coastal_init): done
20241204 003754.850 INFO PET0000 (esmFldsExchange_coastal): done
20241204 003754.850 INFO PET0000 before med_map_RouteHandles_init
20241204 003754.850 INFO PET0000 (module_med_map: med_map_routehandles_initfrom_field): mapname bilnr_nstod
20241204 003754.851 INFO PET0000 ocn to wav srcMask = -987987 dstMask = 0
20241204 003754.897 ERROR PET0000 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Mesh/src/ESMCI_Mesh_Glue.C:6233 ESMCI_meshcreatedual() Internal error: Bad condition - /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/cache/build_stage/spack-stage-esmf-8.6.0-rqrapepmgfb7kpri3ynqlxusquf6npfq/spack-src/src/Infrastructure/Mesh/src/ESMCI_MeshDual.C, line:950: - there was a problem with triangulation (e.g. repeated points, clockwise poly, etc.)
20241204 003754.897 ERROR PET0000 ESMCI_MeshCap.C:258 MeshCap::meshcreatedual() Internal error: Bad condition - Internal subroutine call returned Error
20241204 003754.897 ERROR PET0000 ESMF_Mesh.F90:3707 ESMF_MeshCreateDual() Internal error: Bad condition - Internal subroutine call returned Error
20241204 003754.897 ERROR PET0000 ESMF_FieldRegrid.F90:1414 getMeshWithNodesOnFieldLoc Internal error: Bad condition - Internal subroutine call returned Error

[janahaddad]

@uturuncoglu, it looks like this may be related to the ESMF library interpolation issue right?

[uturuncoglu]

@janahaddad I think I respond this via mail but it would be also nice to add in here. This seems an issue in the mesh. If you look at the error message it has there was a problem with triangulation (e.g. repeated points, clockwise poly, etc.) information. It would be nice to check the meshes in both end.