harminv.c: invalid argument to ZGEEV

[tummasn]

Hi all

I am running a simulation of a ring resonator using a gaussian source and using harminv to analyse the resonant modes.
To simulate the ring, I use cylindrical dimensions and give an m-value so that 'dimensions=mp.CYLINDRICAL,m=mm'. This is the same as in the tutorial for cylindrical coordinates

I start out with a source with a center frequency fcen and a width df. src=mp.GaussianSource(fcen, fwidth=df), which then find many resonant frequencies in harminv.

When investigating one of the frequencies, i.e. I set m, fcen to match one of the found resonant frequencies and then reduce df to only include one resonance, I get the following error:

'harminv.c: invalid argument to ZGEEV' from line 503 in the harminv code.

Increasing the resolution or the size of the cell does not work.

Has anyone had the same issue or have any idea how to fix it?

For reference I am running python 3.10.4, pymeep 1.23.0, numpy 1.22.3, nomkl 3.0, libblas 3.9.0, and harminv 1.4.1.

[smartalecH]

How are you calling Harminv? It doesn't like one of the arguments you are using. (Specifically, LAPACK doesn't like one of the arguments you passed)

[tummasn]

Thanks for the reply!

Yes, I am calling Harminv as
'harmin = mp.Harminv(EE, sourceCenter, fcen, df)'
where EE= mp.Ez, sourceCenter =mp.Vector3(Radius,0,(h_rib+H/2)), and fcen and df are floats.
I then run it as
' sim.run(mp.after_sources(harmin),
until_after_sources=tAfterSource)'
And get frequencies etc. as
'hFreqs = np.array([m.freq for m in harmin.modes])'

The code runs fine for large df, but as when I decrease the bandwidth below a certain point it fails as above.

I should add that I get the following errors in my output file as well:
** On entry to ZGEBAL parameter number 3 had an illegal value
** On entry to ZGEBAL parameter number 3 had an illegal value
** On entry to ZGEHRD parameter number 3 had an illegal value

[tummasn]

I have found a fix for this problem. It seems like there is an upper limit of how many cores Harminv can tolerate. The code runs well with up to 12 cores, but increasing the number of threads above this causes these errors.

I don't know if this is an issue to fix in upcoming releases, but for me this can be considered solved.

[stevengj]

Apparently this is a problem with some LAPACK builds — ZGEEV isn't thread-safe for large numbers of cores unless LAPACK is built with -frecursive, which isn't the case on some platforms. See this discussion thread. So it's nothing to do with Meep or Harminv specifically.

(Interestingly the other discussion thread also arose from a electromagnetics solver, in that case S4.)

The LAPACK calls in Meep are not performance critical, so I would generally just recommend against using multiple cores for your BLAS library. (You won't want to do this if you are parallelizing Meep with MPI anyway.)