Slurm's hetejob issue

[JanStreffing]

I ran into an issue today where I kept getting

sbatch TCO1279-DART-1D-hetjob_compute_20000101-20000101.run 
sbatch.bin: error: Batch job submission failed: Invalid account or account/partition combination specified

After tinkering around for a while I found that slurm crashes when the job name

#SBATCH --job-name=TCO1279-DART-1D-hetjob

contains -hetjob

So the fix was changing the experiment name to something like TCO1279-DART-1D-hetjo

As I'm writing this, I realize it's really more of a bug in slurm, but I ask my question anyway. Do we have a list of prohibited phrases for experiment ids?

[mandresm]

Thanks for reporting this! Good to know. I'm also surprised that SLURM made this kind of mistake...

About what you proposed: on one side, it's likely that SLURM will take time to get this fixed and even longer for the HPC admins to update to the fixed version. On the other side, there are a few places in ESM-Tools where one could introduce the hetjob string into the .run file, so if we would decide to check expid for hetjob then we should probably check other ESM-Tools variables as well. I'd say let's don't add this feature for now, let's switch this to a discussion so that it's available to everyone, and only if people starts to confront this problem more and more often, then we can make a workaround.

[JanStreffing]

I have the feeling HPC centers don't update their slurm. It gets set up and then they run until the system is taken down. Levante, a machine that is 1 year old uses a version that is not even in the supported version I found on the slurm issue tracker. (slurm 20.11.9-Bull.1.3)

I'm not sure how frequent this is, and if other flags are affected.

[mandresm]

I renamed this discussion to be more descriptive of the problem. The original name was Prohibit certain exp names?