Super-scaling from giving an esmvaltool run more tasks on slurm

[alistairsellar]

Hi all,

I'm testing a cut-down version of recipe_spmi.yml either 1 or 2 cores on the Met Office analysis cluster, and using 2 cores with same total memory available gives a 3+ factor speedup (details below). I would have expected less than a factor of two. Is there an obvious explanation for why it might be so much faster? I'm ony asking out of curiosity - not a priority if deeper investigation would be needed.

With ntasks=1 and mem=40GB it completed in 29 mins and used 34 GB.
With ntasks=2 and mem=40GB it completed in 8 mins and used 34 GB.

Recipe: recipe_smpi_filled.yml.txt
Log file: main_log.txt

Slurm directives used:

#SBATCH --time=00:40:00
#SBATCH --mem=80GB    
#SBATCH --ntasks=1   # or ntasks=2

[alistairsellar]

Details of individual tasks:

ntasks=2

Successfully completed task tas/tas (priority 1) in 03:58
Successfully completed task pr/pr (priority 3) in 04:00
Successfully completed task tas/grading (priority 0) in 04:14
Successfully completed task pr/grading (priority 2) in 04:12
Successfully completed task collect/SMPI (priority 4) in 00:03
Time for running the recipe was: 08:17

ntasks=1

Successfully completed task tas/tas (priority 1) in 03:51
Successfully completed task pr/pr (priority 3) in 03:54
Successfully completed task pr/grading (priority 2) in 24:50
Successfully completed task tas/grading (priority 0) in 24:59
Successfully completed task collect/SMPI (priority 4) in 00:04
Time for running the recipe was: 28:57

The grading tasks are ncl diagnostics.

[NParsonsMO]

It must actually have been running the priority 1 and 3 tasks concurrently, and the priority 0 and 2 tasks concurrently, each time, even with the ntasks=1 specified.

[schlunma]

Yes, as far as I know, --ntasks is completely ignored by ESMValTool. You can select the number of concurrent tasks by the config option max_parallel_tasks.

I assume you used 80GB of memory for the 2-tasks case and 40GB of memory for the 1-task case? Then I guess the answer why your run is much faster with 2 tasks is that there is more physical memory available (i.e., less memory spilling to disk).

[bouweandela]

Related issue #3655

[mo-jareddrayton]

Apologies, accidently hit comment mid-way through writing this.

This piqued my interest as I would have thought there would have been no difference in performance due to a lazy assumption I had around the SLURM configuration at the MO. The assumption I had was based on this line describing --ntasks

Number of tasks. By default each task will be assigned 1 CPU, but see --ntasks-per-core below. Only multiples of 2 CPUs are actually allocated, so any odd numbers are rounded up. 1 becomes 2, 3 becomes 4, etc.

• A CPU is a hyperthread in this context.
• Each core has two hyperthreads

So if the only change between two jobs is --ntasks=1 and ntasks=2 you still end up with two CPU's allocated.

             JobName     JobID    Elapsed   TotalCPU NCPUS     ReqMem     MaxRSS  ExitCode      State
------------------------------------------ --------- ---------- ---------- ----- ------ -------------
 esmvaltool-40gb-nt2   9103409   00:08:29  16:22.998     2       40Gn                  0:0  COMPLETED 
               batch 9103409.+   00:08:29  16:22.998     2       40Gn  29572316K       0:0  COMPLETED 
 esmvaltool-40gb-nt1   9105979   00:26:58  16:33.829     2       40Gn                  0:0  COMPLETED 
               batch 9105979.+   00:26:58  16:33.829     2       40Gn  32380588K       0:0  COMPLETED 

But it must be the case that even though you are allocated 2 CPUs when using ntasks=1 SLURM must still restrict the job to one. I had just assumed that you'd still be able to use both.

As @NParsonsMO pointed out, the grading tasks are running concurrently so I would hazard a guess that because the ntasks=1 configuration you have one CPU/Hyperthread trying to juggle two processes it may introduce some amount of extra overhead. So when running with -ntasks=2 you of course get double the compute, but also eliminate the additional overhead.

It would be interesting to rerun the ntasks=1 job and specify max_parallel_task=1 using the config option @schlunma suggested and see if that improves the performance of the grading tasks.

One other thing I've noticed is how ESMValTool/Dask is configuring the number of processes to run. You'll notice when running on SPICE, if you don't specify max_parallel_task it picks up the default by using os.cpu_count(). At least I think so based off of this line

n_processes = session["max_parallel_tasks"] or os.cpu_count()

The problem with this is it sees all of the CPUs on a SPICE node.

Processors: 2 x 12 core Xeon E5-2690 v3 @ 2.60GHz, 2 hyperthreads (CPUs) per core; 48 CPUs per server

Which is why you end up with this.

INFO [24249] Running tasks using at most 48 processes

Not much of a big deal in this case as there are only two variables being processed, but I guess it might cause issues if you had say eight variables and Dask was trying to run eight grading tasks concurrently but you only requested ntasks=2.

[schlunma]

From what I understand, using --ntasks in sbatch has no effect if the underlying application ignores this (and ESMValTool actually ignores ntasks):

-n, --ntasks=<number>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the Slurm controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node, but note that the --cpus-per-task option will change this default.

(quoted from the doc here).

What I do in practice to request resources for ESMValTool is either using --mem or --cpus-per-task.

[alistairsellar]

Thanks. On our system, --ntasks is (one of) the recommended way to control how many CPUs are allocated.

@NParsonsMO and I will run some experiments with different combinations of ntasks and max_parallel_tasks on our system,and report back. I expect from evidence so far and above discussion that it will be important to keep max_parallel_tasks smaller than the number of CPUs/threads.

[valeriupredoi]

core issue to look at is #3569 AFAICSI