[Berzelius-users] New Berzelius features added

[Johan Raber]

Dear Berzelius User,

There are now two more options available on how to run jobs on
Berzelius, the MIG and "devel" reservations. The reservations are named
"3g.20gb" (the MIG option) and "devel", and are accessed by adding the
switch "--reservation=[3g.20gb|devel]" to your allocation command. The
reservations are both aimed at providing an option for the type of work
that is lighter and/or more interactive in nature like for instance
inference dominated workflows (e.g. alphafold) or rapid prototyping and
evaluation, with the goal of providing a fast-track queue for that type
of work and thereby free up resources for jobs requiring more compute
power.

Please direct any questions you may have regarding these features to
berzelius-support at nsc.liu.se.

The MIG Enabled Nodes
---------------------

We have now made a part of Berzelius MIG-enabled (see below and
https://www.nvidia.com/en-us/technologies/multi-instance-gpu/ for
details) to better allow jobs not fully utilizing the compute power of a
single A100 match the compute power to their workload. Using this
Berzelius feature will allow less consumption of a project's compute
time allocation for a significant portion of jobs on Berzelius, hence
potentially keeping the queue priority up for these projects now that
Berzelius is getting full.

The MIG-enabled nodes are put in a SLURM reservation called "3g.20gb"
accessible by adding the flag "--reservation=3g.20gb" to your SLURM
resource allocation command ("interactive", "sbatch", "salloc" or
"srun"). From usage statistics of Berzelius, we know a significant
number of jobs stand to benefit from using these MIG devices without
making compromises on actual performance. Many jobs use less than half
the available GPU performance of a standard, non-MIG A100 and should
thus not notice the difference when running on the MIG devices.

Multi-Instance GPU (MIG) is an NVIDIA technology allowing a logical
split of a physical GPU into 2, 3 or 7 instances. The A100 devices in
the "3g.20gb" Berzelius reservation have been split into 2 even parts
using 3 compute units and half the memory each. Using a MIG instance of
this type will consume half the amount of GPU hours on your allocation
compared to a standard A100.

Running jobs in the "3g.20gb" reservation comes with the limitation that
you cannot run multi-GPU jobs in this reservation, it is a single GPU
per job reservation only, but there is no set quota to how many single
GPU jobs you may run concurrently here. The wall time limit of 3d (72h)
is the same as on the main part of Berzelius.

Example job submission (1 "MIG" GPU for 1h):

$ interactive --gpus 1 --reservation=3g.20gb -t 1:00:00

The "devel" Node(s)
-------------------

The "devel" reservation currently consists of only one single node, but
this may be changed to the extent that it fills its intended purpose,
the purpose being to provide a high-job-turnover queue for small jobs.
This serves to allow interactive work, such as for instance rapid
prototyping and testing of submit scripts for longer batch jobs, with
(hopefully) less need to wait in queue for resources to become
available.

The reservation is active daily between 8:00 and 17:30 and has a wall
time limit of 2h, with a default of 2 minutes (to ensure/encourage that
a specific time is explicitly requested in the resource allocation). The
17:30 o'clock end of the reservation is inclusive, meaning that all jobs
submitted to this reservation must be finished before that time, i.e.
you can for instance *not* start a 2h long job at 15:31 but *can* start
a 1h 59m job. Jobs submitted between 17:30 and 8:00 (next day), are put
in a queue and are eligible to be started when the reservation becomes
active again.

Job allocations can use no more than one GPU, and a user can only have
one such job active at any given time in this reservation. The GPUs are
full A100-40GB, i.e. not MIG split. Access this reservation using the
switch "--reservation=devel" to your SLURM resource allocation command
("interactive", "sbatch", "salloc" or "srun").

Example job submission (1 standard A100-40GB GPU for 1h):

$ interactive --gpus 1 --reservation=devel -t 1:00:00

Best regards,
-- 
Johan Raber

Application Expert -- Quantum Chemistry
National Supercomputer Centre, Linköping University
email: raber at nsc.liu.se
tel.: +46 13 282619