[Bi-users] Changes on Bi queue system tomorrow

[Höglund Anders]

There will be some major changes in the queue system on Bi tomorrow (3
October).

This change have been discussed internally at SMHI.  If you have been
part of those discussions, then this e-mail contains no news for you. 
Otherwise read on.  Also, this change does not concern the group fm.

If you are external to SMHI, then keep in touch with your contact point
at SMHI on how this affects your usage of the machine, especially if it
was based on free nodes for that group.  At the moment they do not have
any more information than what is said here.

The background for the change is that we have a low utilisation of the
machine while at the same time many have the feeling that they can not
run jobs as much as they like.

All commands to submit jobs will be the same as before but there are
new options for new categories of jobs introduced.  Otherwise the only
thing that changes is which job get to run and which job has to wait. 
The details about new options can be found on the user guide for Bi at 
https://www.nsc.liu.se after the change is made.

The new system is introduced as a test and will be evaluated in
February.  We will try to solve problems that arise during the test but
if the problems are major and can not be solved then it will be
possible to revert the changes on short notice and go back to today's
system.  As always, it is important that you report things that doesn't
work.

The base for the new system is FairShare, i.e., the same as is used on
the academic clusters at NSC.  The priority between different jobs in
the queue is based on each groups (not individuals) use in the recent
past compared to their share of the system.  This is a continuous
process and not based on calendar months.  Each groups use at a certain
time will impact the queue priority less as time goes by.  The
different groups share of the system will initially be based on today's
number of nodes.

There are four categories of jobs: Normal jobs, high priority, low
priority and risk jobs.  Normal jobs are the default unless something
else is requested and should be the large bulk of jobs.

High priority jobs have higher priority in the queue than all other
jobs and should be used for interactive sessions, short test jobs or as
an emergency solution if something is urgent or the number of nodes is
not enough at work load peeks.  To monitor that this is not misused,
the jobs will be logged (as well as other jobs but we have specifically
asked it for these ones).  We hope that social conventions and the risk
for shame is enough for this to work in practise.

Low priority jobs and risk jobs have lower priority than all other
jobs.  Risk jobs will work as today, i.e., as normal jobs but will be
interrupted if the nodes are needed by other jobs.  Low priority jobs,
on the other hand, will not be interrupted but are guaranteed to run
the requested time but the maximum wall time is two hours (this limit
is up for discussion).  Usage from low priority and risk jobs are not
included when calculating fairshare priority for normal jobs.

Maximum number of nodes for normal jobs per group is twice today's
limits but at least 20 nodes.  The purpose is to protect against a
single group using up all the nodes, e.g., after a stop when the
machine is empty.

High priority jobs goes beyond this limit but have a total limit of
half of today's node limit per group but with a minimum here too that
is not yet determined.

The maximum time for jobs will remain at seven days but we would like
to encourage everybody running long jobs to try to split them into
smaller pieces.

Some of today's usage of fat nodes are not for the extra memory but
just because they happened to be free.  To improve utilasation the fat
nodes will be used for non-fat jobs but they will be used last.  This
is something we have to monitor to see if it is enough for the fat
nodes to be available when needed.

Trouble tickets for things that does not work should go to NSC as
before but things concerning the choices of the new queue system or how
it is experienced should be sent to the user representatives (me or
Michael Kolax).