[Bi-users] Reminder: Retiring filesystems rossby20, smhid13 and fouo6

[Fredrik Nyström]

Dear Accumulus storage users,


this is a reminder about rossby20, smhid13 and fouo6 being retired on 
June 30 2022. 



Today we have also updated rsync from 3.1.2 to 3.2.4 on Bi login nodes.

- Default checksum method is xxhash instead of md5, higher throughput 
  and less cpu usage. Up to ~3 times faster depending on bandwidth to 
  storage, <30% gain on Bi due to network and to a certain degree I/O 
  bottlenecks.

- Default compression method is zstd instead of zlib. You should not 
  use compression (-z or --compress) when both source and destination 
  are local paths, but doing so by mistake and getting zstd will not 
  incur any performance penalty. zlib on the other hand will cap out 
  at less than 33MB/s and consume lots of cpu.
  Note that when using rsync over ssh, both server and client need to 
  have support for zstd, otherwise zlib will be used and this will most 
  likely be slower than without compression. You can check for 
  available hash and compression methods with "rsync --version".



When you restart rsync after downtime on Tuseday June 12, please do not 
use rsync with the "-c" option unless you know the implications. 

Example: Interrupted after copying half of a 100TB file tree with 
compression ratio 1.5 on Lustre servers. Restarting rsync with "-c", 
rsync will then verify checksums for all files it has already copied, 
on both source and destination before it resumes copying files. In this 
case it means reading 150TB of data over network before resuming.


Kind regards / Fredrik

On 2022-04-07 13:30, Fredrik Nyström wrote:
> Dear Accumulus storage users,
> 
> on Thursday June 30 2022 we will retire the following filesystems:
> 
> /nobackup/rossby20 /nobackup/smhid13 /nobackup/fouo6
> 
> Please copy any data that you wish to keep before this date.
> 
> We would also appreciate if you from now on try to avoid filling up 
> these filesystems with more precious data. Temporary data that does
> not need to be copied is ok. Same goes for rossby25 and smhid18 which
> will be retired in October 2022.
> 
> At this time there is plenty of space available on rossby27 and
> smhid20.
> 
> 
> If you need assistance copying data from rossby20, smhid13 and fouo6,
>  feel free to contact NSC <smhi-support at nsc.liu.se>.
> 
> 
> Kind Regards,

-- 
Fredrik Nyström, National Supercomputer Centre
freny at nsc.liu.se