[Vagnekman-users] Data transfer recommendations between Ekman-Vagn

[Johan Raber]

Dear Users of Ekman and Vagn,

We have recently upgraded the network infrastructure to a level where we now
have 10 Gb Ethernet all the way between Ekman and Vagn. Accompanying this
hardware upgrade is a version of SSH (High Performance Networking, HPN)
which better utilizes the multi-CPU core nature of the systems. This change
has been largely transparent to you, but there are some things worth
considering on your part still. However, if you have no interest in how to
improve transfer rates of your data between computers in general and
Ekman-Vagn in particular, you can stop reading now, the rest will be of no
interest to you.

You can now opt to chose the "none" cipher while using scp and rsync, as in
no encryption. Absolutely *no* sensitive stuff should be transmitted this
way naturally, This should give you close to available bandwidth in transfer
rates. This will vary wildly depending on other peoples usage of the
bandwidth. Authentication is however as safe as always and the integrity of
the data transferred is still verified. By design, you can *not* use the 
"none"
cipher in a regular interactive SSH session, only in batch mode 
transmissions.
This is invoked as:

you at ekman $ scp -oNoneSwitch=yes -oNoneEnabled=yes <your_files> 
username at vagn.nsc.liu.se:/some/path/ # OR using rsync
you at ekman $ rsync -ae 'ssh -oNoneSwitch=yes -oNoneEnabled=yes' 
<your_files> username at vagn.nsc.liu.se:/some/path/

Do note however, this is not standard in SSH and only works because both
server and client are patched. You can't expect this to work to/from/between
systems other than Ekman-Vagn unless they are patched. Same goes for the
parallelized crypto described below.

If you want speed and encryption, you are basically set right now since we
have enabled parallelized encryption/decryption on both Ekman and Vagn which
works for the default crypto, no adjustments should be necessary on your
part. We do not know at this point whether transmissions will be
bottlenecked by network bandwidth, CPU processing or possibly disk I/O (not
unlikely in many scenarios) since we haven't tested this yet on an idle
system. However, transfer rates are more or less as good as they are ever
likely to be right now with respect to the hardware and software side on
these two systems. Some fine tuning may improve things further to an extent
though.

If you need speedy transfer of files to/from other systems than Ekman and
Vagn and feel you are satisfied with a more light weight encryption, both in
terms of absolute encryption security and CPU recourses used, you can invoke
both "rsync" and "scp" with the "arcfour" cipher. This is a *less* secure
encryption than the standard SSH cipher (aes-128) but it doesn't mean that
it is easily cracked, it is officially supported by SSH after all, so unless
you need the absolute top-notch best cryptographic security this is
perfectly acceptable. You can invoke it like this

you at whereever $ scp -c arcfour <your_files> 
username at destination:/some/path/ # OR
you at whereever $ rsync -ae 'ssh -c arcfour' <your_files> 
username at destination:/some/path/

Unscientific tests have provided figures pretty close to available bandwidth
on 1 Gb ethernet (LAN) between non-HPN patched client/servers, ~90MB/s in
this case, whereas the standard crypto yielded ~45 MB/s. Note that these are
ballpark type figures.

Best Regards,
PDC and NSC support