[Vagnekman-users] Pilot testing of ffv - an asynchronous file transfer tool

[Per Lundqvist]

[DESCRIPTION]

ffv is an asynchronous file transfer tool for moving large amounts of
data locally or remotely. It has been developed by NSC for the purpose
of simplifying data moving from ekman to vagn (but of course also
works between other hosts if available). Note that the current version
is a beta version.

 It's usage is very similar to using a batch queueing system in a
 sense that you submit a job that describes what to move and where to
 move it. After the job has been created ffv takes care of the rest
 until the job is finished, an error occurs or authentication is
 needed.

[PILOT TESTING]

We have until now had one user testing it but now we seek for more
pilot users. If you are interested to try out the current beta-version
ffv from ekman then do the following:

   1. load the modules
      $ module load beta-modules
      $ module load ffv/0.9.4b

   2. then contact support with the output from (we need to setup
      certain file permissions at ekman):
      $ ffv

[EXAMPLE]

A man page with examples is also available (man ffv). Example usage
when transferring a file tree from ekman to vagn (requires passphrase
protected public key login to vagn - see
http://www.nsc.liu.se/systems/snic/security.html):

   1. On workstation load private key into key agent:
      $ ssh-add -c

   2. log in to ekman with agent forwarding (-A)
      $ ssh -A -K -o GSSAPIDelegateCredentials=yes -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes perlundq at ekman.pdc.kth.se

   3. load ffv module
      $ module load beta-modules
      $ module load ffv/0.9.4b

   4. submit a ffv job that moves all files within ~/data/ to
      /home/perl on vagn (user perl). A job id is returned:

      $ ffv submit -u perl ~/data/ vagn.nsc.liu.se:/home/perl
      0.perlundq.ekman

      This job id is a unique handle for the job at ekman for the user
      perlundq. 0.perlundq.ekman may be shortened to 0 or 0.perlundq
      when using ffv as perlundq on ekman.

   - by default progress is sent asynchronously by mail to
     perlundq at ekman (which I redirect using ~/.forward), and to a file
     at ~/ffv.0.perl.ekman.out

   - to get status about the job:
        $ ffv status 0
     or: 
        $ ffv status 0.perl.ekman

   - if additional authentication is required:
        $ ffv auth 0

   - if afs tokens have expired:
        $ ffv afslog 0

   - to cancel a job before it has finished or failed (will revert all
     files to its original content):
        $ ffv cancel 0

   - to block until job has finished or failed:
        $ ffv wait 0

   Note that while a job is ongoing, data is temporary moved into a
   spool directory that must not be tampered with or you may have
   incomplete data at the destination host after the job has finished.

(*) ffv is currently also available on vagn and gimle but cannot
transfer files to ekman since they need a updated ssh client (with
certain kerberos patches).

/Per Lundqvist

-- 
Per Lundqvist

National Supercomputer Centre
Linköping University, Sweden

http://www.nsc.liu.se