Transferring Data

All data transfer with the HPC facilities must use one of the SSH-based transfer methods listed below. We do not expose any of the HPC filesystems by native Windows file sharing protocols, so you must use one of the methods listed below to get code and data into or out of the facilities.

• Software Options
  • scp
  • sftp
  • rsync
  • Expandrive
  • WinSCP
  • FileZilla
• Transfers to RDW
• Comparison Table

This is not an exhaustive list of all the available file transfer options, but a selection of the most common tools which are in use. Most file transfer software tools can be used, as long as they support the SSH connection protocol as detailed in the Connecting to HPC (On Campus) - Connection Details table.

scp is used to copy single files (or small numbers of files) to or from the HPC facility (either Rocket or Comet). It is a standard tool that should be provided on all systems which have an SSH client installed (e.g. Mac OS and Linux). Windows users can find scp distributed with versions of Windows 10 which have the ssh command, in MobaXterm, in Cygwin or as the pscp command from PuTTY (as well as in many other software packages).

For example, copying the files file1.txt, file2.txt and jobscript.txt from a machine we are on (called mylaptop) to the Comet HPC facility, directly to our NOBACKUP project folder (assuming our project folder is called myproject):

mylaptops $ scp file1.txt file2.txt jobscript.sh n1234@comet.hpc.ncl.ac.uk:/nobackup/myproject/
password for n1234@comet.hpc.ncl.ac.uk: ************
file1.txt                               100% 10.0MB/s 00:01
file2.txt                               100% 10.0MB/s 00:02
jobscript.sh                            100% 10.0MB/s 00:01
mylaptop $

You can also use scp in the other direction; copying data and files from Rocket or Comet, out to another system, as long as that system you are copying to has SSH running. In the example below copying the files job_result.log and job_data.csv from login node 02 in the Comet HPC facility to the /tmp folder on a Linux laptop named mylaptop.ncl.ac.uk:

login2.comet.hpc $ scp job_result.log job_data.csv n1234@mylaptop.ncl.ac.uk:/tmp
password for n1234@mylaptop.ncl.ac.uk: ************
job_result.log                             100% 10.0MB/s 00:01
job_data.csv                               100% 10.0MB/s 00:10
login2.comet.hpc $

Whilst scp is a simple command to use, and is available everywhere which is already running SSH, it does have downsides:

• You have to pass the name of every file you want to copy as an argument on the command line
• It does not offer any synchronisation or resume facilities in the event that you need to pause or resume a large transfer
• It will blindly overwrite any files of the same name on the destination
• Copying many dozens (or hundreds) or individual files can cause additional load on the login nodes

You can find full information on the use of the scp tool with the following command on one of our HPC facilities:

$ man scp

• Individual files, or small groups of files
• Automated file transfers (scp can use SSH key based authentication, to run without user interaction)
• Quick, one-off transfers where the copy destination is a Linux machine and/or running SSH

• Small groups of files, with an interactive interface

• Individual files to small groups of files
• Large numbers of files or entire directory trees
• Keeping files or directory trees in-sync with another source
• Long file transfers which need to be re-startable

coming soon

Expandrive is a commercial application for Windows, Linux and Mac OS systems. You can use the software to turn a SSH-based connection (such as scp) into a native drive or shared directory. Using this you can integrate the HOME and NOBACKUP filesystems from the Comet facility into your desktop, to enable drag-and-drop data transfer - this may be a useful option for some users.

Expandrive does not offer any file transfer tools itself - it integrates the remote filesystem as a local drive; it is then up to the user to use their normal file copy tools to copy, move or delete data between drives as needed. Allows local tools such as text editors to directly edit files on the remote filesystem.

Please note that Expandrive is not funded by the University (nor can we offer any specific support), and is just one of several similar options; Linux users may choose to investigate sshfs which offers similar functionality.

Whilst logged in to a HPC login node (not a compute node) you may copy data to the RDW filesystem(s) under /rdw/ using standard cp commands. You cannot access RDW from a compute node.

Your RDW project directories are under /rdw/, under one of several numeric folders:

$ ls -l /rdw/
total 452
drwxrwx---  32 root 213509  770 Jul 22  2021 01
drwxrwx---  44 root 213509 1136 Jan 11  2022 02
drwxrwx---  57 root 213509 1549 Feb  5 10:32 03
drwxrwx---  79 root 213509 2205 Apr  3 10:03 04
drwxrwx---  74 root 213509 1997 Feb 12 08:50 05
drwxrwx--- 133 root 213509 3922 Aug 19  2024 06
drwxrwx---  57 root 213509 1522 Feb 20 13:31 07
drwxrwx---  32 root 213509  807 Jul 22  2021 08
$

You will need to find which of those numeric folders contains your specific project directory - we do not have a list of them.

Then, you can access the RDW folder with a path similar to:

• /rdw/04/MyProject/

Remember that you do not get an RDW folder as part of registering for HPC as it is not part of the HPC service - you must register for an RDW space seperately.

Back to Getting Started