====== Transferring Data ====== All data transfer with the HPC facilities must use one of the SSH-based transfer methods listed below. We do not expose any of the [[started:filesystems|HPC filesystems]] by native Windows file sharing protocols, so you must use one of the methods listed below to get code and data into or out of the facilities. * [[#File Transfer Software Options|Software Options]] * [[#scp]] * [[#sftp]] * [[#rsync]] * [[#Expandrive]] * [[#WinSCP]] * [[#FileZilla]] * [[#Comparisons|Comparison Table]] ---- ===== File Transfer Software Options ===== This is not an exhaustive list of all the available file transfer options, but a selection of the most common tools which are in use. Most file transfer software tools can be used, as long as they support the SSH connection protocol as detailed in the [[:started:connecting_onsite#connection_details|Connecting to HPC (On Campus) - Connection Details]] table. ---- ==== scp ==== **scp** is used to copy single files (or small numbers of files) to or from the HPC facility (//either// **Rocket** or **Comet**). It is a standard tool that should be provided on all systems which have an SSH client installed (e.g. Mac OS and Linux). Windows users can find scp distributed with versions of Windows 10 which have the **ssh** command, in [[https://mobaxterm.mobatek.net/|MobaXterm]], in [[http://cygwin.com/|Cygwin]] or as the [[https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html|pscp]] command from PuTTY (as well as in many other software packages). For example, copying the files //file1.txt//, //file2.txt// and //jobscript.txt// from a machine we are on (called //mylaptop//) to the **Comet** HPC facility, directly to our NOBACKUP project folder (assuming our project folder is called //myproject//): mylaptops $ scp file1.txt file2.txt jobscript.sh n1234@comet.hpc.ncl.ac.uk:/nobackup/myproject/ password for n1234@comet.hpc.ncl.ac.uk: ************ file1.txt 100% 10.0MB/s 00:01 file2.txt 100% 10.0MB/s 00:02 jobscript.sh 100% 10.0MB/s 00:01 mylaptop $ You can also use scp in the //other direction//; copying data and files from **Rocket** or **Comet**, out to another system, as long as that system you are copying //to// has SSH running. In the example below copying the files //job_result.log// and //job_data.csv// from login node 02 in the **Comet** HPC facility to the /tmp folder on a Linux laptop named //mylaptop.ncl.ac.uk//: login2.comet.hpc $ scp job_result.log job_data.csv n1234@mylaptop.ncl.ac.uk:/tmp password for n1234@mylaptop.ncl.ac.uk: ************ job_result.log 100% 10.0MB/s 00:01 job_data.csv 100% 10.0MB/s 00:10 login2.comet.hpc $ Whilst **scp** is a simple command to use, and is available everywhere which is already running SSH, it //does// have downsides: * You have to pass the name of every file you want to copy as an argument on the command line * It does not offer any synchronisation or resume facilities * It will blindly overwrite any files of the same name on the destination * Copying many dozens (or hundreds) or individual files can cause additional load on the login nodes You can find full information on the use of the **scp** tool with the following command on one of our HPC facilities: $ man scp === Suggested Use === * Individual files, or small groups of files * Automated file transfers (**scp** can use [[:advanced:ssh_keys|SSH key based authentication]], to run without user interaction) * Quick, one-off transfers where the copy destination is a Linux machine and/or running SSH ---- ==== sftp ==== === Suggested Use === * Small groups of files, with an interactive interface ---- ==== rsync ==== === Suggested Use === * Individual files to small groups of files * Large numbers of files or entire directory trees * Keeping files or directory trees in-sync with another source * Long file transfers which need to be re-startable ---- ==== Expandrive ==== [[https://www.expandrive.com/|Expandrive]] is a commercial application for Windows, Linux and Mac OS systems. You can use the software to turn a SSH-based connection (such as scp) into a native drive or shared directory. Using this you can integrate the HOME and NOBACKUP filesystems from the **Comet** facility into your desktop, to enable drag-and-drop data transfer - this may be a useful option for some users. Expandrive does not offer any file transfer tools itself - it integrates the remote filesystem as a local drive; it is then up to the user to use their normal file copy tools to copy, move or delete data between drives as needed. Allows local tools such as text editors to directly edit files on the remote filesystem. Please note that Expandrive is not funded by the University (nor can we offer any specific support), and is just one of several similar options; Linux users may choose to investigate [[https://github.com/libfuse/sshfs|sshfs]] which offers similar functionality. ---- ==== WinSCP ==== == Suggested Use == ---- ==== FileZilla ==== == Suggested Use == ---- ===== Comparisons ===== ^ Title ^ Suitable For... ^ Scriptable ^ Interface ^ License ^ Other ^ | scp | Few files | Yes | Text / Terminal | Open Source | | | sftp | Few files | No | Text / Terminal | Open Source | | | rsync | Single or bulk transfers | Yes | Text / Terminal | Open Source | Incremental transfers, synchronises files or directories, resume partial transfers | | Expandrive | Persistent connection | N/A | Graphical | Commercial | Does not offer any transfer tools itself - integrates Comet or Rocket as a local drive | | WinSCP | Single or bulk transfers | No | Graphical | Open Source | Single transfers or bulk transfers by queuing up files | | FileZilla | Single or bulk transfers | No | Graphical | Open Source | Single transfers or bulk transfers by queuing up files | ---- [[:started:index|Back to Getting Started]]