Table of Contents
Rclone
Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and –dry-run protection. It is used at the command line, in scripts or via its API.
If you need to interact with Amazon or Google storage from Rocket or Comet, as part of your data workflow on the HPC facilities, Rclone will let you do that.
Installation
Rclone is not installed on our HPC facilities (neither Rocket nor Comet), but the developer fortunately makes a pre-built, read-to-run version available to download for most compatible Linux systems.
The Rclone download page shows a table of available versions:
The version we need, to run from within our own personal area(s) on the HPC systems is the Intel/AMD - 64 Bit row, and the Linux column; not the .deb or .rpm version - you cannot install those versions yourself.
Right click on the download icon of the correct version, and copy the link. In the above case, as of May 2025, it is https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip - but this will differ, and you should always visit the URL above to get the most current link.
Now open up a SSH connection to the HPC and run the following:
$ wget https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip
Saving to: ‘rclone-v1.69.1-linux-amd64.zip’
100%[==========================================================>] 23,314,273 23.9MB/s in 0.9s
$
Once the .zip file has downloaded we should extract it:
$ unzip rclone-v1.69.1-linux-amd64.zip
Archive: rclone-v1.69.1-linux-amd64.zip
creating: rclone-v1.69.1-linux-amd64/
inflating: rclone-v1.69.1-linux-amd64/README.html
inflating: rclone-v1.69.1-linux-amd64/git-log.txt
inflating: rclone-v1.69.1-linux-amd64/rclone
inflating: rclone-v1.69.1-linux-amd64/README.txt
inflating: rclone-v1.69.1-linux-amd64/rclone.1
$
Running
Simply change to the directory that you unzipped Rclone to, and run the rclone executable:
$ cd rclone-v1.69.1-linux-amd64
$ ./rclone version
rclone v1.69.1
- os/version: redhat 7.9 (64 bit)
- os/kernel: 3.10.0-1160.125.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: static
- go/tags: none
$
You can copy the rclone file anywhere in your HOME or NOBACKUP area. If you have a directory for man pages you can copy the rclone.1 to that folder, but it is not essential - the README.txt and README.html files contain the same help information.
Support for the use of Rclone is outside of what we are able to provide - please consult the (extensive) Rclone documentation: