Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 70 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and –dry-run protection. It is used at the command line, in scripts or via its API.
If you need to interact with Amazon or Google storage from Rocket or Comet, as part of your data workflow on the HPC facilities, Rclone will let you do that.
RClone is installed as a system package on Comet, on the login nodes only. You can therefore run rclone without loading any modules:
$ rclone version
rclone v1.57.0-DEV
- os/version: redhat 9.4 (64 bit)
- os/kernel: 5.14.0-427.91.1.el9_4.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: none
Full options:
$ rclone
Usage:
rclone [flags]
rclone [command]
Available Commands:
about Get quota information from the remote.
authorize Remote authorization.
backend Run a backend specific command.
cat Concatenates any files and sends them to stdout.
check Checks the files in the source and destination match.
checksum Checks the files in the source against a SUM file.
cleanup Clean up the remote if possible.
completion generate the autocompletion script for the specified shell
config Enter an interactive configuration session.
copy Copy files from source to dest, skipping identical files.
copyto Copy files from source to dest, skipping identical files.
copyurl Copy url content to dest.
cryptcheck Cryptcheck checks the integrity of a crypted remote.
cryptdecode Cryptdecode returns unencrypted file names.
dedupe Interactively find duplicate filenames and delete/rename them.
delete Remove the files in path.
deletefile Remove a single file from remote.
genautocomplete Output completion script for a given shell.
gendocs Output markdown docs for rclone to the directory supplied.
hashsum Produces a hashsum file for all the objects in the path.
help Show help for rclone commands, flags and backends.
link Generate public link to file/folder.
listremotes List all the remotes in the config file.
ls List the objects in the path with size and path.
lsd List all directories/containers/buckets in the path.
lsf List directories and objects in remote:path formatted for parsing.
lsjson List directories and objects in the path in JSON format.
lsl List the objects in path with modification time, size and path.
md5sum Produces an md5sum file for all the objects in the path.
mkdir Make the path if it doesn't already exist.
mount Mount the remote as file system on a mountpoint.
move Move files from source to dest.
moveto Move file or directory from source to dest.
ncdu Explore a remote with a text based user interface.
obscure Obscure password for use in the rclone config file.
purge Remove the path and all of its contents.
rc Run a command against a running rclone.
rcat Copies standard input to file on remote.
rcd Run rclone listening to remote control commands only.
rmdir Remove the empty directory at path.
rmdirs Remove empty directories under the path.
selfupdate Update the rclone binary.
serve Serve a remote over a protocol.
settier Changes storage class/tier of objects in remote.
sha1sum Produces an sha1sum file for all the objects in the path.
size Prints the total size and number of objects in remote:path.
sync Make source and dest identical, modifying destination only.
test Run a test command
touch Create new file or change file modification time.
tree List the contents of the remote in a tree like fashion.
version Show the version number.
Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Support for the use of Rclone is outside of what we are able to provide - please consult the (extensive) Rclone documentation: