/dokuwiki/dokuwiki/ 2026-08-08T16:13:41+00:00 /dokuwiki/dokuwiki/ /dokuwiki/dokuwiki/lib/exe/fetch.php?media=wiki:dokuwiki.svg text/html 2026-08-06T08:11:43+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:rdw_setup&rev=1786003903&do=diff Setting up a new RDW share for a research project RDW is the best place to store research data, it is accessible as a mount on both Rocket and Comet HPCs and also from campus Linux and Windows computers. To request a new RDW share “Request Something text/html 2026-08-06T08:06:22+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:filesystems&rev=1786003582&do=diff Data And HPC Filesystems HPC Filesystem Areas It is important to understand that the HPC facilities, like most non-desktop based computing environment, does not have a single filesystem or storage area. This is often one of the biggest changes for users coming from a non-HPC environment; even those who may have traditionally used Linux or unix-based systems such as Mac text/html 2026-08-05T08:16:21+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:037&rev=1785917781&do=diff Why is my file operation going so slowly? Most people don't realise that HPCs have a variety of filesystems optimised for different tasks. This blog post highlights the differences that can occur: * <https://walkingrandomly.com/?p=6167> Are you doing file operations like ls or cp on files under /nobackup? text/html 2026-08-04T09:34:42+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:windows_file_explorer&rev=1785836082&do=diff Viewing RDW On Windows Windows File Explorer is NOT suitable for copying data to or from Comet. See File Transfer and RDW for the correct method. Windows File Explorer or Powershell can be a handy way to check that data exists on RDW. * Browse your files under the path text/html 2026-08-04T09:33:27+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:044&rev=1785836007&do=diff Can I move data from my OneDrive to Comet? It is not recommended to store research data on OneDrive because your University OneDrive account is deleted one month after you leave the University and due to GDPR requirements, the University cannot retrieve data for you or your colleagues, even before this happens. text/html 2026-08-04T09:29:34+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:data_transfer&rev=1785835774&do=diff Transferring Data All data transfer with the HPC facilities must use one of the SSH-based transfer methods listed below. We do not expose any of the HPC filesystems by native Windows file sharing protocols, so you must use one of the methods listed below to get code and data into or out of the facilities. text/html 2026-08-04T08:57:46+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:index&rev=1785833866&do=diff Frequently Asked Questions Creating and Managing HPC Projects * What is the HPC Driving Test? Data Storage and Transfer * Where do I put my HPC data? * How can I use Research Data Warehouse (RDW) with HPC? * How can I move my data to where I need it? * Can I move data from OneDrive to Comet? Filesystems * Help! I cannot access RDW * How much does RDW cost? * Why is my gzip / tar / cp operation going SO SLOWLY? Git & Github for managing code and scripts * How do I set u… text/html 2026-08-04T08:52:52+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:031&rev=1785833572&do=diff Why does my MPI job complain about connection errors, too many open files or addresses already in use? Depending on the number of MPI processes you attempt to start in your Slurm jobs, you may see errors such as: Or Or words to the effect of: text/html 2026-07-30T15:13:26+00:00 ncb176_newcastle.ac.uk (ncb176_newcastle.ac.uk@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=faq:043&rev=1785424406&do=diff sacct for resource usage see also FAQ 004 019 027 sacct is helpful for seeing resource usage by job e.g. gives output like filter for start and end time range add `-S` and `-E` for start and end dates in format YYYY-MM-DD checking for pre-empted jobs