/dokuwiki/dokuwiki/ 2026-05-11T07:05:24+00:00 /dokuwiki/dokuwiki/ /dokuwiki/dokuwiki/lib/exe/fetch.php?media=wiki:dokuwiki.svg text/html 2025-08-08T15:31:33+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:comet_progress&rev=1754667093&do=diff Comet HPC Progress August 2025 The RSE team and academic colleagues have moved to testing higher level application functionality; container technology, interactive applications, 3D visualisation and commercial software packages are installed. July 2025 text/html 2026-02-17T09:43:58+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:comet_resources&rev=1771321438&do=diff HPC Resources & Partitions - Comet * Our HPC Resources List - Full technical details of all resource types on the Comet HPC * * * * * * * - This is now obsolete; please read the description to understand why * * * * * text/html 2025-03-11T13:03:32+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:concepts&rev=1741698212&do=diff Basic HPC Concepts What Is An HPC? We have already covered why you may need to use an HPC in an earlier section, Collection of resources, access more than practical on a single computer, long running hours and days at a time, split problems up across multiple computers, many users text/html 2026-03-04T10:41:18+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:connecting_offsite&rev=1772620878&do=diff Connecting to HPC (Off Campus) To connect to our HPC facilities remotely you must sign up for another service, request an access token, as well as configure additional settings in your SSH client application. The steps needed are explained in this guide. text/html 2026-02-17T14:37:08+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:connecting_onsite&rev=1771339028&do=diff Connecting to HPC (On Campus) This guide details how to connect to HPC facilities whilst you are connected to one of the Newcastle University networks (i.e. you are connected to Newcastle WiFi, or are physically connected via an ethernet cable in your office or lab space). text/html 2026-01-06T15:50:37+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:data_transfer&rev=1767714637&do=diff Transferring Data All data transfer with the HPC facilities must use one of the SSH-based transfer methods listed below. We do not expose any of the HPC filesystems by native Windows file sharing protocols, so you must use one of the methods listed below to get code and data into or out of the facilities. text/html 2026-03-20T13:41:06+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:do_i_need_hpc&rev=1774014066&do=diff What Are My Compute Options? Consider the three options below and the questions asked for each: Desktop/Laptop * You only need to run a few tens of computations or they can only be sequential e.g. the next step needs the result of the previous step. text/html 2026-04-29T14:51:56+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:filesystems&rev=1777474316&do=diff Data And HPC Filesystems HPC Filesystem Areas It is important to understand that the HPC facilities, like most non-desktop based computing environment, does not have a single filesystem or storage area. This is often one of the biggest changes for users coming from a non-HPC environment; even those who may have traditionally used Linux or unix-based systems such as Mac text/html 2025-07-30T15:33:28+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:first_job&rev=1753889608&do=diff Your First Slurm Job Job Files / Slurm SBATCH Files Job files are just text files. They can be created with any text editor (but not a word processor like Word). The job file instructs the HPC scheduler what hardware your job requires and contains the commands that your job runs to run your computation workload. text/html 2025-08-20T08:53:40+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:globus_trial_announcement&rev=1755680020&do=diff August 2025 The University has secured a year's licence to trial Globus, a leading research data management platform that addresses many of the large-scale data transfer challenges our research community faces. Globus provides a robust, cloud-based solution for moving, sharing, and managing scientific datasets securely between institutions. This represents a significant opportunity to enhance our research support capabilities. text/html 2026-02-17T15:56:57+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:index&rev=1771343817&do=diff Getting Started Do I Need HPC? Not all types of compute or workflow can benefit from the use of HPC facilities. We provide a description of the main types of compute facilities you can access at Newcastle University, and which of those facilities are the most suitable for various types of computational work. text/html 2025-04-09T14:36:42+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:job_long&rev=1744209402&do=diff Long Running Slurm Job This is a variant of the simple sequential Slurm job, but we move the submission of the job to the long partition which allows for jobs running more than 48 hours: * Uses the HPC Project group myhpcproject; change to use your real HPC Project name text/html 2025-04-09T14:38:15+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:job_parallel&rev=1744209495&do=diff Task Array Slurm Job Where you have multiple sets of data to process, and they can all be processed using the same commands, then it is beneficial to ask Slurm to automate this for us. Instead of submitting 4 jobs, each taking 1 hour each to process a data file, we can submit one job to run all 4 at the same time. text/html 2025-04-09T14:35:02+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:job_ram&rev=1744209302&do=diff Large Slurm Job This is an example of a very large Slurm job which is still running on a single node. A single-node Slurm job can use up to 256 cores and 1.5TB of RAM. * Uses the HPC Project group myhpcproject; change to use your real HPC Project name text/html 2025-04-09T14:35:26+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:job_simple&rev=1744209326&do=diff Simple Sequential Slurm Job This is the basis of most common, sequential Slurm jobs and can be extended easily by adjusting the CPU, RAM and runtime resources. Items to note are: * Uses the HPC Project group myhpcproject; change to use your real HPC Project name text/html 2025-11-18T11:49:32+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:managing&rev=1763466572&do=diff How Do I Manage My HPC Project? Anyone who is a member of an HPC Project may access the HPC Project Management page for that project from their My HPC Projects area. After you select one of your projects from the list that you either own, manage or are a member of, you will see the text/html 2025-04-09T12:29:46+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:module_basics&rev=1744201786&do=diff Introduction to Software Modules This guide will explain the basics of how to load software when working on our HPC facilities. * * * * * ---------- Linux Environment Modules On our HPC facilities we have many hundreds of software applications; text/html 2025-12-17T18:56:25+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:moving_your_data_from_rocket_to_comet&rev=1765997785&do=diff Moving project data from Rocket to Comet At the end of 2025, as Rocket is retired, projects may be transferred to Comet using the HPC Portal, but this will not transfer any data from Rocket to Comet. All data transfer is the responsibility of the project owner and members. text/html 2025-04-02T13:18:12+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:paying&rev=1743599892&do=diff Do I Have To Pay? Historically, access to the Rocket HPC facility has been free at the point of access. This will not change for the remainder of the lifespan of that facility. For Comet, we have introduced a new funding model. All users; both staff and students, will be able to continue to access HPC resources on the new facility free of charge. However, some of the text/html 2026-04-01T12:02:06+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:register&rev=1775044926&do=diff How Do I Register? To access our HPC facilities you must be a member of an HPC Project. Read on to understand what an HPC Project is, and who is eligible to apply. ---------- HPC Projects An HPC project is a record of your request, a grouping of users who you want to access resources on the HPC, as well as a place to view the reports, utilisation data and analysis of historic jobs. We also use your HPC project on the portal in a public directory of users of our facilities. The description of… text/html 2026-03-14T07:47:54+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:resource_overview&rev=1773474474&do=diff Resources and Partitions All Slurm jobs cost resources. Some of the hardware resources we make available are free, whereas some are reserved for paying projects. It is important to understand that the resources you request in your Slurm job directly impact the costs incurred by your project. text/html 2025-03-31T14:34:57+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:rocket_resources&rev=1743431697&do=diff HPC Resources & Partitions - Rocket * Our HPC Resources List - Rocket Partition Node Types text/html 2025-04-09T12:59:43+00:00 Anonymous (anonymous@undisclosed.example.com) /dokuwiki/dokuwiki/doku.php?id=started:slurm_basics&rev=1744203583&do=diff Introduction to Slurm What is Slurm? With an HPC facility, unlike running code on your own desktop or laptop, you do not directly access the various hardware which makes up the system. Instead, in most cases, you instruct a special software service, known as the scheduler, to run applications on your behalf. The scheduler analyses the type of job you want to run, taking note of the type and quantity of resources you indicate it needs (number of processors, amount of memory, etc) and finds the…