====== Why is my file operation going so slowly? ======

Most people don't realise that HPCs have a variety of filesystems optimised for different tasks.  This blog post highlights the differences that can occur:
  * https://walkingrandomly.com/?p=6167

Another factor that's often missed is that the number of files in a directory can have a dramatic effect on some operations on the Lustre (/nobackup) filesystem.  If you have large numbers (thousands) of very small files, it's best to split them into multiple subdirectories to keep numbers per directory in the hundreds.  We like NASA's best practices guide:  
  * https://www.nas.nasa.gov/hecc/support/kb/lustre-best-practices_226.html






----

[[:faq:index|Back to FAQ index]]