====== Why is my file operation going so slowly? ======

Most people don't realise that HPCs have a variety of filesystems optimised for different tasks.  This blog post highlights the differences that can occur:
  * https://walkingrandomly.com/?p=6167

===== Are you doing file operations like ls or cp on files under /nobackup? =====

Users have reported that listing and copying is slow on /nobackup, the 'working' area for Comet.  This is because /nobackup uses the Lustre filesystem, which is optimised for large calculations. File operations like ''ls'' and ''cp'' access every file's metadata and this is something Lustre is not optimised for.

The number of files in a directory can have a dramatic effect on some operations on the Lustre (/nobackup) filesystem.  If you have large numbers (thousands) of very small files, it's best to split them into multiple subdirectories to keep numbers per directory in the hundreds.  We like NASA's best practices guide:  
  * https://www.nas.nasa.gov/hecc/support/kb/lustre-best-practices_226.html

<WRAP round tip 60%>
If you can't avoid listing directory containing thousands of files, **''ls -f'' is much more efficient** than ''ls''
</WRAP>








----

[[:faq:index|Back to FAQ index]]