Most people don't realise that HPCs have a variety of filesystems optimised for different tasks. This blog post highlights the differences that can occur:
Another factor that's often missed is that the number of files in a directory can have a dramatic effect on some operations on the Lustre (/nobackup) filesystem. If you have large numbers (thousands) of very small files, it's best to split them into multiple subdirectories to keep numbers per directory in the hundreds. We like NASA's best practices guide:
Back to FAQ index
Table of Contents
Main Content Sections
Documentation Tools