We limit the number of array task entries by design - each array task is a distinct job submitted to Slurm, so vast arrays can very quickly overwhelm the scheduler with hundreds (or even thousands) of new entries, each of which it has to track, check, schedule and launch.
Whilst launching one array task per input file is the simplest method to ensure each data file is processed, it is not the only way.
In our Building a Parallel Task Array Solution guide, at the end of the walkthrough we detail a method for launching a 250 element task array, with each task processing 1/250th of a 27000 element data set. If you have very large input data sets that you want to process in a task array, this is a valid method you can use which only needs a few extra simple steps involving basic Linux commands and a small amount of shell script adding to your Slurm batch file.