Could the situation: many CPUs simultaneously read the same single file, slow down the speed of reading of every CPU?

Question

I am running cat to separately combine file_X with a numerous number of files e.g. file_1 to file_100000000000.

Due to the large number, I distributed the job to a node with 64 CPUs to run parallelly on each CPU. Each job is ran in a sub-folder, and so there are 64 sub-folders.

To my surprise, the overall speed is much slower than expected.

As the shell script I used just direct each job to the same single file_X located in the parent directory of the 64 sub-folders, I wonder if many CPUs simultaneously read the same single file, would slow down the speed of reading of every CPU?

Answer 1

Yes and no.

The actual reading of the file should happen at the same speed regardless of how many processors are doing it.

However, depending on the operating system and its configuration, there could be file locking that is happening. While multiple processes can have a read lock simultaneously, getting a lock and releasing that lock must happen in a shared mutual exclusion block. If your system is doing those sorts of locks, the processors must line up to get access to the file, and then must line up afterwards to declare their lack of further interest in the file.

Depending on the file system that stores the file_X and the various files being combined with it and the options with which that file system was mounted, the access time of file_X may be getting updated each time cat reads it. If this is the case, it's probable that there will be a write lock made to the file_X inode before each update, which will then be released.

Another possible reason for the reduced speed is the fact that all 64 of these jobs are writing files in parallel, which are necessarily on different points on the disk. Unless you're using a solid state drive (SSD), that could require a substantial amount of moving the write heads around on the disk. Those files are in 64 different directories, so there are 64 places to update besides the files being created.

Doing all of this activity in a shell script means that each file copy is fork-execed. Fork is seen as a fairly expensive system call, but on a system with shared libraries, it pales in comparison to the cost of the exec family of system call, as that requires doing a search for every shared library, and loading all of those shared libraries. This is another place which could potentially have a read lock placed on a file, depending on exactly what unix this is on and possibly what its configuration is.