4

I have a web-site that I'm checking for performance issues and bugs, and i came across a caching code that caches thousands of files into one single directory.

I understand that this is not good and that I/O will degrade and i also heard about potential inode problem.

And i know how to fix the caching code, but the thing is that at this point fixing would be very expensive.

The question: What is the worst case scenario if i live it like it is right now? What will happen to the website? (right now this one single cache directory has 400K files)

I'm new to Ubuntu. And i understand that this is might be an off-topic. But i think this is a "system" question and it does not belong to the 'programming' part of the stackoverflow.

Thanks!

UPDATE: The file system is UFS

rinchik
  • 141
  • 2
    Whether the files are in one directory or 1000 directories does not make any difference to the inode issue. I'm not sure why you think having the files "in one directory" amounts to a problem. – goldilocks Jan 31 '13 at 15:53
  • http://stackoverflow.com/a/6939792/1378267 – rinchik Jan 31 '13 at 15:56
  • What filesystem is that? Will more file be added? Check if stat -f / (or where it's mounted to) to see the inode count. Type ls and go to have a cup of coffee. Or two cups. – ott-- Jan 31 '13 at 16:01
  • 1
    @rinchik The link you posted to makes no mention of the filesystem type either, there is just a blanket statement about I/O and directory size, which IMO is hogwash. If you want to believe a claim that insubstantial, it is up to you, but for myself, I would want to look around for a more serious explanation and justification. If you can't find that, just take it as the paranoid imagining of some netizen... – goldilocks Jan 31 '13 at 16:16
  • @goldilocks That's why i posted this question :) – rinchik Jan 31 '13 at 16:17
  • @ott-- I don't have anaccess to the server at this point. But yes. Files are being added into this directory constantly and there will be a lot more of them.. – rinchik Jan 31 '13 at 16:19
  • I could see someone theorizing that having a lookup table with 400k entries may make for more collisions than a table with 10 entries, but since it sounds like you have done some coding, hopefully you will agree that a 400k table is still not a significant issue for a decent look-up algorithm -- which presumable your OS and UFS have managed that ;) Again, the inode issue is a different one. – goldilocks Jan 31 '13 at 16:21
  • @rinchik The filesystem type is UFS? On Ubuntu/Linux? Really? I think you would be safer with a more typical and modern type of filesystem, like ext4. – Celada Jan 31 '13 at 16:22
  • @goldilocks that actually makes sense :) And what about inode issue? – rinchik Jan 31 '13 at 16:27
  • 1
    I don't actually know anything about UFS, but most likely, you need one inode per file. So when you run out of inodes, you can't store any more files, even if you have free space to put them in. Where the files are does not make a difference to the inode count or number of inodes used (in fact, having more directories will use more inodes, if each directory counts as a file). – goldilocks Jan 31 '13 at 16:36
  • The number of inodes is irrelevant. What happens is that when a file is looked up in a directory, most filesystems just do a sequential search, and that search will take time proportional to the number of files. Distribute the files among directories, and compute the directory to search from the file name (for example, use the first two characters of the name as the directory). – vonbrand Jan 31 '13 at 16:44
  • i think it's related to your client.If i was designer of your project, i use db instead of fs. – PersianGulf Jan 31 '13 at 18:27

2 Answers2

2

The situation is somewhat surprising. UFS is an unusual filesystem for a production Linux install. UFS write-access under Linux normally needs to be explicitly enabled in the kernel, since it has been considered experimental for many years:

CONFIG_UFS_FS_WRITE: UFS file system write support (DANGEROUS)

Say Y here if you want to try writing to UFS partitions. This is experimental, so you should back up your UFS partitions beforehand.

Like many traditional file systems, UFS uses sequential file lookups within directories. This does indeed lead to performance problems for directories with many files, since the search time grows linearly with number of files. In the BSDs, where UFS is often the default file system, this issue lead directly to the creation of Dirhash, a hash table lookup for directories, which significantly improves performance.

As far as I know, UFS support under Linux does not use Dirhash. Therefore, you can expect to experience increasing performance issues as the number of files in your directory grows. In terms of sequential access, 400K files is a lot, and you can expect a significant performance hit.

Splitting files between sub-directories effectively manages the sequential access problem. Alternatively, you could move to a filesystem that supports a more sophisticated file storage structure. For example, XFS implements fast file access for large directories through the use of B+ trees.

Your second concern was about inodes. Usually, the number of inodes on your filesystem is fixed, and this is usually a function of the amount of space available at filesystem creation time. For example, /etc/mke2fs.conf holds the default inode ratio (number of inodes per x bytes) for ext filesystems.

Normally, this number is far larger than the number of files you are likely to create, and is not a cause for concern. However, you can check your inode usage with df -i. If inode limitations actually are likely to be a problem, messing with directories will not help you, since inodes are a filesystem-wide concept, independent of directory. In this case, you would be forced to recreate the filesystem, setting the inode parameter (-i) to mkfs appropriately.

ire_and_curses
  • 12,362
1

On a normal unix (inode-based) filesystem, including UFS, it's a reasonable approximation to say each file or directory you create uses one inode. Having lots of files in one directory doesn't change this.

The usual problems with the approach you describe are:

  • file systems use hashes or tree-type data structures for directory lookups to speed up searching and creation, the more files you have in a single directory the slower it gets. With hashing, this slowdown can be quite pronounced as collisions occur.
  • typical unix commands have trouble (specifically ls sorting, and shell glob expansion), though usually well before a filesystem slowdown.
  • as the directory gains new files, more blocks are allocated, it will become increasingly fragmented, requiring more disk IO to access.

More modern filesystems (ext3/4) use B-tree like data structures to keep directories sorted, as part of their on-disk data. I believe UFS implementation use in-memory hashing (based on FreeBSD use and documentation, I don't have much direct experience with UFS on Linux) as the on-disk format does not use hashes.

This has some good UFS information and links: https://serverfault.com/questions/53416/max-total-files-in-a-directory-in-freebsd-6-ufs

The most likely worst-case is at some point you will experience noticeable and ever-worsening slowdown when accessing that directory. When it gets to that point it will be tedious to fix (based on my experience with exploding sendmail queues).

I encourage you to monitor (and graph) your system's iowait time, and get to know iotop and slabtop if you don't already.

If possible, I also suggest you try some simple experiments to time the creation of 1000 files in your cache directory, and compare with that in an empty directory.

Community
  • 1
mr.spuratic
  • 9,901