9

I always assumed the st_blocks field returned by stat()/lstat()... system calls and which du uses to get the disk usage of files was expressed in 512 bytes units.

Checking the POSIX specification, I now see POSIX makes no guarantee for that. The perl documentation for its own stat() function also warns against making that assumption.

In any case, as indicated by POSIX, that block size is not related to the st_blksize field returned by stat(), so has to be found elsewhere.

Checking the GNU du or GNU find source code, I see evidence HP/UX uses 1024 bytes units instead. GNU find adjusts its -printf %b output to always give a number of 512-byte units which is probably the source of my confusion.

Is there any other Unix-like system, still currently in use where st_blocks is not expressed in 512 byte units? Can that be filesystem dependant (as POSIX suggests)? I guess mounting an HP/UX NFS share could do it.

ilkkachu
  • 138,973
Stéphane Chazelas
  • 544,893

1 Answers1

13

st_blksize is called "the optimum I/O size" and unrelated to the units used for st_blocks. The optimum I/O size of course is filesystem specific. This is a result from the fast filesystem development from Berlekey in 1981/1982. Before, there was no optimum block size in the available filesystem

st_blocks is expressed in units of DEV_BSIZE that indeed is 1024 on HP-UX. DEV_BSIZE is a platform specific constant. Later, when FFS was renamed to UFS, there was a second filesystem in BSD UNIX with different behavior related to indirect blocks and that required this new stat() field. Before, du did just know the algorithm for indirect blocks from the filesystem.

If you run a HP-UX NFS fileserver and other NFS clients, you get wrong reports from the HP-UX NFS server for df unless HP did fix their problem during the past 15 years, where I had no access to recent versions of HP-UX. I know of no other UNIX with similar NFS related bugs.

BTW: up to NFSv3, NFS assumes a blocksize of 512 and HP would need to convert their NFS reports in the server. NFSv4 does not make that implicit assumption, but HP-UX still reports wrong numbers.

I know no other UNIX that is based on a 1024 DEV_BSIZE.

schily
  • 19,173
  • Thanks Jörg. I see GNU tools also refer to the S_BLKSIZE macro to determine the st_blocks size. Do you know more about the history of the stat() API? It seems to me st_blksize and st_blocks should have come in pair, and possibly they got misaligned at some point for historical reason? – Stéphane Chazelas Sep 18 '21 at 08:26
  • It actually looks like st_blksize came first around BSD 4.1 in 1983, and st_blocks in BSD 4.2. They may not have been meant to be related. – Stéphane Chazelas Sep 18 '21 at 11:37
  • 1
    S_BLKSIZE is Linux only and you are right, st_blksize is a result from developing FFS while st_blocks was added later to get better du results for files with indirect blocks and for holey files. – schily Sep 18 '21 at 12:21
  • I don't see S_BLKSIZE mentioned anywhere in the Linux sources. GNU libc and uclibc define both S_BLKSIZE and DEV_BSIZE to 512 and musl only DEV_BSIZE. NetBSD also defines both. BSD 4.2 do define DEV_BSIZE to 512 independently of the BLKDEV_IOSIZE. I'm trying to figure out when HPUX added st_blocks and why it deviated from BSD. Earliest evidence I found so far is from 1985. None of the AT&T systems I looked (from research v1 -> v10 to SysIII -> SVR4.2 had st_blocks), I suppose POSIX took it from BSD – Stéphane Chazelas Sep 18 '21 at 16:23
  • I see some header files in HPUX 5.1 (around the time its HFS file system was introduced) tape images as almost identical as the BSD4.2 one but with DEV_BSIZE changed from 512 to 1024. Also evidence there was collaboration with UCB. I guess that defines the minimum block size supported on the system. – Stéphane Chazelas Sep 18 '21 at 16:38
  • @StéphaneChazelas: if I run $ grep -Irl S_BLKSIZE /usr/include/ at least on Ubuntu 20.04, I get this file: /usr/include/x86_64-linux-gnu/sys/stat.h. EDIT: OK, alright – it's the GNU libc, not Linux kernel stictly speaking. – saulius2 Aug 27 '23 at 17:01