9

I backed up/restored a filesystem from a Ubuntu system to a Fedora 17 one. I noticed that according to df output the used space increased by 30 %. What could be the reasons for this?

At the Fedora system df displays: 78 GB used

At the Ubuntu system df displays: 60 GB used

Differences between the systems:

Ubuntu: ext3 (created years ago)

Fedora 17: ext4 (created with a vanilla mkfs.ext4 call)

Restoring to an XFS filesystem (on Fedora 17) yields 78 GB used space.

Backup and restore was done with GNU tar. The filesystem contains a wide range of different sorts of files (i.e. from source trees, maildirs to ISO's etc.).

maxschlepzig
  • 57,532

2 Answers2

10

The first thing that comes to mind is "sparse files". Traditionally, one could create a file with data at an offset into the file, and then seek to a much greater offset. When writing data to the much greater offset, the filesystem would not allocate disk blocks for the intervening offsets. Programs that read those offsets without allocated disk blocks read zero values.

Tarring up sparse files causes the offsets of a sparse file that does not have allocated disk blocks to allocate disk blocks, both in the tar file (or output stream) and in the re-created file.

I recall that some DBMS created sparse files, as did programs like MSC/NASTRAN (finite element modelling system). Backing up those sparse files ends up using large amounts of offline storage, much to the surprise of everyone involved.

  • Torrent programs have an option that I think will do the same thing, for incomplete downloads. (At least, the description in Azureus sounds like what you're describing) – Izkata Jun 15 '12 at 03:22
  • Would du --apparent-size ... be a way to test if sparse files make the difference? – maxschlepzig Jun 15 '12 at 08:54
  • 1
    Another common usage for sparse files are disk images of virtualization software. – Jochen Lutz Jul 26 '18 at 20:22
4

Another thing to watch out for when you see space discrepancies is space reserved for root (typically 5% on ext{2,3,4}). This space allows the OS to function (write log files, etc...) even if a user fills the disk (so long as that user is not root).

You can view this setting using tune2fs -l:

[root@host ]# tune2fs -l /dev/md0 |grep Reserved
Reserved block count:     1279986
Reserved GDT blocks:      1017
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

You can disable this on your ext filesystems with tune2fs -m 0 /dev/NAME

From looking into it really quickly, I don't believe xfs has reserved space for root

zje
  • 2,311