ext4: How to account for the filesystem space?

Question

I've recently formatted a 1.5 TB drive with the intention of replacing ntfs with ext4.

Then I noticed that the files I saved don't fit on the new partition.

df:

ext4 (ext3 & ext2 show the same behavior)
Filesystem      1K-blocks   Used  Available Use% Mounted on
/dev/sdb1      1442146364   71160 1442075204    1% /media/Seagate

ntfs (similar to all other options that gparted offers):
/dev/sdb1      1465137148  110700 1465026448    1% /media/Seagate

That 1K-blocks difference means a glaring 22 GiB less usable space.

I have already executed

tune2fs -O \^has_journal
tune2fs -r 0
tune2fs -m 0

with, unsurprisingly, no effect as that does not affect blocks that just aren't there.

Still, fdisk reports that the ext4 partition covers the entire disk.

fdisk -l /dev/sdb:

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  2930277167  1465138583+  ee  GPT

And thus e. g. resize2fs reports that there's "Nothing to do!"

dumpe2fs -h /dev/sdb1:
dumpe2fs 1.41.14 (22-Dec-2010)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          d6fc8971-89bd-4c03-a7cd-abdb945d2173
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              91578368
Block count:              366284288
Reserved block count:     0
Free blocks:              360518801
Free inodes:              91578357
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      936
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat May 21 17:12:04 2011
Last mount time:          Sat May 21 17:15:30 2011
Last write time:          Sat May 21 17:24:32 2011
Mount count:              1
Maximum mount count:      32
Last checked:             Sat May 21 17:12:04 2011
Check interval:           15552000 (6 months)
Next check after:         Thu Nov 17 16:12:04 2011
Lifetime writes:          1372 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4
Directory Hash Seed:      c334e6ef-b060-45d2-b65d-4ac94167cb09
Journal backup:           inode blocks

What is using that missing space?

Answer 1

Let's see. The device size is 1,465,138,583½ kB = 1,500,301,909,504 B. The filesystem consists of 366,284,288 blocks of 4096 B each, which is 1,500,300,443,648 B. I don't know what the remaining 1,465,856 B (1.4 MB) are used for (additional copies of the superblock? I know there are a few kB of space at the beginning for the bootloader.).

The filesystem contains 91,578,368 inodes of 256 bytes each, which takes up 23,444,062,208 B (about 22 GB, hint, hint). Then there is 1,442,146,364 kB = 1,476,757,876,736 B for file contents. This accounts for 23,444,062,208 B + 1,476,757,876,736 B = 1,500,201,938,944 B. The remaining size is 98,504,704 B = 24,029 blocks which is in the right range to be the journal size.

As you can see, everything is accounted for. (Ok, almost everything, but we're talking megabytes, not gigabytes.)

Answer 2

First of all, the difference in available space you are seeing doesn't mean at all that there is space "wasted"; it is not wasted because it is of fundamental importance for the filesystem to function. You should not compare Ext4 and NTFS in this way without a very big "but" specifying all the design and structural differences between filesystems, and also specifics of each implementation (e.g. how each driver reports free space to the VFS layer).

Imagine the partition as a huge space where you can put any pieces of data you have. If you have only one piece of data with a size equal to the partition, you could just write it starting at the beginning of the partition and be cool with it. But you don't. Instead you may have thousands of small files, and all these files grouped in different ways, and each file associated with many other small pieces of data (name, date/time and permissions), etc. You have to organize the big space of the partition so that you can reach all these pieces of data quickly and efficiently. Also, you have to be concerned with how to write new pieces of data and discard old pieces of data efficiently. You need data structures.

And there is a lot of data structures. Some of them are very dumb, others allow you to retrieve data more quickly at the expense of more slower writes, other allow writes more quickly at the expense of reads, some still may be very good at both reads and writes but require long pauses and idle overhead while it rearranges the data, etc.

You certainly want a system that:

1. Is very fast to write information on it;
2. Is very fast to retrieve information from it;
3. Is good at organizing and managing the information stored in it;
4. Makes good use of the space (partition) where the filesystem is stored;
5. Is resilient against hardware problems, so that you still get most or all your information back on partial system failures;
6. Is resilient against software problems, so that a bug in an application, or a malicious application installed, will not destroy your data permanently;
7. Is resilient against human errors, so that it forgives you when you accidentally orders the system to delete something you shouldn't (a.k.a. the trash/recycle bin).

High-performance filesystems allow very fast reads and writes at the expense of some space. Some of the fastest data structures used in filesystems, like hash tables and B-trees, are very complex, and they reserve more space than it is actually in use in order to allow very fast accesses.

Ext4 has other important properties. There is no single point-of-failure in the filesystem. There are many copies of critical data spread through the partition, while some other filesystems (I can't say for NTFS) may render all your data unreadable if a failure happens on the right spot. Also, Ext4 reserves a lot of space for your data right at filesystem creation stage, while NTFS grows with your data.

Answer 3

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! 
The util fdisk doesn't support GPT. Use GNU Parted.

This message indicates the disk uses GPT-style partitioning, and this fdisk tool only understands the legacy MBR style.

To prevent accidental reformattings if GPT-partitioned disks are plugged into older non-GPT-aware systems, GPT partitioning scheme includes a "protective MBR": a completely fake partition table that basically tells "this disk is completely in use by a partition type you don't know anything about" to any operating system or tool that only understands MBR partitioning.

To get an accurate display of the partition table of your /dev/sdb, use:

parted /dev/sdb print