How much storage overhead comes along with cryptsetup and ext4?

Question

I want to encrypt the content of a directory in a container with an ext4 filesystem using cryptsetup. The size of the container should be as small as possible and as big as necessary, because I only want to write once and then backup.

First try: setting the size of the container to the size of the content.

dirsize=$(du -s -B512 "$dir" | cut -f 1)
dd if=/dev/zero of=$container count=$dirsize
losetup /dev/loop0 $container
fdisk /dev/loop0 # 1 Partition with max possible size
cryptsetup luksFormat --key-file $keyFile /dev/loop0
cryptsetup luksOpen --key-file $keyFile /dev/loop0 container
mkfs.ext4 -j /dev/mapper/container
mkdir /mnt/container
mount /dev/mapper/container /mnt/container
rsync -r "$dir" /mnt/container

Rsync returns that there is not enough space for the data. Seems reasonable as there has to be some overhead for the encryption and the file system.

I tried it with a relative offset:

dirsize=$(($dirsize + ($dirsize + 8)/9))

This fixes the problem for dirs with > 100 MB, but not for dirs with < 50 MB.

How can I determine the respective amount of bytes the container has to be bigger than the directory?

Answer 1

LUKS by default uses 2 MiB for its header, mainly due to data alignment reasons. You can check this with cryptsetup luksDump (Payload offset: in sectors). If you don't care about alignment, you can use the --align-payload=1 option.

As for ext4, it's complicated. Its overhead depends on the filesystem size, inode size, journal size and such. If you don't need a journal, you might prefer ext2. It may be that other filesystems have less overhead than ext*, might be worth experimenting. Also some of the mkfs flags (like -T largefile or similar) might help, depending on what kind of files you're putting on this thing. E.g. you don't need to create the filesystem with a million inodes if you're only going to put a dozen files in it.

If you want the container to be minimal size, you could start out with a larger container, and then use resize2fs -M to shrink it to the minimum size. You can then truncate the container using that size plus the Payload offset: of LUKS.

That should be pretty close to small, if you need it even smaller, consider using tar.xz instead of a filesystem. While tar isn't that great for hundreds of GB of data (need to extract everything to access a single file), it should be okay for the sizes you mentioned and should be smaller than most filesystems...

Answer 2

"LUKS has a 1032 sectors (a sector being 512 bytes) overhead in which it store the LUKS header. Following the header, encrypted data just takes as much space as its decrypted counterpart, linearly"

See: http://glandium.org/blog/?p=139

Answer 3

It's difficult to predict the exact size of a filesystem, because a filesystem contains more than the files themselves. See Why are there so many different ways to measure disk usage? for some subtleties regarding disk usage. In particular:

• Each file ends with an incomplete block. Running du takes this into account, but you need to omit the -B argument, and this only applies if the filesystem you run it on has the same block size as the one where you'll copy the files.
• The filesystem also needs to contain inodes and other data structures (superblock, journal, …).

If you keep track of the number of files and create that many inodes, and sum up the sizes of the blocks and the sizes of the inodes, you'll get a good idea for the filesystem size. But if you want it to be really tight, expect a bit of trial and error.

The easiest way to find the minimal ext2 image size would be to create a large enough ext4 image, then use resize2fs to shrink it.

If you're making a read-only filesystem to be read back under Linux, squashfs would be a better choice than ext4.

Dm-crypt itself also has an overhead. which is explained in the FAQ. By default, cryptsetup reserves 2MB at the beginning of the volume for metadata (FAQ 2.18); the overhead is independent of the volume size. FAQ 6.12 and 6.13 explain how to reduce the overhead. Choose a 128-bit key size (you need that much and no more for security), which requires 512kB of key slots, in addition to the fixed-size header (544B). Pick the smallest possible alignment, or close thereto; I think that with a filesystem with a 4kB block size, there will be a small performance penalty if you use an alignment that is less than 4kB, and 4kB is negligible on that scale anyway, so I recommend a 4kB alignment, for a total overhead of 516kB. The options when creating the volume are

cryptsetup luksFormat -s 128 --align-payload=8 --key-file $keyFile /dev/loop0