Fastest way to create a filesystem image with cloop/create_compressed_fs

Question

Cloop is a linux software package to mount compressed block device images. create_compressed_fs is the accompanying tool to create a compressed image of a filesystem (or any kind of file or block device). What is the fastest way to create a filesystem image with it? Assume the source is a partition on one harddisk and the destination is a file on a different disk. Also assume that the source harddisk is the limiting factor in throughput speed.

Answer 1

This answer documents what I learned while attempting to optimize the transfer with create_compressed_fs.

create_compressed_fs is available in the repository in the cloop-util package in Ubuntu 20.04, but the cloop kernel driver is not. To build it, use the source from https://git.launchpad.net/ubuntu/+source/cloop, which works for current kernels. The upstream source is out of date. Cloop has been added to Ubuntu 20.10.

create_compressed_fs has a number of options, including ones that try to store the entire disk image in memory, so don't use that. Additionally, create_compressed_fs as a tendency to stop reading for a few seconds every ten or so seconds, probably to do some internal synchronization or whatever. When it is not reading, the source disk is sitting idle for a few seconds.

If your source is a partition, you might want to make sure the unused data is zeroed to improve compression. Use fstrim (if the device supports TRIM) or zerofree (if the device does not support TRIM but is an ext2/3/4 filesystem) for that.

First, find the size of the source partition with

sudo blockdev --getsize64 /dev/my-partition

If the source is a regular file use command ls -l my-imagefile, the size is then the number after the owner and group.

To ensure that the source drive keeps reading at full speed, we want to add a large buffer before create_compressed_fs. The pv tool can do that, and will also show the throughput and buffer level in real time. As create_compressed_fs also generates its own output, we want to run pv and create_compressed_fs in different terminals so their outputs don't get messed up. Create a fifo to have both processes talk to each other:

mkfifo fifo

In one terminal, run

create_compressed_fs -B <blocksize> -s <input_size> fifo <output_filename>.cloop

I like to use 16K for blocksize, but that is up to you.

At the same time in another terminal, run

sudo mbuffer -i <path_to_source_device_or_file> -m 1G >fifo

mbuffer now uses a 1 GB buffer to keep the source drive busy if create_compressed_fs pauses. The buffer should be sized such that it doesn't completely fill up in the pauses, but you should make sure that you have enough memory for the buffer. If you find that the buffer is full all the time and create_compressed_fs does not manage to empty it, your source device is not the bottleneck.

I also tried using pv to do the buffering, but it doesn't work very well. If it reads from the source block device directly it doesn't get the source drive to produce data at full speed, and it also manages to block when create_compressed_fs pauses, even though it should continue reading then. That appears to be a bug.

Once the image is done, don't forget to remove the fifo:

rm fifo