3

I have a system image sparse file which actual size is only a few MBs but the "apparent size" is about 1GB. I'm trying to write it to a block device efficiently (without the holes). Here are some none working solutions I've tried:

  • dd if=sparse_file of=/dev/some_dev processes the whole file including the holes, so at the end I'm getting something like 1007869952 bytes (1,0 GB) copied, 22,0301 s, 45,7 MB/s
  • cp --sparse=always sparse_file /dev/some_dev seems also is not working as it takes long time for few MBs (~13s)
  • ddrescue --sparse --force sparse_file /dev/some_dev fails with a message ddrescue: Only regular files can be sparse. (Note: it works in the opposite direction as covered here).

There are 2 other ways covered here but I'd like to use only the standard tools which are the part of Linux distribution.

So is there a way to write the sparse file to a block image skipping the holes?

把友情留在无盐
  • 841
Artak Begnazaryan
  • 203

3 Answers3

1

This tool written in C may be helpful for you. It is not a standard tool, but it is very simple and easy to compile.

You are right that cp doesn't support sparse output to block devices (confirmed in man page).

LubosD
  • 141
  • 1
    This tool doesn't skip holes, rather it skip zeroes. So target device should be zeroed before(otherwise significant zeroes will be lost). – Yuriy Nazarov Aug 31 '15 at 15:46
0

I encountered a problem that is related to this thread: I created a LUKS crypto block device with integrity. I'd want to wipe each sector (i.e. logical block) so the sector's integrity checks, before I could read from the sector. Upon "luksFormat", cryptsetup(8) offers to wipe all sectors by default. If I'd overwrite a sector, the sector's integrity bits'd be overwritten, too, so I'd no need to care whether its previous integrity checks. However, if a partial write'd cause the kernel to read this sector, and possibly along with neighbor sectors within page size, and the related integrity'd fail to check, the write'd fail. Unfortunately, mke2fs writes partial sectors. So I could not mke2fs on a not wiped such device. So I wrote this "sparseblk" utility. I could create a fs image on a plain file, then write the plain file to the device. This "sparseblk" utility skips holes in its input by lseek(fd, offset, SEEK_DATA), and pads data to the sector boundary when writing. The kernel writes whole sectors after the fs is mounted.

hypothetical scenario: https://gitlab.com/cryptsetup/cryptsetup/issues/335#note_270050959

truncate -s 1G /tmp/1.img /tmp/2.img
truncate -s 980M /tmp/1.img /tmp/2.img
mke2fs /tmp/2.img
cryptsetup luksFormat \
    --key-file /etc/motd \
    --sector-size $(getconf PAGESIZE) \
    --cipher chacha20-plain \
    --integrity poly1305 \
    --integrity-no-wipe \
    /tmp/1.img
cryptsetup open --key-file /etc/motd /tmp/1.img dm-0
argv[0] < /tmp/2.img > /dev/mapper/dm-0
mount /dev/mapper/dm-0 /mnt/1

I've not tested it with e2fsck and resize2fs, etc., yet.

Anyway, if I'd be serious about security, I'd wipe the whole device.

You may use this "sparseblk" utility for other purposes. You may ignore the "logical block size not a multiple of page size" warning if you use this "sparseblk" utility for other purposes.

posix is adopting SEEK_DATA SEEK_HOLE: http://austingroupbugs.net/view.php?id=415

"sparseblk": https://gist.github.com/insulsa/18b7d31bd82ddade14db07f413c0b2d2

把友情留在无盐
  • 841
0

I'd like to use only the standard tools which are the part of Linux distribution.

As in, whatever would be installed by default?

Depending on what you're thinking, two possible answers:

  1. dd supports conv=sparse as a flag. As long as you know the target device is all zero'd out, just dd if=sparse_file conv=sparse of=/path/to/device

  2. qemu-img is installable from most standard distribution repositories, and it can understand sparse formats and writing. If the target block device supports TRIM it will sparse write to the target, and as of version 4 (I think it was?), you can force it to sparse write via the --target-is-zero argument:

qemu-img convert -O raw sparse_file /path/to/device or to force it qemu-img convert -O raw --target-is-zero sparse_file /path/to/device

Twirrim
  • 101