5

I'd like to recreate a feature of Mac OS X called sparse bundles (disk images made out of smaller files, making them easy to backup after a small change). For that I'm looking for a way to 'virtually' create a single file made by concatenation of smaller ones (big.file shouldn't use all this space, just link to .files):

4096 0.file
4096 1.file
4096 2.file
4096 3.file
4096 4.file
20480 big.file

so that I'd be able to mount big.file using loop device, format as btrfs and upon writing to this disk, data should be written only to certain .files, allowing me to backup easily.

Any suggestions how I could accomplish that? Perhaps something FUSE-related?

anon94943
  • 53

2 Answers2

4

I was also looking for this kind of functionality and for similar reasons (remote backup to a cloud storage). I also wanted the disk image to start small and grow in size as data is added to it, just like MacOS sparse bundle disk images do.

Since I could not find anything comparable on Linux, I ended up writing a FUSE driver to accomplish this: MosaicFS.

To create a 300 GB MosaicFS disk image made up of 4 MB tile files (76800 files), you'd do the following:

# Create MosaicFS disk image
# (the individual tile files will be stored in the directory 'disk.img')
mosaicfs create --number 76800 --size 4M disk.img

# Mount disk image
# (the virtual image is mounted as 'myimage' with a size of 300 GB)
mosaicfs mount disk.img myimage

# Create file system 
mkfs.ext4 myimage

# Mount it
mkdir -p /media/mydrive
mount -o loop myimage /media/mydrive

# Unmount everything
umount /media/mydrive
fusermount -u myimage

You can use any file system supported in Linux. Optionally, you can also add a layer of encryption with dm-crypt (underneath the file system) or EncFS (on top of the file system).

For more details and source code, checkout MosaicFS on GitHub

ChrisLazgo
  • 41
  • 1
2

One way to do this would be to make each file an LVM physical volume, and join those physical volumes in a volume group and make an LVM logical volume using that space. But it's cumbersome: you need to associate the file with a loop device.

dd if=/dev/zero of=0.file bs=1024k count=4
losetup /dev/loop0 0.file
pvcreate /dev/loop0
# … repeat for all parts …
vgcreate -s 1m foo /dev/loop0 /dev/loop1 …
lvcreate -l 19 -n big foo
mkfs.btrfs /dev/mapper/foo-big

Reassembling the parts is not likely to be directly supported by your boot scripts, so you'd have to code quite a few things manually.

I don't see the point: how does splitting files facilitate backups? Many changes are likely to be spread over the whole volume (for example, several parts will contain copies of the superblock). You won't gain much by only backing up the parts that have changed: you'll need to look further inside the parts anyway.

If you want to make incremental backups, make them at the filesystem level.

If you want to make full backups of the whole image but ignore empty space, make sure to create a sparse file, use backup tools that manipulate sparse files efficiently, and periodically fill the empty space in the filesystem with zeroes and sparsify it.

Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
  • Unfortunately, that won't do because of the limit of loop devices you can create (can't make more than 9999 I believe). I'd need to create much more - I'm interested in making something like 300 GB image made out of 4MB parts. Is there a way of attaching multiple files to one loop device? – anon94943 Dec 30 '12 at 18:10
  • @anon94943 A loop device corresponds to one file. If you want a device that maps to the concatenation of several files, I think you'd have to code it up. But again, I don't see the point. – Gilles 'SO- stop being evil' Dec 30 '12 at 18:25