How do I create small disk image with large partitions

Question

I am building a disk image for an embedded system (to be placed on an 4GB SD card). I want the system to have two partitions. A 'Root'(200Mb), and a 'Data' partition(800Mb).

I create an empty 1GB file with dd. Then I use parted to set up the partitions. I mount them each in a loop device then format them; ext2 for 'Root' ext4 for 'Data'. Add my root file system to the 'Root' partition and leave 'Data' empty.

Here's where the problem is. I am now stuck with a 1GB image, with only 200MB of data on it. Shouldn't I, in theory, be able to truncate the image down to say.. 201MB and still have the file system mountable? Unfortunately I have not found this to be the case.

I recall in the past having used a build environment from Freescale that used to create 30Mb images, that would have partitions for utilizing an entire 4GB sdcard. Unfortunately, at this time, I can not find how they were doing that.

I have read the on-disk format for the ext file system, and if there is no data in anything past the first super block (except for backup super blocks, and unused block tables) I thought I could truncate there.

Unfortunately, when I do this, the mounting system freaks out. I can then run FSCK, restore the super blocks, and block tables, and can mount it then no problem. I just don't think that should be necessary.

Perhaps a different file system could work? Any ideas?

thanks,

edit

changed partition to read file system. The partition is still there and deoesn't change, but the file system is getting destroyed after truncating the image.

edit

I have found the case to be that when I truncate the file to a size just larger than the first set of 'Data' partition superblock and inode/block tables, (Somewhere in the data-block range) the file system becomes umountable without doing a fsck to restore the rest of the super blocks and block/inode tables

Answer 1

The easiest way to do this is to create your backing file as a sparse file; that is, make it 1GB with truncate -s 1G disk.img instead of dd if=/dev/zero bs=1048576 count=1024 of=disk.img (or whatever). Nicely, truncate is also far faster.

If you do an ls -l on the file, it'll show as 1GB—but that's only its apparent size. du disk.img will give the actual size.

(Note—you'll need to have your image on a filesystem that supports sparse files. All the common Unix ones do. Ext2/3/4 all do. FAT32 does not. HFS+ doesn't, either.)

NOTE: Sparse files still are, logically, the full size. Just the never-written sections aren't physically stored on disk. For the most part, the not-actually-stored part is hidden from programs. A few utilities have support for it, though. (E.g., dd conv=sparse, cp --sparse=auto/always, etc.). An actual USB stick can't be sparse. And if you use dd conv=sparse to write it, it'll probably be much faster, but you'll leave whatever data was there before, instead of the blocks full of NULs (0x00) expected. Should work fine (as its free space), but will leave old data on the USB stick—possibly a security concern.

Answer 2

Firstly, writing a sparse image to a disk will not result in anything but the whole of the size of that image file - holes and all - covering the disk. This is because handling of sparse files is a quality of the filesystem - and a raw device (such as the one to which you write the image) has no such thing yet. A sparse file can be stored safely and securely on a medium controlled by a filesystem which understands sparse files (such as an ext4 device) but as soon as you write it out it will envelop all that you intend it to. And so what you should do is either:

1. Simply store it on an fs which understands sparse files until you are prepared to write it.

2. Make it two layers deep...

   • Which is to say, write out your main image to a file, create another parent image with an fs which understands sparse files, then copy your image to the parent image, and...

   • When it comes time to write the image, first write your parent image, then write your main image.

Here's how to do 2:

• Create a 1GB sparse file...

  dd bs=1kx1k seek=1k of=img </dev/null
  

• Write two ext4 partitions to its partition table: 1 200MB, 2 800MB...

  printf '%b\n\n\n\n' n '+200M\nn\n' 'w\n\c' | fdisk img
  

• Create two ext4 filesystems on a -Partitioned loop device and put a copy of the second on the first...

  sudo sh -c '
      for p in "$(losetup --show -Pf img)p"*        ### the for loop will iterate
      do    mkfs.ext4 "$p"                          ### over fdisks two partitions
            mkdir -p ./mnt/"${p##*/}"               ### and mkfs, then mount each
            mount "$p" ./mnt/"${p##*/}"             ### on dirs created for them
      done; sync; cd ./mnt/*/                       ### next we cp a sparse image
      cp --sparse=always "$p" ./part2               ### of part2 onto part1
      dd bs=1kx1k count=175 </dev/zero >./zero_fill ### fill out part1 w/ zeroes
      sync; cd ..; ls -Rhls .                       ### sync, and list contents
      umount */; losetup -d "${p%p*}"               ### last umount, destroy
      rm -rf loop*p[12]/ '                          ### loop devs and mount dirs
  

  ---

  mke2fs 1.42.12 (29-Aug-2014)
  Discarding device blocks: done
  Creating filesystem with 204800 1k blocks and 51200 inodes
  Filesystem UUID: 2f8ae02f-4422-4456-9a8b-8056a40fab32
  Superblock backups stored on blocks:
      8193, 24577, 40961, 57345, 73729
  
  Allocating group tables: done
  Writing inode tables: done
  Creating journal (4096 blocks): done
  Writing superblocks and filesystem accounting information: done
  
  mke2fs 1.42.12 (29-Aug-2014)
  Discarding device blocks: done
  Creating filesystem with 210688 4k blocks and 52752 inodes
  Filesystem UUID: fa14171c-f591-4067-a39a-e5d0dac1b806
  Superblock backups stored on blocks:
      32768, 98304, 163840
  
  Allocating group tables: done
  Writing inode tables: done
  Creating journal (4096 blocks): done
  Writing superblocks and filesystem accounting information: done
  
  175+0 records in
  175+0 records out
  183500800 bytes (184 MB) copied, 0.365576 s, 502 MB/s
  ./:
  total 1.0K
  1.0K drwxr-xr-x 3 root root 1.0K Jul 16 20:49 loop0p1
     0 drwxr-xr-x 2 root root   40 Jul 16 20:42 loop0p2
  
  ./loop0p1:
  total 176M
   12K drwx------ 2 root root  12K Jul 16 20:49 lost+found
   79K -rw-r----- 1 root root 823M Jul 16 20:49 part2
  176M -rw-r--r-- 1 root root 175M Jul 16 20:49 zero_fill
  
  ./loop0p1/lost+found:
  total 0
  
  ./loop0p2:
  total 0
  

• Now that's a lot of output - mostly from mkfs.ext4 - but notice especially the ls bits at the bottom. ls -s will show the actual -size of a file on disk - and it is always displayed in the first column.

• Now we can basically reduce our image to only the first partition...

  fdisk -l img
  

  ---

  Disk img: 1 GiB, 1073741824 bytes, 2097152 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: dos
  Disk identifier: 0xc455ed35
  
  Device Boot  Start     End Sectors  Size Id Type
  img1          2048  411647  409600  200M 83 Linux
  img2        411648 2097151 1685504  823M 83 Linux
  

• There fdisk tells us there are 411647 +1 512 byte sectors in the first partition of img...

  dd seek=411648 of=img </dev/null
  

• That truncates the img file to only its first partition. See?

  ls -hls img
  

  ---

  181M -rw-r--r-- 1 mikeserv mikeserv 201M Jul 16 21:37 img
  

• ...but we can still mount that partition...

  sudo mount "$(sudo losetup -Pf --show img)p"*1 ./mnt
  

• ...and here are its contents...

  ls -hls ./mnt
  

  ---

  total 176M
   12K drwx------ 2 root root  12K Jul 16 21:34 lost+found
   79K -rw-r----- 1 root root 823M Jul 16 21:34 part2
  176M -rw-r--r-- 1 root root 175M Jul 16 21:34 zero_fill
  

• And we can append the stored image of the second partition to the first...

  sudo sh -c '
      dd seek=411648 if=./mnt/part2 of=img
      umount ./mnt; losetup -D
      mount "$(losetup -Pf --show img)p"*2 ./mnt
      ls ./mnt; umount ./mnt; losetup -D'
  

  ---

  1685504+0 records in
  1685504+0 records out
  862978048 bytes (863 MB) copied, 1.96805 s, 438 MB/s
  lost+found
  

• Now that has grown our img file: it's no longer sparse...

  ls -hls img
  

  ---

  1004M -rw-r--r-- 1 mikeserv mikeserv 1.0G Jul 16 21:58 img
  

• ...but removing that is as simple the second time as it was the first, of course...

  dd seek=411648 of=img </dev/null
  ls -hls img
  

  ---

  181M -rw-r--r-- 1 mikeserv mikeserv 201M Jul 16 22:01 img
  

Answer 3

Are you wanting to create this image with a 800MB of space, so you can copy to sd cards. But not wanting to wast space when storing the image. If so may I suggest compression, such as bzip. It will depend on unused blocks being initialised to zero though.

Answer 4

Hope I understand the question properly:

To shrink a file-system and partition. Use gparted.