100

How to clear unused space with zeros ? (ext3,ext4)

I'm looking for something smarter than

cat /dev/zero > /mnt/X/big_zero ; sync; rm /mnt/X/big_zero

Like FSArchiver is looking for "used space" and ignores unused, but opposite site.

Purpose: I'd like to compress partition images, so filling unused space with zeros is highly recommended.

Btw. For btrfs : Clear unused space with zeros (btrfs)

Community
  • 1
Grzegorz Wierzowiecki
  • 14,264
  • 5
    Check this out: http://superuser.com/questions/19326/how-to-wipe-free-disk-space-in-linux – Mat Jul 29 '12 at 10:18
  • 1
    Two different kind of answer are possible. What are you trying to achieve? Either 1) security, by forbidding someone to read those data, or 2) optimizing compression of the whole partition or [SSD performance](http://en.wikipedia.org/wiki/Trim_(computing)? – Totor Jan 05 '14 at 02:57

6 Answers6

108

Such an utility is zerofree.

From its description:

Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3 file-system and fills them with zeroes. This is useful if the device on which this file-system resides is a disk image. In this case, depending on the type of disk image, a secondary utility may be able to reduce the size of the disk image after zerofree has been run. Zerofree requires the file-system to be unmounted or mounted read-only.

The usual way to achieve the same result (zeroing the unused blocks) is to run "dd" do create a file full of zeroes that takes up the entire free space on the drive, and then delete this file. This has many disadvantages, which zerofree alleviates:

  • it is slow
  • it makes the disk image (temporarily) grow to its maximal extent
  • it (temporarily) uses all free space on the disk, so other concurrent write actions may fail.

Zerofree has been written to be run from GNU/Linux systems installed as guest OSes inside a virtual machine. If this is not your case, you almost certainly don't need this package.

UPDATE #1

The description of the .deb package contains the following paragraph now which would imply this will work fine with ext4 too.

Description: zero free blocks from ext2, ext3 and ext4 file-systems Zerofree finds the unallocated blocks with non-zero value content in an ext2, ext3 or ext4 file-system and fills them with zeroes...

Other uses

Another application this utility is to compress disk images that are a backup of a real disk. A typical example of this is the dump of the SD card in a BeagleBone or a Raspberry Pi. Once empty spaces have been zeroed, backup images can be compressed more efficiently.

Community
  • 1
enzotib
  • 51,661
  • 2
    Is it official page of the tool http://intgat.tigress.co.uk/rmy/uml/index.html ? Do you think it's safe to use with ext4 ? – Grzegorz Wierzowiecki Jul 29 '12 at 14:08
  • 3
    @GrzegorzWierzowiecki: yes, that is the page, but for debian and friends it is already in the repos. I used on a ext4 partition on a virtual disk to successively shrink the disk file image, and had no problem. – enzotib Jul 29 '12 at 14:12
  • 2
    This isn't equivalent to the crude dd method in the original question, since it doesn't work on mounted file systems. – jlh Mar 04 '16 at 10:10
  • 1
    zerofree page talks about a patch that lets you do "filesystem is mounted with the zerofree option" so that it always zeros out deleted files continuously. does this require recompiling the kernel then? is there an easier way to accomplish the same thing? – endolith Oct 14 '16 at 16:33
  • 4
    Be careful - I lost ext4 filesystem using zerofree on Astralinux (Debian based)… – Hubbitus Nov 23 '16 at 22:20
  • 2
    June 2019 - I've successfully used zerofree on an ext4 filesystem mounted read-only. (My Debian version refuses to attempt anything with a filesystem mounted read-write.) – Chris Davies Jun 11 '19 at 12:51
  • I wrote a C program called zerohere that takes the opposite approach and sort of "perfects" the "crude dd method" and works on all operating systems. While the drawbacks mentioned are definitely worth considering, they are also not generally a practical problem with a user that is knowingly running such a tool. My zerohere program immediately deletes the zero file, so there is an extremely small time between all free space being full and being freed again. One last note: I never use ext* filesystems and I don't think anyone should when XFS exists. – Jody Bruchon Jul 31 '21 at 00:46
48

Summary of the methods (as mentioned in this question and elsewhere) to clear unused space on ext2/ext3/ext4:

Zeroing unused space

File system is not mounted

  • If the "disk" your filesystem is on is thin provisioned (e.g. a modern SSD supporting TRIM, a VM file whose format supports sparseness etc.) and your kernel says the block device understands it, you can use e2fsck -E discard src_fs to discard unused space (requires e2fsprogs 1.42.2 or higher).
  • Using zerofree (e.g. zerofree src_fs) to explicitly write zeros over unused blocks.
  • Using e2image -rap src_fs dest_fs to only copy blocks in use (new filesystem should be on an otherwise zeroed "disk", requires e2fsprogs 1.42.9 or higher).

File system is mounted

  • If the "disk" your filesystem is on is thin provisioned (e.g. a modern SSD supporting TRIM, a VM file whose format supports sparseness etc.), your kernel says the block device understands it and finally the ext filesystem driver supports it you can use fstrim /mnt/fs/ to ask the filesystem to discard unused space.
  • Using cat /dev/zero > /mnt/fs/zeros; sync; rm /mnt/fs/zeros (sfill from secure-delete uses this technique). This method is inefficient, not recommended by Ted Ts'o (author of ext4), may not zero certain things and can slow down future fscks.

Having the filesystem unmounted will give better results than having it mounted. Discarding tends to be the fastest method when a lot of previously used space needs to be zeroed but using zerofree after the discard process can sometimes zero a little bit extra (depending on how discard is implemented on the "disk").

Making the image file smaller

Image is in a dedicated VM format

You will need to use an appropriate disk image tool (such as qemu-img convert src_image dst_image) to enable the zeroed space to be reclaimed and to allow the file representing the image to become smaller.

Image is a raw file

One of the following techniques can be used to make the file sparse (so runs of zero stop taking up space):

  • cp --sparse=always src_image dst_image.
  • fallocate -d src_image (requires util-linux v2.25 or higher).

These days it might easier to use a tool like virt-sparsify to do these steps and more in one go.

 Sources

Community
  • 1
Anon
  • 3,794
  • 3
    These days, you might want to lead with virt-sparsify! I tried all these steps and that one was the real deal - chopped an ubuntu vm in half and fit in great in our workflow. – Peter Turner Aug 29 '22 at 18:32
17

sfill from secure-delete can do this and several other related jobs.

e.g.

sfill -l -l -z /mnt/X

UPDATE #1

There is a source tree that appears to be used by the ArchLinux project on github that contains the source for sfill which is a tool included in the package Secure-Delete.

Also a copy of sfill's man page is here:

Josip Rodin
  • 1,138
cas
  • 78,579
  • that URL is obsolete. no idea where its home page is now (or even if it still has one), but it's packaged for debian and ubuntu. probably other distros too. if you need source code, that can be found in the debian archives if you can't find it anywhere else. – cas Jul 29 '12 at 12:04
  • The obsolete manpage URL is fixed now. Looks like "Digipedia" is no longer a thing. – mwfearnley Jul 31 '17 at 13:04
8

If you have e2fsprogs 1.42.9, then you can use e2image to create the partition image without the free space in the first place, so you can skip the zeroing step.

psusi
  • 17,303
  • I couldn't (easily) find any info online about these parameters, but they are indeed given in the 1.42.9 release notes: http://e2fsprogs.sf.net/e2fsprogs-release.html#1.42.9 – mwfearnley Mar 03 '17 at 13:36
  • 1
    That's a valuable tool! However, your answer would be better if it also explained how to use it. It's e2image -ar /dev/foo1 | gzip > fs.img.gz, or some variation thereof. Include -f to use it on a mounted fs. – marcelm Oct 03 '18 at 16:04
3

if you don't want to resort to additional tools this method should be faster than cat /dev/zero > /mnt/fs/zeros:

dd if=/dev/zero of="/mnt/fs/filler" bs=1048576
dd if=/dev/zero of="/mnt/fs/filler1" bs=1
sync
rm /mnt/fs/filler
rm /mnt/fs/filler1
arauzo
  • 182
eadmaster
  • 1,643
  • Why would that be faster than cat? (No, the block size argument of dd doesn't help) – marcelm Oct 03 '18 at 15:49
  • the block size used by cat is smaller. – eadmaster Oct 04 '18 at 11:03
  • 6
    Yes, but that doesn't make dd faster. In fact, it probably makes cat faster. Block size is only important to minimize the overhead of system calls, and optimize CPU cache usage. With the syscall overhead, you enter the area of diminishing returns. Cache starts to hurt above, say, 1MiB. And when operating on real disks, you're I/O bound anyway and the point is mostly moot. Try benchmarking it yourself. – marcelm Oct 04 '18 at 14:13
  • Might want to add a "sync" command between the "dd" and "rm" commands. – Gunar Gessner Oct 07 '21 at 11:27
  • I started dd and cat concurrently and mostly the same file size has been reached by both methods. So both approaches achieved similar speed, at least on my system (SSD disk). – arauzo Jul 27 '22 at 09:44
1

I do it like:

#!/bin/sh
# take available size from string
DISKSIZE=`df / | grep / | cut -d " " -f 5`

#minus 10mb (you can do 1Mb)
TODELETE=$(( (DISKSIZE - 10000)/1000 ))

dd if=/dev/zero of=/zero.dump bs=1M count=$TODELETE 2> /dev/null
sync
rm -f /zero.dump
Sun Rise
  • 11
  • 1
  • 1
    Welcome to the site, and thank you for your contribution. Please note that the "backtick"-style for command substitutions is deprecated and the $( ... ) notation is now the recommended syntax. Also, would you mind editing your post to add some explanations on how your approach works, as answers should if possible be understandable to a wide range of audiences with different levels of expertise in shell scripting and filesystem manipulation. – AdminBee May 06 '20 at 11:17
  • I had to change the df / to df -P / due to my root having a long device name, since df splits long output to multiple lines otherwise. -P makes it output posix-style which lacks that feature. – David Feb 23 '21 at 18:54
  • note that the df "free space" stat is not strictly true--there is usually a percentage of free space reserved for the root user which is not included in the free space shown by df. In other words, to zero all of the free space, you need to go past that amount (as the root user) – Chris Combs Oct 18 '23 at 17:11