Fast Way to Randomize HD?

Question

I've read about how to make hard drives secure for encryption, and one of the steps is to write random bits to the drive, in order to make the encrypted data indistinguishable from the rest of the data on the hard drive.

However, when I tried using dd if=/dev/urandom of=/dev/sda in the past, the ETA was looking to be on the order of days. I saw something about using badblocks in lieu of urandom, but that didn't seem to help a whole lot. I would just like to know if there are any ways that might help me speed this up, such as options for dd or something else I may be missing, or if the speed is just a limitation of the HD.

Answer 1

dd if=/dev/urandom of=/dev/sda, or simply cat /dev/urandom >/dev/sda, isn't the fastest way to fill a disk with random data. Linux's /dev/urandom isn't the fastest cryptographic RNG around. Is there an alternative to /dev/urandom? has some suggestions. In particular, OpenSSL contains a faster cryptographic PRNG:

openssl rand $(</proc/partitions awk '$4=="sda" {print $3*1024}') >/dev/sda

Note that in the end, whether there is an improvement or not depends on which part is the bottleneck: the CPU or the disk.

The good news is that filling the disk with random data is mostly useless. First, to dispel a common myth, wiping with zeroes is just as good on today's hardware. With 1980s hard disk technology, overwriting a hard disk with zeroes left a small residual charge which could be recovered with somewhat expensive hardware; multiple passes of overwrite with random data (the “Gutmann wipe”) were necessary. Today even a single pass of overwriting with zeroes leaves data that cannot realistically be recovered even in laboratory conditions.

When you're encrypting a partition, filling the disk with random data is not necessary for the confidentiality of the encrypted data. It is only useful if you need to make space used by encrypted data indistinguishable from unused space. Building an encrypted volume on top of a non-randomized container reveals which disk blocks have ever been used by the encrypted volume. This gives a good hint as to the maximum size of the filesystem (though as time goes by it will become a worse and worse approximation), and little more.

Answer 2

You can get OpenSSL to encrypt /dev/zero with a randomized password, giving decent pseudorandom data very fast (if your CPU supports accelerating it).

openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero | dd of=/dev/sda

You could pipe this through pv to get progress/ETA. The commands I'm running right now (in a root shell) are:

DISK="sda"
DISKSIZE=$(</proc/partitions awk '$4=="'"$DISK"'" {print sprintf("%.0f",$3*1024)}')
apt-get install pv
openssl enc -aes-256-ctr -nosalt \
  -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" \
  < /dev/zero |
  pv --progress --eta --rate --bytes --size "$DISKSIZE" |
  dd of=/dev/"$DISK" bs=2M

I got this idea from this answer, after having the same problem as irrational John, who commented on Gilles's answer above. This increased my wipe speed to my new RAID array from 11 MB/s to around 300 MB/s, taking what was going to take a week down to 10 hours.

I'll add that you should be able to use openssl rand #of_bytes rather than the more complicated openssl enc ... statement above, but there is a bug which will allow ssl to produce only 16 MB of output. (This bug has been filed, Jan 2016.)

And, as per the answer to this question, and continuing to assume that the CPU is the bottleneck, it may be possible to increase speed further by running multiple parallel openssl processes on separate cores, combining them using a FIFO.

Answer 3

The openssl did not seem to work for me. I got "unknown options" and other issues with the provided solutions. So I ended up going with the program fio.

fio -name="fill" -ioengine=libaio -direct=1 -bs=512m -rw=write -iodepth=4 -size=100% -filename=/dev/md0

Which seems to be taking 3 hours to do 19TB across 24 HDDs. So roughly 1,800 MB/s

smp-016:~ # fdisk -l /dev/md0
Disk /dev/md0: 18890.1 GB, 18890060464128 bytes

smp-016:~ # fio -name="fill" -ioengine=libaio -direct=1 -bs=512m -rw=write -iodepth=4 -size=100% -filename=/dev/md0
fill: (g=0): rw=write, bs=512M-512M/512M-512M/512M-512M, ioengine=libaio, iodepth=4
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [W(1)] [2.7% done] [0KB/1536MB/0KB /s] [0/3/0 iops] [eta 03h:01m:11s]

I hope this is actually random data. The man page says fio "Default: fill buffers with random data." http://linux.die.net/man/1/fio

I'm not doing it for secure/encryption purposes, just trying to be sure my later read tests are actual data and not just 0's. This same fio command could be used for SSD/NVMe preconditioning. As just using /dev/zero can lead to disk level compression "cheating" how much is actually written. Although I would add a -loops=2 flag to it, if it is a fresh SSD for benchmarking.

If you did want it to be secure you may be able to use the -randrepeat=bool option, as that will toggle "Seed the random number generator in a predictable way so results are repeatable across runs. Default: true.", but I'm still not certain how secure that would be.

Additionally some enterprise class HDDs out there are SED (Self Encrypting Drives) and will allow you to spin the encryption key to instantly and securely erasing all the data written.

Lastly, I have in the past used DBAN (aka Darik's Boot and Nuke), which has CD and USB bootable options and "is an open source project hosted on SourceForge. The program is designed to securely erase a hard disk until its data is permanently removed and no longer recoverable"

Answer 4

I was trying to fill a 4TB external USB HDD with dd if=/dev/urandom of=/dev/sdX status=progress which was way too slow (irrespective of bs= settings), and it seems that openssl has a cap on how much random data it will output (at least for version 1.0.2p). The best option I found was the comment from frostschutz to use shred, as in:

shred -v -n 1 /dev/sdX

Make sure to use the -n 1 otherwise it will default to writing the device 3 times over (plus the -v which shows progress). I don't think the quality of the pseudo-random numbers is that high, but it's enough for me to prepare a large capacity portable HDD for encryption.

Answer 5

The bottleneck if neither the block size nor the hard drive, but the slow generation of pseudo-random numbers. /dev/urandom is by magnitudes faster compared to /dev/random since it's not blocking on a low entropy pool.

You can confirm this by measuring the raw output of your pseudo-random numbers:

pv /dev/urandom >/dev/null

This rate will be much slower than the write rate of your hard drive. A correct solution totally depends on your required level of security. If you require high security either use a fast hardware random generator or accept the slow speed. If your security needs are not that high, you could capture a few dozen MiB of data and write that string repeatedly to the drive. Or maybe even writing zeros from /dev/zero is an option.

Summary

/dev/random - secure, very slow
/dev/urandom - less secure¹, slow
hardware RNG - secure, fast, very expensive
(/dev/zero - not random at all, very fast)

¹_{According to Is a rand from /dev/urandom secure for a login key? /dev/urandom is as secure as /dev/random. Thanks to Gilles for pointing this out.}

Answer 6

Completing Marco's answer, what you need is faster random number generator.

You use a simple program that echos random numbers from a good library like boost::random and use that one in dd.

If you choose boost, you can use this example, changing the experiment function to your needs.