2

I am using @tremby's great idea to fill a disk with random data.

This involves piping openssl, which is encrypting a lot of zeros, to dd (bs=4M).

I'm maxing out the single core on which this is being run (I have 7 more), and I'm nowhere near maxing out my I/O.

I'm looking for a way to parallelize the input to dd.

I suppose I could do it like this, but what I'm really looking for is a way to parallelize openssl and write that to dd so that the write to the disk is sequential.

Does anyone have a suggestion?

Community
  • 1
Diagon
  • 630
  • PLease, take a look at this question and see if it fits on your needs :) - http://unix.stackexchange.com/questions/160469/how-to-parallelize-dd –  Jan 05 '16 at 19:25
  • Not quite, thanks. I was clarifying the question, but you were too fast for me! :) – Diagon Jan 05 '16 at 19:29
  • Instead of encrypting zeros why not just use random data: openssl rand 123456789 – Ole Tange Jan 05 '16 at 23:07
  • @Ole_Tange - Not sure what's faster. The complaint over at that first link (to which tremby was responding) was that /dev/urandom was too slow; so I was taking it that the openssl method he proposed was faster and just as dependably random. Perhaps this would work just as well? – Diagon Jan 06 '16 at 06:07
  • @Ole_Tange - I tried your suggestion with openssl rand 2trillion > fifo_file &. Once I start reading from the fifo, the openssl process dies after about 16MB. I didn't have through problem with @tremby's approach (linked above). – Diagon Jan 07 '16 at 17:16
  • @Diagon great: I reported the bug (which is due to a 32 bit counter). – Ole Tange Jan 09 '16 at 04:14

1 Answers1

3

The important part is having a way to merge the output of your several openssl commands. I believe a FIFO would solve your problem. Try this

mkfifo foo
openssl <whatever your command is> > foo &
openssl <whatever your command is> > foo &
openssl <whatever your command is> > foo &
dd if=foo of=/dev/sda bs=4M

EDIT: Add as many of the openssl lines as you need to max out your system; you can even add them after dd invocation.

As mentioned by the OP in the comments below, it is possible to cat foo | pv | dd of=/dev/sda to monitor progress.

Diagon
  • 630
David King
  • 3,147
  • 9
  • 23
  • Awwwesome! There is no end to learning unix. Thanks. – Diagon Jan 05 '16 at 19:59
  • IMHO named pipes (aka FIFO's) are one of the most under utilized features of Linux. I'm here to spread the good word :) – David King Jan 05 '16 at 20:07
  • can I cat foo | dd ? I ask because if so, I can use pv to see how far I've go to go. – Diagon Jan 05 '16 at 20:08
  • Sure. You can treat foo just like any regular file except that it won't get to the end until the openssl commands terminate. – David King Jan 05 '16 at 20:09
  • Now if only the debian installer would use this method during cryptsetup initialization... – Dominik R Jan 05 '16 at 21:28
  • @Dominik - If you don't do this already, you'll need to prepare the disks yourself, do the install, and then chroot in to fix the fstab/crypttab & update-initramfs/update-grub. I do that on all my installs. The installers are just not flexible enough. – Diagon Jan 06 '16 at 06:03
  • @Diagon: using the text installer and manual partitioning, I don't usually prepare the disks manually. Should I? Only thing I never did was filling the disk with pseudorandom data using the slow d-i method. – Dominik R Jan 06 '16 at 08:28
  • 1
    @Dominik - Well, I usually do that because I want to use specific options with cryptsetup (like whirlpool or detached headers). Also, I usually want to install using btrfs with subvolumes set up in some particular way. It's not hard. Just use the live usb, prepare the disks and then run the installer. Then you can install to the prepared disks, without doing any formatting. – Diagon Jan 06 '16 at 10:35