9

I was wondering if it would be possible to write a disk image file directly to a partition without saving it as a file first. Something like

dd if="http://diskimages.com/i_am_a_disk_image.img" of=/dev/sdb1 bs=2M

I would also accept an answer in C or Python because I know how to compile them.

Jens
  • 1,752
  • 4
  • 18
  • 36
person the human
  • 229
  • 1
  • 6
  • 1
    The point is how you can verify the correctness of the download. The other thing is resuming an aborted download. With today's disk sizes you should afford to save it in a file first. – U. Windl Jan 07 '22 at 01:17
  • @U.Windl You can do both just fine, whether you write to a file or a partition. – wizzwizz4 Jan 07 '22 at 15:59
  • @U.Windl The problem with phrases like "today's disk sizes" is that it assumes a desktop, laptop or server. There are many contexts even today where you're not so lucky with space such as SBCs. – Philip Couling Jan 07 '22 at 19:34

3 Answers3

25

This is actually trivial. You can write to the device just like it's a file, and there are commands for directly downloading content and either writing it to a file or writing it to "stdout".

As the user root you can simply:

curl https://www.example.com/some/file.img > /dev/sdb

Where /dev/sdb is your hard drive.

This is not generally recommended but will work just fine and is useful in very small devices without much disk space.

Incidently it would be more normal to write a disk image to a disk /dev/sdb not a partition /dev/sdb1.

Philip Couling
  • 18,733
15

You can use wget -O option to print to disk directly:

wget -O /dev/sdb http://diskimages.com/i_am_a_disk_image.img

You don't really need to use dd.

Arkadiusz Drabczyk
  • 25,539
  • 1
    Consider rephrasing your answer - in its current form it suggests -O has something to do with accessing the disk, while it is simply a flag for output file. – ciamej Jan 07 '22 at 20:28
4

Yes, you can actually. Use something like this:

curl 'http://diskimages.com/i_am_a_disk_image.img' | dd conv=sync,noerror bs=2M of=/dev/sdX
Alex
  • 1,958
  • 1
    Out of curiosity, why bother with dd in this context? – Philip Couling Jan 05 '22 at 23:56
  • 3
    well two reasons, main is that OP gave an example with dd in the initial question, second is, the advantages of block size specifications (if required) and to avoid stopping due to read errors (high latency over network in this case, etc) with conv=sync,noerror. Given the question, they are not mandatory options but if one is to use dd, they would be nice to have – Alex Jan 06 '22 at 00:14
  • Curious. I'm not convinced that's desirable when reading from a pipe. but hey. Thanks for the info. – Philip Couling Jan 06 '22 at 00:27
  • 3
    It's also handy to split into two processes if you want them to run as different users. A cautious operator would download as non-root user, but may need to be root to write to the disk partition. – Toby Speight Jan 06 '22 at 09:33
  • 1
    @TobySpeight interestingly you don't need need dd for this since with redirection the stdout file descriptor is opened by the shell and passed in. Though I take your point about it being marginally simpler to achieve. – Philip Couling Jan 06 '22 at 11:33
  • 1
    I was thinking of curl … | sudo dd … when the invoking user can't write to the device. – Toby Speight Jan 06 '22 at 11:36
  • @TobySpeight: Right, that's easier to type than what Philip was getting at. But it is possible. Given a root shell, you redirect stdout to disk, then drop privileges and exec curl or wget so it runs as nobody or some other non-root account. e.g. sudo sh -c "sudo -u nobody wget ... > /dev/sdX" – Peter Cordes Jan 06 '22 at 14:19
  • 4
    @Alex: What actual (re)blocking do you want dd to do? If that actually happened, e.g. writing a full output block after a short read of an input block from the pipe, it would munge your data. When is dd suitable for copying data? (or, when are read() and write() partial) / Is it better to use cat, dd, pv or another procedure to copy a CD/DVD?. – Peter Cordes Jan 06 '22 at 14:25
  • 3
    Most programs other than dd pick some reasonable block size for their output, like at least 4k or 8k. 64k or so would be better to trade off system call overhead with CPU L2 cache staying hot for the kernel to copy the data into a buffer queue, but since this isn't O_DIRECT it's not an actual hardware write block size. Specifying bs= something non-default is important for dd only because its default is tiny, 512 bytes, so system-call overhead is a killer. – Peter Cordes Jan 06 '22 at 14:26
  • 2
    -1 because of the conv=sync possible data corruption issues. Using tee or input redirection is safer. – Josh Jan 06 '22 at 15:42
  • @Josh it's right the opposite in this case. If I wanted en example of data corruption from an image transfered over the network with ssh or curl or whatever, you wouldn't be able to give one to me. The funny thing is I've done this on lots of machines in a real production environment, and a real world example shows that there was no data corruption. Somebody will google this question someday and they will come accross the answers. I will never offer a "quirky test man cave linux environment solution" to anyone. If somebody chooses dd for this, my answer will do the job. – Alex Jan 06 '22 at 16:09
  • 4
    I disagree. conv=sync,noerror means that if curl writes less than 2 MiB, dd will pad that 2 MiB block with zeros, corrupting the data. This is unlikely to happen over a reliable, fast link, but it's possible. Review the questions that Peter Cordes posted -- I, too, used to use dd all the time, and answers on this site made me reconsider my views. – Josh Jan 06 '22 at 16:25
  • @Alex "... to avoid stopping due to read errors (high latency over network in this case, etc)" - That is not a thing. curl handles the download. If there are network issues, either curl will stop producing data for a bit, and resume outputting data when the network side recovers. In that case, dd adds nothing. Or the network issues are so bad that TCP timeouts occur, curl aborts, the pipe is torn down, and dd is terminated as well. Again, dd adds nothing. I also concur with the other commenters warning of dd problems. dd is usually harmful. – marcelm Jan 06 '22 at 20:58