dd shows a random number of records in/out

Question

I'm writing some images back to block devices, with dd, and getting very strange output I've never seen before.

# xz -dc goren.img.xz | dd bs=1M of=/dev/storage2/goren
35+2475166 records in
35+2475166 records out
21474836480 bytes (21 GB) copied, 222.912 s, 96.3 MB/s
# xz -dc gronn.img.xz | dd bs=1M of=/dev/storage2/gronn
50+2413782 records in
50+2413782 records out
21474836480 bytes (21 GB) copied, 233.478 s, 92.0 MB/s
# xz -dc grummle.img.xz | dd bs=1M of=/dev/storage2/grummle
63+2443466 records in
63+2443466 records out
21474836480 bytes (21 GB) copied, 222.898 s, 96.3 MB/s
# xz -dc hozen.img.xz | dd bs=1M of=/dev/storage2/hozen
19+2556787 records in
19+2556787 records out
21474836480 bytes (21 GB) copied, 250.989 s, 85.6 MB/s

The output I expected to see in each case (and what I got when creating the image files) is:

20480+0 records in
20480+0 records out

As far as I can tell the images are being written correctly, though I am concerned with the number of records shown. This is obviously not correct in any of the cases. Though, as I said, the written images match the originals, pass filesystem checks, etc.

I'm using Fedora 21 x86_64 with coreutils 8.22.

I suspect dd isn't reading 1M blocks reliably from the pipe but I'm too busy to dig up the source code and look. Try ... | dd ibs=1024 obs=1m ... to change the input block size to a smaller value and see what you get. — Andrew Henle, May 06 '15 at 21:39
Use the right tool for the right job. If the job doesn't involve tapes or seeking, dd isn't the right tool. — Gilles 'SO- stop being evil', May 06 '15 at 22:43
@Gilles What relevance is the marked duplicate to this question? It discusses a completely different issue. — Michael Hampton, May 06 '15 at 22:53
@MichaelHampton Same issue: dd does not, in general, just copy its input to its output. It copies bits and pieces, here and there, in a difficult-to-control fashion, unless you use the right parameters (absolutely avoid bs) with the right file types (plain files are block devices can be ok, forget about pipes and character devices). — Gilles 'SO- stop being evil', May 06 '15 at 22:54

score 4 · Accepted Answer · answered May 06 '15 at 21:37

4

Those are incomplete reads. It should go away if you add iflag=fullblock.

By default, dd will happily accept smaller blocks from a pipe, if there isn't more data readily available. With the iflag, dd will wait until a full block of data has been gathered, or EOF.

In regards to data consistency there should be no issue, so you should be getting correct results either way.

The question is why are you using dd at all, your example can just as well be reduced to:

xz -dc goren.img.xz > /dev/storage2/goren

answered May 06 '15 at 21:37

frostschutz

48,978

Because > gives me no feedback. And this will eventually be part of a larger backup script. – Michael Hampton May 06 '15 at 21:39
1

@MichaelHampton perhaps consider pv instead of dd – Chris Davies May 06 '15 at 22:35
Hm. iflag=fullblock makes dd report the expected numbers, but then it's almost 20% slower. – Michael Hampton May 06 '15 at 22:52
With a larger block size and iflag=fullblock there will be longer gaps between writes to disk hence the slow down. I see no need for fullblock here. Note the next version of dd (version 8.24) will have a status=progress option to update the status line every second. – Pádraig Brady May 07 '15 at 09:26
@MichaelHampton you could try bs=64k or something and see if that helps ... dd blocksize is a bit of a mystery, see also http://unix.stackexchange.com/a/183920/30851 – frostschutz May 07 '15 at 10:12

dd shows a random number of records in/out

1 Answers1

Linked